The FBI Virtual Case File System

The initiative

The FBI began developing a case management software system called the Virtual Case File (VCF) in 2000, as part of a wider IT upgrade project called Trilogy. “The Trilogy project centred on upgrading the agency's 56 field offices and 22,000 agents and support staff with new desktops and servers, web-enabling a number of the most important investigative database systems, and, most importantly, a VCF system that would automate the antiquated paper-based Automated Case Support (ACS) system.”[4]

The new case file system was essential to improve the efficiency of the agency's information management. “The VCF was envisioned to help FBI agents efficiently share data about cases in progress, especially terrorist investigations. The system would also enable agents anywhere in the United States quickly to search various documents and allow them to connect possible leads from different sources. In addition, the VCF would include a case management system, an evidence management system, and a records management system. The intention was to eliminate the need for FBI employees to scan hard-copy documents into computer files."[5]

The FBI's estimates for completing the full Trilogy infrastructure were revised several times. “The first delivery of VCF was targeted for completion in December 2003. The second and third deliveries, which were intended to upgrade and add additional investigative applications to the VCF, were targeted for completion in June 2004.”[6]

The VCF contractor, Science Applications International Corp (SAIC), provided the first delivery in December 2003. However, the application was not fully functional and the FBI did not accept it. At this time, the FBI revised the VCF deployment schedule again. “Rather than having the VCF implemented and enhanced in three deliveries, the FBI Director announced in June 2004 that development of the project would be split into two parallel tracks, Initial Operational Capability (IOC) and Full Operational Capability (FOC).”[7]

The project was eventually abandoned in April 2005, after costs estimated at over USD170 million.[8]

The challenge

The FBI spent many years developing its information systems without an overarching organisational view, and found itself with an "improvised" IT infrastructure that had more than “50 independent application systems written in different programming languages and running on disparate platforms".[1] There was a need for a customised system, since there were no existing commercial software packages available that met the agency's needs.

By 2000, ageing infrastructure - including i386-based desktop PCs and 12-year-old interoffice networks - were becoming a handicap for the FBI.[2] Both the agency's IT infrastructure and its case management system were in dire need of modernisation. Without an upgrade to the infrastructure, it would be impossible to implement a modern, fully functional case management system with information-sharing capabilities. At the same time, without an effective case management system the FBI could not identify and capitalise on all of the information in its possession.[3]

The public impact

Although the FBI made some progress in modernising its IT infrastructure by installing modern computers and providing a secure network with Trilogy, it was not able to achieve the deployment of VCF, which was one of its critical objectives.

In February 2005, a month after VCF was discarded, the head of the FBI made the following statement: “We have invested approximately 170 million dollars in VCF to date. It is my understanding our vendors have delivered services and reusable equipment worth USD53.3 million and that we have USD12.2 million in unspent obligations on our VCF contracts. This results in a loss of USD104.5 million”.[9] Therefore, only a minimal portion of the investment was recoverable, and the goal of improving the agency's case management system was stalled.

Stakeholder engagement

The main stakeholders in the VCF system were the FBI, principally the system's users, the contractors, principally SAIC, and the federal government.

After reviewing and recognising the limitations of the web front end for the ACS system, two special agents convinced the FBI director in 2001 that the FBI needed a new system with an improved user interface and database management system that could improve storage and analysis capabilities of “relationships on everything from witnesses, suspects, and informants, to evidence such as documents, photos, and audio recordings”(Goldstein, 2005). In December 2001, SAIC was told to stop building a web front end to the FBI's existing systems and instead begin development of a new application to replace the old ACS.”[10]

The FBI refined the VCF concept through Joint Application Development (JAD) sessions held with SAIC engineers and other experts between January and June 2002. “The JAD sessions brought together FBI representatives to determine what applications were needed to support the case management and information requirements of FBI agents, analysts, and support personnel; [user application component] contractor representatives to determine what applications could be created; and infrastructure contractor representatives to ensure that the applications could be supported by the groundwork that was being developed.”[11]

Political commitment

The FBI executes critical and confidential operations that require the management of great amounts of information, and therefore the implementation of a modern and efficient system to do this was widely recognised by the agency and Congress as a necessary investment.

Congress provided the financial support requested for the project - and more funding later on as it began to overrun. “In September 2000, Congress approved USD379.8 million for a three-year project that was called the FBI Information Technology Upgrade project under the direction of FBI Director Louis Freeh.”[12]

As the VCF project spend exceeded its estimates, its deadline was brought forward. “In January 2002, the FBI requested an additional USD70 million from Congress to accelerate the Trilogy project, but received USD78 million. SAIC agreed to deliver the VCF system in December 2003 rather than June 2004.”[13]

The FBI director at the time, Robert Mueller, also expressed his support for the project, even after it had "gone wrong". “With regard to the funding of VCF, this Committee has been supportive of our efforts and has generously provided the funding we have needed to overcome obstacles and attempt to move forward... I am disheartened by this result but remain confident in our ability to deliver a case management system to our employees' desktops in the future."[14]

Public confidence

It is difficult to establish the level of public confidence in such a project, as its details at the time were confidential, so that knowledge of its details was probably limited to the few individuals managing the institution and budget. There is, however, evidence of public pressure on the FBI as a whole to improve its efficiency after the September 11 attacks in 2001.

The development of Trilogy and VCF was started shortly before the attacks in 2001, and this situation had a major impact on the pressure on the FBI to improve its administrative efficiency. “Due to intense public and congressional pressure, Mueller instructed that the three-year schedule be put into overdrive so that the Trilogy project could be completed 'as soon as technically possible'.”[15]

Clarity of objectives

The objectives of the VCF project were very high level at its inception, and there is no evidence that measurable targets were set to accompany the project's goals.

The FBI did not have a set of defined VCF requirements when the original contract was signed in 2001. “The development of the VCF application started with a very simple concept - the FBI's need for a modern case management system. As the FBI's mission evolved over the past several years, so did our technological needs. As a result of these changes and other issues, the FBI faced obstacles in a number of key areas relating to the VCF programme.”[16]

The VCF was intended to:

• “Help FBI agents efficiently share data about cases in progress, especially terrorist investigations
• “Enable agents anywhere in the US quickly to search various documents and allow them to connect possible leads from different sources
• “Provide a case management system, an evidence management system, and a records management system to eliminate the need for FBI employees to scan hard-copy documents into computer files.”[17]

The overall Trilogy project was in a similar situation. “During the initial years of the project, the FBI had no firm design baseline or road map for Trilogy. According to one FBI Trilogy project manager, Trilogy's scope grew by about 80 percent [from the] initiation of the project. Such large changes in the requirements meant that the specific detailed guidance for the project was not established, and as a result a final cost and schedule was not established.”[18]

Strength of evidence

This project was meant to replace an outdated document management system. However, although there had been previous IT projects implemented within the FBI, there is no evidence that experience from them was used in the implementation of the VCF project. The FBI relied on pilots (after the first stage was delivered) to test the project. However, this seems to have happened too late into the project to lead towards a successful implementation.

The VCF contractor, SAIC, delivered the first of three planned system deliveries for the VCF in December 2003, which was not fully functional and was rejected. The delivery of the project was then split into two parallel tracks, IOC and FOC, and by 31 December 2004, SAIC provided the FBI with a "proof-of-concept" or prototype VCF in line with the JAD approach to software development. “This prototype is being used by agents and analysts in a test field office to demonstrate an electronic workflow process in ongoing investigations. Over a 6-week pilot test period scheduled between January and March 2005, the FBI's New Orleans Field Office and the Baton Rouge Resident Agency are entering actual investigative lead and case data into the VCF. This information will be uploaded into the ACS to create paper case files upon electronic approval of the electronic information in the VCF.”[19]

Feasibility

The project faced several feasibility difficulties, including lack of qualified staff, unrealistic timelines, and inappropriate infrastructure. Although sufficient funding had been approved for the project, these difficulties resulted in significant overruns.

Some of the issues acknowledged by the FBI after the project was cancelled included the limited engineering capabilities of the FBI and SAIC in creating the VCF system. “We lacked skill sets in our personnel such as qualified software engineering, programme management, and contract management. We also experienced a high turnover in Trilogy programme managers and chief information officers (CIOs).”[20] The agency also stated that it underestimated the complexity of interfacing with its legacy systems, of addressing its security needs, and of establishing an enterprise architecture.

There were also statements that the IT infrastructure of the FBI at the time was not appropriate for supporting such a project. According to a 2002 report from the Department of Justice's Office of the Inspector General, 13,000 of its computers could not run modern software. “Most of the 400 resident agency offices were connected to the FBI intranet with links about the speed of a 56-kilobit-per-second modem. Many of the bureau's network components were no longer manufactured or supported. And agents couldn't email US Attorney offices, federal agencies, local law enforcement, or each other; instead, they typically faxed case-related information.”[21]

The schedules and budget were greatly underestimated. Although enough funds were allocated to the VCF project, the estimated budget was exceeded by around USD50 million, ultimately reaching USD170 million.[22] Additionally, the timeline commitments from developers were considered unrealistic. “Not getting the right resources when they are needed may create a risk that the schedule will not be met. Unrealistic schedules may doom the project before it even starts.”[23]

Management

The FBI and SAIC, the contractor executing the project, were the organisations principally responsible for the outcome of the VCF system. However, there were several weaknesses in the planning and monitoring of the programme as a whole. Both sides blamed each other for the mismanagement.

For the FBI, the head of the agency was the top person accountable for the project, and he designated several people to manage it more directly. Larry Depew, who was one of the first proponents of the project - although he had no IT project management experience - was named the VCF project manager. His team communicated the FBI's investigative and administrative processes to the SAIC engineers. C. Z. (Sherry) Higgins, a seasoned IT professional, was hired to create the Office of Programme Management, and was in charge of centralising IT management and overseeing "the FBI's most expensive, complex, and risky projects", as well as managing Trilogy.[24]

When SAIC'S work was first evaluated after the delivery in 2003, it was concluded that “lack of effective engineering discipline has led to inadequate specification, design and development of VCF... There was ‘no assurance‘ that the requirements were satisfied, nor that the architecture, Concept of Operations, and requirements were correct and complete."[25]

On the other hand, SAIC criticised the FBI's management. Duane Andrews, SAIC's chief operating officer, said that: “The FBI modernisation effort involved a massive technological and cultural change, agency-wide. To add to that complexity, in the time that SAIC has been working on the Trilogy project, the FBI has had four different CIOs and 14 different managers. Establishing and setting system requirements in this environment has been incredibly challenging.”[26]

Finally, an audit report from the Office of the Inspector General found that: “Despite the use of two contractors to provide three major project components, the FBI did not hire a professional project integrator to manage contractor interfaces and take responsibility for the overall integrity of the final product until the end of 2003. According to FBI IT managers, FBI officials performed the project integrator function even though they had no experience performing such a role.”[27]

Measurement

There were no monitoring guidelines or tracking indicators included in the project's plan at the outset, and the project was first tested and measured only a few years after inception, which was arguably too late to recover from the initial mistakes.

According to FBI IT and contract managers, the cost-plus-award-fee type of contract used for Trilogy did not specify key items to guide the implementation, which also led to some of the challenges mentioned. “It did not require specific completion milestones, did not include critical decision review points, and did not provide for penalties if the milestones were not met.”[28]

The Office of the Inspector General also noted in its audit report that had monitoring taken place from the outset, many of the problems might have been avoided  “If monitoring the scheduling of the project had been a priority, the FBI could have taken more timely action to effectively address Trilogy's problems. At the end of 2003 — well over two years into the project — the FBI hired a contractor to perform these project integration duties when it became apparent that a professional project integrator was needed to effectively complete the project.”[29]

Alignment

A lack of clarity regarding the project's specific objectives and timelines led to several disagreements between the FBI personnel leading the project and the contractors. This eventually led to both parties blaming each other for the issues with the project.

Initially, the FBI did not have a clear vision of what the FBI's Trilogy project should achieve or of specific design requirements, and the contractors were not held to a firm series of achievable milestones. “The FBI's investment management process also left it ill-equipped to ensure that all three components of Trilogy were developed in an integrated fashion. Moreover, at the outset, the FBI and others did not provide consistent or effective management of Trilogy, leading to technical and scheduling problems.”[30]

The tensions between SAIC developers and the FBI increased over the winter and spring of 2003, as the first part of the VCF project was being delivered. A number of problems were blamed on the contractors but were also compounded by the FBI's "sloppy inventories" of existing networks and an underestimation of the difficulties of such a complex implementation.

The acting CIO for the FBI made the decision to reject the deliverable, and the FBI documented 17 functional deficiencies that SAIC would have to fix before the system could be accepted. SAIC argued that many of these deficiencies were caused by requirements changes from the FBI team, so an arbitrator had to be called in. In March 2004, the arbitrator released a report that found fault with both the FBI and SAIC. “According to the report, 59 issues and sub-issues were derived from the original 17 deficiencies - 19 were due to requirements changes and the FBI's fault, while the remaining 40 were SAIC problems. SAIC offered to make all the changes if given one more year and USD56 million. [The FBI's CIO] rejected the offer."[31]

Bibliography

Anatomy of an IT disaster: How the FBI blew it, Eric Knorr, 21 March 2005, InfoWorld

The Federal Bureau of Investigation's Management of the Trilogy Information Technology Modernization Project, February 2005, Office of the Inspector General

The FBI Virtual Case File: A Case Study, Jack T. Marchewka, 2010, Northern Illinois University

Testimony - FBI's Virtual Case File System, 03 February 2005, Robert S. Mueller, III, The Federal Bureau of Investigations

Who Killed the Virtual Case File? Harry Goldstein, 1 September 2005, IEEE Spectrum