Case Study: Borderline Success at the Department of Homeland Security

By Edward Cone  |  Posted 10-20-2006

Case Study: Borderline Success at the Department of Homeland Security

At first glance, the Department of Homeland Security's US-VISIT program may seem an unlikely model for future government information-technology projects. That may be true on second glance, too.

But beyond the project's very real shortcomings lies a possible template for smarter, faster work on important public-sector jobs. Steven Cooper, the founding chief information officer of DHS, says the project's methodology of incremental releases and refinements, rather than the massive development-and-release cycle common to federal jobs, has "the potential to save hundreds of millions of dollars across agencies for taxpayers." Robert Mocny, US-VISIT's acting director, agrees, saying, "This is a model for rolling out such a large-scale system."

For more on technology in government, read Three Big Government IT Projects That Struggled

Bold claims for a program that has spent $1 billion and counting, but has not yet delivered a key element of its original design, or a done-by date for that critical missing piece.

US-VISIT is a network of biometric-screening systems, such as fingerprint and ocular scanners, that ties into government databases and watch lists to check the identities of millions of people coming into the United States. Created in the wake of the Sept. 11 terror attacks, and implemented in a hurry to meet the suddenly-pressing demands of national security, the project is now operational in some 300 locations, including major international points of entry by air and sea; land borders are next. The Department of Homeland Security credits the program with keeping more than 1,350 criminals from entering the country.

Among those nabbed: a murderer from Germany who had resettled in Canada; a Bulgarian national who had slipped into Costa Rica with somebody else's $13 million; and an escaped detainee from Iraq.

Yet the system still lacks a vital (and long-promised) component: the capability to notify authorities when visitors subsequently leave the U.S. And beyond that missing piece, US-VISIT has been criticized repeatedly over its three-year life by its overseers in Congress and the Government Accountability Office for not setting specific timetables and metrics of success, shortcomings that are acknowledged by senior officials close to the project. Even the 9/11 Commission got in a few licks, noting that US-VISIT is cobbled together from several older systems and will need to be updated in short order with newer technology, including up-to-date biometric scanners.

"You can't manage what you can't measure, and there was never a definition of what the program would accomplish by a certain time," says Randolph "Randy" Hite, director of information technology architecture and systems issues at the Government Accountability Office, who has monitored US-VISIT for years. "From the beginning we said, like any program, it needs to define and commit on what it will deliver in terms of capability, value, cost and timeline. Our reports kept saying the expenditure plans [the appropriations that fund the project] are a contract with Congress, so you need to disclose and commit and report, and be held accountable. That has not occurred. We were never able to link cost and capability. They got an annual appropriation, and spent it, but we could not see if a particular functionality was tied to the cost."

Cooper, now CIO at the American Red Cross, does not disagree with Hite's assessment. But he says the haste required to protect national security justified taking certain shortcuts on scheduling and system development along the way. "The team well understood what GAO and Congress were pointing out," he says. "We simply didn't have the time and resources to do it all. We did not set out to ignore good management practice [e.g., scheduling and executing on the exit system] and made tradeoff decisions to get capability on line to protect Americans. If time and national security were not involved, we might have done it differently." For example, says Cooper, "We pushed to roll out the entrance portion of the system and didn't focus on the exit portion, because screening people coming into the U.S. seemed more crucial. We decided that with X-number of action items that needed to be done, we couldn't do them all in parallel, so we would try to do the first half-X, then pick up the rest."

The original system checked only two fingerprints per person, rather than all ten. "We get criticized for that, but we've identified 20,000 people as immigration violators because we chose to move in a sequenced fashion," says Mocny. The program is now moving toward acquiring new equipment that will allow checks of all ten digits, which is becoming the international standard for fingerprint identification.

There were choices made on how to do the work, as well. "We made a tradeoff on the amount of time spent on tests before turning things over to production," says Cooper. "You always want to reduce defects as close to zero as possible; you can find 80 to 90 percent of defects in any software very quickly, but it takes much more time, proportionately, to find the rest. So if you release and then test in parallel to production, you can move a lot faster and realize tremendous value."

Despite the project's shortcomings, however, the release-and-refine methodology forced upon US-VISIT by circumstance has real promise for future projects.

Next page: Piece-by-Piece Project Planning

1 | 2 | 3 | 4

Piece

-by-Piece Project Planning">

The staged development-and-release methods behind US-VISIT set it apart from typical federal IT project planning, which has tended toward long timetables that promise huge systems to be delivered at a single stroke, says Cooper. He says it is possible to recognize the project's shortcomings in terms of both management and deployment, and still discern in its history a new model for building federal systems.

Maybe, he suggests, it's time to ditch the traditional big-bang approach for a release-and-refine method. Change of some sort seems a necessity, given the miserable record of bloated government IT jobs, which overrun deadlines and budgets with numbing frequency.

"It's probably true in a lot of places where the government makes critical policy decisions that the most important priority should be followed by the next most important, to start on a prioritized list and see where it goes," agrees Jim Stolarski, the US-VISIT project manager for lead integrator Accenture. While DHS has made most of the decisions on the project, he says, "We tend to favor a phased approach, especially in an environment like this one where there is still learning going on."

Private companies, notes Cooper, are already less likely than in the past to engage in "zillion-dollar programs." Instead, he says, "they are defining smaller chunks that fit together into larger systems. It's a model worth exploring." He looks to the Office of Management and Budget, which guides many federal departments, along with the Federal CIO Council, a group of top technology executives at government agencies, for leadership on the way government programs are deployed. Nobody would recommend that the hurry-up methodology behind US-VISIT be copied exactly, but there is a case to be made for learning from it in positive ways, as well as negative ones.

DHS backed into its iterative strategy on US-VISIT. There was just no way, says Cooper, to do everything at once and still hope for a timely release of the most important piece—the entry-screening system. So that became the priority: Tie key immigration databases and watch lists together, and get some biometrics to U.S. Customs checkpoints, pronto; the latest and greatest screening systems could replace available tools at a later date. Then, roll out things like RFID readers for new, chip-bearing passports. And only then address the exit portion of the system, which was seen as less critical to national security, and which involved infrastructure and regulatory challenges as well as technology solutions. The benefit of catching miscreants entering the U.S. was obvious, and the costs of waiting on the exit system seemed bearable.

By rolling the project out in stages, DHS was able to rapidly deploy a working version of the system most critical to national security. The biometric entry-check network isn't perfect. It will have to be upgraded, and its companion pieces must be added over time. But as the saying goes, The perfect is the enemy of the good; US-VISIT got something good done, fast.

Next page: Development on the Run

1 | 2 | 3 | 4

Development on the Run


The mandate facing U.S. immigration officials in the wake of the 2001 terrorist attacks on New York City and Washington, D.C., was clear: Secure the country's entry points and improve the government's understanding of who was coming and going across the borders. Congress wanted those processes automated, quickly.

The U.S. Immigration and Naturalization Service, which had responsibility for those tasks, was one of several agencies folded into the newly created Department of Homeland Security in late 2002. Soon afterward, in April 2003, then-Secretary of DHS Tom Ridge announced one of the new super-agency's first big projects: the U.S. Visitor and Immigrant Status Indication Technology system, a.k.a. US-VISIT. The plan was to create a database of photographs and biometric information, including ocular scans and fingerprints, that would be used to check people in and out of the country by matching their information against watch lists maintained by the government.

Homeland Security announced that the new system would meet Congressional requirements, and said it would start to come on line by the end of 2003. The challenge was huge. DHS was trying to integrate 22 component offices and agencies, and the hodgepodge of technology they used, on the fly. The program began bringing together disparate networks and data centers from previously independent agencies, along with various mainframes and client-server and Web-based environments. Beyond the legacy systems at INS and U.S. Customs, connections were needed to systems at the State Department.

In addition to the immigration and watch-list data to be shared, DHS needed to make itself into a unified entity. More than 3,000 desktop computers in offices across the newly formed agency had to be refreshed in order to work together. And all the while the new agency was trying to meld different cultures, training regimens, development strategies—work that is still going on in some broad areas across DHS.

US-VISIT's primary integrator, Accenture, didn't come on board until June 2004, when the project was already underway. "They were off and running on a number of things, especially the entry capability," says Stolarski. "The government retained a tremendous amount of the decision-making. We tried to support their decision-making with our best analysis."

The GAO had concerns from the start. In a 2003 report, the agency said it had "concluded that the program is a very risky endeavor. Some risk factors are inherent to the program, such as its mission criticality, its size and complexity, and its enormous potential costs. Others, however, arise from the program's relatively immature state of governance and management. For example, although the program has governmentwide scope, an accountable governance structure to direct and oversee the program that reflects this scope is not yet established."

Part of the complexity involved the dive into an alphabet soup of government systems. US-VISIT would replace a program called NSEERS (National Security Entry Exit Registration System) run by the INS; overall, seven legacy systems would be cobbled together and given new, enhanced functionality. Included on the list: the Arrival Departure Information System, or ADIS, for information on individual travelers; the Advance Passenger Information System, or APIS, for information from air and sea carriers; the Computer Linked Application Information Management System 3, or Claims 3, which dealt with benefit requests by foreign nationals; the Interagency Border Inspection System, or IBIS, a watch system for known domestic and international bad guys; the Automated Biometric Identification System, or Ident, for information on visitors to the U.S.; the Student and Exchange Visitor Information System, or SEVIS; and the Consular Consolidated Database, or CCD, which stores visa information.

Three years later, the biometric check-in system is operational, and add-ons such as RFID-readers to scan new-model passports are being tested. Few deny there's been some success. "There is some demonstrable value in what's been put out there," says Hite. In 2005, a successor to the 9/11 Commission, the 9/11 Public Discourse Project, gave US-VISIT a B in its grading system, the best such mark given to a DHS effort.

But the checkout system is still missing, and the GAO worries it may never work as intended. It has recommended that dhs, which it says has "taken actions to expand the scope and time frames of the pilot," reassess its plans for the exit capability. The watchdog agency cited a $33.5 million budget for 2006 for testing a pilot version of the exit system, even as development of a comprehensive plan went on at the same time. "Until the US-VISIT program office adequately evaluates the exit alternatives and knows whether the alternative to be selected will be effective, the program office will not be in a position to select the solution that is in the best interest of the program," says one GAO report.

None of the strategies under consideration for an exit-tracking system look very effective, says the GAO, and not all the problems have to do with technology. For example, no enforcement mechanisms for noncompliance with exit rules have been formally evaluated or benchmarked for success. Meanwhile, Mocny points to factors beyond US-VISIT's reach as part of the problem: The United States does not have a mandatory checkout rule for people leaving the country, and DHS has to scramble for space for any checkout facilities at airports, to which it must pay rent for the square footage it uses. "We're still in pilot mode," he concedes. The exit system is being tested now at twelve airports and two seaports.

The deployed systems have had their share of problems. Documents obtained under the Freedom of Information Act by Wired News show that the US-VISIT system succumbed to problems related to a virus in 2005, leading to long delays at several large airports. DHS officials had a security patch that could have prevented the problem, but delayed using it for fear of glitching the scanners, cameras and other devices attached to the aging Windows 2000 workstations used by US-VISIT. The workstations have been criticized internally at DHS as weak points in the network's security, while the lack of a comprehensive security plan for the system was singled out by the GAO.

Personnel issues have also plagued the program. In September, project CIO Scott Hastings resigned for health reasons. Preceding him out the door were Cooper, who moved on in April 2005, and Asa Hutchinson, the one-time congressman who served as the first undersecretary of Border and Transportation Security Directorate at the agency and was a champion of the project.

Hite says he sees some signs of progress on the deeper management issues GAO has raised from the start. "I do believe that the leadership of the program has taken our recommendations on things like scheduling to heart. We hope to see the results in terms of explicit measurements and commitment to goals." Mocny says several specific goals have been met, such as hitting the target date for interoperability of some systems with the Boston Police Department, thus better securing a major port of entry. When it comes to finishing the exit systems, though, he says, "We don't have a date."

Next page: A New Way of Doing Business?

1 | 2 | 3 | 4

A New Way of

Doing Business?">

Changing the culture of government IT is no easy task. Beyond questions about methods and schedules, says Hite, there is often a lack of accountability on many federal projects that gives agencies little impetus to change their ways. "It's a consistent pattern when it comes to government IT programs," he says. "Not just this one. Pick one, pick the program du jour. People act in their own perceived best interests. There are criteria out there on how you are supposed to do it, but the bottom line is that there are no consequences if you don't." On US-VISIT, he says, "Congress took an active role and said, 'we want this and we want this,' but DHS didn't do it. They didn't set the schedules, or build the exit system. You can't put it all on the CIO. The responsibility and accountability are shared by a lot of people inside and outside the program office. There's plenty of blame to go around."

Mocny says the successes are replicable, but that strong leadership is required. "It will take a champion who believes what you are doing is necessary and important," he says. Techniques used at US-VISIT, such as integrated project teams that regularly gathered together users and developers, can be duplicated with executive help. Other federal projects have moved in similar directions. Earlier in the decade, the Federal Aviation Administration used an iterative methodology known as "spiral development" to create a successful air-traffic control system, amid a much larger project that saw epic cost overruns and disappointing deliverables. But the overall culture is still waiting for a reason to change.

Cooper hopes that Karen Evans, the Office of Management and Budget's administrator for e-government and IT, can help lead that change. Speaking to Congress in May, she said, "We have been collecting 'best practices' and 'promising practices' and they are documented both in the public and private sector. However, it is now time to actually execute some of these practices and achieve results for the American citizens."

Indeed it is. And US-VISIT, for all its flaws, may help show the way.

1 | 2 | 3 | 4