Rethinking RiskBy Dave Lindorff
Case Study: New York Shipping Association and Risk Management
On an average day, a dozen giant freighters ply their way in and out of New York City's busy harbor to load and unload some 150,000 tons of goods, from electronics to automobiles. But Sept. 11 was not an average day, and for the New York Shipping Association, Inc., a cooperative headquartered on the 19th and 20th floors of the World Trade Center, it proved the ultimate test of the nonprofit's eight-year-old disaster recovery plan. In 1993, when terrorists with a truck bomb first tried to destroy the twin towers, NYSAwhich coordinates the work assignments of the city's 2,900 longshoremenwas nearly forced to shut its doors for good: The nonprofit hadn't saved its computer data, some of its employees panicked and momentarily disappeared during the evacuation and some ships languished at their piers, others idled at anchor outside the harbor, and still others were diverted elsewhere, costing local shipping companies millions of dollars.
Not this time. Within 24 hours of the attack on the World Trade Center, all of NYSA's some 160 employees were back on the job in emergency offices in Jersey City, N.J., just across the Hudson River, and in Philadelphia. By Thursday morning, 48 hours after the attack, when the harbor was reopened to commercial shipping, NYSA was once again helping such companies as the Bermuda Container Line Ltd., China Shipping Agency Co., Essex Cement Co. and more than 70 other member organizations load and unload freighters. NYSA is still searching for permanent quarters, but the results so far look good. Says Executive Vice President James Melia: "After the 1993 bombing, it took us five weeks to get back in full operation. After that, we changed the way we do disasters."
How did the association do it? Having a good disaster plan in place certainly contributed much to NYSA's success. But nobody could anticipate the complete collapse of both towers. "I remember sitting in on a discussion this summer on how we'd handle a floor fire," says Kenneth Lepczyk, NYSA's IT manager for applications, who led the association's evacuation and is helping it sprint back into business. "Someone asked, 'What if the whole building blew up?' and I said, 'Yeah, right.'"
Indeed, just as critical, in the view of NYSA executives, was the group's culture: its employees, toughened by their experience in the 1993 attack, and its management, whose single-minded goal was to protect its people first, then worry about the technology. Some companies may let their workers take lunch during a disaster drill, but not NYSA. "We made attendance mandatory and made sure people knew what to do if they ever had to get out fast," Lepczyk says.
The day began routinely enough. NYSA's first shift of workersthose who track ships in the harbor and then determine which workers are needed at which terminalswere already up on the 20th floor of the World Trade Center, working the early shift since 7 a.m. Some 120 more back-office workers, who process benefit checks, vacation pay, pension and welfare payments to longshoremen, were just starting to arrive on the 19th floor to start work at 9 a.m., when Lypczyk felt a sharp jolt.
"I was in someone else's office on the 19th floor on the east side of Tower Two when it happened," he recalls. "I ran around to my office on the north side and saw Tower One engulfed in flames." Lepczyk, the association's highest-ranking executive on the floor at the time, sprang into action on the 19th floor. He called his colleague, IT systems manager Joseph Strcich, working on the floor above, and both began ordering all of NYSA's employees out of the building, turning away those just arriving at their desks. It was the first step in a disaster drill NYSA had run through countless times as part of its regular emergency operation plan. "We had everyone into the stairwell within 90 seconds," Lepczyk says.
Once down in the mezzanine-level lobby, Lepczyk and Strcich ordered the group to ignore World Trade Center security personnel, who were telling people that with debris and bodies falling from Tower One, they should return to their offices and stay there. Lypczyk recalls hearing a voice on the public address system urging people to "Go back up to your offices." As they searched for the safest lobby exit, Lepczyk recalls, "There was another huge explosion and the building shook even harder." It was about 9:15 a.m., and the second plane had just hit Tower Two. "I looked at Joe and said, 'Let's get our people out of herenow.'"
Safe and Sound
Safe and Sound
They found an exit off the mezzanine. Lepczyk peeked out of the door, where a fallen body lay on the pavement. "I looked up and nothing was falling at the moment. I'm a pilot, and I knew that meant we'd have 20 seconds before anything new that started down would hit the ground, so I had everyone run," he says.
He directed them to flee to the Post Office across the streetbecause it had a wall that would offer them some cover from flying debris. He then ordered them to line up, single file, against the wall and edge their way around it, then make their way uptown as fast as possible. By the time the WTC buildings collapsed, less than an hour later, all of NYSA's employees had been evacuated, out of harm's way.
That's when Phase Two of the plan kicked in, and Lepczyk's and Strcich's thoughts turned to restoring the company's data and operations. Once out of the building, Lepczyk realized that he'd forgotten to grab a clipboard and eight critical data CDs from his desk. But they only contained records of the company's work in the hours that Tuesday morning before the attack. As part of NYSA's disaster plan, everything that had been done prior to Sept. 11 had been recorded on backup computer tapes andin accordance with a daily routineshipped every night to an emergency records storage facility in Rosedale, N.Y., run by Boston-based Iron Mountain Inc.
Then, unable to find a cab or use their cell phonesand hampered by attack-related long-distance phone outagesLepczyk and Strcich walked some 40 blocks north to a friend's midtown apartment, where they called Strcich's nephew in his midtown office. His company's Centrex phone system was able to patch Strcich through to New Jersey, to notify NYSA emergency operators there to set aside some desks, chairs, phones and computers for the NYSA staff in two prearranged emergency office locations in Jersey City and Philadelphia, both hosted by SunGard Recovery Services L.P. Then Strcich, still patched into the Centrex line, alerted Iron Mountain to ship the backup tapes down to the Philadelphia facility.
It was only then that both men notified their families that they were safe, on cordless phones being offered by helpful New Yorkers lined up on the streets, hoping to help dust-covered tower escapees as they made their way north from the blast site. Lepczyk and Strcich were finally able to flag down a limo driver to take them further north, to the Tappan Zee Bridge, where their families were waiting to meet them.
Once home, they learned that New York Harbor would be closed to commercial shipping the next day. So they began contacting employees at home by phone, telling them whether they'd be needed Wednesday or Thursdayand to which facility they should report. In some cases, special cars were dispatched to pick up people and take them, amid traffic tie-ups and bridge closings, to the sites. In other cases, due to widespread telephone and cellphone service outages, evacuated executives, once home, drove to some individual employees' homes to convey scheduling information in person and check up on workers' welfare.
Wednesday was spent readying the emergency offices for basic capabilitiesa step also spelled out in the company's IT disaster recovery plan. "Joe and I worked through the night Wednesday, making sure we had all the emergency sets of PCs up and running. There were only 40 onsite, so we had to arrange for 40 more to be drop-shipped into the site at 2 a.m.," Lepczyk recalls. The two, working with SunGard personnel, tested the machines, loaded them with data and hooked them up to the site's LAN lines, ensuring that the Jersey City location would be able to communicate flawlessly with Philadelphia. "By Thursday morning, we were functioning," Lepczyk says.
There was a nervous moment Thursday, the first day of emergency operations, when the Jersey City facility received a bomb threat. "Everyone had to evacuate the building," Lepczyk recalls. "It was a walk down from the 20th floor for all of us, again." But the threat proved to be a prank, and everyone returned to their desks. Lepczyk then assigned various staffers to begin answering a surge of calls from longshoremen worried that their pension data and health and benefit information might have been lost in the blast. By the end of the day, the only complaints were coming from customers back in the harbor concerned that some longshoremen were late showing up to their assigned posts. "We have 12,000 pensioners and widows who receive millions of dollars a year," he says. "No one missed a payment."
Back to Business
Back to Business
By Friday, Phase Three of the disaster recovery plan was launched: a search for new offices and permanent IT equipment and services. Negotiations were started for a temporary site in New Jersey, at a dock facility owned by Maersk Sealand, a Madison, N.J.-based international shipping and terminal operating company. "We'll be on the third floor this time, and it won't be a landmark building," Lepczyk says. NYSA also launched negotiations to find a more permanent site, and all the while, the job of keeping the cargos moving must continue without letup. "Right now we're making arrangements to buy all the new computers and furniture and phones we'll need at the Maersk site," says Lepczyk. "Then we'll need to set up the new communications lines to all our end-users, so we can make the move. Then when we make the move to a permanent location, we'll have to set up new T-1 lines and link them to the temporary site for the transition."
Looking back at the crisis, Lepczyk and Strcich say they wouldn't do much differently, but Strcich acknowledges that he now wants to have additional data backups that will include duplicate copies of the backup tapes he sends every day to Iron Mountainjust in case something happens to the first set. NYSA is also considering altering its disaster plan to include near-site transportation options for employees who would need to leave the vicinity of the buildingand is reviewing the company's cellphone policies to see if the association might equip each worker with a cellphone at the association's expense.
Lepczyk and Strcich also cite a number of lessons for other companies to consider when they review their own disaster plans in the wake of the attack and in the face of the prolonged new threat of terrorism.
First, says Strcich, protect your people. That may sound obvious, but it can't be stressed enough, he says. "You can have all these grand disaster recovery plans, but if you lose your people, they're not good for anything and your company is out of business," he says.
Second, back up data continuously and store it off-site. Strcich says many companies in New York's financial district were crippled by the disaster for days even though their buildings had not been hit and even though they routinely backed up all their data. "The problem was that they didn't store their backup data off-site," he says. When, for security reasons, police and National Guard troops barred entry to the entire financial district for several days, those companies were stuckin much the same way NYSA had been stuck during the 1993 truck bomb attack.
Third, line up in advance an alternate location, complete with compatible computers and phones, including some cellphones and updated contact numbers for employees, customers, partners and vendors. "You've got to have an alternate site where you can go on a moment's notice to keep operating," says Lepczyk. "It doesn't do you any good to have your data tapes and nothing to run them on."
In that respect, larger companies have generally fared better. Ford Motor Company, for example, has a permanent emergency facility located 20 miles from its executive offices in Dearborn, Mich. The company's entire treasury department simply went there and continued operations uninterrupted on Sept. 11 and 12, when there were concerns that terrorists might be planning other attacks. "It's the little companies that have the hardest time," says Lepczyk. "If you were a mom-and-pop pizza place across the street from the World Trade Center, you're out of business." Still, he says, smaller companies that may not be able to afford to maintain a permanent emergency office site can always contract for space at disaster recovery firms like SunGard or Comdisco Inc.
Finally, emergency systems need to be testednot just partial tests, but full-scale runs. Once a year, the association has what it calls "hot-site drills" at its emergency offices. Ironically, though, this year the company postponed its annual full-scale drillscheduled for late August but delayed due to logistics scheduling problems between NYSA and SunGard. "We were actually about to run it when the planes hit the World Trade Center," Lepcyzk recalls. "Turns out our drill this year was the real thing."
Not all companies fared as wellor were as lucky or as prepared as NYSA. According to a CIO Insight poll of 258 IT executives conducted online Sept. 20-24, 27.8 percent said they now plan to back up data more regularly to off-site locations, and 50.5 percent said they also plan to do a better job training staff on what to do in the event of a disaster.
John McArthur, group vice president for storage research at International Data Corp., a research company, says it's hard to know at this point how well most companies' emergency recovery plans worked following the attack. "You don't get a lot of people raising their hands and saying they screwed up when you ask a question like that," McArthur says. Anecdotally, he says, it appears that many companies had problems, but McArthur says he doesn't expect this to lead to a wave of spending on IT emergency budgets. "A disaster like this tests the limits of anyone's disaster preparedness planning," he says. "But few companies can afford to protect against all eventualities. With companies already struggling for profits and market share, how much more can they spend on emergency preparedness?" McArthur suspects some companies will "go out and investigate the idea of having alternate sites on standby, and the answer will come back that most people don't have the money to do that."
For its part, NYSA has spent an estimated $36,000 on the WTC evacuation effort so far, and spends $72,000 annually just to maintain its disaster recovery planand is considering raising that budget to include additional backup tapes, new employee data reporting requirements and, possibly, a range of specific transportation options upon evacuation.
But Lepczyk insists that NYSA can't afford not to consider additional precautions. And now, Lepcyzk's problem is guarding against any over-confidence employees may have in their disaster recovery plans. The reason? "No plan is going to save you," he says. "You have to be calm, but it's dangerous to feel too calm."
Indeed, if anything has changed for NYSA, it's probably a new confidence that the company can handle whatever may be coming next. When employees got word of a bomb threat at their emergency site during the first day of emergency operations, Linda Hutchinson, a project manager who had fled the World Trade Center tower with the rest of the staff, says everyone was calm and collected. "We all figured: We've already seen our office collapse," she says. "What else can they do to us?"
Whatever it is, Lepcyzk says, he hopes NYSA will be ever-ready.
Dave Lindorff is a Philadelphia-area writer who has written for publications including The Atlantic Monthly and BusinessWeek magazine. He is the author of Marketplace Medicine: The Rise of the For-Profit Hospital Chains.
Ready for the Worst
Ready for the Worst
Key decisions that kept NYSA afloatand back on the job within 24 hours:
- Backing up data at the end of each business day and transferring it each day to an offsite storage location.
- Having a well-rehearsed evacuation plan.
- Having an alternate location,arranged for in advance, complete with ample, up-to-date computer and telecom equipment needed to continue daily operations.
- Holding regular disaster drills and dry runs of the company's emergency evacuation plan, which enabled all employees on the day of the disaster to feel confident about ignoring WTC security officials' advice to return to their offices and stay put.
- Empowering local managers to take charge if necessary, with only general guidance about which priorities to consider first when thrown into a crisis.
Charting a New Course
What 258 information technology executives say they'll do differently now.
14.8%: ESTABLISH MULTIPLE GEOGRAPHICALLY-DISPERSED WEB SERVERS
27.8%: BE MORE DILIGENT ABOUT BACKING UP CORPORATE DATA
50.5%: BETTER TRAIN EMPLOYEES IN CRISIS RESPONSE
Disaster Recovery Information Exchange
Disaster Recovery Journal
The Information Systems Audit and Control Association & Foundation
Disaster Recovery Planning:Strategies for Protecting Critical Information Assets
2nd Edition, by Jon William Toigo. Prentice Hall PTR, 2000
Disaster Recovery Planning:For Computers and Communication Resources
by Jon William Toigo. John Wiley & Sons Inc., 1995
A Primer for Disaster Recovery Planning in an IT Environment
by Charlotte J. Hiatt. Idea Group Publishing, 2000