Disaster Recovery Plans Fight Chaos

Hurricane Katrina wreaked havoc on communications and data networks across the Gulf Coast region last week, sending IT managers scurrying to implement their disaster recovery plans and patch together makeshift networks to keep their businesses running as best they could.

For the region’s businesses, from metropolitan law firms to the Gulf Coast’s plethora of waterfront casinos, meeting the challenges of continuity and disaster recovery plans was predicated on people and resources being available at multiple locations. The situation was exacerbated by mandatory evacuations.

Shortly before Katrina hit, David Erwin, CIO of New Orleans-based Adams and Reese LLP, took his family to stay with relatives in Dallas.

In the midst of executing his disaster recovery plan, Erwin said a hurricane’s full-potential might was unimaginable to many.

“We always talked about a Category 5 hurricane coming; it was talk for 20 years. Sure enough, it turned out to be more risk versus [reality], but it’s happened to us,” Erwin said.

“The surprise is the extent of the damage and the things people don’t think of. I think a Category 5 was modeled, but I don’t know too many people that actually thought it would be real.”

Read details here about how communications providers are struggling to restore service in the wake of Hurricane Katrina.

Operating from the tallest building in New Orleans, Erwin’s law firm prepared its IT systems for the storm’s onslaught by scheduling backup and moving its tapes to be managed off-site by a third party.

Following the Sept. 11, 2001, terrorist attacks, Adams and Reese decided to help mitigate risk by enlisting the aid of MessageOne Inc.’s EMS (Emergency Mail System). Erwin decided to power down his Hewlett-Packard Co. servers and fail over his e-mail systems to MessageOne’s hosted service after his office’s motion detectors were set off by Katrina’s powerful wind gusts shaking the building. He said his e-mail system was down for only 15 minutes.

“The most important thing, if anything, is you have a way to communicate effectively [in a disaster]. It makes things go a lot smoother because we have people all over the place,” said Erwin.

“Our people are productive, they can communicate, and our clients that can e-mail us have no problem getting to us, and that’s important to continue business as normal as possible.”

The law firm’s employees are dispersed among Baton Rouge, La.; Nashville, Tenn.; Birmingham, Ala.; Houston; Dallas; and New York. Erwin said the EMS service has proved crucial for performing roll call and keeping track of where everyone is at any given time.

Refusing to stop the presses, a New Orleans paper takes to the Web. Click here to read more.

Erwin said he plans to switch his e-mail systems back to the company’s Birmingham alternate site by this week. He’s expecting his Mobile, Ala., office to be reopened in about two weeks. However, the New Orleans office could be closed for two months.

“A lot of people who don’t work in the technology business don’t understand it’s not just power but it’s about water. Our [IT environment] needs water to [cool] the servers. So you actually need both,” said Erwin.

For the time being, the law firm will operate out of its Baton Rouge office, and all technical equipment will be housed in Birmingham.

It is crucial for customers to update their disaster recovery contracts with their vendors whenever a change is made as any incongruity could affect cost, recovery levels or services rendered, said Bob DiLossi, crisis management manager for SunGard Availability Services, in Wayne, Pa., a division of SunGard Data Systems Inc.

“If you make a change to your technology and how you do your backups and how systems are run and you don’t make a [contractual] change with that, you’re out of sync,” said DiLossi. “I’ve seen that happen a couple of times during this event.”

Officials at Harrah’s Entertainment Inc., which operates several casinos in the hardest-hit areas, said the company is still assessing the situation, although its Grand Casino Biloxi and Grand Casino Gulfport appear to have suffered extensive damage. IT resources from the company’s six other properties in Mississippi and Louisiana were being marshaled to assist in the recovery effort, according to a company statement.

Mike Rosenfelt, executive vice president for MessageOne, in Austin, Texas, said his company so far has had 36 clients that have been directly affected or are in the process of recovery from Katrina.

He said people in affected areas have no sense of when they can return to their businesses or when they may become operational again.

“I think that’s something we didn’t see with [hurricanes] Frances, Charley and Dennis,” which ripped through Florida in 2004 and 2005. “I don’t think we’ve even begun to see how devastating this is yet. I think there’s a feeling amongst our customers [that] there are second and third parachutes to drop here, and everyone is frightened to know what’s still to come.”