For some IT users caught in the direct path of Hurricane Katrina, lack of time and lack of power were two big roadblocks on the path to recovery.
"The hurricane crept up on us so quickly," said Venkata Mahadevan, systems manager for the computer science department at the University of New Orleans. "It's not possible to move giant racks of servers, etc., to other locations without major resources, which we don't have. Even if the servers could be moved, where could they be set up again? This boils down to a lack of resources."
Like Mahadevan, most system administrators are not trained to effectively deal with such a crisis. He said the best course of action is usually to shut systems down before the storm and set up temporary sites and Web portals.
Although he has all his department's data on backup tapes, with which he evacuated, Mahadevan finds himself lacking servers, a tape drive, and a data center or remote site to begin the rebuilding process. Still, he must formulate a plan to recover when the opportunity eventually comes around.
"I expect that the first thing we will do when we get back to campus, which is not accessible right now but remained mostly dry, would be to come up with a more effective hurricane/ emergency response plan for our department," Mahadevan said in an e-mail message. "Hindsight is always 20/ 20, but this may include purchasing a generator and installing it on the rooftop of the building and establishing a replicated computing infrastructure at a site outside of New Orleans that mirrors the original data center."
For many people charged with running IT systems, Katrina proved the ultimate testing ground for disaster recovery and business continuity plans. Tim Babco, senior director of IT for SCP Pool Corp., in Covington, La., said his company designed a core set of disaster recovery documentation several years ago to mitigate hurricane risk. Babco said Katrina will force future revisions.
Three years ago, SCP Pool, a large pool-supplies wholesaler, selected what is now a Vericenter Inc. facility in Dallas as its primary data center for hosting the company's mission-critical, AIX-based ERP (enterprise resource planning) systems. SCP Pool's second-tier applications, such as its human resources, benefits and payroll software, typically reside in Covington.
To safeguard its IT systems from Katrina, SCP Pool relocated the staff necessary for shifting its call center, support operations and phone numbers from Covington to Dallas. "My biggest challenge was two things: get my staff and support capabilities to Dallas, and, two, move [secondary] and tertiary systems to Dallas," said Babco. "Any [disaster recovery] plan has to identify priority. Along with priority, how long can something stay out and unavailable and not impact the business?"
Babco employed what he called a makeshift "Noah's Ark" strategy in redeploying IT staff to Dallas, choosing a handful of experienced staffersincluding an AIX administrator, a network administrator, a Microsoft Windows administrator, a help desk agent and a software specialistto be sent out in a first phase, followed by more staff in later phases.
"By Monday morning, when the hurricane was hitting, I had a core IT staff already positioned in Dallas, with all of our IT help desk calls already forwarding to us. Therefore, all 200 [global SCP Pool] locations needing IT services were uninterrupted," Babco said.
Although many areas near Covington are still without power and there is limited connectivity at SCP Pool's headquarters, Babco said his team will leave the Vericenter facility this week.
Many businesses without the luxury of leaving areas damaged by Katrina must rely on more localized technology offerings for their needs. For instance, HP has shipped two of its mobile trailers equipped with the amenities of the company's data recovery centers.
According to HP's Wilson, a New Orleans bank is using one trailer as a temporary operations location, featuring 100 seats, tellers and even a drive-through window open for business. She said the mobile units most in demand are those with satellite communications capabilities. "Given what's happened in New Orleans and [that] all phone lines are down, it would behoove us to make the investment there and add satellite [phones] to all of them," Wilson said.
The hurricane's unexpected fury and aftermath have left many IT userssome of whom had left in advance with only a few days' worth of clothing or personal itemsforced to balance moving ahead with their personal lives with keeping their companies' technology and business needs uninterrupted.
"People in New Orleans and affected areas, they're worried about where they're going to get their eyeglasses prescription or where their kids are going to go to school. The last thing they want to worry about is [IT]," said Agility's Boyd.