What About Next TimeBy Wayne Rash | Posted 09-30-2005
Ships Systems: Surviving the Storm, and the Recovery
Fighting against the ravages of nature and the extremes of weather has become part of the job for Jan Rideout, the vice president and sector CIO for Ships Systems at Northrop Grumman, the company that builds the latest, most powerful destroyers for the U.S. Navy.
Rideout's IT operation has been based at the company's shipyard in Pascagoula, Miss., where she supports that shipyard and another that the company runs in nearby Avondale.
Last year, Rideout found herself preparing for Hurricane Ivana storm that struck the Gulf Coast once, and circled back for another try. She successfully implemented her company's business continuity plan that kept the shipyard operating.
This year, with Hurricane Katrina, things were different. "With this storm we had less time," Rideout said. "It was Friday morning when they started to change the predicted track and said it was going to New Orleans We didn't have a lot of time to react."
Still, Rideout had a plan. "Our prep was pretty much the same as we do for every storm," she said, describing how her staff backed up multiple terabytes of critical data, and then prepared it for transport. "We sent our backups to Dallas. That made a big difference. We used to send them to Mobile," she said.
Just as in the previous year, the backups went to a Northrop Grumman Corp. data center in the Dallas area where they would be safe from the storm that was bearing down on the Gulf Coast. She had to move fast.
"This was very quick," she said. As in the previous storm, a few critical systems were backed up, but kept running in hopes that they would survive the storm. "We left e-mail and Blackberry servers running," Rideout said, "but took everything else down, including the DD(X)."
Then the IT staff had to protect the servers against rain that might come in through a broken window or damaged roof. "We covered all of the servers that were not running with plastic wrap," she said.
Next, the problem of providing power to keep those servers running.
Last year, Rideout's IT staff had rented a standby generator, then switched to the shipyard's internal power during the storm when the rented generator gave out. This year, things were better. "We did have generators this year," Rideout said.
"At Avondale we had the generator installed so it would start automatically," she said. "In Pascagoula we had to do a manual transfer to the generator."
For more coverage of Katrina, Rita and the IT world's recovery,
By then, the Northrop Grumman IT staff had done all it could do to protect the data center against the ravages of Hurricane Katrina. It was time to evacuate to Dallas, hoping that the steps that had worked so well last year would work again.
No such luck.
"When the storm hit it was very bad. Several buildings in the shipyard were destroyed. The IT building was one of those buildings," Rideout said.
She and her staff could only watch the servers they had carefully protected drop off line as the storm surge reached the generators.
"The data center was an outlying building," she said. "Those buildings had an 8-foot waterline inside."
One side of the data center building was completely blown away, and Rideout discovered that things were even worse than she'd initially feared. "We're condemning those buildings."
With the building went all the other assets that the IT department needed to come alive again. "Everything in those buildings was destroyed," Rideout said, describing the devastation where her department used to be. "We lost about 200 servers, the network and telecom infrastructure was destroyed. The public infrastructure wasn't working."
As the winds began to die down, Jan Rideout started the rebuilding effort. "The very first thing we did was send in a person from Dallas," she said. "He arrived the day after the storm and assessed that the data center was really bad. This started several things rolling."
They couldn't use their existing offices, which for all practical purposes didn't exist. But, unlike some companies, they had another option.
"Ship systems business people set up operations on one of the ships we're building," she said, describing the frantic move to a nearly complete Navy destroyer. "It already had some Navy people on it," she said. She noted that while the ship had actually been turned over to the Navy, her staff was welcomed aboard as the sailors helped them adjust to their new, post-Katrina digs.
But there was still the problem of getting the entire IT infrastructure back up and running. "The first thing was to reestablish communications. We couldn't communicate with people, or with each other, so we had to establish some degree of communications." First the IT staff sent in dozens of satellite phones so they could at least talk with each other. Only then could they take the next step.
Rebuilding in Dallas, Mississippi
"We executed on our disaster plan, and started to rebuild our data center in Dallas," Rideout explained.
She said the next step was to replace the destroyed IT assets in Mississippi. "We brought in suppliers to Dallas and started placing orders to rebuild our data center."
She said the suppliers included everyone from server and client makers to telecom companies and companies that made the infrastructure. Everything had to be replaced.
As soon as the first parts of the infrastructure arrived in the Dallas data center, the IT staff got to work. "We brought back the e-mail and Blackberry environments. First for people here [in Dallas] and then everyone," she said.
Next was a critical data system called DDX that is devoted to developing a new type of destroyer for the Navy. The system was due for a major program review within a few days, one that could not be missed for any reason. The government was unwilling to waiteven for a company that had to rebuild after Katrina.
But getting everything underway took more than bringing up a few servers. The entire infrastructure had been destroyed.
"We had to reroute the WAN through Dallas," she said. Even so, it took a week to restore all the systems for the DDX program, a week the team spent finding ways to function even without the IT structure on which it was accustomed to depend.
"They managed to do things without the systems, and they were able to meet everything they needed to meet for the review. It was a closure of the phase we were in," Rideout said, "it was very critical."
Then it was time for everything else. There was no question about moving back to the condemned facilities in Pascagoula because there was nothing there to move back to.
"We brought upa new data center in Dallas," Rideout said. Before the storm, Northrop Grumman had started a data center consolidation effort. The new, bigger data centers would be located in Pascagoula and New Orleans. Not now.
"We're staying in Dallas," she said.
Rideout sad she realized, after executing emergency plans and having to evacuate two years in a row, that staying on the coast was simply too risky. "We already have our mainframe in Dallas, and it's as the only data center where we had enough floor space for the 200 servers. We won't move it back now," Rideout said.
Still, there had to be IT support in Mississippi, so there was work to be done there as well. "The public infrastructure was pretty bad. It was restored in about two weeks," she said.
For more coverage of Katrina, Rita, and the IT world's recovery,
Rideout added that there needed to be more bandwidth available between Mississippi and the new data center in Dallas now that the servers were being relocated. Fortunately, much of the new bandwidth arrived last week. "Bell South was able to put something together that would normally take a lot longer," she said.
But the missing infrastructure continued to vex restoration efforts. "We also have issues with local network connectivitythe infrastructure that was in the yard. We have to move to leased buildings in the community that don't have network infrastructure," she explained, but added that there was a silver lining to this particular cloud.
"We're taking advantage of the wireless implementation plan that we had prior to the storm. We were going to outfit the yard with wireless infrastructure over three years. Now we're bringing it back immediately with wireless," Rideout said, describing her plan to implement Wi-Fi throughout the shipyard. "We've had to replace approximately 2,100 computers, and they're all coming in with Wi-Fi cards."
Some systems were recoverable, and some functioned with some manual help; but damage to the power grid, which was submerged, kept recovery efforts from closing in many gaps.
Rideout said the shipyard is well on its way to a full recovery, but there's still a long way to go. "We're only about 50 percent complete," she said.
What About Next Time
Now that she's had to lead the evacuation of her IT staff and her data center twice in as many years, and had to rebuild it once, what would Rideout do differently?
"There's no doubt that having a hot site where we could just flip the switch would have been beneficial in this situation," she said. Until Katrina, it was difficult to make a case for such a hot site, but now Rideout thinks that will be revisited.
"Having some kind of communication devices more readily available" is next on Rideout's list. "Maybe satellite phones."
And Rideout said she thinks that the human aspect needs more thought and better planning.
For more coverage of Katrina, Rita, and the IT world's recovery,
One of Rideout's employees died during the disaster, and a quarter of the employees remain homeless. "When you think of all they've done in the face of such personal loss it is an amazing thing," she said, recounting the terrible price some of her staff paid. "We need to make sure we take care of these people."
"We have over 50 people in Dallas to bring back the servers and operations," Rideout said, thinking of her staff. "Some of those people have homes that were wiped out here. It's an amazing human story."
And taking care of her people is tops on Rideout's agenda.
She will not require her employees to relocate when their hearts and families are still in Mississippi. "No one will lose their jobs because of this. But there are people who would like to move to Dallas, and we will make that option available," she said.
Either way, she will accommodate the needs and desires of her staff. "They won't need to relocate. It's a nice thing about technology and virtualness," she said.