Ships Systems: Surviving the Storm, and the Recovery

Fighting against the ravages of nature and the extremes of weather has become part of the job for Jan Rideout, the vice president and sector CIO for Ships Systems at Northrop Grumman, the company that builds the latest, most powerful destroyers for the U.S. Navy.

Rideout’s IT operation has been based at the company’s shipyard in Pascagoula, Miss., where she supports that shipyard and another that the company runs in nearby Avondale.

Last year, Rideout found herself preparing for Hurricane Ivan—a storm that struck the Gulf Coast once, and circled back for another try. She successfully implemented her company’s business continuity plan that kept the shipyard operating.

This year, with Hurricane Katrina, things were different. “With this storm we had less time,” Rideout said. “It was Friday morning when they started to change the predicted track and said it was going to New Orleans … We didn’t have a lot of time to react.”

Still, Rideout had a plan. “Our prep was pretty much the same as we do for every storm,” she said, describing how her staff backed up multiple terabytes of critical data, and then prepared it for transport. “We sent our backups to Dallas. That made a big difference. We used to send them to Mobile,” she said.

Just as in the previous year, the backups went to a Northrop Grumman Corp. data center in the Dallas area where they would be safe from the storm that was bearing down on the Gulf Coast. She had to move fast.

“This was very quick,” she said. As in the previous storm, a few critical systems were backed up, but kept running in hopes that they would survive the storm. “We left e-mail and Blackberry servers running,” Rideout said, “but took everything else down, including the DD(X).”

Then the IT staff had to protect the servers against rain that might come in through a broken window or damaged roof. “We covered all of the servers that were not running with plastic wrap,” she said.

Next, the problem of providing power to keep those servers running.

Last year, Rideout’s IT staff had rented a standby generator, then switched to the shipyard’s internal power during the storm when the rented generator gave out. This year, things were better. “We did have generators this year,” Rideout said.

“At Avondale we had the generator installed so it would start automatically,” she said. “In Pascagoula we had to do a manual transfer to the generator.”

For more coverage of Katrina, Rita and the IT world’s recovery,

By then, the Northrop Grumman IT staff had done all it could do to protect the data center against the ravages of Hurricane Katrina. It was time to evacuate to Dallas, hoping that the steps that had worked so well last year would work again.

No such luck.

“When the storm hit it was very bad. Several buildings in the shipyard were destroyed. The IT building was one of those buildings,” Rideout said.

She and her staff could only watch the servers they had carefully protected drop off line as the storm surge reached the generators.

“The data center was an outlying building,” she said. “Those buildings had an 8-foot waterline inside.”

One side of the data center building was completely blown away, and Rideout discovered that things were even worse than she’d initially feared. “We’re condemning those buildings.”

With the building went all the other assets that the IT department needed to come alive again. “Everything in those buildings was destroyed,” Rideout said, describing the devastation where her department used to be. “We lost about 200 servers, the network and telecom infrastructure was destroyed. The public infrastructure wasn’t working.”

As the winds began to die down, Jan Rideout started the rebuilding effort. “The very first thing we did was send in a person from Dallas,” she said. “He arrived the day after the storm and assessed that the data center was really bad. This started several things rolling.”

They couldn’t use their existing offices, which for all practical purposes didn’t exist. But, unlike some companies, they had another option.

“Ship systems business people set up operations on one of the ships we’re building,” she said, describing the frantic move to a nearly complete Navy destroyer. “It already had some Navy people on it,” she said. She noted that while the ship had actually been turned over to the Navy, her staff was welcomed aboard as the sailors helped them adjust to their new, post-Katrina digs.

But there was still the problem of getting the entire IT infrastructure back up and running. “The first thing was to reestablish communications. We couldn’t communicate with people, or with each other, so we had to establish some degree of communications.” First the IT staff sent in dozens of satellite phones so they could at least talk with each other. Only then could they take the next step.

Next page: Rebuilding in Dallas and Mississippi.

Get the Free Newsletter!

Subscribe to Daily Tech Insider for top news, trends, and analysis.

Latest Articles