Disaster Recovery 2001By Terry Kirkpatrick
Disaster Recovery 2001
In the wake of the recent terrorist attacks, disaster recovery is on everybody's minds. But judging from the results of a survey of more than 250 top IT executives conducted after Sept. 11, most businesses are already involved in the process to some degree. A significant percentage of respondents already have DR plans, and CIOs are reasonably dedicated to reviewing and testing them. Still, there's work to be done. Many companies' DR activities have significant holes. And some companies still treat certain key areas, such as preventing malicious security breaches, creating redundant systems and lining up backup network providers, as a relatively low priority.
Of the 258 technology executives we surveyedall of them CIOs, CTOs or vice presidents of IS, IT, networking or communications79 percent already have disaster recovery plans in place. That number rises to 84 percent among companies with reported annual revenues greater than $20 million, while 70 percent of their smaller-company counterparts have plans.
Given the importance of the topic, DR budgets aren't exactly fat. About two thirds of companies with annual revenues of more than $20 million reported DR budgets of less than $500,000 per year. However, considering that more than half of such firms reported daily potential losses in excess of $100,000, a healthy DR budget seems a small price to pay to avoid major outages.
Andrew Lippo was at a conference with about 200 other CIOs on a cruise ship off the coast of Atlantic City, N.J., on Sept. 11, when the BBC broadcast a bulletin: An airliner had struck one of the World Trade Center towers.
"My daughter was flying from Boston to New York City that morning, and I thought it was her plane," says the director of MIS at Standex International Corp., a diversified manufacturing company based in Salem, N.H. "I was kind of panicky." He would eventually learn that she was safe, but another passenger got word that a relative was on board one of the ill-fated airliners. All over the ship that day, the CIO Forum's program of meetings and speeches on IT issues yielded to anguished huddles of executives, many of whom knew someone who worked in lower Manhattan.
In Orlando, Robert Kreiger, vice president of information technologies at the Hilton Grand Vacations Co., a unit of Hilton Hotels, was in an IT planning and strategy meeting when the news broke. His first thought was of colleagues at the Millennium Hilton Hotel, which stood in the shadow of the World Trade Center towers. "Frankly, I was thinking more about the people than the systems," he says.
In Detroit, Donald Ledwith, manager of information security and disaster recovery planning for General Motors' North American operations, immediately began planning the relocation of people in GM's corporate headquarters at the Renaissance Center, which features a 77-story tower. "It appeared to be the kind of target the hijackers were aiming for," he says. GM's employees were given the option to evacuate, and they did; an alternate site was designated and equipped with laptops and phones. "Where," Ledwith had to figure out, "would we put people if they were out a second or third day?"
In New York City, in the days following the attack, Larry Tabb, vice president of securities and investment research at the TowerGroup, a research firm that focuses on technology in the financial services industry, talked to CIOs in the financial district. In addition to the myriad technical issues they faced, "they were trying to make sure their people were okay," he says. "They were trying to make sure that even if their people were okay physically, that they were also okay mentallywhich a lot of folks weren't."
As we talked with CIOs who participated in our disaster recovery survey, what emerged as the single largest issue in the aftermath of the terrorist attacks was not systems or technology, but people.
"The technology executives I've had an opportunity to discuss this with are still focused on the human and emotional element of this disaster," says Larry Henderson, senior vice president of operations at SunGard Recovery Services L.P., a Wayne, Pa.-based supplier of disaster recovery services. Soon after the attack, the company was working on 22 disaster declarations. "They're not talking about how to improve their DR plans right now; they're still focused on the human side."
Something fundamental changed for technology executives on Sept. 11: "Even if you get data and systems backed up, the people who know how to run them might be gone," says Robert Enderle, research fellow at Giga Information Group. "People were prepared for some kind of outage, but they weren't prepared for loss of life."
At Standex, Lippo is revising his DR plan. "Our approach is employee awareness. Employees need to know all of the potential threats." As for his own MIS department, "The human equation is the most important piece of disaster recovery. I can find machinery. I have my backups in a fireproof vault at a bank. But I have to pull together multiple people to make this work. That's the issue: You have to identify who the people are and make them aware of their part in recovery."
All of the CIOs we spoke to have some kind of DR plan in place, but they are reviewing them, revising them in some cases, driving harder to wrap up parts that are unfinished and anticipating that their spending might rise. "This hasn't changed what I want to do," says GM's Ledwith, "but I'll probably get more resources to make it happen. People don't seem to respond until something like this happens." His near-term focus: bringing the company's distributed applicationsengineering and operations programsup to the same level of readiness as the mainframe and Web apps.
"In the short term, I think spending may actually go up a little bit," says John Lambeth, CIO of Xerox Connect Inc., a unit of Xerox Corp. that provides IT and knowledge-management systems. "I don't think your average company will cancel ongoing projects. A lot of CIOs may go back to their boards and say they need to carve out a little incremental investment. Non-technical C-level executives will have the awareness right now, so CIOs will be successful in making their point."
Lambeth's own DR plan was well along when the attacks occurred, although he didn't need it that day. "We initially focused on business continuity, looking at things like diesel generators, self-healing networks and the like. Now we're implementing phase two'the building disappears'and we need to go to hot backup or redundant backup. This began unfolding two or three months ago," he says, "but everyone has more energy around it now."
CIOs are now thinking about risk in entirely new and unfamiliar ways. "Corporations here, unlike those in other parts of the world, have not thought about the possibility of targeted destructive action," says Giga's Enderle. This means, for one thing, that a company does not have to have a high profile to be vulnerable. "Those attacks didn't just take out the towers; they took out the surrounding buildings as well. You might be a small company near a government facility or a high-profile brand. You may not be able to get employees in if, for example, a bridge gets hit."
In that sense, no location is immune, and that's part of the new reality every CIO is facing following the attacks. No historical context exists to help comprehend the magnitude of events. Says TowerGroup's Tabb: "I don't think anybody's prepared for what happened, unless they're in Iraq or Afghanistan. I don't think you can prepare for this. You can make as many plans as you want, but what do you do when 6,000 people are missing and lower Manhattan is closed?"
Even the vocabulary of disaster recovery has been affected. "We used to talk about the 'plane dropping through the roof' scenario," says Xerox' Lambeth. "Now, sadly, it's real."
Research results are available in Adobe Acrobat PDF format:
While the great majority of companies have plans for disaster recovery in place, 67 percent of those without a DR plan say they intend to create one. Of the other 33 percent, two thirds have annual revenues of less than $20 million.
When asked to consider their business continuation planning efforts prior to the recent terrorist attacks, 73 percent of respondents said they were very or somewhat prepared. Yet fewer than 30 percent described themselves as being "very prepared." Is "somewhat prepared" good enough to make sure the critical parts of your business will be able to continue to function in the event of a catastrophe? You will have to take the time to determine, in partnership with your executive committee, exactly what it means for your firm to be "very prepared."
Some 22 percent of those answering said that they either had made or planned to make significant changes in their disaster recovery plans as a result of the events of Sept. 11. But most76 percentsaid they had made or planned to make only minor changes or none at all.
Staff training is clearly the greatest missing link in disaster recovery preparations. Just over half of those responding said they will increase their focus on training. The next most important issue was backing up corporate data more frequently, mentioned by 28 percent of respondents.
The great majority of our respondents see their disaster recovery efforts as being focused primarily on their IT departments. IT people are in the lead in sponsoring and managing their DR plans, and relatively few companies involve line-of-business staff and partners in designing and testing such plans at all.
Not surprisingly, the person most frequently cited as being responsible for the management of a disaster recovery plan is the company's CIO or another IT manager, at 74 percent. The CEO and "another senior vice president" were each mentioned by less than 7 percent of those responding, with such roles as the chief operating officer and the chief financial officer trailing behind. When asked about the executive sponsor of the DR program, nearly 61 percent cited the CIO alone, with 14 percent pointing to the CEO.
When it comes to testing Disaster recovery plans, fully 91 percent involve IT employees. But 42 percent involve line-of-business staff, and only 34 percent or less involve anyone outside the company, such as outsourcing partners or key customers.
In 27 percent of the cases, non-IT managers aren't even involved in disaster recovery processes.
There's general agreement on what should be covered in a DR plan. Network outages were the number-one issue for smaller companies, and high on the list for larger companies. This seems to put a premium on reliable networking hardware and software. Natural disasters also ranked high. At the bottom of the list were attacks on company Web sites, employee-initiated outages and service provider failures.
Larger companiesthose with $20 million or more in annual revenueswere more likely than smaller companies to prepare for events such as hardware component failure (89 percent versus 79 percent), natural disasters (91 percent versus 78 percent) and accidental employee-initiated outages (61 percent versus 47 percent). In most other categories, though, smaller and larger companies were similar in their priorities.
The most frequent services included regular offsite data backups and virus detection and protection, at 90 percent and 88 percent, respectively. But the average company had prepared for or planned to prepare for such services as offsite locations, backup network providers and onsite replacement equipment 54 percent of the time.
As for components that are or will be part of their DR plan, at 67 percent, larger businesses were more likely to perform or plan to perform a business impact analysis than smaller firms, at 60 percent. But 63 percent of all companies cite components such as a process for administering the DR plan, setting out what individuals should do in the aftermath of a disaster and recovery strategies.
A majority58 percentsaid they'd reviewed their DR plans within the past quarter, with 19 percent having done so since Sept. 11. But a quarter hadn't reviewed their plans in the past yearor hadn't reviewed their plans at all. Testing, however, appears to be done less often; 19 percent said they tested quarterly or more frequently prior to Sept 11. And 51 percent said they either tested annually or less often, or never tested at all.
The data center is the most frequently tested plan component, cited by 71 percent of those who test. The voice network came in last, at 38 percent, with service providersincluding Internet and application servicesmentioned by 43 percent of respondents.
A few businesses are showing increased interest in testing their DR plans more often than they have in the past. Forty percent reported testing twice a year or more frequently before the events of Sept. 11, and 54 percent say they'll test more often going forward.
The impact of the loss of mission-critical systems naturally varies depending on a company's size. But of companies with more than $20 million in annual revenues, 50 percent reported they'd have losses in excess of $100,000 per day. Twenty percent of larger companies predicted a daily price tag north of $1 million, with a small group2.5 percentpegging potential losses at the breathtaking level of $100 million to $250 million per day.
Also tracking to company size were budgets for IT-related DR services. Sixty-two percent of respondents said that their DR budget is under $100,000 per year, while 5 percent said their budgets topped $5 million or more. And about the same percentage said they had no budget for DR services at all.
What's the most likely problem to occur? Fifty-four percent of CIOs surveyed reported a major hardware component failure within the previous five years, and 46 percent cited a network failure. That's strong support for their reported focus on the kinds of problems they're most intent on avoiding. But most of these can't have been major breaches: Only 33 percent of those answering said their DR plans were activated because of one of these problems.
Most believed they didn't have to worry about being offline for too long. Sixty percent thought they could have their mission-critical systems back up within 24 hours.
Even before the events of Sept. 11, most companies were taking disaster recovery planning seriously. The majority seemed to be following reasonably good planning practices, focusing on major potential sources of problems. Large companies, certainly, have more incentive to plan and test more completely, as well as the resources to do so, but even smaller companies had given at least some thought to the problem. Yet only about 30 percent of the group describe themselves as "very prepared," while 76 percent say they planned to make minor changes, or none at all, to their plans. Thus, there's a potential disconnect in terms of perception. And IT organizations must focus far more on involving partners, suppliers and other members of the organization in any future plan.
How the survey was done: CIO Insight designed the disaster recovery survey in partnership with Survey.com, a San Jose-based supplier of custom online research services. CIOs, chief technology officers and vice presidents of information technology and services from a number of sources, including third-party lists and other Ziff Davis Media publications, were invited to participate in the study by e-mail. The survey was then posted on a password-protected site, and 258 people responded between Sept. 20 and Sept. 24.