12-19-14 | Blog Post
After 30 years in the disaster recovery field, Lance Thompson has some good stories to tell. The founder and president of Baseline Data Services shared a few of them – along with plenty more pertinent facts and advice – during a recent installment of the Online Tech “Tuesdays at Two” webinar series.
Listed below are three paraphrased stories from “Lessons Learned From the Disaster Recovery Trenches,” and a quick recap of the messages behind each story. If you’re interested in what you see here, we suggest checking out the full replay of the 52-minute session.
HURRICANE KATRINA
A software partner of Baseline Data Services called to see if Thompson could help a bank client that was devastated by Hurricane Katrina. The bank was not a Baseline client. On the phone was the husband of a bank manager – retired from the Army – who said he was standing on the fourth floor of the bank building, one wall had crumbled and he was overlooking the city of New Orleans and watching boats float through the streets. The bank’s backup tapes were in the trunk of the IT manager’s car, which was washed away in the surge. The tapes – and, sadly, the IT manager himself – were lost to the storm. The bank’s computer system was on the fourth floor, weighed more than 600 pounds and was rendered essentially unmovable given that the steps of the building had been washed away. Baseline engineers walked the man through removing four hard drives; he wrapped them in furnace filters and boxed them up. Shipping them to Baseline was an issue since airports were closed. The owner of the software company that initially contacted Thompson called in some political favors. The package was sent overnight mail and Baseline installed the hard drives in their own computers. The system was up and running the next day and Baseline ran the institution from its data center for six weeks.
Two lessons learned from this story:
First, you can never be over-prepared for disaster recovery. The bank certainly had no way of knowing its backup tapes wouldn’t survive a storm.
Second, the bank had a disaster recovery contract with a national firm that was oversubscribed in the area and could not back up everybody that had subscribed to its services. “You want to be careful that, no matter who you’re getting your disaster recovery agreements with, ask them questions like, ‘How many subscribers do you have in our area? What is your density? What are the ratios?’ They are all fair questions, in my opinion, to try to find out,” Thompson said.
BANK ON AN INVESTMENT
Many years ago, a bank president told Thompson he liked the idea of disaster recovery (as did the FDIC), but wasn’t comfortable paying for it monthly if he there was a chance he’d never use the services. “Here’s what I’d rather do,” he said. “I’m going to put $40,000 into an escrow account. We’ll write an agreement that if we have a disaster, that $40,000 is yours.” Thompson agreed, and said, “I’ll tell you how we’re going to do it. I am going to make sure the equipment, the machines and disk storage and all the things I need for your bank are specifically in my data center for you and I’m going to borrow the money from your bank to pay for that equipment. I’ll start making payments on that loan when you have a disaster and I get the $40,000.” He said, “Well I can’t do that.” I said, “Bingo. Neither can I.”
The message: Quite frankly, there’s a lot of money involved with disaster recovery, whether it’s traditional disaster recovery (recovering and resuming mission critical applications at a remote site in the event of a disaster) or high availability (the ability to withstand all outages to provide continuous processing for all mission-critical applications). When talking to top management and trying to justify the need for disaster recovery, Thompson says his best line is: “How important is it? It’s only your business.”
Break down the business and figure out how much it costs to be shut down per hour. If you want to take the “it won’t happen to me” approach, understand what you’re gambling.
But you also want to make sure you’re spending that budget to the best value possible. If you have data that’s only used in archived situations, archiving that data is much cheaper than keeping it close at hand. Prioritize the data – not just the application – and archive what you can. Keep it rolling so whatever you need to provide disaster recovery failover capability is protected and the rest is archived.
CAN YOU REALLY HANDLE THAT IN-HOUSE?
An East Coast company asked for a disaster recovery proposal, but warned Thompson going into the process that they would most likely handle the project in-house. As foreshadowed, Thompson submitted a proposal but was told the client decided to handle it internally. Thompson asked if he could be of service reviewing the proposed architecture of that internal system. A few days later, Thompson was in a board room with the company president and CFO as the CIO whiteboarded his methodology. Thompson told him it was very solid architecture, but he did have one question: “With your scenario that you’ve written here, which looks really good, who exactly is going to do the recovery with the systems and make sure the software that you’ve designed here all flows through and installs it and recreates it and reboots it and does all the connectivity? Who’s going to do that?” The CIO replied, “Well I am.” Thompson had one more question: “When that tornado rips through town and it clips your house and takes off the roof, you’re going to tell your wife, ‘Honey, I’m going to go to work. I have to get this done. You and the kids are on your own.’?” There was silence in the room for a minute until the president said, “Did you bring the paperwork?”
The message: Staffing is critically important. Do you have the staff to handle a disaster? Typically, larger businesses can do so. “But, please, think through that when a disaster strikes how busy will you be recreating your production environment and will you have time recreating your production environment to run and handle the disaster recovery environment as well,” Thompson said. “Think about how busy you are today. Do you have time to kill? I doubt it. Just think it through: Are you able to handle both situations at once? As much as you’d like to think everything is automated in your own in-house failover environment, it may not be quite as automated as you think and it will take time.”
DISASTER RECOVERY WORKSHOPS
You can hear more from Lance Thompson and other disaster recovery experts at our upcoming half-day workshops scheduled at our Indianapolis Data Center (Thursday, Jan. 22) and our Metro Detroit Data Center (Thursday, Feb. 19). Attendance at both workshops is limited to 50, so register now!
Thompson will be joined by Steven Aiello, a technical architect for Chicago-based IT consulting firm AHEAD, and Online Tech Director of Infrastructure Nick Lumsden. They’ll present the following agenda at both workshops:
RELATED CONTENT
White paper: Disaster Recovery
Webinar: Backup is the ‘sweet spot’ on the data protection spectrum
