Disaster Recovery: Learning From Painful Experience
Sometimes the only way you know whether your disaster recovery plan is any good is when you face a real disaster. After 2008's Hurricane Ike hit, insurance wholesaler Myron Steves & Co. had a difficult time getting its systems back on track just when customers needed the company's help the most. That motivated its IT team to set the stage for a faster, cheaper and more robust DR capability.
04/23/12 5:00 AM PT
When Hurricane Ike struck Texas in 2008, it became the second-costliest hurricane ever to make landfall in the U.S. It was also a wake-up call for Houston-based insurance wholesaler Myron Steves & Co., which was not struck directly but nonetheless realized its IT disaster recovery (DR) approach was woefully inadequate.
Supporting some 3,000 independent insurance agencies in the Gulf Coast region, with many insured properties in that active hurricane zone, Myron Steves must have all it resources up and available, if and when severe storms strike.
The next BriefingsDirect discussion then centers on how Myron Steves, a small- to medium-sized business (SMB), developed and implemented a modern disaster recovery and business continuity strategy based on a high degree of server and client virtualization.
Learn how Tim Moudry, associate director of IT; and William Chambers, IT operations manager, both at Myron Steves, made a bold choice to go essentially 100 percent server virtualized in 90 days. That then set the stage for a faster, cheaper and more robust DR capability. It also helped them improve their desktop-virtualization delivery, another important aspect of maintaining constant availability no mater what.
The discussion is moderated by Dana Gardner, principal analyst at Interarbor Solutions.
Listen to the podcast (31:45 minutes).
Here are some excerpts:
Tim Moudry: When Hurricane Ike came, we were using another DR support company, and they gave us facilities to recover our data. They were also doing our backups.
We went to that site to recover systems, and we had a hard time recovering anything. We were testing it, and it was really cumbersome. We tried to get servers up and running. We stayed there to recover one whole day and never got even a data center recovered.
So William and I were chatting and thinking that there's got to be a better way. That's when we started testing a lot of the other virtualization software. We came to VMware, and it was just so easy to deploy.
We made a proposal to our executive committee, and it was an easy sell. We did the whole project for the price of one year of our old DR system.
Dana Gardner: William, what were your top concerns about change?
William Chambers: Our top concerns were just avoiding what happened during Ike. In the building we're in in Houston, we were without power for about a week. So that was the number one cause for virtualization.
Number two was just the amount of hardware. Somebody actually called us and said, "Can you take these servers somewhere else and plug them in and make them run?" Our response was no.
That was the lead into virtualization. If we wanted everything to be mobile like that, we had to go with a different route.
Then, once you get into virtualization, you think, "Well, OK, this is going to make us mobile, and we'll be able to recover somewhere else quicker," but then you start seeing other features that you can use that would benefit what you are doing at smaller physical size. It's just the mobility of the data itself, if you've got storage in place that will do it for you. Recovery times were cut down to nothing.
There was ease of backups, everything that you have to do on a daily maintenance schedule. It just made everything simpler to manage, faster to manage, and so on.
Gardner: And so for you as an SMB with 200 employees, what requirements were involved? You obviously don't have unlimited resources and you don't have a huge IT staff.
Chambers: It's probably what any other IT shop wants. They want stability, up-time, manageability, and flexibility. That's what any IT shop would want, but we're a small shop. So we had to do that with fewer resources than some of the bigger Exxons and stuff like that.
Moudry: And it can't cost an arm and leg either. We're an insurance broker. We're not a carrier. We are between the carriers and agents. With our people being on the phone, up-time is essential, because they're on the phone quoting all the time. That means if we can't answer our phones, the insurance agent down the street is going to go pick up the phone, and they're going to get the business somewhere else.
Also, we do have claims. We don't process all claims, but we do some claims, mainly for our stuff that's on the coast. After a hurricane, that's when people are going to want that.
We have to be up all the time. When a disaster strikes, they are going to say, "I need to get my policy," and then they are going to want to go to our website to download that policy, and we have to be up.
Gardner: Why did you go 100 percent virtualized in such a short time?
Chambers: We did that because we've got applications running on our servers, things like rating applications, emails, our core applications. A while back, we separated the data volumes from the physical server itself. So the data volume is stored on a storage area network (SAN) that we get through an iSCSI.
That made it so easy for us to do a physical-to-virtual (P2V) conversion on the physical server. Then in the evenings, during our maintenance period, we shut that physical server down and brought up the virtual connected to the SAN one, and we were good. That's how we got through it so quickly.
Moudry: William moved us to VMware first, and then after we saw how VMware worked so well, we tried out VMware View and it was just a no-brainer, because of the issues that we had before with Citrix and because of the way Citrix works. One session affects all the others. That's where VMware shines, because everybody is on their independent session.
Gardner: Where are your data centers?
Moudry: Right now it's Houston and San Antonio, but we are moving all of our equipment to colos, and we are going to be in Phoenix and Houston.
Gardner: So that's even another layer of protection, wider geographic spread, and just reducing your risk in general. Let's take a moment and look at what you've done and see in a bit more detail what it's gotten for you. Return on investment (ROI), do you have any sense, having gone through this, what you are doing now that perhaps covered the cost of doing it in the first place?
Moudry: We spent about $350,000 a year in our past DR solution. We didn't renew that, and the VMware DR paid for itself in the year.
We're working with automation. We're getting less of a footprint for our employees. You just don't hire as many.
And we are not buying equipment like we used to. We had 70 servers and four racks. It compressed down to one rack. How many blades are we running, William?
Chambers: We're running 12 blades, and the per-year maintenance cost on every server that we had compared to what we have now is 10 percent now of what it was.