Buncefield – the site of this winter’s huge oil refinery fire near Hemel Hempstead – was, like all disasters, completely unexpected. But this is the nature of the world we live in, where global warming, international unease and good old human stupidity can swiftly wreak havoc on all plans, personal and organisational.
In response to these facts of life, IT managers have either quietly warmed up what in many cases are very long-standing disaster recovery contracts or looked to what the industry has labelled business continuity.
The latter’s philosophy is that technology should be linked with business processes, so if something disruptive occurs, the system can, like the internet, route around the problem and carry on as if nothing has happened.
This is said to be achievable by a combination of modern comms, storage and disk mirroring techniques. But this only works as part of a strategy bedded into the business as a whole, where all participants take responsibility for getting the company back up and running. It is proactive as opposed to reactive, and can mean the difference between no time lost at all and several hours to get the disaster recovery site online.
The concept is elaborated on by Neal Watkings, treasurer of the Storage Networking Industry Association (SNIA), a group composed of large users of storage and some suppliers. “Disaster recovery is really a technology-based recovery path, while business continuity is about people and processes. In fact, disaster recovery should just be a component of the business continuity programme. It can’t only be seen as administrative conformance, it has to be constantly evaluated and tested to keep it alive or the organisation can easily lose sight of it – a policy that can have terrible consequences if disaster does strike.”
But the users MIS UK talked to about coping with disasters showed a range of reactions to the business continuity message.
"Disaster recovery is really a technology-based recovery path, while business continuity is about people and processes"
Neal Watkings, treasurer, Storage Networking Industry Association
Many, for instance, acknowledge the value of business continuity as a goal, but either see it as a target they are still working towards or a by-product of their more pragmatic disaster recovery or back-up and restore policies.
For instance, Richard Bates, network and communications manager at Warwick District Council, says the council has identified a need for a disaster recovery plan this financial year. He has installed a new SAN-based replication back-up system for his HP kit, a move he says has protected the environment from one possible serious issue, local flooding. But, he says, it is just a step along the path: “We are definitely not done yet, and the next stage is looking at fibre for more protection.”
Lance Edwards is head of IT at GB Airways, an independent airline servicing southern Europe and north Africa as part of the BA system, taking around 2.5 million passengers a year. “Our issue was over questions about the reliability of our existing software,” he says. “We were extensive tape users but there are problems with that technology – data can be easily corrupted by daft things like knocking the tape or leaving it in a less than ideal environment.
“We also had a 20-slot tape library which would have to be upgraded – we were backing up 350Gb a night and we were genuinely worried that soon there might not be enough hours in the night to do this.” Edwards also began to weary of the disciplines around tape management, such as scheduling and incremental versus full back-ups.
Aiming for complete security
Even so he has not totally dropped tape, just adopted an extra element of online back-up too. “Using a SAN-based back-up approach instead means that we can restore lost data to any location using the internet, get 26 servers backed up on a nightly basis and as it’s distributed we get real peace of mind too.”
That said, again the job is not totally done; there are still “issues to be resolved before we can be completely secure,” Edwards says. Certainly electronic alternatives to tape as a back-up option seems to be winning hearts and minds.
Another customer who has reached the end of his tape tolerance is Michael Boscic, IT manager at 125-strong firm of solicitors, Fisher Meredith. “As we grew from one to five servers, the amount of back-up work we had to do just started getting ridiculous,” he says.
“I had one too many headaches about ‘did I put the right tape in this time’ and all that. So as part of a move to implement a disaster recovery plan we have put in a comprehensive electronic back-up and off-site mirroring facility. Now if one of the servers breaks down we are fine, if the building blows up we can have machines with the same operating software and state of data as the night before, ready for people to start work with next morning. It’s a big relief.”
"The best you can do is take all reasonable steps and keep on upgrading and testing on a permanent basis"
Mark Smith, head of ICT, Great Ormond Street Hospital
But some managers say they are definitely further toward business continuity than disaster recovery these days.
Mark Smith, head of ICT at Great Ormond Street Hospital in London, says he has solved some availability and confidentiality issues around the electronic patient records system.
“We’ve had a disaster recovery contract with SunGard for six years, but really that’s the last link in the chain in a highly business continuity-friendly structure which has been our game ‹ plan for quite some time.” This is delivered by a combination of clusters and SAN that he thinks offer resilience all the way along.
“It’s all about making sure, about having both belt and braces. You can never say all your business continuity work is done, just as you don’t know what disaster could be waiting round the corner. So the best you can do is to take all reasonable steps you can and keep on upgrading and testing on a permanent basis.”
Predicting the risks
Another executive who has very much switched on to the business continuity message is Stephen Segel, finance director at Blue Group International, a £50 million turnover recruitment company in the healthcare and technology markets.
“Business continuity really is the prediction of the potential risks you might face. That could be around either processes or systems, and it’s our job to plan for those. The issue of course, is to what level to pitch the response – it has to be appropriate, as it’s very cheap to do nothing but equally very expensive to do too much.”
He says he has also installed overnight electronic back-ups to help service both his front and back-office systems.
"Lights on is all about back-ups and that sort of thing. Lights off is more complicated, it’s about covering for disasters like power failure"
Stephen Segel, finance director, Blue Group International
“The business continuity ethic has to be either lights on or lights off,” he thinks. “Lights on is all about back-ups and that sort of thing. Lights off is more complicated, it’s about covering for disasters like power failure, having an off-site mirror location you can go to, being able to upload your systems there and be ready to work with essentially the same system as soon as you and your staff arrive.”
Of course, some organisations feel they cannot afford not to have anything like disaster recovery, business continuity or whatever it will be called next year. Take PayPoint, a UK payment network used for mobile phone credits or bill payments as well as electricity and gas bills, which also has a web presence for doing the same.
Ready for any disaster
The value of transactions going through its system is fairly impressive – £2.9 billion, of which around £1bn is being done in cyberspace. To protect this, its service delivery manager Mike Holden has brought in a business continuity infrastructure based around Stratus hardware, a move he believes “gives us the resilience we need to cope with any disasters we are likely to face”.
Thus the reality of IT disaster proofing in 2006 is that many aim for business continuity perfection but are getting there by slow and steady increments, adapting technology and processes as appropriate.
That sounds like a good plan – until next week, when that nice local oil refinery next door blows up while you are still in the middle of your back-up evolution.