This Article is About
air conditioning units
watching the news
internet connectivity
valuables
computer room
ups
Plan For Everything, “the Worst Can Still Happen”
Join 1000's of Authors at StreetArticles Today!

Plan for Everything, “the Worst Can Still Happen”

Watching the news recently reminded me that whatever you do, life still bites you in the bottom when you least expect it. A village in England had been flooded after a local river burst its banks. Properties flooded and more misery for those that live there. However, the notable fact was not the flooding, this is normal and had sadly happened there many times. The fact was that after years of protesting by the residents, the authorities finally installed flood defences and flooding was supposed to be a thing of the past. So this time when the river rose, instead of moving furniture and valuables to safety local residents continued with their lives as normal. Why shouldn’t they, a multi-million pound pump had been installed to prevent flooding. The river flooded but the pump did not start. So whilst everyone slept in their beds thinking they were safe, no-one knew they would discover that they were worse off than ever.

Most sensible businesses these days especially the large ones plan for the worst. This includes fire, theft, flooding, power issues, telephone problems, internet connectivity, terrorism and disease etc. I have been involved with a few which to one degree or another have spent large sums of money and time on chasing “an ideal” which is never achievable.

Back when the internet was just emerging from a laboratory somewhere, my employer decided to build a purpose designed computer room. At the time the company had more money than sense so the IT department were given a blank cheque to do it. So a room was built on the first floor (no flooding), with special fire retardant materials and fire suppression systems (no fire), multiple air-conditioning units (make sure computers are nice and cosy) and special security to keep everyone out. To make sure there were no power problems two enormous UPS (uninterruptible power supply) systems were installed to make sure electricity would not be a problem. Each had its own three phase power supply and batteries which would not only keep all the “big” servers running for a few hours but also the telephone system, lights on in various parts of the building, pc’s and terminals in sales and customer service departments, so that if the worst happened business would continue albeit at a reduced level. Of course we also had a big diesel generator which would automatically start and provide electricity to the UPS systems if there was a power failure.

One day I sat in my office working and all of a sudden the lights went out and my computer screen went blank. “Strange I thought, that’s not supposed to happen”. I was using a protected system. It wasn’t long before myself and colleagues realised we had a big problem. A quick inspection of the computer room revealed complete silence and control panels completely blank on everything including the main UPS and its twin sister. A colleague was despatched to the basement car park and soon they revealed the generator was as quiet as a mouse too. Soon we were all gathered together trying to work out what had happened, lots of “that is impossible” and “it shouldn’t happen” conversations. Looking out the window revealed all the other buildings nearby had power.

A few minutes later word arrived from the warehouse that an ambulance was needed. A workman had been electrocuted and blown off his feet. Amazingly he was ok but needed checking over in hospital. The request was for a phone that worked, we had no telephone system at that time! We soon realised that there was a connection. He had been drilling a hole through the main riser wall where the electric cables were. Bad luck meant that the drill (a big industrial one) was inserted between both cables and cut through the cable casings and shorted them both.

It transpired that when the UPS experienced this event it did not know what to do and decided to shutdown completely. A short between the power supply and its backup had not been considered by the manufacturer. As a result everything else shutdown too. Nobody considered the “workman with a big drill what-if” possibility, and therein is the problem with disaster planning and recovery…..

Of course we all should do what we can to protect our businesses but the day you think that you have all bases covered that is probably the time to be worried! So, implement your disaster recovery and prevention procedures but never go to bed feeling overly confident, you might wake up with wet feet.


Street Talk

Love the feedback, very encouraging. More articles will follow

Reply
  about 9 years ago

Hi Rob, This is a very interesting article. Also very informative. Thank you for the great job.

Reply
  about 9 years ago
You May Also Like
Server Hosting Providers, Please Think About Distributed Risk
For the last 15 years I have worked with many clients and their hosting providers. Sometimes clients have a server, sometimes just a website hosting account and often a combination of various solutions from multiple providers. In that time I have not found any company which you could describe as…
By:  in  Business  >  Continuity Disaster Recovery   Dec 09, 2012  
2
  Likes: 0

Article Views: 1305    Report this Article