Lost power, lost business: How to avoid this overlooked tech risk
One sometimes overlooked element of an organisation’s technology infrastructure is something a non-technology person can understand: electricity.
No matter how fast a company’s computer systems work, and no matter how new its servers may be, a company is grounded if it doesn’t have a plan that accounts for backup power in the event of an outage, and regular testing of backups.
That was the case recently at major US-based airlines Southwest and Delta. In July, Southwest cancelled 2,300 flights in a four-day period when a failed router at the company’s data center in Dallas did not “fail over” to a backup.
Delta suffered a similar issue in the second week of August, when, according to the company, a “massive failure” at Delta’s Technology Command Center resulted in critical systems and network equipment failing to switch over to backups. The repercussions of an outage of a few hours: passengers in airports around the world were stranded, Delta’s CEO apologised, the airline waived rebooking fees, and it offered a $200 travel voucher to customers who experienced cancellations or delays of more than three hours. It took several days for Delta’s flight schedules to return to normal.
A major airline could be more adversely affected by a system outage than, say, an independent, brick-and-mortar store, although a big company is likely better equipped to absorb a financial hit. But the lessons of the airline outages are universal. Contingency plans for power failures should be part of an organisation’s technology and risk-management discussions, where CFOs often take the lead.
“It’s not actually a technical problem; it’s a fundamental risk-analysis problem,” said Robin Hirsch, FCMA, CGMA, managing director of UK consultancy Kingdom Technology Partners. “The mitigations will often be technical, but the due diligence in terms of ensuring that you actually quantify your risks accurately and undertake some type of mitigation is not a technical issue.”
Here are three tips for companies seeking to strengthen their technology infrastructure:
Engage in scenario planning. Companies should have discussions that lay out scenarios for loss of power and the impact of such losses. Torpey White, CPA/CITP, CGMA, a partner at Wipfli LLP’s risk advisory and forensic services practice, recommended starting with this question: “If we were without power for two hours, who is affected, and would we lose any revenue?” Then ask the same question for a four-hour outage, then six, etc. “The answers will be very different the longer you go,” he said.
CFOs’ roles can include oversight of information technology, which means finance chiefs need to be able to speak the language of IT, or at least know the most important concepts. Combining that knowledge with strategic thinking and cost allocation helps an organisation prepare for an outage scenario.
“The IT group isn’t necessarily figuring out dollars and cents,” said Byron Patrick, CPA/CITP, CGMA, managing director, CPA practice at Network Alliance in the Washington, D.C., area. “The discussion has to start from the CFO perspective of what the cost would be if there was a catastrophic failure.”
Determine the best fit. A multimillion-dollar backup system is not for everyone; some businesses can withstand an occasional loss of power. But paying $5 million to $10 million, Southwest’s initial cost estimate, is not an option for smaller businesses, either. “Even a few thousand dollars to a small business can be detrimental to next week’s payroll,” Patrick said.
That’s why an organisation’s leaders must determine how much they want to spend for peace of mind. White pointed out that a smaller entity that pays $1,000 a month to a third-party provider to ensure a fast response to a loss of power might not suffer a power loss. But if that $12,000 service saves a company $75,000 in potential lost revenue when, say, its e-commerce site is down because of a power loss, that’s money well-spent.
And, as companies outsource more of their technology infrastructure to data centres – providers that specialise in keeping systems cool and data secure – they can take a hybrid approach to such outsourcing. For example, an organisation might manage some data in house but run its email through a data centre that can guarantee close to 100% “uptime” – meaning it has backups in place in case of a power loss. CFOs, Patrick said, should work with IT and other departments to prioritise an organisation’s most critical systems and the associated costs.
Trust, but verify. Whether your company’s systems run in company-owned physical space or in a third-party location, testing should be done, White said. At least once a year, organisations should ensure their backup power systems work as designed. The more critical the specific system, the more regular the testing. And third-party providers should be required to provide evidence of test results.
Related CGMA Magazine content:
“Expectations and Reality Vastly Different for IT Departments”: IT departments with a solid infrastructure are seen as more strategic and give companies a better chance at getting applications to market ahead of competitors.
“Retailers Struggle to Capitalise on Omni-Channel Opportunity”: Providing consumers with a seamless shopping experience across a variety of channels gives retailers a golden opportunity to differentiate their brand, but technical, operational, and organisational barriers have prevented many from achieving this goal, posing a threat to their long-term sustainability.
—Neil Amato (namato@aicpa.org) is a CGMA Magazine senior editor.