Most crises are avoidable.
Around ten years ago, my boss and I used to carry home back-up tapes of our server data. That way, if the office burnt down over night we’d still have two copies of the back-up. While fortunately systems for backing up data have moved on, the basic premise still stands.
You can’t always prevent something from catching fire. You can control your level of preparation and your response.
British Airways and the half-term meltdown
Saturday 27th
Bank holiday weekend, just as thousands of families were heading to the airport for the half-term break, British Airways’ computer system crashed.
People shouldn’t still be talking about this, but they are because it wasn’t just a quick glitch while BA’s systems kicked-over to back-up servers.
It affected more than 1000 flights over the bank holiday weekend.
It ruined holidays and weddings. People separated from their luggage found that they couldn’t access vital medication.
Enter the CEO
That same day, BA released its first video of the CEO addressing the ongoing crisis. He apologised, and mentioned that they thought the computer system crashing was caused by a power issue.
Comments were disabled on YouTube, but BA also posted it to Facebook, where angry passengers responded with criticisms, questions and complaints.
Many addressed towards the CEO and the video.
It also posted the video to Twitter, with similar results.
— British Airways (@British_Airways) May 27, 2017
Sunday 28th May
Aviation analysts were commenting on the issues, with one predicting that the consequences of the crash would be felt for up to two weeks – as planes, crew and luggage needed to be routed to the right place.
Another aviation expert – Julian Bray – highlighted the role of the CEO. He told the Mail Online that Alex Cruz (who has been CEO for just over a year) has a reputation as an “outsourcer and cost-cutter” and that he was “not particularly suited to an upmarket, people-intensive, luxury-price brand.”
Meanwhile, the Telegraph reported that hotels near Heathrow were charging stranded customers between £1000 and £2500 per night for a room.
It also highlighted BA’s “unreliable new IT system” that had suffered a worldwide crash for the sixth time this year.
CEO video #2
A second video came the next day.
Although there was another apology, and it included the CEO reciting some of the issues BA customers were facing, many people remained angry.
Again, comments were banned on YouTube, but BA posted the video to Facebook and Twitter.
— British Airways (@British_Airways) May 28, 2017
Monday 29th
A whistleblower spoke to thesun.co.uk – saying that there were known issues with the computer system, and that management refused to fix the problems.
“We started using the new system in October. Training aside, the whole thing has been a disaster.
“The Chief Executive Alex Cruz, when he was warned about the system told us that it was the staff’s fault not the system.
“We are making more money than ever and all this has happened because they are being tight.
A second article was published with the headline:
DAMAGE LIMITATION British Airways boss ‘tries to gag staff’ over IT failure which hit 300,000 passengers after ‘inexperienced staff outsourced to India didn’t know to launch back up system’
It linked to an internal email sent by the CEO, published by the Mail Online.
“Guys, either you are part of the team working to fix this or you aren’t. We are not in the mode of ‘debriefing on what happened’ but rather ‘let’s fix this mode’.”
“In the meantime, if you do not want to get involved or cannot get involved, I would kindly ask you to refrain from live commentary, unless it is a message of support to the thousands of colleagues that love BA as much as you do.”
Over on The Guardian’s site, the GMB union were accusing the executive team of hiding while its members had to deal with thousands of angry passengers. It also refuted claims made by the CEO that the crisis had nothing to do with outsourcing or cost-cutting.
However, the CEOs statement saying that the issues were from the local data centre and were “managed and fixed by local resources” was met with the union pointing out that members of the equipment and facilities team at this location had, in fact, been made redundant.
The media also started to report that the CEO was refusing to resign.
CEO video #3
The third CEO video was more of an in-house interview and went over the same messages as the previous videos.
The social media response was mixed.
Our Chairman and CEO, Alex Cruz, apologises for the disruption caused by the recent IT system issues and… https://t.co/DRqwuM3SvF
— British Airways (@British_Airways) May 29, 2017
31st May
BA’s statement on the cause
The Guardian reported that the IT shutdown was caused by an “uncontrolled return of power” following a power failure that damaged servers at its data centre.
BA’s statement said:
“There was a loss of power to the UK data centre which was compounded by the uncontrolled return of power which caused a power surge taking out our IT systems. So we know what happened, we just need to find out why,” the airline said in a statement.
“It was not an IT failure and had nothing to do with outsourcing of IT, it was an electrical power supply which was interrupted.
“We are undertaking an exhaustive investigation to find out the exact circumstances and most importantly ensure that this can never happen again.”
The expert’s view on the cause
The media turned to the experts. Scottish and Southern Electricity, (which runs the local grid) and the National Grid said that there had been no issues or power surges in the area on Saturday morning.
Data centre experts pointed to a possible design flaw, as all data centres should have good surge protection built-in. There should also be procedures for incidents like power outages, and a quick way to start the back-up systems.
As of 2nd June, the Times reports that the crash was caused by an IT worker accidently turning off the power supply.
More than 75,000 passengers were affected by the system crash due to cancelled flights.
BA’s parent company, IAG, saw shares fall by 4% on Tuesday as markets opened after the Bank Holiday Weekend.
Whether or not this was caused by a power surge, a power outage, or human error, there will be questions asked of BA as to why it didn’t have a backup plan in place, whether there were lessons missed from previous failures, and whether human error could have been prevented. Meanwhile the cost to the airline in fixing the problem and paying passengers compensation is mounting.
