Three straight days of system outages cost Southwest Airlines more than $10 million in lost fares this week. The company's COO Mike Van de Ven said that the router crashes which started the meltdown are not uncommon. But then the routers triggered Web server crashes. Finally, the company's disaster recovery plan failed to save the IT operations. Social media posts from customers complained of delayed flight departures and arrivals and an inability to check in for flights on Southwest's website. The running count by Friday morning was 700 canceled flights, with another 1,300 delayed. People could not get to gates without boarding passes.
Customers running 3000s through the 1990s might remember Southwest as a shining star in the MPE/iX galaxy. The system came online with ticketless travel using MPE/iX software developed at Morris Air. When Southwest started to skip the paper, it was one of the very first major airlines to do so. Dispensing with paper tickets was possible because of the 3000's unparalleled reliability.
Stranding an estimate 4,000 customers was never a part of the 3000's history at Southwest. The computer was the dominant ticketing tool in an era before the elaborate security checks in the US. From Wednesday through today, customers on thousands of its flights could not check in at kiosks or via those web servers. The IT failure happened as the Republican National Convention closed out its Cleveland circus.
It's commonplace for a system vendor who's been shown the door, like the 3000 group was in the first decade of this century, to say "It wasn't on our watch" when a crash like this hits. But being commonplace won't recover those millions of dollars of revenues. Maybe they were a small fraction of the overall savings while leaving the 3000. The reliability of an airline is worth a lot more than delivery of a product, though, like an auto. Hertz was a 3000 shop for many years, and their portion of the travel business didn't suffer these woes, either.
Both companies made their IT 3000-free while the worst fact about the system was that HP stopped selling it. They both had plans to expand, strategies MPE/iX wasn't going to be able to handle easily, too. When a vendor ends their business plans for a server, the sweater of coverage unravels one thread at a time. Mission-critical systems are never supposed to leave a publicly traded company naked from the waist up, however.
The company's CEO Gary Kelly had to tell reporters something that founding CEO Herb Kelleher never was faced with. "We have significant redundancies built into our mission-critical systems, and those redundancies did not work," Kelly said in a conference call. "We need to understand why and make sure that that doesn't happen again." Southwest's chief commercial officer said every customer affected on Wednesday or Thursday would be contacted. The company extended for a week a fare sale scheduled that was supposed to end July 21.
Southwest also had to contact the travelers affected Friday, too. The contacting of vendors involved was not part of the stories this week. This would not be a good week to be the CIO at Southwest. Randy Sloan got the job this year, inheriting decisions like making Southwest 3000-free. Until Wednesday, that decision didn't seem like a risk.
In related news, Southwest extended its July fare sale by one week.