O2 CUSTOMERS are back online this morning after one of the most spectacular telecoms outages in recent years.
Yesterday, 32m customers were directly affected by the outage at O2, either through its own network, or the MVNOs such as its subsiduary GiffGaff and third parties like Sky and Tesco that piggyback it.
Further disruption was caused by outages on services dependent on O2 data, such as London bus timetable displays, Oyster card readers, and taxi control systems. GSM based security will also have been affected, as will smart meters.
As the inquest into what happens begins, O2 and Ericsson which provides the software at the heart of the problem have issued joint apologies, but is still leaving customers guessing about compensation, for the moment.
Internally, it’s likely that O2 and other providers will be seeking reperations from Ericsson.
The two firms said: “We fully appreciate it’s been a poor experience and we are really sorry.”
Ericsson adds: “The cause of today’s network issue is in certain nodes in the core network resulting in network disturbances for a limited number of customers across the world, including in the UK. We have been working hard on resolving the UK data issue since early this morning. The faulty software that has caused these issues is being decommissioned.”
It seems that the problem stemmed from the smallest detail – the security certificate in some Ericsson run data centres had expired, causing anything running on its systems to come to a screeching halt as the system did exactly what it was supposed to – block the invalidly encrypted data.
3G services were restored last night, whilst 4G was back by this morning, though some customers are still reporting problems (try turning your phone off and on again if this is you).
Ericsson has promised “a complete and comprehensive root cause analysis” to find out how the security certificate expiries were missed.
Ironically, the outage came as Huawei, the main alternative provider of infrastructure came under the increased weight of bad publicity following the arrest of their CTO.
The next question facing O2 is whether it did enough to prevent the outage, and if its systems should have had some sort of fallback.
For the rest of us, it’s a reminder of how much we depend on data networks, even when there is no direct relationship for us, and maybe, just maybe, we’ll all start to think of some analogue failsafes. You know – for old times’ sake. μ
Source : Inquirer