Faulty Update Causes Microsoft Cloud Outage

Last week, users of Microsoft's Hotmail, Outlook.com and SkyDrive experienced interruptions to the service. It was last Tuesday at 4:35 p.m. Eastern time that the outage began, and problems were not solved until 8.45 a.m. Eastern time the next morning.

A vice president of Microsoft, Arthur de Haan, said that the problem was caused by an attempted update of firmware.

De Haan stated: "This is an update that had been done successfully previously, but failed in this specific instance in an unexpected way. This failure resulted in a rapid and substantial temperature spike in the datacenter. This spike was significant enough ... that it caused our safeguards to come in to place for a large number of servers in this part of the datacenter."

He added: "These safeguards prevented access to mailboxes housed on these servers and also prevented any other pieces of our infrastructure to automatically failover and allow continued access. This area of the datacenter houses parts of the Hotmail.com, Outlook.com, and SkyDrive infrastructure, and so some people trying to access those services were impacted."

Once the team was made aware of the outage, work began immediately to restore the service. De Haan added: "Based on the failure scenario, there was a mix of infrastructure software and human intervention that was needed to bring the core infrastructure back online. Requiring this kind of human intervention is not the norm for our services and added significant time to the restoration."

Most inboxes were restored before midnight, and service was restored on Tuesday through to Wednesday on an incremental basis.

On February 23 of this year, an error involving an expired SSL certificate caused over 50 services to be deployed through Microsoft Azure, and last week's outage is therefore the second in recent weeks. Microsoft has promised a refund for affected customers for the Azure incident. (CY) Link