Yesterday was not a good day for Google's usually reliable Gmail service. Google says that due to a dual network failure, some users experienced partial service interruptions, slow performance, or total outages for up to 11 hours.
The company says that almost one-third of all emails routed through it servers were affected and that about 1.5 percent of all emails sent or received through the service were delayed by as much as two hours. Google has apologized via an official statement, and says that it will be implementing steps over the next few weeks to make sure these issues do not happen again.
Google plans to beef up its network and its backup capacity for Gmail, and says that it will make email delivery more resilient even in the event of a dual network failure. For many of us who use Gmail as our exclusive email provider, yesterday's issues were a little more serious. However, I am sure that the service will not see a loss of users over the recent issues. Google's complete statement can be found below.
The message delivery delays were triggered by a dual network failure. This is a very rare event in which two separate, redundant network paths both stop working at the same time. The two network failures were unrelated, but in combination they reduced Gmail's capacity to deliver messages to users, and beginning at 5:54 a.m. PST messages started piling up.
Google's automated monitoring alerted the Gmail engineering team within minutes, and they began investigating immediately. Together with the networking team, the Gmail team restored some of the network capacity that was lost and worked to repurpose additional capacity, clearing much of accumulated message backlog by 1:00 p.m. PST and the remainder by shortly before 4:00 p.m. PST.