TechWeb

Google Gmail Disruption Caused By Network Load, Code Update

Sep 28, 2009 (12:09 PM EDT)

Read the Original Article at http://www.informationweek.com/news/showArticle.jhtml?articleID=220300128


The Gmail outage last week was the product of a Gmail code change that inadvertently increased the load placed on Google's Contacts service, in conjunction with heavy usage of the Contacts service and with a data center network issue.

Those findings were included in a report that Google published on Friday detailing what happened during the service disruption on Thursday morning.

"After an internal investigation and review, the Engineering team has determined that the root cause of the Contacts issue was a high load on the service," the report says.

Between 7:00 AM and 9:50 AM Pacific Time last Thursday, users of Google Apps were unable to access Contacts through the Gmail interface. The report says that during this period, most customers couldn't access Google Talk or add users to their Google Apps accounts.

Until about 8:30 AM that morning, an undisclosed number users could not access Gmail, except through an IMAP client. Google last week characterized those affected as "a small subset of users."

In a statement last week, Google apologized for the inconvenience and said that the slowness and service degradation lasted about an hour. The company said that Gmail was fully operations for everyone by about 8:00 AM on Thursday, and that Contacts functionality was restored shortly after that.

The modest apology appears to correspond with Google's assessment the disruption as relatively minor.

A 100-minute Gmail outage at the beginning of the month was characterized by Google engineering VP and "Site Reliability Czar" Ben Treynor as a "big deal" in his apology.

In early May, a more significant outage brought an apology from someone even higher in the Google hierarchy, SVP of operations Urs Hoelzle. That incident caused a 5% drop in Internet traffic, according to Arbor Networks, and affected 14% of Google users worldwide, according to Hoelzle.


InformationWeek has published an in-depth report on the current state of IT salaries. Download the report here (registration required).