Fastly, a major content delivery network, triggered a major internet blackout on Tuesday morning has blamed a software bug.
We first noticed the problem a little after 0600 ET Tuesday when countless websites, including Reddit, Financial Times, PayPal, and other websites, went down.
"We experienced a global outage due to an undiscovered software bug that surfaced on June 8 when it was triggered by a valid customer configuration change," Nick Rockwell, Fastly's senior vice president of engineering and infrastructure, wrote in a blog post late Tuesday.
Rockwell said, "the outage was broad and severe, and we're truly sorry for the impact to our customers and everyone who relies on them." He said the company "detected the disruption within one minute, then identified and isolated the cause, and disabled the configuration," adding that "within 49 minutes, 95% of our network was operating as normal."
"Even though there were specific conditions that triggered this outage, we should have anticipated it," Rockwell said.
The company operates servers at strategic points worldwide to support customers moving and storing content.
But when a customer switched their settings, they had accidentally exposed a bug in a software update issued in May, which triggered widespread outages.
Here's a timeline of yesterday's disruption (all times are in UTC):
09:47 Initial onset of global disruption
09:48 Global disruption identified by Fastly monitoring
09:58 Status post is published
10:27 Fastly Engineering identified the customer configuration
10:36 Impacted services began to recover
11:00 Majority of services recovered
12:35 Incident mitigated
12:44 Status post resolved
17:25 Bug fix deployment began
We noted yesterday that incidents like this underline the fragility of the internet and how the global internet is dependent on a handful of companies like Fastly and Cloudflare for vital infrastructure. On the flip side, the incident also highlights how quickly an outrage of this magnitude can be resolved.
