A significant Amazon Web Services (AWS) outage disrupted internet services worldwide, impacting thousands of applications, websites, and online platforms. The widespread disruption, which lasted for several hours, affected a broad range of services including social media, gaming, streaming, and financial platforms. Amazon has since confirmed that its systems are back online and has identified the root cause of the incident.
Key Takeaways
-
Human error during a routine maintenance task is the identified cause of the AWS outage.
-
The error led to the removal of more servers than intended, impacting critical network subsystems.
-
The outage affected numerous high-profile services globally, including Snapchat, Roblox, and Fortnite.
-
Amazon has pledged to implement changes to prevent similar incidents in the future.
The Incident Unfolds
The outage began early Monday morning, with users reporting widespread issues around 8 AM UK time. Downdetector showed a sharp increase in reported problems, with over 15,000 users affected across various regions. Amazon’s status page confirmed increased error rates and latencies in the US-EAST-1 Region, which is a critical hub for many AWS services.
Services that rely on AWS infrastructure experienced significant downtime. This included popular platforms like Snapchat, Roblox, Fortnite, Duolingo, and Amazon’s own services such as Alexa and Ring security cameras. The disruption highlighted the deep reliance of the modern internet on cloud computing providers like AWS.
Root Cause Identified
Amazon attributed the outage to an error with its EC2 internal network. Specifically, the problem stemmed from an underlying internal subsystem responsible for monitoring network load balancers. According to reports, an authorized S3 team member, while attempting to remove a small number of servers for the S3 billing process, incorrectly entered a command. This resulted in a much larger set of servers being removed than intended.
These affected servers were crucial for other S3 subsystems, including those responsible for metadata and location information in the US-EAST-1 data centers. The subsequent need to restart these systems and perform safety checks took several hours, leading to the prolonged outage.
Recovery and Future Changes
Amazon began observing signs of recovery approximately three hours after the outage began. By 6 PM ET, services had returned to normal operations. The company stated that it is implementing changes to speed up the recovery time for S3 systems and has created new safeguards to prevent excessive server capacity from being taken offline during maintenance.
Additionally, Amazon is making improvements to its service health dashboard, which itself was affected by the outage. The company has apologized for the impact on its customers and emphasized its commitment to learning from the event to further enhance availability.
