Some EC2 instances in AWS us-east-2(Ohio) may experienced the loss of power issue this morning due to affected AWS Availability Zone. Currently, power has been restored and the majority of the affected EC2 instances have recovered and the rest is expected to recover within one hour.
Attached is the Instance Impairments Statement from AWS:
[12:45 PM PDT] Starting at 9:57 AM PDT some EC2 instances and EBS volumes experienced a loss of power within a single Availability Zone in the US-EAST-2 Region. Power was restored at 10:19 AM PDT and EC2 instances and EBS volumes began to recover. By 10:23 AM PDT, the vast majority of EC2 instances and EBS volumes has fully recovered and by 11:37 AM PDT, all but a very small number of EC2 instances and EBS volumes had recovered. Elastic Load Balancing shifted traffic away from the affected Availability Zone, which has now been shifted back. RDS impact for single-AZ databases, which have also been recovered. Other services (tagged below) saw impact during the event, but most have fully recovered. Those that are still seeing impact, will provide updates via the Personal Health Dashboard as they work towards full recovery. The issue has been resolved and the service is operating normally.
[11:45 AM PDT] At this stage, the vast majority of EC2 instances and EBS volumes, affected by the loss of power in a single Availability Zone in the US-EAST-2 Region, have fully recovered. A small number of EC2 instances and EBS volumes are on hardware that was adversely affected by the loss of power. Engineers continue to work on recovering the EC2 instances and EBS volumes on this hardware and will provide updates via the Personal Health Dashboard if any of these could not be recovered. Elastic Load Balancers affected by the issue have recovered and traffic has been shifted back into the affected Availability Zone. The vast majority of single-AZ databases have also recovered and the remaining databases are running on hardware that was affected by the event. We will provide updates via the Personal Health Dashboard if any of these databases can not be recovered. At this stage, if your EC2 instance or EBS volumes that has still not recovered, attempting a reboot of the EC2 instance could resolve the issue. If not, we recommend relaunching the affected EC2 instance or recreating the affected EBS volume.
[11:25 AM PDT] We continue to make progress in recovering the remaining EC2 instances and EBS volumes affected by the loss of power in a single Availability Zone in the US-EAST-2 Region. The vast majority of EC2 instances are now healthy, but we continue to work on recovering the remaining EBS volumes affected by the issue. EC2 API error rates and latencies have returned to normal levels. Elastic Load Balancing remains weighted away from the affected Availability Zone. Error rates and latencies for Lambda function invocations have now returned to normal levels. Power has been restored to all affected resources and remains stable. We expect the recovery of EC2 instances and EBS volumes to continue to improve over the next 30 minutes. For customers that need immediate recovery, we recommend failing away from the affected Availability Zone as other Availability Zones are not affected by this issue.
[10:49 AM PDT] We continue to see recovery of EC2 instances that were affected by the loss of power in a single Availability Zone in the US-EAST-2 Region. At this stage, the vast majority of affected EC2 instances and EBS volumes have returned to a healthy state and we continue to work on the remaining EC2 instances and EBS volumes. Elastic Load Balancing has shifted traffic away from the affected Availability Zone. Single-AZ RDS databases were also affected and will recover as the underlying EC2 instance recovers. Multi-AZ RDS databases would have mitigated impact by failing away from the affected Availability Zone. While the vast majority of Lambda functions continue operating normally, some functions are experiencing invocation failures and latencies, but we expect this to improve over the next 30 minutes. Power has been restored to all affected resources and remains stable. We expect the recovery of EC2 instances and EBS volumes to continue to improve over the next 45 minutes. For customers that need immediate recovery, we recommend failing away from the affected Availability Zone as other Availability Zones are not affected by this issue.
[10:25 AM PDT] We can confirm that some instances within a single Availability Zone (USE2-AZ1) in the US-EAST-2 Region have experienced a loss of power. The loss of power is affecting part of a single data center within the affected Availability Zone. Power has been restored to the affected facility and at this stage the majority of the affected EC2 instances have recovered. We expect to recover the vast majority of EC2 instances within the next hour. For customers that need immediate recovery, we recommend failing away from the affected Availability Zone as other Availability Zones are not affected by this issue.
[10:11 AM PDT] We are investigating network connectivity issues for some instances and increased error rates and latencies for the EC2 APIs within the US-EAST-2 Region.
Posted Jul 28, 2022 - 10:45 PDT
This incident affected: AWS Platform (US East (Ohio)).