At a glance…
• What was affected?
Impact on Customers
UI unreachable / Console and CheckoutAPI unreachable.
VPC Peering configuration corrupted the link between MySQL and various services.
This lasted for 28 minutes from 11:37 am UTC to 12:05 am UTC.
• What was the root cause?
We identified a total disconnect from various services and the console MySQL database. This was initially raised with us from our internal team using the Console and from various merchants accessing the Checkout UI.
• What preventative actions are being taken?
Process changes have been implemented such that all production infrastructure changes will need to be reviewed by at least one Tech Lead to ensure more than one person has context of changes.