Checkout API elevated 5xx rate in us-east-1
Checkout API is returning 5xx errors at 1.4% baseline after a canary deployment. Impacting ~3% of payment attempts in us-east-1.
Incident Timeline
Chronological log of all events during this incident
Posted public update: 'We are investigating elevated checkout failures in US-East. Some payment attempts may fail. Please retry.'
Linked runbook: Checkout 3DS Timeout
Internal note: error rate started climbing 4 minutes after canary rollout began. Pattern matches a known issue with the new 3DS retry logic.
Linked deployment DEP-2024-006 (canary v2.4.1-rc1)
Added service: checkout-api, billing-service
Added Sofia Bianchi (payments on-call) and Priya Raman (reliability on-call) as responders
Marcus Anderson acknowledged and took commander role
Alert ALR-1248 linked to incident
Incident auto-created from alert ALR-1248 (checkout 5xx > 1%)