Shipping through the storm: the infra story.

Gifting has a heartbeat. It spikes on Valentine's Day, on Eid, on birthdays that cluster around school-year rhythms, on last-minute campaign pushes from corporate clients who decide to send 10,000 gift cards on a Tuesday afternoon with no advance notice. Our infrastructure has to breathe with it.

This is the story of how we went from an infrastructure that survived spikes to one that barely notices them.

The architecture we inherited

When the team was small, everything lived in a single AWS region. Auto-scaling was configured, but not well-tuned — the scale-out trigger was too conservative, and by the time new instances were warm, the spike had already caused queuing. Database connections were pooled, but pool exhaustion under load was a recurring incident.

The symptom was always the same: response times would climb, then errors would start, then the on-call engineer's phone would buzz at 2am. We got good at incident response. We needed to get better at incident prevention.

"The goal isn't to respond faster to incidents. The goal is to make your system so boring that incidents stop being interesting."

Multi-region and the failover problem

Our first major architectural change was introducing a standby region. Active-passive across two AWS regions: primary traffic runs in me-central-1 (UAE), with a warm standby in a secondary region. RDS read replicas, S3 replication, and Route 53 health checks form the backbone of the failover mechanism.

The tricky part was DNS TTL. We had TTLs set to five minutes — reasonable in theory, but in practice, a five-minute failover window when payments are in flight is a long time. We reduced TTLs to 30 seconds and added circuit-breaker patterns at the application layer so transient failures during failover don't cascade into user-visible errors.

Taming peak traffic

Peak-season traffic is predictable in timing but not in magnitude. We know Eid is coming; we don't know if this year will be 2x or 4x normal traffic. Our approach:

• Pre-warming. We schedule capacity increases 48 hours before known peak windows. Auto-scaling handles the unexpected; pre-warming handles the expected.
• CDN-first architecture. Product catalogue, brand assets, and any response that doesn't change per-user is aggressively cached at the edge. Peak traffic never reaches the origin for these requests.
• Queue-based workload offloading. Non-blocking operations — notification delivery, analytics events, PDF generation for vouchers — are queued via SQS and processed async. The checkout path stays thin and fast.
• Load shedding. If we detect saturation, non-critical read paths gracefully degrade before the critical purchase and redemption paths are affected.

Observability as a first-class concern

We can't fix what we can't see. We moved to a unified observability stack: distributed tracing with OpenTelemetry, metrics in CloudWatch with custom dashboards, and structured logging that makes log queries fast enough to be useful during an incident rather than after it.

Our SLOs are defined at the service level and reported weekly. If we're degrading toward a threshold, the alert fires before the breach, not after. This shift from reactive monitoring to proactive SLO tracking has been the single biggest cultural change in how the team relates to production.

What a calm night on-call feels like

The measure of good infrastructure work is not dramatic war stories — it's the absence of them. In the last two peak seasons, our on-call rotation has been quiet. Not because nothing happened, but because the systems handled it. Traffic spiked, auto-scaling responded, CDN absorbed the read load, and the databases never blinked. The engineers slept.

That's the goal. Infrastructure work done well is invisible.