Infrastructure resilience dashboard

Deploys pausedFoundationDB upgrade in flight on lon-a.

Failover readiness

94 / 100

Last drill recovered in 3:41, budget 5:00

Proven capacity

6.2× load

Target 10×

Deploy time p50

11 min

Target 7 min

Needs attention · 2

dub-a has not failed over in 34 days

Budget is 30. Two config differences from lon-b.

Run drill

DataDog spend up 24% since Tuesday's release

Custom metrics from payments-ingest.

Review

01Failover readiness

Can we lose a site today?

lon-a AWS eu-west-2Upgrading

lag 0 ms (primary)last drill 9d agodrift 0

lon-b Colo, owned hardwareReady

lag 42 mslast drill 9d agodrift 0

dub-a AWS eu-west-1Drill overdue

lag 118 mslast drill 34d agodrift 2: TLS cert, routing weight

How the score is built

40% drill recency · 30% replica lag · 20% config drift · 10% cert and PKI expiry.

02Database upgrade · fdb-prod-lon-a

7.3.43 → 7.3.51

In progress

All steps and hosts

03Deploys

Paused during upgrade

Down from 19 to 11 min since Q2. Rollback rate 1.4%.

Stage timings

Stage	p50
CI	3:10
Bazel build	2:25
Rollout	5:20

Bazel cache hit 87%.

04Chaos & capacity

1 partial, 1 overdue

Partition lon-a ↔ dub-aPartial

2 retries exceeded SLO

Region loss drill · dub-aOverdue

Passed and scheduled · 2

Kill 1 coordinator · lon-bPassed

Load test 7× · payments APIThu 10:00

05Cost & network

£71.9k a month

DataDog£18.7k +24%

All services and network

Service	£ / mo
AWS compute	41.2k
DataDog	18.7k
Colo hardware	12.0k

3 service-discovery flaps in 24h on lon-a → dub-a.