Why did checkout p95 regress?

astralyx-large · 14 messages

Context used41k / 200k
Checkout p95 doubled overnight. What happened?
Ada Lovelace
Investigator
Input
json
{
  "query": "histogram_quantile(0.95, checkout_request_seconds)",
  "window": "7d",
  "step": "5m"
}
Output
json
{
  "before": 0.21,
  "after": 0.48,
  "changedAt": "2026-09-02T02:00:00Z"
}
Input
tsx
{ "repo": "astralyxdev/checkout", "number": 4182 }
Output
tsx
- const address = await lookup(line)\n+ const addresses = await lookupBatch(lines, { size: 200 })

PR #4182 batched the address lookup, and the address service started shedding load: its 429 rate went from 0.4% to 11.2% at 02:00 UTC — the same twenty minutes checkout p95 moved from 210ms to 480ms. The batch size in the diff is 200. The address service documents a limit of 50 per request and returns 429 above it without a Retry-After, so the client's backoff never engages and every batch costs a full timeout.

Sources

  1. 1checkout-service — p95 latency, 7 daysGrafana
  2. 2PR #4182 — batch the address lookupGitHub
  3. 3Runbook: checkout latencyNotion
checkout-servicemetricsPR #4182diff
Navigate
Go to Threads
Go to Runs
Go to Prompts
Go to Tools
Go to Evals
Go to Policy
Runs
Why did checkout p95 regress?
Draft the 0.15.0 release notes
Migrate the billing schema
Summarise last week’s incidents
Tools
grafana.query
github.diff
issues.create
db.migrate
Back to examples