zitadel/benchmark/postmortems/2026-08-27-503-burst.md

Follow-up: the 503 bursts of 2026-08-27

Two bursts, six hours apart, on different targets, during the v4.17.1 benchmark sweep. Burst 1 was found first and most of this note was written about it; burst 2 is seven times larger and is what makes the pair worth a post-mortem rather than a shrug.

Both are HTTP 503 Service error -27 — an HTML page from the Google frontend, not a Zitadel JSON error. They are recorded here because a page that reports them as a plain failure rate would be attributing an infrastructure event to Zitadel, and the difference matters more than the number does.

Status as of 2026-08-28

The metric half of the investigation is done. The log half was run with broken filters and must be repeated. Nothing here is concluded. The sections below are the chronological record and this block is the summary.

Established

• Two bursts on 2026-08-27, both 503 Service error -27. Burst 1: manipulate_user, 13:43:19–13:44:13, 150 requests. Burst 2: human_password_login, 19:37–19:44, 1,044 requests. Both bounded, then silence.
• No container startups in any of the eleven test windows. The Cloud Run container/startup_latencies metric returned empty distributions throughout, and instance count was constant at 7.
• The two bursts have different server-side signatures. In burst 1's window there were zero log entries at severity>=WARNING. In burst 2's there were at least 200, and that 200 was the query limit, reached within the first ~2 seconds of a 20-minute window.
• The 2026-08-26 sweep produced no 503s at all, on any of its eleven targets. Same instance, same deployment, roughly 24 hours earlier.

Ruled out, provisionally

• Container restart or replacement. This was the original working hypothesis, on the grounds that a bounded burst followed by silence looks like something being replaced. The startup metric does not support it: a replaced container starts, and a start would appear. Provisional only because it rests on one metric being both correct and complete.
• Panic and crash. The stack traces in the collected logs are not crash dumps: they begin runtime/debug.Stack() called from zerrors.CreateZitadelError, which is Zitadel's error type capturing a stack when it is constructed. recover_interceptor.go appears because it is in the call chain, not because it recovered anything. Combined with zero container startups, the crash-and-restart family of explanations is firmly out.

Open questions, in the order worth answering

1. What do burst 2's errors actually say? Still open after two attempts. The first failed on textPayload filters against a service that logs structured JSON. The second fixed that and then spent its whole 5,000-entry budget on the seven minutes of padding before the burst, at a baseline rate of 800–1,300 errors/minute. Both failures were in the collection, not in the data, and both produced output that looked like an answer.
2. How many errors are there really? 200 in two seconds, truncated. Over twenty minutes the true figure could be in the thousands, and 1,044 client-observed 503s may be a small fraction of what the server logged — or unrelated to it.
3. Did the 503s reach Cloud Run at all? The queries filtering on httpRequest.status=503 came back empty, which invites the reading "the frontend rejected them before they arrived". Do not accept that yet. If request logging is not enabled on this service the query returns nothing regardless. Establish the control first: confirm the service emits any httpRequest entry, in any window. Only then is an empty result evidence.
4. Why is burst 1 silent and burst 2 noisy? Same HTTP symptom, opposite server-side signature. Two causes wearing the same error page is at least as likely as one, and this note's earlier framing — that the pair is the finding — assumed a single mechanism it had not established.
5. Which container instances were involved? The instance-id label rendered empty for every entry. Unknown whether the label path is wrong for this service or the value is genuinely unset. Check one known-good entry's full JSON before trusting any instance-level conclusion.
6. What is Service error -27? Used throughout this note as an identifier and nowhere explained. It is a Google frontend string; if it names the layer that rejected the request, it answers question 3 without any log query at all.
7. Why only the 27th? The 26 August sweep was clean. The instance's feature flags were changed at 13:35:31Z on the 27th, eight minutes before burst 1. That is close enough to be worth checking and far too neat to trust: burst 2 came six hours later, so a one-off configuration change does not cover both. Treat as a lead, not a cause.
8. Does it recur? Two bursts on one day and none the day before. Whether this reproduces is unknown, and it determines whether this is worth an issue or a footnote.

Weak signal, recorded so it is not rediscovered as though it were new: both bursts fell on the two targets that create users in bulk. With n=2 that is barely distinguishable from coincidence.

Burst 1 — manipulate_user @ 200 VUs, 13:43:19–13:44:13

Run: manipulate_user, 200 VUs, 1800s, started 2026-08-27 13:41:09 UTC. Feature flags: improvedPerformance = PROJECT_GRANT, PROJECT, USER_GRANT, ORG_DOMAIN_VERIFIED, captured before the run; sequence 7, set 13:35:31Z.

29 errors surfaced as thrown k6 errors, all of them on user creation. The CSV shows the real figure:

first   2026-08-27T13:43:19Z  (CSV; k6 error lines showed 13:43:21)
last    2026-08-27T13:44:13Z  (CSV; k6 error lines showed 13:43:57)
count   150 failed requests (29 surfaced as thrown errors on user creation)
window  55 seconds, ~2 minutes into the run
other error classes: 75 request timeouts, 14:08:48-14:11:02 - UNRELATED, see below

A bounded burst followed by silence. This is not the shape of a service under sustained load; it is the shape of something being restarted or replaced — which is what made the absence of any container startup in the metrics the surprise it was. It is also a different failure class from every other failure seen during the sweep, all of which are application or harness level.

Burst 2 — human_password_login @ 600 VUs, 19:37–19:44

Found after burst 1. It is the larger of the two by a wide margin:

	burst 1	burst 2
target	manipulate_user (200 VUs)	human_password_login (600 VUs)
window (UTC)	13:43:19 – 13:44:13	19:37 – 19:44
duration	55 s	~7 min
count	150	1,044
position in run	~2 min in	~13 min in
recurrence	none in the following 25 min	none after

Same symptom, six hours and four intervening targets apart.

Note also what burst 2 is not: human_password_login reported 8,178 request timeouts spread evenly across the whole run at 125–360/min. Those are Zitadel under load and are the number to read. The 1,044 platform 503s are a separate population, and the page reports them separately. Do not let the post-mortem collapse them back together.

What the 2026-08-28 collection added — read the caveat before the findings

The log half of that collection was partly invalid and must be re-run. The filters matched only textPayload, and Zitadel logs structured JSON, so every such term matched nothing and the display column rendered blank even where entries existed. The corrected queries then hit a second problem: at 800–1,300 errors per minute on this target, a window padded to 19:30–19:50 and ordered ascending exhausted its 5,000-entry cap at 19:34:51, before the burst began at 19:37. Everything it retrieved is the ordinary pre-burst error rate.

What survives that caveat, because severity>=WARNING is payload-agnostic:

	burst 1 (13:38–13:50)	burst 2 (19:30–19:50)
entries at severity >= WARNING	0	>= 200, truncated at the limit
span of the 200 sampled	—	19:30:00–19:30:0x, i.e. ~2 seconds

The two bursts do not have the same signature, which contradicts the working assumption that one mechanism produced both. Burst 1 produced no warnings or errors at all; burst 2 produced at least 200 in the first two seconds of the window alone, so the true count over 20 minutes is likely in the thousands. Their content is unknown — that is exactly what the broken filter cost.

From the metric side, which was not affected: no container startups in any window, and instance count constant at 7.

Do not conclude "the frontend rejected the requests before Cloud Run saw them" from the empty 503 request logs. See open question 3: without the control, that query cannot distinguish an infrastructure finding from a disabled log.

Two unrelated defects, found by accident

The baseline error mix under 600 VUs is worth recording independently of the burst: 2,901 bare context canceled, 1,239 QUERY-AWx52 Errors.Query.SQLStatement (parent: context canceled), 232 LOG-ii6Pi with parent sql: transaction has already been committed or rolled back, and 66 occurrences of a missing i18n key, "unable to get user by id" not found in language "en".

context canceled is a consequence of the client giving up, not a cause. But sql: transaction has already been committed or rolled back is a genuine defect — code using a transaction after it has finished — and the missing translation key is a smaller one. Neither is related to the 503s; both should not be lost.

Why it matters for the page

If this was a container restart, the failures are a deployment event and must be reported as such rather than as a Zitadel failure rate. If there was a panic, that is a defect and worth an issue of its own. Either way the page states what happened; an unexplained 503 burst rendered as a plain failure count would be true and misleading, which is the worst combination available.

Addendum: the two populations in burst 1's run

The full status breakdown, taken while the CSV still existed, shows the run contained two unrelated failure populations, separated by 25 minutes of clean traffic:

error_code  reqs  first_utc             last_utc
1503         150  2026-08-27 13:43:19   2026-08-27 13:44:13   <- infrastructure
1050          75  2026-08-27 14:08:48   2026-08-27 14:11:02   <- the known wall

Only the first population is the subject of this note. The second is the documented degradation-over-sustained-churn behaviour described on the manipulate_user page, and is expected.

The burst was also larger than the k6 error lines suggested: 150 failed requests, of which only 29 surfaced as thrown errors (user creation). The other 121 hit update, lock and delete, where a failed check does not throw. This is why failure counts are taken from the CSV and not from grep -c level=error on the log, which gave 29.

---

The Cloud Shell queries used for the collection, and the identity of the deployment they run against, are not in this repository.