mirror of
https://github.com/andwati/zitadel.git
synced 2026-10-18 05:52:51 +00:00
docs(perf): v4.17.1 results and harness fixes (#12651)
Publishes the v4.17.1 benchmark sweep and the harness fixes the sweep uncovered. ## Results All 11 published targets ran for 30 minutes each against a fixed 7-container deployment (vCPU 6 / 6 Gi, psql v17.4, US1 client and server), at 600 VUs except where noted. Latencies are `http_req_duration` over the test phase. | target | VUs | iter/s | p50 | p95 | p99 | failed requests | |:---|---:|---:|---:|---:|---:|:---| | machine_pat_login | 600 | 3452 | 24.53ms | 872ms | 1.00s | 29 / 6,335,633 (0.00%) | | user_info | 600 | 3217 | 29.32ms | 943ms | 1.13s | 66 / 5,800,788 (0.00%) | | introspect | 600 | 2297 | 46.49ms | 1.16s | 1.36s | 1,400 / 4,159,863 (0.03%) | | machine_jwt_profile_grant | 600 | 1076 | 85.34ms | 2.57s | 2.93s | 97 / 1,986,156 (0.00%) | | machine_client_credentials_login | 600 | 834 | 56.78ms | 1.74s | 2.42s | 1,789 / 3,073,497 (0.05%) | | add_session | 600 | 767 | 95.25ms | 3.53s | 4.15s | 1 / 1,416,546 (0.00%) | | oidc_session | 600 | 326 | 77.47ms | 3.41s | 4.43s | 3 / 1,767,162 (0.00%) | | otp_session | 600 | 84 | 47.80ms | 13.83s | 20.22s | 182 / 466,756 (0.03%) | | password_session | 600 | 68 | 408.63ms | 20.00s | 24.05s | 1 / 253,280 (0.00%) | | human_password_login | 600 | 49 | 146.67ms | 4.08s | 1m0s | 9,227 / 683,256 (1.35%) | | manipulate_user | **200** | 19 | 304.58ms | 10.23s | 38.43s | 225 / 171,987 (0.13%) | Against the goals: - **#8352 is met.** `machine_jwt_profile_grant` reached **1076/s** against the 1000/s goal, with p50 at 85ms. - **#4424 is not met.** `human_password_login` runs at **49/s** against a 1200/s goal. The sweep ran with `improvedPerformance` enabled (`PROJECT_GRANT`, `PROJECT`, `USER_GRANT`, `ORG_DOMAIN_VERIFIED`), captured from the instance before the first target rather than assumed; the full flag set is on every page. ## What the observability rows show The containers were never the constraint. ZITADEL CPU sat at 9–50% (p50), memory at 6–7% (p50, peaking at 16%) of 6 GiB, and instance count was flat at 7 with no container startups in any window. Database CPU was at **86–98% median on nine of eleven targets**. Any "ZITADEL achieves N/s" figure that omits which resource ran out is true and misleading, so the pages say. The query breakdown then shows what the CPU numbers cannot: **advisory lock wait is the single largest category of database time across the sweep (91,857 s accumulated)**, ahead of event reads (64,805 s), event push (26,411 s) and projection reads (14,625 s). The three locks are the projection handler lock (`handler/v2/handler.go`), the eventstore read barrier (`repository/sql/query.go`) and the shared lock every push takes (`eventstore/v3/push.go`). That is what sets the `flowchart outcome` on each page, and it is not uniform: - **Scale** on the seven targets where a resource genuinely ran out — database CPU at 92–98% with 0% lock share. - **Potential issues** on `human_password_login`, `password_session`, `otp_session` and `manipulate_user`, where 45–74% of the top query time is lock wait. `password_session` runs at 51% database CPU and 50% container CPU and still only manages 68 iter/s; adding vCPU there buys nothing. Autovacuum is cleared as a contributor: 93.9% median DB CPU in buckets containing a vacuum against 93.1% without, within noise. The new v4.17 eventstore autovacuum is barely triggered — `events2` is not in the top twenty vacuumed tables. ## Harness fixes Defects found during the sweep, all of which corrupted published numbers or aborted runs: 1. **Colliding check names.** `updateHuman`, `lockUser` and `deleteUser` all registered checks as `update user is status ok`, two of them asserting `status === 201` against endpoints returning 200. k6 aggregates checks by name, so this produced a permanent, fictional 28% failure rate. Each check now has its own name and accepts 2xx. 2. **User-creation read-after-write race**, in three variants, each found after the previous fix. The read-back resolved `undefined` when the projection was behind, surfacing much later as `Cannot read property '0' of undefined`; then an unguarded `res.json()` turned every request timeout into a killed iteration; then a *partial* read — a valid user object with `loginNames` not yet projected — passed an `if (user)` guard and failed identically. `readUserAfterCreate()` now guards status and body before parsing, requires the field rather than the object, and retries with backoff, rejecting with the user id and username. 3. **Metric cardinality.** Id-bearing URLs produced 800k+ time series against k6's suggested 100k, driving k6 to 6.8 GB and an OOM mid-run. All 16 such call sites now set a grouped `name` tag. 4. **`password_session` wrote its CSV to `otp_session`'s filename** in the Makefile, silently overwriting the previous target's output. The read-back race is not only a harness defect: at 600 VUs it measures real load-induced projection lag, exceeding 25 s for login-name projection and 100 s for the user projection, against instant reads on an unloaded instance. ## Stable v2 user API Four calls in `benchmark/src/user.ts` still went to `/v2beta/users` — update, lock, delete and the machine read-back. They date from the original load-test bootstrap in April 2024 (#7635), when the user service was still v2beta, and were never moved when it was promoted; they surfaced in this diff only because the read-after-write fix rewrote the surrounding functions. All four are now on stable v2. `updateHuman` is the one that changed shape: `PUT /v2beta/users/{id}` is `UpdateHumanUser`, whose stable-v2 counterpart `PUT /v2/users/human/{id}` is deprecated in the proto, so it now calls `PATCH /v2/users/{id}` (`UpdateUser`) and `manipulate_user` nests the profile under `human`. Both RPCs land on `command.ChangeUserHuman`, so the measured work is unchanged. `lockUser`, `deleteUser` and the `createMachine` read-back move over unchanged in method and payload. No published result is affected: per-endpoint latencies come from custom Trends, and `run-bench.sh` drops the `name` system tag, so no artefact ever recorded a v2beta URL. The remaining `/management/v1` and `/admin/v1` calls are left alone — migrating those changes which API surface is benchmarked and would need a re-run. ## Timestamps Pages reported `Test start` as a bare `HH:MM`, so a run could not be correlated against monitoring the next day. Pages now carry the full UTC date and time and a `Test end` row, both derived from the first and last per-second row of `output.json`. `_template.mdx` is updated to match. ## Post-mortem `benchmark/postmortems/2026-08-27-503-burst.md` records two bursts of HTTP 503 `Service error -27` — an HTML page from the Google frontend, not a ZITADEL error — on `manipulate_user` and `human_password_login`, six hours apart. **The investigation is open**, and the note is explicit about what is established (no container startups; the two bursts have opposite server-side signatures), what is provisionally ruled out (restart, panic), and what is not yet known. It is included because both affected pages report the 503s as a separate population from the ZITADEL failures beside them, and that split needs its evidence somewhere. It also records two defects found by accident in the collected logs, unrelated to the 503s: `sql: transaction has already been committed or rolled back` (232 occurrences under load) and a missing i18n key, `"unable to get user by id" not found in language "en"`. ## Notes for review - **`manipulate_user` ran at 200 VUs, not 600, and the page says so.** Throughput saturates near 100 VUs; at 600 the same target produced 4,283 failed requests for identical throughput against 225 at 200 VUs. Its failure mode is degradation over sustained single-org churn — per-operation p50 climbs 1.7x–6x between the first and last five minutes of a run — not a concurrency threshold. - **Two `Observed errors` rows are hand-corrected** and deliberately differ from the generated text, on `manipulate_user` and `human_password_login`. Both cases are one generated number covering two unrelated failure populations; each page says it was corrected and why. - Org deletion failing with a 500 (`V3-C8l3V`) after heavy user churn is already tracked as #8510 and was hit repeatedly during cleanup. - **The run/publish tooling is no longer part of this PR.** It moved to a private repository, along with the cloud collection scripts, because deployment identity does not belong here. What remains is the published pages, the post-mortem findings and the harness fixes. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
Tim Möhlmann
co-authored by
Claude Opus 5
Claude Opus 5
parent
8de625dc48
commit
3be21ccbf0
@@ -1,43 +1,45 @@
|
||||
import OutputSource from "./output.json";
|
||||
import { BenchmarkChart } from '@/components/benchmark_chart';
|
||||
import { BenchmarkChart } from "@/components/benchmark_chart";
|
||||
|
||||
TODO: describe the outcome of the test?
|
||||
|
||||
## Performance test results
|
||||
|
||||
| Metric | Value |
|
||||
|-:-------------------------------------|-:-----|
|
||||
| Baseline | none |
|
||||
| Purpose | |
|
||||
| Test start | UTC |
|
||||
| Test duration | 30min |
|
||||
| Executed test | |
|
||||
| k6 version | |
|
||||
| VUs | |
|
||||
| Client location | |
|
||||
| Client machine specification | vCPU: <br/> memory: Gb |
|
||||
| ZITADEL location | |
|
||||
| Metric | Value |
|
||||
| :------------------------------------ | :------------------------------------------- |
|
||||
| Baseline | none |
|
||||
| Purpose | |
|
||||
| Test start | YYYY-MM-DD HH:MM:SS UTC |
|
||||
| Test end | YYYY-MM-DD HH:MM:SS UTC |
|
||||
| Test duration | 30min |
|
||||
| Executed test | |
|
||||
| k6 version | |
|
||||
| VUs | |
|
||||
| Client location | |
|
||||
| Client machine specification | vCPU: <br/> memory: Gb |
|
||||
| ZITADEL location | |
|
||||
| ZITADEL container specification | vCPU: <br/> Memory: Gb <br/>Container count: |
|
||||
| ZITADEL Version | |
|
||||
| ZITADEL Settings | |
|
||||
| ZITADEL feature flags | |
|
||||
| Database | type: psql<br />version: |
|
||||
| Database location | |
|
||||
| Database specification | vCPU: <br/> memory: Gb |
|
||||
| ZITADEL metrics during test | |
|
||||
| Observed errors | |
|
||||
| Top 3 most expensive database queries | |
|
||||
| Database metrics during test | |
|
||||
| k6 Iterations per second | |
|
||||
| k6 overview | |
|
||||
| k6 output | |
|
||||
| flowchart outcome | |
|
||||
| ZITADEL Version | |
|
||||
| ZITADEL Settings | |
|
||||
| ZITADEL feature flags | |
|
||||
| Database | type: psql<br />version: |
|
||||
| Database location | |
|
||||
| Database specification | vCPU: <br/> memory: Gb |
|
||||
| ZITADEL metrics during test | |
|
||||
| Observed errors | |
|
||||
| Top 3 most expensive database queries | |
|
||||
| Database metrics during test | |
|
||||
| k6 Iterations per second | |
|
||||
| k6 overview | |
|
||||
| k6 output | |
|
||||
| flowchart outcome | |
|
||||
|
||||
## Endpoint latencies
|
||||
|
||||
<BenchmarkChart testResults={OutputSource} />
|
||||
|
||||
## k6 output
|
||||
|
||||
```bash
|
||||
TODO: add summary of k6
|
||||
```
|
||||
|
||||
@@ -0,0 +1,99 @@
|
||||
---
|
||||
title: add session benchmark of zitadel v4.17.1
|
||||
sidebar_label: add session
|
||||
---
|
||||
|
||||
import OutputSource from "./output.json";
|
||||
import { BenchmarkChart } from "@/components/benchmark_chart";
|
||||
|
||||
Benchmark results of the v4.17.1 release of Zitadel.
|
||||
|
||||
## Performance test results
|
||||
|
||||
| Metric | Value |
|
||||
| :------------------------------------ | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| Baseline | none |
|
||||
| Purpose | Test current performance |
|
||||
| Test start | 2026-08-27 14:58:50 UTC |
|
||||
| Test end | 2026-08-27 15:28:51 UTC |
|
||||
| Test duration | 30min |
|
||||
| Executed test | add_session |
|
||||
| k6 version | v2.1.0 |
|
||||
| VUs | 600 |
|
||||
| Client location | US1 |
|
||||
| ZITADEL location | US1 |
|
||||
| ZITADEL container specification | vCPU: 6<br/> Memory: 6 Gi <br/>Container min scale: 7<br/>Container max scale: 7 |
|
||||
| ZITADEL Version | v4.17.1 |
|
||||
| ZITADEL Settings | Eventstore autovacuum (new in v4.17): Enabled: true<br/>VacuumThreshold: 1000000<br/>AnalyzeThreshold: 1000000<br/>Tuned for the ~4,000 events/s expected during the benchmark, per the [tuning guide](https://zitadel.com/docs/self-hosting/manage/tuning#reading-events). |
|
||||
| ZITADEL feature flags | Enabled: loginDefaultOrg (instance), oidcTokenExchange (system), enableBackChannelLogout (system), consoleUseV2UserApi (instance)<br/>improvedPerformance (instance): IMPROVED_PERFORMANCE_PROJECT_GRANT, IMPROVED_PERFORMANCE_PROJECT, IMPROVED_PERFORMANCE_USER_GRANT, IMPROVED_PERFORMANCE_ORG_DOMAIN_VERIFIED<br/>Disabled: loginV2 (explicitly false at instance), userSchema, debugOidcParentError, oidcSingleV1SessionTermination, permissionCheckV2<br/>Captured from the instance **before** the run (sequence 7, set 2026-08-27 13:35:31 UTC); the same capture covers `manipulate_user` at 13:41 and the rest of the sweep from 14:56, so all eleven targets ran the same configuration.<br/>This restores like-for-like comparison with the v4 pages, which ran five improvedPerformance options. The fifth, IMPROVED_PERFORMANCE_ORG_BY_ID, is `reserved` in the proto and no longer exists. |
|
||||
| Database | type: psql<br />version: v17.4 |
|
||||
| Database location | US1 |
|
||||
| Database specification | vCPU: 8<br/> memory: 32Gib |
|
||||
| ZITADEL metrics during test | CPU: p50 18%, p99 26% (peak 28%)<br/>Memory: p50 6%, peak 12%<br/>Instances: 7 (constant), no container startups recorded<br/>Server-side: 790 req/s, latency p50 80ms / p95 3.5s / p99 3.9s |
|
||||
| Observed errors | 1 of 1,416,546 requests failed (0.00%): none of the 1 individually logged (see k6 output). 1 of 2,831,284 checks failed (0.00%): add Session status ok (1 failed) |
|
||||
| Top 3 most expensive database queries | 1: Push events (`eventstore.push`) (5,059 s)<br/>2: Query events by instance_id, aggregate_types, event_types, position (paged) (3,497 s)<br/>3: Shared advisory lock on `events2`, taken by every push — **lock wait** (585 s)<br/>Ranked by total accumulated execution time summed across concurrent sessions (Cloud SQL Query Insights) — not CPU time; it can exceed the wall-clock window. |
|
||||
| Database metrics during test | CPU: 97% median, 98% peak<br/>Memory: 40%<br/>Connections: 46<br/>Transactions: 9,895/s<br/>IOPS: read 0, write 787<br/>**CPU-saturated — the bottleneck for this target.** |
|
||||
| k6 Iterations per second | 767 |
|
||||
| k6 output | [output](#k6-output) |
|
||||
| flowchart outcome | **Scale.** Database CPU is the limit: 97% median against ZITADEL at 18%, with advisory lock wait only 6% of the top-5 query time. Raising the database beyond 8 vCPU is the next step, and throughput should be re-measured against the increase. |
|
||||
|
||||
## Endpoint latencies
|
||||
|
||||
<BenchmarkChart testResults={OutputSource} />
|
||||
|
||||
## k6 output
|
||||
|
||||
```bash
|
||||
█ TOTAL RESULTS
|
||||
|
||||
checks_total.......: 2831284 1533.737688/s
|
||||
checks_succeeded...: 99.99% 2831283 out of 2831284
|
||||
checks_failed......: 0.00% 1 out of 2831284
|
||||
|
||||
✓ user defined
|
||||
✓ authorize status ok
|
||||
✓ login name status ok
|
||||
✓ login shows password page
|
||||
✓ password status ok
|
||||
✓ password callback
|
||||
✓ code set
|
||||
✓ token status ok
|
||||
✓ access token created
|
||||
✓ id token created
|
||||
✓ info created
|
||||
✓ org created
|
||||
✓ create user is status ok
|
||||
✗ add Session status ok
|
||||
↳ 99% — ✓ 1415335 / ✗ 1
|
||||
✓ add session is status ok
|
||||
✓ org removed
|
||||
|
||||
CUSTOM
|
||||
add_session_duration.................: min=23ms avg=763.11ms max=7.69s p(50)=96ms p(95)=3.53s p(99)=4.15s
|
||||
login_ui_enter_login_name_duration...: min=210.14ms avg=210.14ms max=210.14ms p(50)=210.14ms p(95)=210.14ms p(99)=210.14ms
|
||||
login_ui_enter_password_duration.....: min=18.75ms avg=18.75ms max=18.75ms p(50)=18.75ms p(95)=18.75ms p(99)=18.75ms
|
||||
login_ui_init_login_duration.........: min=230.06ms avg=230.06ms max=230.06ms p(50)=230.06ms p(95)=230.06ms p(99)=230.06ms
|
||||
login_ui_token_duration..............: min=137.93ms avg=137.93ms max=137.93ms p(50)=137.93ms p(95)=137.93ms p(99)=137.93ms
|
||||
org_create_org_duration..............: min=57.15ms avg=57.15ms max=57.15ms p(50)=57.15ms p(95)=57.15ms p(99)=57.15ms
|
||||
session_add_session_duration.........: min=21.8ms avg=761.9ms max=7.69s p(50)=95.24ms p(95)=3.53s p(99)=4.15s
|
||||
user_create_human_duration...........: min=464.18ms avg=3.23s max=5.82s p(50)=3.06s p(95)=5.61s p(99)=5.75s
|
||||
|
||||
HTTP
|
||||
http_req_duration....................: min=18.75ms avg=762.65ms max=12.57s p(50)=95.25ms p(95)=3.53s p(99)=4.15s
|
||||
{ expected_response:true }.........: min=18.75ms avg=762.65ms max=12.57s p(50)=95.25ms p(95)=3.53s p(99)=4.15s
|
||||
http_req_failed......................: 0.00% 1 out of 1416546
|
||||
http_reqs............................: 1416546 767.358551/s
|
||||
|
||||
EXECUTION
|
||||
iteration_duration...................: min=23.47ms avg=763.22ms max=7.69s p(50)=96.45ms p(95)=3.53s p(99)=4.15s
|
||||
iterations...........................: 1415336 766.703081/s
|
||||
vus..................................: 0 min=0 max=600
|
||||
vus_max..............................: 600 min=600 max=600
|
||||
|
||||
NETWORK
|
||||
data_received........................: 1.1 GB 584 kB/s
|
||||
data_sent............................: 235 MB 127 kB/s
|
||||
|
||||
running (30m46.0s), 000/600 VUs, 1415336 complete and 0 interrupted iterations
|
||||
default ✓ [ 100% ] 600 VUs 30m0s
|
||||
```
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,104 @@
|
||||
---
|
||||
title: human password login benchmark of zitadel v4.17.1
|
||||
sidebar_label: human password login
|
||||
---
|
||||
|
||||
import OutputSource from "./output.json";
|
||||
import { BenchmarkChart } from "@/components/benchmark_chart";
|
||||
|
||||
Benchmark results of the v4.17.1 release of Zitadel.
|
||||
|
||||
## Performance test results
|
||||
|
||||
| Metric | Value |
|
||||
| :------------------------------------ | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| Baseline | none |
|
||||
| Purpose | Test current performance |
|
||||
| Test start | 2026-08-27 19:24:36 UTC |
|
||||
| Test end | 2026-08-27 19:54:52 UTC |
|
||||
| Test duration | 30min |
|
||||
| Executed test | human_password_login |
|
||||
| k6 version | v2.1.0 |
|
||||
| VUs | 600 |
|
||||
| Client location | US1 |
|
||||
| ZITADEL location | US1 |
|
||||
| ZITADEL container specification | vCPU: 6<br/> Memory: 6 Gi <br/>Container min scale: 7<br/>Container max scale: 7 |
|
||||
| ZITADEL Version | v4.17.1 |
|
||||
| ZITADEL Settings | Eventstore autovacuum (new in v4.17): Enabled: true<br/>VacuumThreshold: 1000000<br/>AnalyzeThreshold: 1000000<br/>Tuned for the ~4,000 events/s expected during the benchmark, per the [tuning guide](https://zitadel.com/docs/self-hosting/manage/tuning#reading-events). |
|
||||
| ZITADEL feature flags | Enabled: loginDefaultOrg (instance), oidcTokenExchange (system), enableBackChannelLogout (system), consoleUseV2UserApi (instance)<br/>improvedPerformance (instance): IMPROVED_PERFORMANCE_PROJECT_GRANT, IMPROVED_PERFORMANCE_PROJECT, IMPROVED_PERFORMANCE_USER_GRANT, IMPROVED_PERFORMANCE_ORG_DOMAIN_VERIFIED<br/>Disabled: loginV2 (explicitly false at instance), userSchema, debugOidcParentError, oidcSingleV1SessionTermination, permissionCheckV2<br/>Captured from the instance **before** the run (sequence 7, set 2026-08-27 13:35:31 UTC); the same capture covers `manipulate_user` at 13:41 and the rest of the sweep from 14:56, so all eleven targets ran the same configuration.<br/>This restores like-for-like comparison with the v4 pages, which ran five improvedPerformance options. The fifth, IMPROVED_PERFORMANCE_ORG_BY_ID, is `reserved` in the proto and no longer exists. |
|
||||
| Database | type: psql<br />version: v17.4 |
|
||||
| Database location | US1 |
|
||||
| Database specification | vCPU: 8<br/> memory: 32Gib |
|
||||
| ZITADEL metrics during test | CPU: p50 48%, p99 60% (peak 64%)<br/>Memory: p50 6%, peak 16%<br/>Instances: 7 (constant), no container startups recorded<br/>Server-side: 338 req/s, latency p50 207ms / p95 4.9s / p99 32.9s |
|
||||
| Observed errors | 9,227 of 683,256 requests failed (1.35%). **Corrected by hand from the generated row, which reported these as one population; the status breakdown shows two with different causes and different shapes.** (1) **8,178 request timeouts (1.20%), sustained across the whole run** — not a burst: they run from 19:26 to 19:54 at a steady 125–360 per minute, with no onset and no recovery. This is ZITADEL under 600 VUs on this target and is the number to read. (2) **1,044 HTTP 503 platform errors, confined to 19:37–19:44** — `Service error -27`, an HTML page from the Google frontend rather than a ZITADEL JSON error, in the same bounded-burst shape seen on `manipulate_user` at 13:43 the same day and pending the same GCP post-mortem (see `benchmark/postmortems/2026-08-27-503-burst.md`). Plus 4x HTTP 502 and 1x connection reset, scattered singles. 9,093 of 1,007,953 checks failed (0.90%): login name status ok (8,345), authorize status ok (333), token status ok (138), password status ok (137), userinfo status ok (119), password callback (21). The failed checks track the timeouts rather than the 503 burst. |
|
||||
| Top 3 most expensive database queries | 1: Projection handler advisory lock, one per projection per instance — **lock wait, not query work** (17,902 s)<br/>2: Eventstore read barrier: advisory lock + unlock on `events2` — **lock wait** (8,920 s)<br/>3: Query events by instance_id, aggregate_types, event_types, position (paged) (3,810 s)<br/>Ranked by total accumulated execution time summed across concurrent sessions (Cloud SQL Query Insights) — not CPU time; it can exceed the wall-clock window. |
|
||||
| Database metrics during test | CPU: 87% median, 90% peak<br/>Memory: 40%<br/>Connections: 46<br/>Transactions: 12,351/s<br/>IOPS: read 0, write 711<br/>**CPU-saturated — the bottleneck for this target.** |
|
||||
| k6 Iterations per second | 49 |
|
||||
| k6 output | [output](#k6-output) |
|
||||
| flowchart outcome | **Potential issues.** Not resource-bound: database CPU 87%, ZITADEL CPU 48%, yet **86% of the top-5 query time is advisory lock wait** — predominantly the projection handler lock, which serialises this workload. Additional vCPU will not help. Logged as an issue rather than blocking further scale testing. The run also contains a 1,044-request platform 503 burst (19:37-19:44) under separate investigation in `benchmark/postmortems/2026-08-27-503-burst.md`; it does not affect this verdict. |
|
||||
|
||||
## Endpoint latencies
|
||||
|
||||
<BenchmarkChart testResults={OutputSource} />
|
||||
|
||||
## k6 output
|
||||
|
||||
```bash
|
||||
█ TOTAL RESULTS
|
||||
|
||||
checks_total.......: 1007953 542.050139/s
|
||||
checks_succeeded...: 99.09% 998860 out of 1007953
|
||||
checks_failed......: 0.90% 9093 out of 1007953
|
||||
|
||||
✓ user defined
|
||||
✗ authorize status ok
|
||||
↳ 99% — ✓ 90339 / ✗ 333
|
||||
✗ login name status ok
|
||||
↳ 90% — ✓ 81994 / ✗ 8345
|
||||
✓ login shows password page
|
||||
✗ password status ok
|
||||
↳ 99% — ✓ 81702 / ✗ 137
|
||||
✗ password callback
|
||||
↳ 99% — ✓ 81681 / ✗ 21
|
||||
✓ code set
|
||||
✗ token status ok
|
||||
↳ 99% — ✓ 81543 / ✗ 138
|
||||
✓ access token created
|
||||
✓ id token created
|
||||
✓ info created
|
||||
✓ org created
|
||||
✓ create user is status ok
|
||||
✓ openid configuration
|
||||
✗ userinfo status ok
|
||||
↳ 99% — ✓ 81423 / ✗ 119
|
||||
✓ org removed
|
||||
|
||||
CUSTOM
|
||||
human_password_login_duration........: min=652ms avg=7.16s max=1m9s p(50)=4.76s p(95)=16.62s p(99)=1m2s
|
||||
login_ui_enter_login_name_duration...: min=65.11ms avg=3.4s max=1m0s p(50)=400.69ms p(95)=11.5s p(99)=59.35s
|
||||
login_ui_enter_password_duration.....: min=597.54µs avg=25.21ms max=8.43s p(50)=2.77ms p(95)=6.61ms p(99)=459.33ms
|
||||
login_ui_init_login_duration.........: min=29.53ms avg=505.93ms max=6.4s p(50)=94.64ms p(95)=1.77s p(99)=2.52s
|
||||
login_ui_token_duration..............: min=41.07ms avg=651.58ms max=7.5s p(50)=161ms p(95)=2.2s p(99)=3.04s
|
||||
oidc_user_info_duration..............: min=11.81ms avg=103.78ms max=2.79s p(50)=31.68ms p(95)=334.09ms p(99)=489.53ms
|
||||
org_create_org_duration..............: min=47.19ms avg=47.19ms max=47.19ms p(50)=47.19ms p(95)=47.19ms p(99)=47.19ms
|
||||
user_create_human_duration...........: min=349.88ms avg=3.46s max=6.17s p(50)=3.42s p(95)=5.8s p(99)=6.14s
|
||||
|
||||
HTTP
|
||||
http_req_duration....................: min=597.54µs avg=1.58s max=1m0s p(50)=146.67ms p(95)=4.08s p(99)=1m0s
|
||||
{ expected_response:true }.........: min=1.3ms avg=879.85ms max=1m0s p(50)=141.02ms p(95)=3.46s p(99)=9.21s
|
||||
http_req_failed......................: 1.35% 9227 out of 683256
|
||||
http_reqs............................: 683256 367.436785/s
|
||||
|
||||
EXECUTION
|
||||
iteration_duration...................: min=652.1ms avg=11.96s max=1m9s p(50)=5.2s p(95)=1m0s p(99)=1m2s
|
||||
iterations...........................: 90671 48.760436/s
|
||||
vus..................................: 0 min=0 max=600
|
||||
vus_max..............................: 600 min=600 max=600
|
||||
|
||||
NETWORK
|
||||
data_received........................: 1.5 GB 810 kB/s
|
||||
data_sent............................: 325 MB 175 kB/s
|
||||
|
||||
running (30m59.5s), 000/600 VUs, 90671 complete and 0 interrupted iterations
|
||||
default ✓ [ 100% ] 600 VUs 30m0s
|
||||
```
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,104 @@
|
||||
---
|
||||
title: introspect benchmark of zitadel v4.17.1
|
||||
sidebar_label: introspect
|
||||
---
|
||||
|
||||
import OutputSource from "./output.json";
|
||||
import { BenchmarkChart } from "@/components/benchmark_chart";
|
||||
|
||||
Benchmark results of the v4.17.1 release of Zitadel.
|
||||
|
||||
## Performance test results
|
||||
|
||||
| Metric | Value |
|
||||
| :------------------------------------ | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| Baseline | none |
|
||||
| Purpose | Test current performance |
|
||||
| Test start | 2026-08-27 15:31:22 UTC |
|
||||
| Test end | 2026-08-27 16:01:22 UTC |
|
||||
| Test duration | 30min |
|
||||
| Executed test | introspect |
|
||||
| k6 version | v2.1.0 |
|
||||
| VUs | 600 |
|
||||
| Client location | US1 |
|
||||
| ZITADEL location | US1 |
|
||||
| ZITADEL container specification | vCPU: 6<br/> Memory: 6 Gi <br/>Container min scale: 7<br/>Container max scale: 7 |
|
||||
| ZITADEL Version | v4.17.1 |
|
||||
| ZITADEL Settings | Eventstore autovacuum (new in v4.17): Enabled: true<br/>VacuumThreshold: 1000000<br/>AnalyzeThreshold: 1000000<br/>Tuned for the ~4,000 events/s expected during the benchmark, per the [tuning guide](https://zitadel.com/docs/self-hosting/manage/tuning#reading-events). |
|
||||
| ZITADEL feature flags | Enabled: loginDefaultOrg (instance), oidcTokenExchange (system), enableBackChannelLogout (system), consoleUseV2UserApi (instance)<br/>improvedPerformance (instance): IMPROVED_PERFORMANCE_PROJECT_GRANT, IMPROVED_PERFORMANCE_PROJECT, IMPROVED_PERFORMANCE_USER_GRANT, IMPROVED_PERFORMANCE_ORG_DOMAIN_VERIFIED<br/>Disabled: loginV2 (explicitly false at instance), userSchema, debugOidcParentError, oidcSingleV1SessionTermination, permissionCheckV2<br/>Captured from the instance **before** the run (sequence 7, set 2026-08-27 13:35:31 UTC); the same capture covers `manipulate_user` at 13:41 and the rest of the sweep from 14:56, so all eleven targets ran the same configuration.<br/>This restores like-for-like comparison with the v4 pages, which ran five improvedPerformance options. The fifth, IMPROVED_PERFORMANCE_ORG_BY_ID, is `reserved` in the proto and no longer exists. |
|
||||
| Database | type: psql<br />version: v17.4 |
|
||||
| Database location | US1 |
|
||||
| Database specification | vCPU: 8<br/> memory: 32Gib |
|
||||
| ZITADEL metrics during test | CPU: p50 39%, p99 50% (peak 58%)<br/>Memory: p50 6%, peak 15%<br/>Instances: 7 (constant), no container startups recorded<br/>Server-side: 2,325 req/s, latency p50 33ms / p95 1.1s / p99 1.3s |
|
||||
| Observed errors | 1,400 of 4,159,863 requests failed (0.03%): 2x connection reset, 1,398x not individually logged (see k6 output). 2,800 of 8,317,316 checks failed (0.03%): introspect status ok (1400 failed), introspect active (1400 failed) |
|
||||
| Top 3 most expensive database queries | 1: Query events by aggregate_id and event_types, ordered by sequence (4,034 s)<br/>2: Query events by instance_id, aggregate_types, event_types, position (paged) (1,765 s)<br/>3: Load user with primary login name and human profile (1,497 s)<br/>Ranked by total accumulated execution time summed across concurrent sessions (Cloud SQL Query Insights) — not CPU time; it can exceed the wall-clock window. |
|
||||
| Database metrics during test | CPU: 98% median, 99% peak<br/>Memory: 40%<br/>Connections: 46<br/>Transactions: 11,597/s<br/>IOPS: read 0, write 10<br/>**CPU-saturated — the bottleneck for this target.** |
|
||||
| k6 Iterations per second | 2297 |
|
||||
| k6 output | [output](#k6-output) |
|
||||
| flowchart outcome | **Scale.** Database CPU is the limit: 98% median against ZITADEL at 39%, with advisory lock wait only 0% of the top-5 query time. Raising the database beyond 8 vCPU is the next step, and throughput should be re-measured against the increase. |
|
||||
|
||||
## Endpoint latencies
|
||||
|
||||
<BenchmarkChart testResults={OutputSource} />
|
||||
|
||||
## k6 output
|
||||
|
||||
```bash
|
||||
█ TOTAL RESULTS
|
||||
|
||||
checks_total.......: 8317316 4594.588805/s
|
||||
checks_succeeded...: 99.96% 8314516 out of 8317316
|
||||
checks_failed......: 0.03% 2800 out of 8317316
|
||||
|
||||
✓ user defined
|
||||
✓ authorize status ok
|
||||
✓ login name status ok
|
||||
✓ login shows password page
|
||||
✓ password status ok
|
||||
✓ password callback
|
||||
✓ code set
|
||||
✓ token status ok
|
||||
✓ access token created
|
||||
✓ id token created
|
||||
✓ info created
|
||||
✓ org created
|
||||
✓ add project status ok
|
||||
✓ add api status ok
|
||||
✓ add app key status ok
|
||||
✓ openid configuration
|
||||
✗ introspect status ok
|
||||
↳ 99% — ✓ 4156046 / ✗ 1400
|
||||
✗ introspect active
|
||||
↳ 99% — ✓ 4156046 / ✗ 1400
|
||||
✓ org removed
|
||||
|
||||
CUSTOM
|
||||
app_add_app_duration.................: min=38.25ms avg=444.17ms max=964.29ms p(50)=442.27ms p(95)=874.88ms p(99)=906.87ms
|
||||
app_add_app_key_duration.............: min=110.68ms avg=1.1s max=2.09s p(50)=1.13s p(95)=1.87s p(99)=1.93s
|
||||
login_ui_enter_login_name_duration...: min=153.7ms avg=159.94ms max=166.17ms p(50)=159.94ms p(95)=165.55ms p(99)=166.05ms
|
||||
login_ui_enter_password_duration.....: min=5.47ms avg=8.75ms max=12.03ms p(50)=8.75ms p(95)=11.7ms p(99)=11.96ms
|
||||
login_ui_init_login_duration.........: min=64.27ms avg=78.53ms max=92.79ms p(50)=78.53ms p(95)=91.37ms p(99)=92.51ms
|
||||
login_ui_token_duration..............: min=157.39ms avg=205.82ms max=254.26ms p(50)=205.82ms p(95)=249.41ms p(99)=253.29ms
|
||||
oidc_introspect_duration.............: min=12.32ms avg=259.25ms max=23.99s p(50)=46.46ms p(95)=1.16s p(99)=1.36s
|
||||
org_create_org_duration..............: min=56.57ms avg=56.57ms max=56.57ms p(50)=56.57ms p(95)=56.57ms p(99)=56.57ms
|
||||
project_add_project_duration.........: min=41.65ms avg=355.6ms max=749.58ms p(50)=352.72ms p(95)=626.33ms p(99)=682.79ms
|
||||
|
||||
HTTP
|
||||
http_req_duration....................: min=5.47ms avg=259.38ms max=23.99s p(50)=46.49ms p(95)=1.16s p(99)=1.36s
|
||||
{ expected_response:true }.........: min=5.47ms avg=258.8ms max=23.99s p(50)=46.47ms p(95)=1.16s p(99)=1.36s
|
||||
http_req_failed......................: 0.03% 1400 out of 4159863
|
||||
http_reqs............................: 4159863 2297.960059/s
|
||||
|
||||
EXECUTION
|
||||
iteration_duration...................: min=12.55ms avg=259.71ms max=23.99s p(50)=46.87ms p(95)=1.16s p(99)=1.36s
|
||||
iterations...........................: 4157446 2296.624878/s
|
||||
vus..................................: 0 min=0 max=600
|
||||
vus_max..............................: 600 min=600 max=600
|
||||
|
||||
NETWORK
|
||||
data_received........................: 195 GB 108 MB/s
|
||||
data_sent............................: 5.1 GB 2.8 MB/s
|
||||
|
||||
running (30m10.2s), 000/600 VUs, 4157446 complete and 0 interrupted iterations
|
||||
default ✓ [ 100% ] 600 VUs 30m0s
|
||||
```
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,104 @@
|
||||
---
|
||||
title: machine client credentials login benchmark of zitadel v4.17.1
|
||||
sidebar_label: machine client credentials login
|
||||
---
|
||||
|
||||
import OutputSource from "./output.json";
|
||||
import { BenchmarkChart } from "@/components/benchmark_chart";
|
||||
|
||||
Benchmark results of the v4.17.1 release of Zitadel.
|
||||
|
||||
## Performance test results
|
||||
|
||||
| Metric | Value |
|
||||
| :------------------------------------ | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| Baseline | none |
|
||||
| Purpose | Test current performance |
|
||||
| Test start | 2026-08-27 16:03:44 UTC |
|
||||
| Test end | 2026-08-27 16:33:45 UTC |
|
||||
| Test duration | 30min |
|
||||
| Executed test | machine_client_credentials_login |
|
||||
| k6 version | v2.1.0 |
|
||||
| VUs | 600 |
|
||||
| Client location | US1 |
|
||||
| ZITADEL location | US1 |
|
||||
| ZITADEL container specification | vCPU: 6<br/> Memory: 6 Gi <br/>Container min scale: 7<br/>Container max scale: 7 |
|
||||
| ZITADEL Version | v4.17.1 |
|
||||
| ZITADEL Settings | Eventstore autovacuum (new in v4.17): Enabled: true<br/>VacuumThreshold: 1000000<br/>AnalyzeThreshold: 1000000<br/>Tuned for the ~4,000 events/s expected during the benchmark, per the [tuning guide](https://zitadel.com/docs/self-hosting/manage/tuning#reading-events). |
|
||||
| ZITADEL feature flags | Enabled: loginDefaultOrg (instance), oidcTokenExchange (system), enableBackChannelLogout (system), consoleUseV2UserApi (instance)<br/>improvedPerformance (instance): IMPROVED_PERFORMANCE_PROJECT_GRANT, IMPROVED_PERFORMANCE_PROJECT, IMPROVED_PERFORMANCE_USER_GRANT, IMPROVED_PERFORMANCE_ORG_DOMAIN_VERIFIED<br/>Disabled: loginV2 (explicitly false at instance), userSchema, debugOidcParentError, oidcSingleV1SessionTermination, permissionCheckV2<br/>Captured from the instance **before** the run (sequence 7, set 2026-08-27 13:35:31 UTC); the same capture covers `manipulate_user` at 13:41 and the rest of the sweep from 14:56, so all eleven targets ran the same configuration.<br/>This restores like-for-like comparison with the v4 pages, which ran five improvedPerformance options. The fifth, IMPROVED_PERFORMANCE_ORG_BY_ID, is `reserved` in the proto and no longer exists. |
|
||||
| Database | type: psql<br />version: v17.4 |
|
||||
| Database location | US1 |
|
||||
| Database specification | vCPU: 8<br/> memory: 32Gib |
|
||||
| ZITADEL metrics during test | CPU: p50 26%, p99 36% (peak 44%)<br/>Memory: p50 6%, peak 12%<br/>Instances: 7 (constant), no container startups recorded<br/>Server-side: 1,674 req/s, latency p50 46ms / p95 1.8s / p99 2.3s |
|
||||
| Observed errors | 1,789 of 3,073,497 requests failed (0.05%): 3x connection reset, 1,786x not individually logged (see k6 output). 1 iteration aborted on a null response body. 1,789 of 4,607,991 checks failed (0.03%): client credentials status ok (905 failed), userinfo status ok (884 failed) |
|
||||
| Top 3 most expensive database queries | 1: Query events by aggregate_id and event_types, ordered by sequence (4,057 s)<br/>2: Push events (`eventstore.push`) (3,810 s)<br/>3: Query events by instance_id, aggregate_types, event_types, position (paged) (3,573 s)<br/>Ranked by total accumulated execution time summed across concurrent sessions (Cloud SQL Query Insights) — not CPU time; it can exceed the wall-clock window. |
|
||||
| Database metrics during test | CPU: 96% median, 98% peak<br/>Memory: 40%<br/>Connections: 46<br/>Transactions: 10,804/s<br/>IOPS: read 0, write 857<br/>**CPU-saturated — the bottleneck for this target.** |
|
||||
| k6 Iterations per second | 834 |
|
||||
| k6 output | [output](#k6-output) |
|
||||
| flowchart outcome | **Scale.** Database CPU is the limit: 96% median against ZITADEL at 26%, with advisory lock wait only 0% of the top-5 query time. Raising the database beyond 8 vCPU is the next step, and throughput should be re-measured against the increase. |
|
||||
|
||||
## Endpoint latencies
|
||||
|
||||
<BenchmarkChart testResults={OutputSource} />
|
||||
|
||||
## k6 output
|
||||
|
||||
```bash
|
||||
█ TOTAL RESULTS
|
||||
|
||||
checks_total.......: 4607991 2501.878593/s
|
||||
checks_succeeded...: 99.96% 4606202 out of 4607991
|
||||
checks_failed......: 0.03% 1789 out of 4607991
|
||||
|
||||
✓ user defined
|
||||
✓ authorize status ok
|
||||
✓ login name status ok
|
||||
✓ login shows password page
|
||||
✓ password status ok
|
||||
✓ password callback
|
||||
✓ code set
|
||||
✓ token status ok
|
||||
✓ access token created
|
||||
✓ id token created
|
||||
✓ info created
|
||||
✓ org created
|
||||
✓ create user is status ok
|
||||
✓ generate machine secret status ok
|
||||
✓ openid configuration
|
||||
✗ client credentials status ok
|
||||
↳ 99% — ✓ 1535091 / ✗ 905
|
||||
✓ client credentials token ok
|
||||
✗ userinfo status ok
|
||||
↳ 99% — ✓ 1534207 / ✗ 884
|
||||
✓ org removed
|
||||
|
||||
CUSTOM
|
||||
login_ui_enter_login_name_duration...: min=140.15ms avg=140.15ms max=140.15ms p(50)=140.15ms p(95)=140.15ms p(99)=140.15ms
|
||||
login_ui_enter_password_duration.....: min=17.02ms avg=17.02ms max=17.02ms p(50)=17.02ms p(95)=17.02ms p(99)=17.02ms
|
||||
login_ui_init_login_duration.........: min=73.97ms avg=73.97ms max=73.97ms p(50)=73.97ms p(95)=73.97ms p(99)=73.97ms
|
||||
login_ui_token_duration..............: min=188.55ms avg=188.55ms max=188.55ms p(50)=188.55ms p(95)=188.55ms p(99)=188.55ms
|
||||
oidc_client_credentials_duration.....: min=0s avg=434.03ms max=7.96s p(50)=69.73ms p(95)=2.16s p(99)=2.56s
|
||||
oidc_user_info_duration..............: min=0s avg=268.43ms max=8.81s p(50)=39.35ms p(95)=1.35s p(99)=1.61s
|
||||
org_create_org_duration..............: min=65.77ms avg=65.77ms max=65.77ms p(50)=65.77ms p(95)=65.77ms p(99)=65.77ms
|
||||
user_add_machine_secret_duration.....: min=47.04ms avg=3.8s max=7.63s p(50)=3.75s p(95)=7.26s p(99)=7.54s
|
||||
user_create_machine_duration.........: min=45.5ms avg=631.9ms max=946.99ms p(50)=637.66ms p(95)=867.81ms p(99)=881.99ms
|
||||
|
||||
HTTP
|
||||
http_req_duration....................: min=0s avg=351.86ms max=8.81s p(50)=56.78ms p(95)=1.74s p(99)=2.42s
|
||||
{ expected_response:true }.........: min=9.43ms avg=350.9ms max=8.81s p(50)=56.73ms p(95)=1.74s p(99)=2.42s
|
||||
http_req_failed......................: 0.05% 1789 out of 3073497
|
||||
http_reqs............................: 3073497 1668.735106/s
|
||||
|
||||
EXECUTION
|
||||
iteration_duration...................: min=833.79µs avg=703.15ms max=9.19s p(50)=332.64ms p(95)=2.39s p(99)=3.27s
|
||||
iterations...........................: 1535996 833.958988/s
|
||||
vus..................................: 0 min=0 max=600
|
||||
vus_max..............................: 600 min=600 max=600
|
||||
|
||||
NETWORK
|
||||
data_received........................: 3.4 GB 1.9 MB/s
|
||||
data_sent............................: 1.3 GB 692 kB/s
|
||||
|
||||
running (30m41.8s), 000/600 VUs, 1535996 complete and 0 interrupted iterations
|
||||
default ✓ [ 100% ] 600 VUs 30m0s
|
||||
```
|
||||
+3606
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,101 @@
|
||||
---
|
||||
title: machine jwt profile grant benchmark of zitadel v4.17.1
|
||||
sidebar_label: machine jwt profile grant
|
||||
---
|
||||
|
||||
import OutputSource from "./output.json";
|
||||
import { BenchmarkChart } from "@/components/benchmark_chart";
|
||||
|
||||
Benchmark results of the v4.17.1 release of Zitadel.
|
||||
|
||||
## Performance test results
|
||||
|
||||
| Metric | Value |
|
||||
| :------------------------------------ | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| Baseline | none |
|
||||
| Purpose | Test current performance |
|
||||
| Test start | 2026-08-27 16:34:57 UTC |
|
||||
| Test end | 2026-08-27 17:04:57 UTC |
|
||||
| Test duration | 30min |
|
||||
| Executed test | machine_jwt_profile_grant |
|
||||
| k6 version | v2.1.0 |
|
||||
| VUs | 600 |
|
||||
| Client location | US1 |
|
||||
| ZITADEL location | US1 |
|
||||
| ZITADEL container specification | vCPU: 6<br/> Memory: 6 Gi <br/>Container min scale: 7<br/>Container max scale: 7 |
|
||||
| ZITADEL Version | v4.17.1 |
|
||||
| ZITADEL Settings | Eventstore autovacuum (new in v4.17): Enabled: true<br/>VacuumThreshold: 1000000<br/>AnalyzeThreshold: 1000000<br/>Tuned for the ~4,000 events/s expected during the benchmark, per the [tuning guide](https://zitadel.com/docs/self-hosting/manage/tuning#reading-events). |
|
||||
| ZITADEL feature flags | Enabled: loginDefaultOrg (instance), oidcTokenExchange (system), enableBackChannelLogout (system), consoleUseV2UserApi (instance)<br/>improvedPerformance (instance): IMPROVED_PERFORMANCE_PROJECT_GRANT, IMPROVED_PERFORMANCE_PROJECT, IMPROVED_PERFORMANCE_USER_GRANT, IMPROVED_PERFORMANCE_ORG_DOMAIN_VERIFIED<br/>Disabled: loginV2 (explicitly false at instance), userSchema, debugOidcParentError, oidcSingleV1SessionTermination, permissionCheckV2<br/>Captured from the instance **before** the run (sequence 7, set 2026-08-27 13:35:31 UTC); the same capture covers `manipulate_user` at 13:41 and the rest of the sweep from 14:56, so all eleven targets ran the same configuration.<br/>This restores like-for-like comparison with the v4 pages, which ran five improvedPerformance options. The fifth, IMPROVED_PERFORMANCE_ORG_BY_ID, is `reserved` in the proto and no longer exists. |
|
||||
| Database | type: psql<br />version: v17.4 |
|
||||
| Database location | US1 |
|
||||
| Database specification | vCPU: 8<br/> memory: 32Gib |
|
||||
| ZITADEL metrics during test | CPU: p50 22%, p99 31% (peak 36%)<br/>Memory: p50 6%, peak 13%<br/>Instances: 7 (constant), no container startups recorded<br/>Server-side: 1,108 req/s, latency p50 70ms / p95 2.6s / p99 2.8s |
|
||||
| Observed errors | 97 of 1,986,156 requests failed (0.00%): none of the 97 individually logged (see k6 output). 194 of 3,969,305 checks failed (0.00%): token status ok (97 failed), access token returned (97 failed) |
|
||||
| Top 3 most expensive database queries | 1: Push events (`eventstore.push`) (6,040 s)<br/>2: Query events by instance_id, aggregate_types, event_types, position (paged) (1,773 s)<br/>3: Load user with primary login name and human profile (1,022 s)<br/>Ranked by total accumulated execution time summed across concurrent sessions (Cloud SQL Query Insights) — not CPU time; it can exceed the wall-clock window. |
|
||||
| Database metrics during test | CPU: 96% median, 97% peak<br/>Memory: 40%<br/>Connections: 46<br/>Transactions: 8,909/s<br/>IOPS: read 0, write 887<br/>**CPU-saturated — the bottleneck for this target.** |
|
||||
| k6 Iterations per second | 1076 |
|
||||
| k6 output | [output](#k6-output) |
|
||||
| flowchart outcome | **Scale.** Database CPU is the limit: 96% median against ZITADEL at 22%, with advisory lock wait only 7% of the top-5 query time. Raising the database beyond 8 vCPU is the next step, and throughput should be re-measured against the increase. Lock share fell from 64.8% to 7.3% when `improvedPerformance` was enabled, moving this target from lock-bound to CPU-bound. |
|
||||
|
||||
## Endpoint latencies
|
||||
|
||||
<BenchmarkChart testResults={OutputSource} />
|
||||
|
||||
## k6 output
|
||||
|
||||
```bash
|
||||
█ TOTAL RESULTS
|
||||
|
||||
checks_total.......: 3969305 2152.313897/s
|
||||
checks_succeeded...: 99.99% 3969111 out of 3969305
|
||||
checks_failed......: 0.00% 194 out of 3969305
|
||||
|
||||
✓ user defined
|
||||
✓ authorize status ok
|
||||
✓ login name status ok
|
||||
✓ login shows password page
|
||||
✓ password status ok
|
||||
✓ password callback
|
||||
✓ code set
|
||||
✗ token status ok
|
||||
↳ 99% — ✓ 1983650 / ✗ 97
|
||||
✓ access token created
|
||||
✓ id token created
|
||||
✓ info created
|
||||
✓ org created
|
||||
✓ create user is status ok
|
||||
✓ generate machine key status ok
|
||||
✓ openid configuration
|
||||
✗ access token returned
|
||||
↳ 99% — ✓ 1983649 / ✗ 97
|
||||
✓ org removed
|
||||
|
||||
CUSTOM
|
||||
login_ui_enter_login_name_duration...: min=136.8ms avg=136.8ms max=136.8ms p(50)=136.8ms p(95)=136.8ms p(99)=136.8ms
|
||||
login_ui_enter_password_duration.....: min=20.15ms avg=20.15ms max=20.15ms p(50)=20.15ms p(95)=20.15ms p(99)=20.15ms
|
||||
login_ui_init_login_duration.........: min=96.66ms avg=96.66ms max=96.66ms p(50)=96.66ms p(95)=96.66ms p(99)=96.66ms
|
||||
login_ui_token_duration..............: min=166.72ms avg=166.72ms max=166.72ms p(50)=166.72ms p(95)=166.72ms p(99)=166.72ms
|
||||
oidc_token_duration..................: min=19.56ms avg=541.42ms max=5.37s p(50)=85.31ms p(95)=2.57s p(99)=2.93s
|
||||
org_create_org_duration..............: min=63.41ms avg=63.41ms max=63.41ms p(50)=63.41ms p(95)=63.41ms p(99)=63.41ms
|
||||
user_add_machine_key_duration........: min=28.22ms avg=167.45ms max=375.9ms p(50)=174.58ms p(95)=304.34ms p(99)=355.68ms
|
||||
user_create_machine_duration.........: min=112.78ms avg=534.68ms max=798.43ms p(50)=541.02ms p(95)=722.05ms p(99)=769.68ms
|
||||
|
||||
HTTP
|
||||
http_req_duration....................: min=12.65ms avg=541.03ms max=12.01s p(50)=85.34ms p(95)=2.57s p(99)=2.93s
|
||||
{ expected_response:true }.........: min=12.65ms avg=541.04ms max=12.01s p(50)=85.33ms p(95)=2.57s p(99)=2.93s
|
||||
http_req_failed......................: 0.00% 97 out of 1986156
|
||||
http_reqs............................: 1986156 1076.972206/s
|
||||
|
||||
EXECUTION
|
||||
iteration_duration...................: min=21.94ms avg=544.43ms max=5.38s p(50)=88.19ms p(95)=2.57s p(99)=2.93s
|
||||
iterations...........................: 1983746 1075.665408/s
|
||||
vus..................................: 0 min=0 max=600
|
||||
vus_max..............................: 600 min=600 max=600
|
||||
|
||||
NETWORK
|
||||
data_received........................: 3.2 GB 1.7 MB/s
|
||||
data_sent............................: 1.7 GB 911 kB/s
|
||||
|
||||
running (30m44.2s), 000/600 VUs, 1983746 complete and 0 interrupted iterations
|
||||
default ✓ [ 100% ] 600 VUs 30m0s
|
||||
```
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,100 @@
|
||||
---
|
||||
title: machine pat login benchmark of zitadel v4.17.1
|
||||
sidebar_label: machine pat login
|
||||
---
|
||||
|
||||
import OutputSource from "./output.json";
|
||||
import { BenchmarkChart } from "@/components/benchmark_chart";
|
||||
|
||||
Benchmark results of the v4.17.1 release of Zitadel.
|
||||
|
||||
## Performance test results
|
||||
|
||||
| Metric | Value |
|
||||
| :------------------------------------ | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| Baseline | none |
|
||||
| Purpose | Test current performance |
|
||||
| Test start | 2026-08-27 17:06:10 UTC |
|
||||
| Test end | 2026-08-27 17:36:10 UTC |
|
||||
| Test duration | 30min |
|
||||
| Executed test | machine_pat_login |
|
||||
| k6 version | v2.1.0 |
|
||||
| VUs | 600 |
|
||||
| Client location | US1 |
|
||||
| ZITADEL location | US1 |
|
||||
| ZITADEL container specification | vCPU: 6<br/> Memory: 6 Gi <br/>Container min scale: 7<br/>Container max scale: 7 |
|
||||
| ZITADEL Version | v4.17.1 |
|
||||
| ZITADEL Settings | Eventstore autovacuum (new in v4.17): Enabled: true<br/>VacuumThreshold: 1000000<br/>AnalyzeThreshold: 1000000<br/>Tuned for the ~4,000 events/s expected during the benchmark, per the [tuning guide](https://zitadel.com/docs/self-hosting/manage/tuning#reading-events). |
|
||||
| ZITADEL feature flags | Enabled: loginDefaultOrg (instance), oidcTokenExchange (system), enableBackChannelLogout (system), consoleUseV2UserApi (instance)<br/>improvedPerformance (instance): IMPROVED_PERFORMANCE_PROJECT_GRANT, IMPROVED_PERFORMANCE_PROJECT, IMPROVED_PERFORMANCE_USER_GRANT, IMPROVED_PERFORMANCE_ORG_DOMAIN_VERIFIED<br/>Disabled: loginV2 (explicitly false at instance), userSchema, debugOidcParentError, oidcSingleV1SessionTermination, permissionCheckV2<br/>Captured from the instance **before** the run (sequence 7, set 2026-08-27 13:35:31 UTC); the same capture covers `manipulate_user` at 13:41 and the rest of the sweep from 14:56, so all eleven targets ran the same configuration.<br/>This restores like-for-like comparison with the v4 pages, which ran five improvedPerformance options. The fifth, IMPROVED_PERFORMANCE_ORG_BY_ID, is `reserved` in the proto and no longer exists. |
|
||||
| Database | type: psql<br />version: v17.4 |
|
||||
| Database location | US1 |
|
||||
| Database specification | vCPU: 8<br/> memory: 32Gib |
|
||||
| ZITADEL metrics during test | CPU: p50 29%, p99 46% (peak 50%)<br/>Memory: p50 6%, peak 14%<br/>Instances: 7 (constant), no container startups recorded<br/>Server-side: 3,562 req/s, latency p50 11ms / p95 863ms / p99 1.0s |
|
||||
| Observed errors | 29 of 6,335,633 requests failed (0.00%): 29x connection reset. 29 of 6,335,036 checks failed (0.00%): userinfo status ok (29 failed) |
|
||||
| Top 3 most expensive database queries | 1: Load user with primary login name and human profile (2,135 s)<br/>2: Query events by instance_id, aggregate_types, event_types, position (paged) (1,778 s)<br/>3: Query events by aggregate_id, event_types and created_at (474 s)<br/>Ranked by total accumulated execution time summed across concurrent sessions (Cloud SQL Query Insights) — not CPU time; it can exceed the wall-clock window. |
|
||||
| Database metrics during test | CPU: 86% median, 95% peak<br/>Memory: 40%<br/>Connections: 46<br/>Transactions: 21,329/s<br/>IOPS: read 0, write 12<br/>**CPU-saturated — the bottleneck for this target.** |
|
||||
| k6 Iterations per second | 3452 |
|
||||
| k6 output | [output](#k6-output) |
|
||||
| flowchart outcome | **Scale.** Database CPU is the limit: 86% median against ZITADEL at 29%, with advisory lock wait only 0% of the top-5 query time. Raising the database beyond 8 vCPU is the next step, and throughput should be re-measured against the increase. |
|
||||
|
||||
## Endpoint latencies
|
||||
|
||||
<BenchmarkChart testResults={OutputSource} />
|
||||
|
||||
## k6 output
|
||||
|
||||
```bash
|
||||
█ TOTAL RESULTS
|
||||
|
||||
checks_total.......: 6335036 3453.251902/s
|
||||
checks_succeeded...: 99.99% 6335007 out of 6335036
|
||||
checks_failed......: 0.00% 29 out of 6335036
|
||||
|
||||
✓ user defined
|
||||
✓ authorize status ok
|
||||
✓ login name status ok
|
||||
✓ login shows password page
|
||||
✓ password status ok
|
||||
✓ password callback
|
||||
✓ code set
|
||||
✓ token status ok
|
||||
✓ access token created
|
||||
✓ id token created
|
||||
✓ info created
|
||||
✓ org created
|
||||
✓ create user is status ok
|
||||
✓ add pat status ok
|
||||
✓ openid configuration
|
||||
✗ userinfo status ok
|
||||
↳ 99% — ✓ 6333194 / ✗ 29
|
||||
✓ org removed
|
||||
|
||||
CUSTOM
|
||||
login_ui_enter_login_name_duration...: min=258.33ms avg=258.33ms max=258.33ms p(50)=258.33ms p(95)=258.33ms p(99)=258.33ms
|
||||
login_ui_enter_password_duration.....: min=15.09ms avg=15.09ms max=15.09ms p(50)=15.09ms p(95)=15.09ms p(99)=15.09ms
|
||||
login_ui_init_login_duration.........: min=77.89ms avg=77.89ms max=77.89ms p(50)=77.89ms p(95)=77.89ms p(99)=77.89ms
|
||||
login_ui_token_duration..............: min=115.52ms avg=115.52ms max=115.52ms p(50)=115.52ms p(95)=115.52ms p(99)=115.52ms
|
||||
oidc_user_info_duration..............: min=0s avg=170.27ms max=5.6s p(50)=24.52ms p(95)=872.17ms p(99)=1s
|
||||
org_create_org_duration..............: min=85.33ms avg=85.33ms max=85.33ms p(50)=85.33ms p(95)=85.33ms p(99)=85.33ms
|
||||
user_add_machine_pat_duration........: min=32.07ms avg=290.84ms max=519.66ms p(50)=321.25ms p(95)=460.73ms p(99)=480.62ms
|
||||
user_create_machine_duration.........: min=62.02ms avg=529.19ms max=757.86ms p(50)=580.59ms p(95)=686.54ms p(99)=718.02ms
|
||||
|
||||
HTTP
|
||||
http_req_duration....................: min=0s avg=170.29ms max=5.6s p(50)=24.53ms p(95)=872.12ms p(99)=1s
|
||||
{ expected_response:true }.........: min=9.31ms avg=170.29ms max=5.6s p(50)=24.53ms p(95)=872.12ms p(99)=1s
|
||||
http_req_failed......................: 0.00% 29 out of 6335633
|
||||
http_reqs............................: 6335633 3453.577329/s
|
||||
|
||||
EXECUTION
|
||||
iteration_duration...................: min=349.8µs avg=170.49ms max=5.6s p(50)=24.71ms p(95)=872.38ms p(99)=1s
|
||||
iterations...........................: 6333223 3452.263629/s
|
||||
vus..................................: 0 min=0 max=600
|
||||
vus_max..............................: 600 min=600 max=600
|
||||
|
||||
NETWORK
|
||||
data_received........................: 5.4 GB 2.9 MB/s
|
||||
data_sent............................: 508 MB 277 kB/s
|
||||
|
||||
running (30m34.5s), 000/600 VUs, 6333223 complete and 0 interrupted iterations
|
||||
default ✓ [ 100% ] 600 VUs 30m0s
|
||||
```
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,104 @@
|
||||
---
|
||||
title: manipulate user benchmark of zitadel v4.17.1
|
||||
sidebar_label: manipulate user
|
||||
---
|
||||
|
||||
import OutputSource from "./output.json";
|
||||
import { BenchmarkChart } from "@/components/benchmark_chart";
|
||||
|
||||
Benchmark results of the v4.17.1 release of Zitadel.
|
||||
|
||||
## Performance test results
|
||||
|
||||
| Metric | Value |
|
||||
| :------------------------------------ | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| Baseline | none |
|
||||
| Purpose | Test current performance |
|
||||
| Test start | 2026-08-27 13:41:19 UTC |
|
||||
| Test end | 2026-08-27 14:11:30 UTC |
|
||||
| Test duration | 30min |
|
||||
| Executed test | manipulate_user |
|
||||
| k6 version | v2.1.0 |
|
||||
| VUs | 200 |
|
||||
| Client location | US1 |
|
||||
| ZITADEL location | US1 |
|
||||
| ZITADEL container specification | vCPU: 6<br/> Memory: 6 Gi <br/>Container min scale: 7<br/>Container max scale: 7 |
|
||||
| ZITADEL Version | v4.17.1 |
|
||||
| ZITADEL Settings | Eventstore autovacuum (new in v4.17): Enabled: true<br/>VacuumThreshold: 1000000<br/>AnalyzeThreshold: 1000000<br/>Tuned for the ~4,000 events/s expected during the benchmark, per the [tuning guide](https://zitadel.com/docs/self-hosting/manage/tuning#reading-events). |
|
||||
| ZITADEL feature flags | Enabled: loginDefaultOrg (instance), oidcTokenExchange (system), enableBackChannelLogout (system), consoleUseV2UserApi (instance)<br/>improvedPerformance (instance): IMPROVED_PERFORMANCE_PROJECT_GRANT, IMPROVED_PERFORMANCE_PROJECT, IMPROVED_PERFORMANCE_USER_GRANT, IMPROVED_PERFORMANCE_ORG_DOMAIN_VERIFIED<br/>Disabled: loginV2 (explicitly false at instance), userSchema, debugOidcParentError, oidcSingleV1SessionTermination, permissionCheckV2<br/>Captured from the instance **before** the run (sequence 7, set 2026-08-27 13:35:31 UTC); the same capture covers `manipulate_user` at 13:41 and the rest of the sweep from 14:56, so all eleven targets ran the same configuration.<br/>This restores like-for-like comparison with the v4 pages, which ran five improvedPerformance options. The fifth, IMPROVED_PERFORMANCE_ORG_BY_ID, is `reserved` in the proto and no longer exists. |
|
||||
| Database | type: psql<br />version: v17.4 |
|
||||
| Database location | US1 |
|
||||
| Database specification | vCPU: 8<br/> memory: 32Gib |
|
||||
| ZITADEL metrics during test | CPU: p50 16%, p99 23% (peak 41%)<br/>Memory: p50 7%, peak 12%<br/>Instances: 7 (constant), no container startups recorded<br/>Server-side: 89 req/s, latency p50 300ms / p95 12.9s / p99 36.6s |
|
||||
| Observed errors | 225 of 171,987 requests failed (0.13%). **Corrected by hand from the generated row, which reported these as a single population; they are two unrelated ones, 25 minutes apart.** (1) **150 platform errors, not ZITADEL errors**: HTTP 503 `Service error -27`, an HTML page from the Google frontend rather than a ZITADEL JSON error, confined entirely to a 55-second burst from 13:43:19 to 13:44:13 UTC, two minutes into the run, with no recurrence in the following 25 minutes. The shape is a container restart or replacement; a GCP post-mortem (restarts, panics, instance count) is pending and its findings so far are recorded in `benchmark/postmortems/2026-08-27-503-burst.md`. (2) **75 request timeouts** from 14:08:48 to 14:11:02 UTC, the documented degradation-over-sustained-churn behaviour of this target: the first arrives 1,658s (~28min) into the run and they cluster in the final minutes alongside a steady latency climb. The previous 200-VU run saw the equivalent 77 timeouts beginning at 1,255s. 127 of 137,522 checks failed (0.09%): create user (29), update user (41), lock user (25), delete user (32) — all of them within the two windows above. **The failed-check count is not comparable to the previous run's zero**: before commit `cdde3ab3e` a timeout killed the iteration before its check could be recorded (77 timeouts, 77 aborted iterations, 0 recorded check failures); with the response body now guarded, 75 timeouts aborted **0** iterations and the failures are recorded instead of being silently fatal. **This target ran at 200 VUs, not the 600 used for every other target**: at 600 VUs the same failure begins after 179s and produces a far higher error rate for identical throughput. |
|
||||
| Top 3 most expensive database queries | 1: Query events by a multi-aggregate / event_type filter set (13,352 s)<br/>2: Projection handler advisory lock, one per projection per instance — **lock wait, not query work** (6,438 s)<br/>3: Query events by instance_id, aggregate_types, event_types, position (paged) (3,970 s)<br/>Ranked by total accumulated execution time summed across concurrent sessions (Cloud SQL Query Insights) — not CPU time; it can exceed the wall-clock window. |
|
||||
| Database metrics during test | CPU: 98% median, 100% peak<br/>Memory: 40%<br/>Connections: 46<br/>Transactions: 1,487/s<br/>IOPS: read 0, write 539<br/>**CPU-saturated — the bottleneck for this target.** |
|
||||
| k6 Iterations per second | 19 |
|
||||
| k6 output | [output](#k6-output) |
|
||||
| flowchart outcome | **Potential issues.** Not resource-bound: database CPU 98%, ZITADEL CPU 16%, yet **35% of the top-5 query time is advisory lock wait** — predominantly the projection handler lock, which serialises this workload. Additional vCPU will not help. Logged as an issue rather than blocking further scale testing. This target also shows the documented degradation over sustained single-org churn, with per-operation latency climbing across the run. |
|
||||
|
||||
## Endpoint latencies
|
||||
|
||||
<BenchmarkChart testResults={OutputSource} />
|
||||
|
||||
## k6 output
|
||||
|
||||
```bash
|
||||
█ TOTAL RESULTS
|
||||
|
||||
checks_total.......: 137522 75.453808/s
|
||||
checks_succeeded...: 99.90% 137395 out of 137522
|
||||
checks_failed......: 0.09% 127 out of 137522
|
||||
|
||||
✓ user defined
|
||||
✓ authorize status ok
|
||||
✓ login name status ok
|
||||
✓ login shows password page
|
||||
✓ password status ok
|
||||
✓ password callback
|
||||
✓ code set
|
||||
✓ token status ok
|
||||
✓ access token created
|
||||
✓ id token created
|
||||
✓ info created
|
||||
✓ org created
|
||||
✗ create user is status ok
|
||||
↳ 99% — ✓ 34370 / ✗ 29
|
||||
✗ update user is status ok
|
||||
↳ 99% — ✓ 34329 / ✗ 41
|
||||
✗ lock user is status ok
|
||||
↳ 99% — ✓ 34345 / ✗ 25
|
||||
✗ delete user is status ok
|
||||
↳ 99% — ✓ 34338 / ✗ 32
|
||||
✓ org removed
|
||||
|
||||
CUSTOM
|
||||
delete_user_duration.................: min=43.5ms avg=987.45ms max=17.06s p(50)=322.03ms p(95)=5.07s p(99)=8.54s
|
||||
lock_user_duration...................: min=24.92ms avg=138.46ms max=4.11s p(50)=108.29ms p(95)=329.45ms p(99)=502.43ms
|
||||
login_ui_enter_login_name_duration...: min=154.9ms avg=154.9ms max=154.9ms p(50)=154.9ms p(95)=154.9ms p(99)=154.9ms
|
||||
login_ui_enter_password_duration.....: min=14.56ms avg=14.56ms max=14.56ms p(50)=14.56ms p(95)=14.56ms p(99)=14.56ms
|
||||
login_ui_init_login_duration.........: min=109.41ms avg=109.41ms max=109.41ms p(50)=109.41ms p(95)=109.41ms p(99)=109.41ms
|
||||
login_ui_token_duration..............: min=85.43ms avg=85.43ms max=85.43ms p(50)=85.43ms p(95)=85.43ms p(99)=85.43ms
|
||||
org_create_org_duration..............: min=57.64ms avg=57.64ms max=57.64ms p(50)=57.64ms p(95)=57.64ms p(99)=57.64ms
|
||||
update_human_duration................: min=17.01ms avg=116.71ms max=2.02s p(50)=87.97ms p(95)=291.12ms p(99)=441.72ms
|
||||
user_create_human_duration...........: min=295.4ms avg=461.62ms max=2.28s p(50)=428.92ms p(95)=679.59ms p(99)=966.08ms
|
||||
|
||||
HTTP
|
||||
http_req_duration....................: min=14.56ms avg=2.09s max=1m0s p(50)=304.58ms p(95)=10.23s p(99)=38.43s
|
||||
{ expected_response:true }.........: min=14.56ms avg=2.07s max=59.92s p(50)=303.74ms p(95)=10.08s p(99)=38.08s
|
||||
http_req_failed......................: 0.13% 225 out of 171987
|
||||
http_reqs............................: 171987 94.363623/s
|
||||
|
||||
EXECUTION
|
||||
iteration_duration...................: min=497.6ms avg=10.5s max=1m57s p(50)=4.22s p(95)=40.23s p(99)=55.66s
|
||||
iterations...........................: 34399 18.873602/s
|
||||
vus..................................: 0 min=0 max=200
|
||||
vus_max..............................: 200 min=200 max=200
|
||||
|
||||
NETWORK
|
||||
data_received........................: 98 MB 54 kB/s
|
||||
data_sent............................: 31 MB 17 kB/s
|
||||
|
||||
running (30m22.6s), 000/200 VUs, 34399 complete and 0 interrupted iterations
|
||||
default ✓ [ 100% ] 200 VUs 30m0s
|
||||
```
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,109 @@
|
||||
---
|
||||
title: oidc session benchmark of zitadel v4.17.1
|
||||
sidebar_label: oidc session
|
||||
---
|
||||
|
||||
import OutputSource from "./output.json";
|
||||
import { BenchmarkChart } from "@/components/benchmark_chart";
|
||||
|
||||
Benchmark results of the v4.17.1 release of Zitadel.
|
||||
|
||||
## Performance test results
|
||||
|
||||
| Metric | Value |
|
||||
| :------------------------------------ | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| Baseline | none |
|
||||
| Purpose | Test current performance |
|
||||
| Test start | 2026-08-27 17:38:43 UTC |
|
||||
| Test end | 2026-08-27 18:08:45 UTC |
|
||||
| Test duration | 30min |
|
||||
| Executed test | oidc_session |
|
||||
| k6 version | v2.1.0 |
|
||||
| VUs | 600 |
|
||||
| Client location | US1 |
|
||||
| ZITADEL location | US1 |
|
||||
| ZITADEL container specification | vCPU: 6<br/> Memory: 6 Gi <br/>Container min scale: 7<br/>Container max scale: 7 |
|
||||
| ZITADEL Version | v4.17.1 |
|
||||
| ZITADEL Settings | Eventstore autovacuum (new in v4.17): Enabled: true<br/>VacuumThreshold: 1000000<br/>AnalyzeThreshold: 1000000<br/>Tuned for the ~4,000 events/s expected during the benchmark, per the [tuning guide](https://zitadel.com/docs/self-hosting/manage/tuning#reading-events). |
|
||||
| ZITADEL feature flags | Enabled: loginDefaultOrg (instance), oidcTokenExchange (system), enableBackChannelLogout (system), consoleUseV2UserApi (instance)<br/>improvedPerformance (instance): IMPROVED_PERFORMANCE_PROJECT_GRANT, IMPROVED_PERFORMANCE_PROJECT, IMPROVED_PERFORMANCE_USER_GRANT, IMPROVED_PERFORMANCE_ORG_DOMAIN_VERIFIED<br/>Disabled: loginV2 (explicitly false at instance), userSchema, debugOidcParentError, oidcSingleV1SessionTermination, permissionCheckV2<br/>Captured from the instance **before** the run (sequence 7, set 2026-08-27 13:35:31 UTC); the same capture covers `manipulate_user` at 13:41 and the rest of the sweep from 14:56, so all eleven targets ran the same configuration.<br/>This restores like-for-like comparison with the v4 pages, which ran five improvedPerformance options. The fifth, IMPROVED_PERFORMANCE_ORG_BY_ID, is `reserved` in the proto and no longer exists. |
|
||||
| Database | type: psql<br />version: v17.4 |
|
||||
| Database location | US1 |
|
||||
| Database specification | vCPU: 8<br/> memory: 32Gib |
|
||||
| ZITADEL metrics during test | CPU: p50 20%, p99 34% (peak 37%)<br/>Memory: p50 6%, peak 12%<br/>Instances: 7 (constant), no container startups recorded<br/>Server-side: 1,001 req/s, latency p50 59ms / p95 2.8s / p99 3.3s |
|
||||
| Observed errors | 3 of 1,767,162 requests failed (0.00%): 2x connection reset, 1x not individually logged (see k6 output). 3 of 2,355,414 checks failed (0.00%): add Session status ok (1 failed), finalize auth request status ok (2 failed) |
|
||||
| Top 3 most expensive database queries | 1: Push events (`eventstore.push`) (3,945 s)<br/>2: Query events by instance_id, aggregate_types, event_types, position (paged) (3,572 s)<br/>3: Shared advisory lock on `events2`, taken by every push — **lock wait** (1,973 s)<br/>Ranked by total accumulated execution time summed across concurrent sessions (Cloud SQL Query Insights) — not CPU time; it can exceed the wall-clock window. |
|
||||
| Database metrics during test | CPU: 92% median, 94% peak<br/>Memory: 40%<br/>Connections: 46<br/>Transactions: 10,721/s<br/>IOPS: read 1, write 1,086<br/>**CPU-saturated — the bottleneck for this target.** |
|
||||
| k6 Iterations per second | 326 |
|
||||
| k6 output | [output](#k6-output) |
|
||||
| flowchart outcome | **Scale.** Database CPU is the limit: 92% median against ZITADEL at 20%, with advisory lock wait only 20% of the top-5 query time. Raising the database beyond 8 vCPU is the next step, and throughput should be re-measured against the increase. |
|
||||
|
||||
## Endpoint latencies
|
||||
|
||||
<BenchmarkChart testResults={OutputSource} />
|
||||
|
||||
## k6 output
|
||||
|
||||
```bash
|
||||
█ TOTAL RESULTS
|
||||
|
||||
checks_total.......: 2355414 1305.538291/s
|
||||
checks_succeeded...: 99.99% 2355411 out of 2355414
|
||||
checks_failed......: 0.00% 3 out of 2355414
|
||||
|
||||
✓ user defined
|
||||
✓ authorize status ok
|
||||
✓ login name status ok
|
||||
✓ login shows password page
|
||||
✓ password status ok
|
||||
✓ password callback
|
||||
✓ code set
|
||||
✓ token status ok
|
||||
✓ access token created
|
||||
✓ id token created
|
||||
✓ info created
|
||||
✓ org created
|
||||
✓ create user is status ok
|
||||
✓ generate machine key status ok
|
||||
✓ member added successful
|
||||
✓ openid configuration
|
||||
✓ access token returned
|
||||
✓ auth request id returned
|
||||
✗ add Session status ok
|
||||
↳ 99% — ✓ 588848 / ✗ 1
|
||||
✗ finalize auth request status ok
|
||||
↳ 99% — ✓ 588846 / ✗ 2
|
||||
✓ org removed
|
||||
|
||||
CUSTOM
|
||||
login_ui_enter_login_name_duration...: min=114.02ms avg=114.02ms max=114.02ms p(50)=114.02ms p(95)=114.02ms p(99)=114.02ms
|
||||
login_ui_enter_password_duration.....: min=23.78ms avg=23.78ms max=23.78ms p(50)=23.78ms p(95)=23.78ms p(99)=23.78ms
|
||||
login_ui_init_login_duration.........: min=14.73ms avg=272.65ms max=4.22s p(50)=49.2ms p(95)=1.11s p(99)=1.37s
|
||||
login_ui_token_duration..............: min=128.22ms avg=128.22ms max=128.22ms p(50)=128.22ms p(95)=128.22ms p(99)=128.22ms
|
||||
membership_iam_member................: min=70.51ms avg=70.51ms max=70.51ms p(50)=70.51ms p(95)=70.51ms p(99)=70.51ms
|
||||
oidc_auth_request_finalize...........: min=27.3ms avg=833.77ms max=9s p(50)=97.44ms p(95)=3.96s p(99)=4.92s
|
||||
oidc_session_duration................: min=83ms avg=1.83s max=24.62s p(50)=1.21s p(95)=5.21s p(99)=7.4s
|
||||
oidc_token_duration..................: min=47.51ms avg=47.51ms max=47.51ms p(50)=47.51ms p(95)=47.51ms p(99)=47.51ms
|
||||
org_create_org_duration..............: min=82.6ms avg=82.6ms max=82.6ms p(50)=82.6ms p(95)=82.6ms p(99)=82.6ms
|
||||
session_add_session_duration.........: min=22.85ms avg=723.73ms max=23.09s p(50)=79.29ms p(95)=3.51s p(99)=4.36s
|
||||
user_add_machine_key_duration........: min=55.76ms avg=55.76ms max=55.76ms p(50)=55.76ms p(95)=55.76ms p(99)=55.76ms
|
||||
user_create_machine_duration.........: min=75.7ms avg=75.7ms max=75.7ms p(50)=75.7ms p(95)=75.7ms p(99)=75.7ms
|
||||
|
||||
HTTP
|
||||
http_req_duration....................: min=0s avg=609.87ms max=23.09s p(50)=77.47ms p(95)=3.41s p(99)=4.43s
|
||||
{ expected_response:true }.........: min=11.94ms avg=609.87ms max=23.09s p(50)=77.47ms p(95)=3.41s p(99)=4.43s
|
||||
http_req_failed......................: 0.00% 3 out of 1767162
|
||||
http_reqs............................: 1767162 979.487113/s
|
||||
|
||||
EXECUTION
|
||||
iteration_duration...................: min=83.46ms avg=1.83s max=24.62s p(50)=1.21s p(95)=5.21s p(99)=7.4s
|
||||
iterations...........................: 588849 326.382079/s
|
||||
vus..................................: 13 min=0 max=600
|
||||
vus_max..............................: 600 min=600 max=600
|
||||
|
||||
NETWORK
|
||||
data_received........................: 1.3 GB 704 kB/s
|
||||
data_sent............................: 576 MB 319 kB/s
|
||||
|
||||
running (30m04.2s), 000/600 VUs, 588849 complete and 0 interrupted iterations
|
||||
default ✓ [ 100% ] 600 VUs 30m0s
|
||||
```
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,103 @@
|
||||
---
|
||||
title: otp session benchmark of zitadel v4.17.1
|
||||
sidebar_label: otp session
|
||||
---
|
||||
|
||||
import OutputSource from "./output.json";
|
||||
import { BenchmarkChart } from "@/components/benchmark_chart";
|
||||
|
||||
Benchmark results of the v4.17.1 release of Zitadel.
|
||||
|
||||
## Performance test results
|
||||
|
||||
| Metric | Value |
|
||||
| :------------------------------------ | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| Baseline | none |
|
||||
| Purpose | Test current performance |
|
||||
| Test start | 2026-08-27 18:49:09 UTC |
|
||||
| Test end | 2026-08-27 19:19:16 UTC |
|
||||
| Test duration | 30min |
|
||||
| Executed test | otp_session |
|
||||
| k6 version | v2.1.0 |
|
||||
| VUs | 600 |
|
||||
| Client location | US1 |
|
||||
| ZITADEL location | US1 |
|
||||
| ZITADEL container specification | vCPU: 6<br/> Memory: 6 Gi <br/>Container min scale: 7<br/>Container max scale: 7 |
|
||||
| ZITADEL Version | v4.17.1 |
|
||||
| ZITADEL Settings | Eventstore autovacuum (new in v4.17): Enabled: true<br/>VacuumThreshold: 1000000<br/>AnalyzeThreshold: 1000000<br/>Tuned for the ~4,000 events/s expected during the benchmark, per the [tuning guide](https://zitadel.com/docs/self-hosting/manage/tuning#reading-events). |
|
||||
| ZITADEL feature flags | Enabled: loginDefaultOrg (instance), oidcTokenExchange (system), enableBackChannelLogout (system), consoleUseV2UserApi (instance)<br/>improvedPerformance (instance): IMPROVED_PERFORMANCE_PROJECT_GRANT, IMPROVED_PERFORMANCE_PROJECT, IMPROVED_PERFORMANCE_USER_GRANT, IMPROVED_PERFORMANCE_ORG_DOMAIN_VERIFIED<br/>Disabled: loginV2 (explicitly false at instance), userSchema, debugOidcParentError, oidcSingleV1SessionTermination, permissionCheckV2<br/>Captured from the instance **before** the run (sequence 7, set 2026-08-27 13:35:31 UTC); the same capture covers `manipulate_user` at 13:41 and the rest of the sweep from 14:56, so all eleven targets ran the same configuration.<br/>This restores like-for-like comparison with the v4 pages, which ran five improvedPerformance options. The fifth, IMPROVED_PERFORMANCE_ORG_BY_ID, is `reserved` in the proto and no longer exists. |
|
||||
| Database | type: psql<br />version: v17.4 |
|
||||
| Database location | US1 |
|
||||
| Database specification | vCPU: 8<br/> memory: 32Gib |
|
||||
| ZITADEL metrics during test | CPU: p50 9%, p99 20% (peak 22%)<br/>Memory: p50 6%, peak 14%<br/>Instances: 7 (constant), no container startups recorded<br/>Server-side: 256 req/s, latency p50 2.8s / p95 8.4s / p99 9.3s |
|
||||
| Observed errors | 182 of 466,756 requests failed (0.03%): none of the 182 individually logged (see k6 output). 182 of 466,159 checks failed (0.03%): add Session status ok (56 failed), set Session status ok (126 failed) |
|
||||
| Top 3 most expensive database queries | 1: Projection handler advisory lock, one per projection per instance — **lock wait, not query work** (13,149 s)<br/>2: Query events by instance_id, aggregate_types, event_types, position (paged) (3,760 s)<br/>3: Load user with primary login name and human profile (1,615 s)<br/>Ranked by total accumulated execution time summed across concurrent sessions (Cloud SQL Query Insights) — not CPU time; it can exceed the wall-clock window. |
|
||||
| Database metrics during test | CPU: 63% median, 64% peak<br/>Memory: 40%<br/>Connections: 46<br/>Transactions: 3,554/s<br/>IOPS: read 3, write 860 |
|
||||
| k6 Iterations per second | 84 |
|
||||
| k6 output | [output](#k6-output) |
|
||||
| flowchart outcome | **Potential issues.** Not resource-bound: database CPU 63%, ZITADEL CPU 9%, yet **69% of the top-5 query time is advisory lock wait** — predominantly the projection handler lock, which serialises this workload. Additional vCPU will not help. Logged as an issue rather than blocking further scale testing. |
|
||||
|
||||
## Endpoint latencies
|
||||
|
||||
<BenchmarkChart testResults={OutputSource} />
|
||||
|
||||
## k6 output
|
||||
|
||||
```bash
|
||||
█ TOTAL RESULTS
|
||||
|
||||
checks_total.......: 466159 252.308936/s
|
||||
checks_succeeded...: 99.96% 465977 out of 466159
|
||||
checks_failed......: 0.03% 182 out of 466159
|
||||
|
||||
✓ user defined
|
||||
✓ authorize status ok
|
||||
✓ login name status ok
|
||||
✓ login shows password page
|
||||
✓ password status ok
|
||||
✓ password callback
|
||||
✓ code set
|
||||
✓ token status ok
|
||||
✓ access token created
|
||||
✓ id token created
|
||||
✓ info created
|
||||
✓ org created
|
||||
✓ create user is status ok
|
||||
✓ set email otp status ok
|
||||
✗ add Session status ok
|
||||
↳ 99% — ✓ 154981 / ✗ 56
|
||||
✗ set Session status ok
|
||||
↳ 99% — ✓ 309783 / ✗ 126
|
||||
✓ org removed
|
||||
|
||||
CUSTOM
|
||||
login_ui_enter_login_name_duration...: min=111.91ms avg=111.91ms max=111.91ms p(50)=111.91ms p(95)=111.91ms p(99)=111.91ms
|
||||
login_ui_enter_password_duration.....: min=19.51ms avg=19.51ms max=19.51ms p(50)=19.51ms p(95)=19.51ms p(99)=19.51ms
|
||||
login_ui_init_login_duration.........: min=72.77ms avg=72.77ms max=72.77ms p(50)=72.77ms p(95)=72.77ms p(99)=72.77ms
|
||||
login_ui_token_duration..............: min=112.4ms avg=112.4ms max=112.4ms p(50)=112.4ms p(95)=112.4ms p(99)=112.4ms
|
||||
org_create_org_duration..............: min=57.4ms avg=57.4ms max=57.4ms p(50)=57.4ms p(95)=57.4ms p(99)=57.4ms
|
||||
otp_session_duration.................: min=125ms avg=6.97s max=36.37s p(50)=5.63s p(95)=18.87s p(99)=23s
|
||||
session_add_session_duration.........: min=52.96ms avg=6.77s max=30.32s p(50)=5.35s p(95)=18.76s p(99)=22.29s
|
||||
session_set_session_duration.........: min=20.57ms avg=96.93ms max=8.56s p(50)=39.41ms p(95)=64.73ms p(99)=2.29s
|
||||
set_human_email_otp_duration.........: min=27.64ms avg=241.81ms max=549.68ms p(50)=256.75ms p(95)=481.96ms p(99)=535.73ms
|
||||
user_create_human_duration...........: min=378.51ms avg=3.51s max=6.81s p(50)=3.82s p(95)=5.86s p(99)=6.72s
|
||||
|
||||
HTTP
|
||||
http_req_duration....................: min=17.42ms avg=2.32s max=30.32s p(50)=47.8ms p(95)=13.83s p(99)=20.22s
|
||||
{ expected_response:true }.........: min=17.42ms avg=2.32s max=30.32s p(50)=47.79ms p(95)=13.83s p(99)=20.22s
|
||||
http_req_failed......................: 0.03% 182 out of 466756
|
||||
http_reqs............................: 466756 252.632063/s
|
||||
|
||||
EXECUTION
|
||||
iteration_duration...................: min=124.68ms avg=6.97s max=36.37s p(50)=5.62s p(95)=18.86s p(99)=23s
|
||||
iterations...........................: 155037 83.913902/s
|
||||
vus..................................: 0 min=0 max=600
|
||||
vus_max..............................: 600 min=600 max=600
|
||||
|
||||
NETWORK
|
||||
data_received........................: 351 MB 190 kB/s
|
||||
data_sent............................: 178 MB 96 kB/s
|
||||
|
||||
running (30m47.6s), 000/600 VUs, 155037 complete and 0 interrupted iterations
|
||||
default ✓ [ 100% ] 600 VUs 30m0s
|
||||
```
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,102 @@
|
||||
---
|
||||
title: password session benchmark of zitadel v4.17.1
|
||||
sidebar_label: password session
|
||||
---
|
||||
|
||||
import OutputSource from "./output.json";
|
||||
import { BenchmarkChart } from "@/components/benchmark_chart";
|
||||
|
||||
Benchmark results of the v4.17.1 release of Zitadel.
|
||||
|
||||
## Performance test results
|
||||
|
||||
| Metric | Value |
|
||||
| :------------------------------------ | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| Baseline | none |
|
||||
| Purpose | Test current performance |
|
||||
| Test start | 2026-08-27 20:02:39 UTC |
|
||||
| Test end | 2026-08-27 20:32:49 UTC |
|
||||
| Test duration | 30min |
|
||||
| Executed test | password_session |
|
||||
| k6 version | v2.1.0 |
|
||||
| VUs | 600 |
|
||||
| Client location | US1 |
|
||||
| ZITADEL location | US1 |
|
||||
| ZITADEL container specification | vCPU: 6<br/> Memory: 6 Gi <br/>Container min scale: 7<br/>Container max scale: 7 |
|
||||
| ZITADEL Version | v4.17.1 |
|
||||
| ZITADEL Settings | Eventstore autovacuum (new in v4.17): Enabled: true<br/>VacuumThreshold: 1000000<br/>AnalyzeThreshold: 1000000<br/>Tuned for the ~4,000 events/s expected during the benchmark, per the [tuning guide](https://zitadel.com/docs/self-hosting/manage/tuning#reading-events). |
|
||||
| ZITADEL feature flags | Enabled: loginDefaultOrg (instance), oidcTokenExchange (system), enableBackChannelLogout (system), consoleUseV2UserApi (instance)<br/>improvedPerformance (instance): IMPROVED_PERFORMANCE_PROJECT_GRANT, IMPROVED_PERFORMANCE_PROJECT, IMPROVED_PERFORMANCE_USER_GRANT, IMPROVED_PERFORMANCE_ORG_DOMAIN_VERIFIED<br/>Disabled: loginV2 (explicitly false at instance), userSchema, debugOidcParentError, oidcSingleV1SessionTermination, permissionCheckV2<br/>Captured from the instance **before** the run (sequence 7, set 2026-08-27 13:35:31 UTC); the same capture covers `manipulate_user` at 13:41 and the rest of the sweep from 14:56, so all eleven targets ran the same configuration.<br/>This restores like-for-like comparison with the v4 pages, which ran five improvedPerformance options. The fifth, IMPROVED_PERFORMANCE_ORG_BY_ID, is `reserved` in the proto and no longer exists. |
|
||||
| Database | type: psql<br />version: v17.4 |
|
||||
| Database location | US1 |
|
||||
| Database specification | vCPU: 8<br/> memory: 32Gib |
|
||||
| ZITADEL metrics during test | CPU: p50 50%, p99 69% (peak 75%)<br/>Memory: p50 7%, peak 16%<br/>Instances: 7 (constant), no container startups recorded<br/>Server-side: 140 req/s, latency p50 2.6s / p95 10.7s / p99 11.5s |
|
||||
| Observed errors | 1 of 253,280 requests failed (0.00%): 1x connection reset. 1 iteration aborted on a null response body. 1 of 252,683 checks failed (0.00%): set Session status ok (1 failed) |
|
||||
| Top 3 most expensive database queries | 1: Projection handler advisory lock, one per projection per instance — **lock wait, not query work** (18,303 s)<br/>2: Eventstore read barrier: advisory lock + unlock on `events2` — **lock wait** (6,904 s)<br/>3: Query events by instance_id, aggregate_types, event_types, position (paged) (3,794 s)<br/>Ranked by total accumulated execution time summed across concurrent sessions (Cloud SQL Query Insights) — not CPU time; it can exceed the wall-clock window. |
|
||||
| Database metrics during test | CPU: 51% median, 52% peak<br/>Memory: 40%<br/>Connections: 46<br/>Transactions: 2,029/s<br/>IOPS: read 3, write 597 |
|
||||
| k6 Iterations per second | 68 |
|
||||
| k6 output | [output](#k6-output) |
|
||||
| flowchart outcome | **Potential issues.** Not resource-bound: database CPU 51%, ZITADEL CPU 50%, yet **86% of the top-5 query time is advisory lock wait** — predominantly the projection handler lock, which serialises this workload. Additional vCPU will not help. Logged as an issue rather than blocking further scale testing. |
|
||||
|
||||
## Endpoint latencies
|
||||
|
||||
<BenchmarkChart testResults={OutputSource} />
|
||||
|
||||
## k6 output
|
||||
|
||||
```bash
|
||||
█ TOTAL RESULTS
|
||||
|
||||
checks_total.......: 252683 136.744628/s
|
||||
checks_succeeded...: 99.99% 252682 out of 252683
|
||||
checks_failed......: 0.00% 1 out of 252683
|
||||
|
||||
✓ user defined
|
||||
✓ authorize status ok
|
||||
✓ login name status ok
|
||||
✓ login shows password page
|
||||
✓ password status ok
|
||||
✓ password callback
|
||||
✓ code set
|
||||
✓ token status ok
|
||||
✓ access token created
|
||||
✓ id token created
|
||||
✓ info created
|
||||
✓ org created
|
||||
✓ create user is status ok
|
||||
✓ set email otp status ok
|
||||
✓ add Session status ok
|
||||
✗ set Session status ok
|
||||
↳ 99% — ✓ 125734 / ✗ 1
|
||||
✓ org removed
|
||||
|
||||
CUSTOM
|
||||
login_ui_enter_login_name_duration...: min=163.44ms avg=163.44ms max=163.44ms p(50)=163.44ms p(95)=163.44ms p(99)=163.44ms
|
||||
login_ui_enter_password_duration.....: min=12.83ms avg=12.83ms max=12.83ms p(50)=12.83ms p(95)=12.83ms p(99)=12.83ms
|
||||
login_ui_init_login_duration.........: min=55.61ms avg=55.61ms max=55.61ms p(50)=55.61ms p(95)=55.61ms p(99)=55.61ms
|
||||
login_ui_token_duration..............: min=97.03ms avg=97.03ms max=97.03ms p(50)=97.03ms p(95)=97.03ms p(99)=97.03ms
|
||||
org_create_org_duration..............: min=56.23ms avg=56.23ms max=56.23ms p(50)=56.23ms p(95)=56.23ms p(99)=56.23ms
|
||||
password_session_duration............: min=361ms avg=8.61s max=36.61s p(50)=5.67s p(95)=22.32s p(99)=25.76s
|
||||
session_add_session_duration.........: min=48.7ms avg=8.23s max=33.11s p(50)=5.29s p(95)=21.94s p(99)=25.31s
|
||||
session_set_session_duration.........: min=619.72µs avg=372.12ms max=8.7s p(50)=339.05ms p(95)=530.41ms p(99)=711.46ms
|
||||
set_human_email_otp_duration.........: min=24.46ms avg=158.74ms max=552.96ms p(50)=118.89ms p(95)=445.68ms p(99)=528.23ms
|
||||
user_create_human_duration...........: min=350.17ms avg=3.56s max=7.21s p(50)=3.7s p(95)=6.72s p(99)=6.95s
|
||||
|
||||
HTTP
|
||||
http_req_duration....................: min=619.72µs avg=4.28s max=33.11s p(50)=408.63ms p(95)=20s p(99)=24.05s
|
||||
{ expected_response:true }.........: min=12.83ms avg=4.28s max=33.11s p(50)=408.64ms p(95)=20s p(99)=24.05s
|
||||
http_req_failed......................: 0.00% 1 out of 253280
|
||||
http_reqs............................: 253280 137.067706/s
|
||||
|
||||
EXECUTION
|
||||
iteration_duration...................: min=361.04ms avg=8.61s max=36.61s p(50)=5.67s p(95)=22.32s p(99)=25.76s
|
||||
iterations...........................: 125735 68.044094/s
|
||||
vus..................................: 0 min=0 max=600
|
||||
vus_max..............................: 600 min=600 max=600
|
||||
|
||||
NETWORK
|
||||
data_received........................: 192 MB 104 kB/s
|
||||
data_sent............................: 86 MB 47 kB/s
|
||||
|
||||
running (30m47.8s), 000/600 VUs, 125735 complete and 0 interrupted iterations
|
||||
default ✓ [ 100% ] 600 VUs 30m0s
|
||||
```
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,99 @@
|
||||
---
|
||||
title: user info benchmark of zitadel v4.17.1
|
||||
sidebar_label: user info
|
||||
---
|
||||
|
||||
import OutputSource from "./output.json";
|
||||
import { BenchmarkChart } from "@/components/benchmark_chart";
|
||||
|
||||
Benchmark results of the v4.17.1 release of Zitadel.
|
||||
|
||||
## Performance test results
|
||||
|
||||
| Metric | Value |
|
||||
| :------------------------------------ | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| Baseline | none |
|
||||
| Purpose | Test current performance |
|
||||
| Test start | 2026-08-27 18:11:39 UTC |
|
||||
| Test end | 2026-08-27 18:41:40 UTC |
|
||||
| Test duration | 30min |
|
||||
| Executed test | user_info |
|
||||
| k6 version | v2.1.0 |
|
||||
| VUs | 600 |
|
||||
| Client location | US1 |
|
||||
| ZITADEL location | US1 |
|
||||
| ZITADEL container specification | vCPU: 6<br/> Memory: 6 Gi <br/>Container min scale: 7<br/>Container max scale: 7 |
|
||||
| ZITADEL Version | v4.17.1 |
|
||||
| ZITADEL Settings | Eventstore autovacuum (new in v4.17): Enabled: true<br/>VacuumThreshold: 1000000<br/>AnalyzeThreshold: 1000000<br/>Tuned for the ~4,000 events/s expected during the benchmark, per the [tuning guide](https://zitadel.com/docs/self-hosting/manage/tuning#reading-events). |
|
||||
| ZITADEL feature flags | Enabled: loginDefaultOrg (instance), oidcTokenExchange (system), enableBackChannelLogout (system), consoleUseV2UserApi (instance)<br/>improvedPerformance (instance): IMPROVED_PERFORMANCE_PROJECT_GRANT, IMPROVED_PERFORMANCE_PROJECT, IMPROVED_PERFORMANCE_USER_GRANT, IMPROVED_PERFORMANCE_ORG_DOMAIN_VERIFIED<br/>Disabled: loginV2 (explicitly false at instance), userSchema, debugOidcParentError, oidcSingleV1SessionTermination, permissionCheckV2<br/>Captured from the instance **before** the run (sequence 7, set 2026-08-27 13:35:31 UTC); the same capture covers `manipulate_user` at 13:41 and the rest of the sweep from 14:56, so all eleven targets ran the same configuration.<br/>This restores like-for-like comparison with the v4 pages, which ran five improvedPerformance options. The fifth, IMPROVED_PERFORMANCE_ORG_BY_ID, is `reserved` in the proto and no longer exists. |
|
||||
| Database | type: psql<br />version: v17.4 |
|
||||
| Database location | US1 |
|
||||
| Database specification | vCPU: 8<br/> memory: 32Gib |
|
||||
| ZITADEL metrics during test | CPU: p50 26%, p99 42% (peak 44%)<br/>Memory: p50 6%, peak 14%<br/>Instances: 7 (constant), no container startups recorded<br/>Server-side: 3,303 req/s, latency p50 16ms / p95 939ms / p99 1.1s |
|
||||
| Observed errors | 66 of 5,800,788 requests failed (0.00%): 1x connection reset, 65x not individually logged (see k6 output). 66 of 5,800,793 checks failed (0.00%): userinfo status ok (65 failed), org removed (1 failed) |
|
||||
| Top 3 most expensive database queries | 1: Push events (`eventstore.push`) (3,583 s)<br/>2: Query events by instance_id, aggregate_types, event_types, position (paged) (3,576 s)<br/>3: Load user with primary login name and human profile (2,149 s)<br/>Ranked by total accumulated execution time summed across concurrent sessions (Cloud SQL Query Insights) — not CPU time; it can exceed the wall-clock window. |
|
||||
| Database metrics during test | CPU: 96% median, 97% peak<br/>Memory: 40%<br/>Connections: 46<br/>Transactions: 16,545/s<br/>IOPS: read 0, write 12<br/>**CPU-saturated — the bottleneck for this target.** |
|
||||
| k6 Iterations per second | 3217 |
|
||||
| k6 output | [output](#k6-output) |
|
||||
| flowchart outcome | **Scale.** Database CPU is the limit: 96% median against ZITADEL at 26%, with advisory lock wait only 15% of the top-5 query time. Raising the database beyond 8 vCPU is the next step, and throughput should be re-measured against the increase. |
|
||||
|
||||
## Endpoint latencies
|
||||
|
||||
<BenchmarkChart testResults={OutputSource} />
|
||||
|
||||
## k6 output
|
||||
|
||||
```bash
|
||||
█ TOTAL RESULTS
|
||||
|
||||
checks_total.......: 5800793 3217.417923/s
|
||||
checks_succeeded...: 99.99% 5800727 out of 5800793
|
||||
checks_failed......: 0.00% 66 out of 5800793
|
||||
|
||||
✓ user defined
|
||||
✓ authorize status ok
|
||||
✓ login name status ok
|
||||
✓ login shows password page
|
||||
✓ password status ok
|
||||
✓ password callback
|
||||
✓ code set
|
||||
✓ token status ok
|
||||
✓ access token created
|
||||
✓ id token created
|
||||
✓ info created
|
||||
✓ org created
|
||||
✓ create user is status ok
|
||||
✓ openid configuration
|
||||
✗ userinfo status ok
|
||||
↳ 99% — ✓ 5800103 / ✗ 65
|
||||
✗ org removed
|
||||
↳ 0% — ✓ 0 / ✗ 1
|
||||
|
||||
CUSTOM
|
||||
login_ui_enter_login_name_duration...: min=118.36ms avg=118.43ms max=118.51ms p(50)=118.43ms p(95)=118.5ms p(99)=118.51ms
|
||||
login_ui_enter_password_duration.....: min=21.98ms avg=243.15ms max=464.31ms p(50)=243.15ms p(95)=442.2ms p(99)=459.89ms
|
||||
login_ui_init_login_duration.........: min=78.54ms avg=81.56ms max=84.59ms p(50)=81.56ms p(95)=84.29ms p(99)=84.53ms
|
||||
login_ui_token_duration..............: min=75.08ms avg=88.39ms max=101.7ms p(50)=88.39ms p(95)=100.37ms p(99)=101.44ms
|
||||
oidc_user_info_duration..............: min=1.77ms avg=185.96ms max=3.47s p(50)=29.32ms p(95)=942.89ms p(99)=1.13s
|
||||
org_create_org_duration..............: min=52.63ms avg=52.63ms max=52.63ms p(50)=52.63ms p(95)=52.63ms p(99)=52.63ms
|
||||
user_create_human_duration...........: min=364.77ms avg=364.77ms max=364.77ms p(50)=364.77ms p(95)=364.77ms p(99)=364.77ms
|
||||
|
||||
HTTP
|
||||
http_req_duration....................: min=1.77ms avg=185.95ms max=3.47s p(50)=29.32ms p(95)=942.88ms p(99)=1.13s
|
||||
{ expected_response:true }.........: min=4.1ms avg=185.93ms max=3.47s p(50)=29.32ms p(95)=942.85ms p(99)=1.13s
|
||||
http_req_failed......................: 0.00% 66 out of 5800788
|
||||
http_reqs............................: 5800788 3217.41515/s
|
||||
|
||||
EXECUTION
|
||||
iteration_duration...................: min=2ms avg=186.18ms max=3.47s p(50)=29.51ms p(95)=943.09ms p(99)=1.13s
|
||||
iterations...........................: 5800168 3217.071266/s
|
||||
vus..................................: 600 min=0 max=600
|
||||
vus_max..............................: 600 min=600 max=600
|
||||
|
||||
NETWORK
|
||||
data_received........................: 4.0 GB 2.2 MB/s
|
||||
data_sent............................: 458 MB 254 kB/s
|
||||
|
||||
running (30m02.9s), 000/600 VUs, 5800168 complete and 0 interrupted iterations
|
||||
default ✓ [ 100% ] 600 VUs 30m0s
|
||||
```
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1021,6 +1021,23 @@ export const apisSidebar: readonly SidebarItem[] = [
|
||||
"apis/benchmarks/v4/user_info",
|
||||
],
|
||||
},
|
||||
{
|
||||
type: "category",
|
||||
label: "v4.17.1",
|
||||
items: [
|
||||
"apis/benchmarks/v4.17.1/add_session",
|
||||
"apis/benchmarks/v4.17.1/human_password_login",
|
||||
"apis/benchmarks/v4.17.1/introspect",
|
||||
"apis/benchmarks/v4.17.1/machine_client_credentials_login",
|
||||
"apis/benchmarks/v4.17.1/machine_jwt_profile_grant",
|
||||
"apis/benchmarks/v4.17.1/machine_pat_login",
|
||||
"apis/benchmarks/v4.17.1/manipulate_user",
|
||||
"apis/benchmarks/v4.17.1/oidc_session",
|
||||
"apis/benchmarks/v4.17.1/otp_session",
|
||||
"apis/benchmarks/v4.17.1/password_session",
|
||||
"apis/benchmarks/v4.17.1/user_info",
|
||||
],
|
||||
},
|
||||
],
|
||||
},
|
||||
] as const;
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
+1
-1
@@ -43,7 +43,7 @@ otp_session: ensure_key_pair ensure_modules bundle
|
||||
|
||||
.PHONY: password_session
|
||||
password_session: ensure_key_pair ensure_modules bundle
|
||||
${K6} run --summary-trend-stats "min,avg,max,p(50),p(95),p(99)" dist/password_session.js --vus ${VUS} --duration ${DURATION} --out csv=output/otp_session_${DATE}.csv
|
||||
${K6} run --summary-trend-stats "min,avg,max,p(50),p(95),p(99)" dist/password_session.js --vus ${VUS} --duration ${DURATION} --out csv=output/password_session_${DATE}.csv
|
||||
|
||||
.PHONY: add_session
|
||||
add_session: ensure_key_pair ensure_modules bundle
|
||||
|
||||
@@ -0,0 +1,186 @@
|
||||
# Follow-up: the 503 bursts of 2026-08-27
|
||||
|
||||
Two bursts, six hours apart, on different targets, during the v4.17.1 benchmark sweep. Burst 1
|
||||
was found first and most of this note was written about it; burst 2 is seven times larger and
|
||||
is what makes the pair worth a post-mortem rather than a shrug.
|
||||
|
||||
Both are **HTTP 503 `Service error -27`** — an HTML page from the Google frontend, not a Zitadel
|
||||
JSON error. They are recorded here because a page that reports them as a plain failure rate would
|
||||
be attributing an infrastructure event to Zitadel, and the difference matters more than the
|
||||
number does.
|
||||
|
||||
## Status as of 2026-08-28
|
||||
|
||||
The metric half of the investigation is done. The log half was run with broken filters and must
|
||||
be repeated. **Nothing here is concluded.** The sections below are the chronological record and
|
||||
this block is the summary.
|
||||
|
||||
### Established
|
||||
|
||||
- Two bursts on 2026-08-27, both 503 `Service error -27`. Burst 1: `manipulate_user`,
|
||||
13:43:19–13:44:13, 150 requests. Burst 2: `human_password_login`, 19:37–19:44, 1,044 requests.
|
||||
Both bounded, then silence.
|
||||
- **No container startups in any of the eleven test windows.** The Cloud Run
|
||||
`container/startup_latencies` metric returned empty distributions throughout, and instance
|
||||
count was constant at 7.
|
||||
- **The two bursts have different server-side signatures.** In burst 1's window there were
|
||||
**zero** log entries at `severity>=WARNING`. In burst 2's there were **at least 200**, and that
|
||||
200 was the query limit, reached within the first ~2 seconds of a 20-minute window.
|
||||
- **The 2026-08-26 sweep produced no 503s at all**, on any of its eleven targets. Same instance,
|
||||
same deployment, roughly 24 hours earlier.
|
||||
|
||||
### Ruled out, provisionally
|
||||
|
||||
- **Container restart or replacement.** This was the original working hypothesis, on the grounds
|
||||
that a bounded burst followed by silence looks like something being replaced. The startup metric
|
||||
does not support it: a replaced container starts, and a start would appear. Provisional only
|
||||
because it rests on one metric being both correct and complete.
|
||||
- **Panic and crash.** The stack traces in the collected logs are not crash dumps: they begin
|
||||
`runtime/debug.Stack()` called from `zerrors.CreateZitadelError`, which is Zitadel's error type
|
||||
capturing a stack when it is constructed. `recover_interceptor.go` appears because it is in the
|
||||
call chain, not because it recovered anything. Combined with zero container startups, the
|
||||
crash-and-restart family of explanations is firmly out.
|
||||
|
||||
### Open questions, in the order worth answering
|
||||
|
||||
1. **What do burst 2's errors actually say?** Still open after two attempts. The first failed on
|
||||
`textPayload` filters against a service that logs structured JSON. The second fixed that and
|
||||
then spent its whole 5,000-entry budget on the seven minutes of padding *before* the burst, at
|
||||
a baseline rate of 800–1,300 errors/minute. Both failures were in the collection, not in the
|
||||
data, and both produced output that looked like an answer.
|
||||
2. **How many errors are there really?** 200 in two seconds, truncated. Over twenty minutes the
|
||||
true figure could be in the thousands, and 1,044 client-observed 503s may be a small fraction
|
||||
of what the server logged — or unrelated to it.
|
||||
3. **Did the 503s reach Cloud Run at all?** The queries filtering on `httpRequest.status=503` came
|
||||
back empty, which invites the reading "the frontend rejected them before they arrived". **Do
|
||||
not accept that yet.** If request logging is not enabled on this service the query returns
|
||||
nothing regardless. Establish the control first: confirm the service emits *any* `httpRequest`
|
||||
entry, in any window. Only then is an empty result evidence.
|
||||
4. **Why is burst 1 silent and burst 2 noisy?** Same HTTP symptom, opposite server-side signature.
|
||||
Two causes wearing the same error page is at least as likely as one, and this note's earlier
|
||||
framing — that the pair is the finding — assumed a single mechanism it had not established.
|
||||
5. **Which container instances were involved?** The instance-id label rendered empty for every
|
||||
entry. Unknown whether the label path is wrong for this service or the value is genuinely
|
||||
unset. Check one known-good entry's full JSON before trusting any instance-level conclusion.
|
||||
6. **What is `Service error -27`?** Used throughout this note as an identifier and nowhere
|
||||
explained. It is a Google frontend string; if it names the layer that rejected the request, it
|
||||
answers question 3 without any log query at all.
|
||||
7. **Why only the 27th?** The 26 August sweep was clean. The instance's feature flags were changed
|
||||
at **13:35:31Z on the 27th, eight minutes before burst 1**. That is close enough to be worth
|
||||
checking and far too neat to trust: burst 2 came six hours later, so a one-off configuration
|
||||
change does not cover both. Treat as a lead, not a cause.
|
||||
8. **Does it recur?** Two bursts on one day and none the day before. Whether this reproduces is
|
||||
unknown, and it determines whether this is worth an issue or a footnote.
|
||||
|
||||
Weak signal, recorded so it is not rediscovered as though it were new: both bursts fell on the two
|
||||
targets that create users in bulk. With n=2 that is barely distinguishable from coincidence.
|
||||
|
||||
## Burst 1 — `manipulate_user` @ 200 VUs, 13:43:19–13:44:13
|
||||
|
||||
Run: `manipulate_user`, 200 VUs, 1800s, started 2026-08-27 13:41:09 UTC.
|
||||
Feature flags: `improvedPerformance` = PROJECT_GRANT, PROJECT, USER_GRANT, ORG_DOMAIN_VERIFIED,
|
||||
captured before the run; sequence 7, set 13:35:31Z.
|
||||
|
||||
29 errors surfaced as thrown k6 errors, all of them on user creation. The CSV shows the real
|
||||
figure:
|
||||
|
||||
first 2026-08-27T13:43:19Z (CSV; k6 error lines showed 13:43:21)
|
||||
last 2026-08-27T13:44:13Z (CSV; k6 error lines showed 13:43:57)
|
||||
count 150 failed requests (29 surfaced as thrown errors on user creation)
|
||||
window 55 seconds, ~2 minutes into the run
|
||||
other error classes: 75 request timeouts, 14:08:48-14:11:02 - UNRELATED, see below
|
||||
|
||||
A bounded burst followed by silence. This is not the shape of a service under sustained load; it
|
||||
is the shape of something being restarted or replaced — which is what made the absence of any
|
||||
container startup in the metrics the surprise it was. It is also a different failure class from
|
||||
every other failure seen during the sweep, all of which are application or harness level.
|
||||
|
||||
## Burst 2 — `human_password_login` @ 600 VUs, 19:37–19:44
|
||||
|
||||
Found after burst 1. It is the larger of the two by a wide margin:
|
||||
|
||||
| | burst 1 | burst 2 |
|
||||
| --- | --- | --- |
|
||||
| target | `manipulate_user` (200 VUs) | `human_password_login` (600 VUs) |
|
||||
| window (UTC) | 13:43:19 – 13:44:13 | 19:37 – 19:44 |
|
||||
| duration | 55 s | ~7 min |
|
||||
| count | 150 | **1,044** |
|
||||
| position in run | ~2 min in | ~13 min in |
|
||||
| recurrence | none in the following 25 min | none after |
|
||||
|
||||
Same symptom, six hours and four intervening targets apart.
|
||||
|
||||
Note also what burst 2 is **not**: `human_password_login` reported 8,178 request timeouts spread
|
||||
evenly across the whole run at 125–360/min. Those are Zitadel under load and are the number to
|
||||
read. The 1,044 platform 503s are a separate population, and the page reports them separately. Do
|
||||
not let the post-mortem collapse them back together.
|
||||
|
||||
## What the 2026-08-28 collection added — read the caveat before the findings
|
||||
|
||||
The log half of that collection was **partly invalid** and must be re-run. The filters matched
|
||||
only `textPayload`, and Zitadel logs structured JSON, so every such term matched nothing and the
|
||||
display column rendered blank even where entries existed. The corrected queries then hit a second
|
||||
problem: at 800–1,300 errors per minute on this target, a window padded to 19:30–19:50 and ordered
|
||||
ascending exhausted its 5,000-entry cap at **19:34:51**, before the burst began at 19:37.
|
||||
Everything it retrieved is the ordinary pre-burst error rate.
|
||||
|
||||
What survives that caveat, because `severity>=WARNING` is payload-agnostic:
|
||||
|
||||
| | burst 1 (13:38–13:50) | burst 2 (19:30–19:50) |
|
||||
| --- | --- | --- |
|
||||
| entries at severity >= WARNING | **0** | **>= 200, truncated at the limit** |
|
||||
| span of the 200 sampled | — | 19:30:00–19:30:0x, i.e. ~2 seconds |
|
||||
|
||||
**The two bursts do not have the same signature**, which contradicts the working assumption that
|
||||
one mechanism produced both. Burst 1 produced no warnings or errors at all; burst 2 produced at
|
||||
least 200 in the first two seconds of the window alone, so the true count over 20 minutes is
|
||||
likely in the thousands. Their content is unknown — that is exactly what the broken filter cost.
|
||||
|
||||
From the metric side, which was not affected: no container startups in any window, and instance
|
||||
count constant at 7.
|
||||
|
||||
Do not conclude "the frontend rejected the requests before Cloud Run saw them" from the empty
|
||||
503 request logs. See open question 3: without the control, that query cannot distinguish an
|
||||
infrastructure finding from a disabled log.
|
||||
|
||||
### Two unrelated defects, found by accident
|
||||
|
||||
The baseline error mix under 600 VUs is worth recording independently of the burst: 2,901 bare
|
||||
`context canceled`, 1,239 `QUERY-AWx52 Errors.Query.SQLStatement` (parent: context canceled), 232
|
||||
`LOG-ii6Pi` with parent `sql: transaction has already been committed or rolled back`, and 66
|
||||
occurrences of a missing i18n key, `"unable to get user by id" not found in language "en"`.
|
||||
|
||||
`context canceled` is a consequence of the client giving up, not a cause. But **`sql: transaction
|
||||
has already been committed or rolled back` is a genuine defect** — code using a transaction after
|
||||
it has finished — and the missing translation key is a smaller one. Neither is related to the
|
||||
503s; both should not be lost.
|
||||
|
||||
## Why it matters for the page
|
||||
|
||||
If this was a container restart, the failures are a deployment event and must be reported as such
|
||||
rather than as a Zitadel failure rate. If there was a panic, that is a defect and worth an issue
|
||||
of its own. Either way the page states what happened; an unexplained 503 burst rendered as a plain
|
||||
failure count would be true and misleading, which is the worst combination available.
|
||||
|
||||
## Addendum: the two populations in burst 1's run
|
||||
|
||||
The full status breakdown, taken while the CSV still existed, shows the run contained **two
|
||||
unrelated failure populations**, separated by 25 minutes of clean traffic:
|
||||
|
||||
error_code reqs first_utc last_utc
|
||||
1503 150 2026-08-27 13:43:19 2026-08-27 13:44:13 <- infrastructure
|
||||
1050 75 2026-08-27 14:08:48 2026-08-27 14:11:02 <- the known wall
|
||||
|
||||
Only the first population is the subject of this note. The second is the documented
|
||||
degradation-over-sustained-churn behaviour described on the `manipulate_user` page, and is
|
||||
expected.
|
||||
|
||||
The burst was also larger than the k6 error lines suggested: 150 failed requests, of which only 29
|
||||
surfaced as thrown errors (user creation). The other 121 hit update, lock and delete, where a
|
||||
failed check does not throw. This is why failure counts are taken from the CSV and not from
|
||||
`grep -c level=error` on the log, which gave 29.
|
||||
|
||||
---
|
||||
|
||||
The Cloud Shell queries used for the collection, and the identity of the deployment they run
|
||||
against, are not in this repository.
|
||||
@@ -19,6 +19,7 @@ export function createAPI(name: string, projectId: string, org: Org, accessToken
|
||||
authMethodType: 'API_AUTH_METHOD_TYPE_PRIVATE_KEY_JWT',
|
||||
}),
|
||||
{
|
||||
tags: { name: '/management/v1/projects/{projectId}/apps/api' },
|
||||
headers: {
|
||||
authorization: `Bearer ${accessToken}`,
|
||||
'Content-Type': 'application/json',
|
||||
@@ -51,6 +52,7 @@ export function createAppKey(appId: string, projectId: string, org: Org, accessT
|
||||
type: 'KEY_TYPE_JSON',
|
||||
}),
|
||||
{
|
||||
tags: { name: '/management/v1/projects/{projectId}/apps/{appId}/keys' },
|
||||
headers: {
|
||||
authorization: `Bearer ${accessToken}`,
|
||||
'Content-Type': 'application/json',
|
||||
|
||||
@@ -172,6 +172,7 @@ export async function token(request: TokenRequest): Promise<Tokens> {
|
||||
const authRequestByIDTrend = new Trend('oidc_auth_request_by_id_duration', true);
|
||||
export async function authRequestByID(id: string, tokens: any): Promise<Response> {
|
||||
const response = http.get(url(`/v2/oidc/auth_requests/${id}`), {
|
||||
tags: { name: '/v2/oidc/auth_requests/{id}' },
|
||||
headers: {
|
||||
Authorization: `Bearer ${tokens.accessToken}`,
|
||||
},
|
||||
@@ -195,6 +196,7 @@ export async function finalizeAuthRequest(id: string, session: any, tokens: any)
|
||||
},
|
||||
}),
|
||||
{
|
||||
tags: { name: '/v2/oidc/auth_requests/{id}' },
|
||||
headers: {
|
||||
Authorization: `Bearer ${tokens.accessToken}`,
|
||||
'Content-Type': 'application/json',
|
||||
|
||||
@@ -40,6 +40,7 @@ export function setSession(id: string, session: any, accessToken: string, challe
|
||||
};
|
||||
return new Promise((resolve, reject) => {
|
||||
let response = http.asyncRequest('PATCH', url(`/v2/sessions/${id}`), JSON.stringify(body), {
|
||||
tags: { name: '/v2/sessions/{id}' },
|
||||
headers: {
|
||||
authorization: `Bearer ${accessToken}`,
|
||||
'Content-Type': 'application/json',
|
||||
|
||||
@@ -2,7 +2,6 @@ import { loginByUsernamePassword } from '../login_ui';
|
||||
import { createOrg, removeOrg } from '../org';
|
||||
import { createHuman, updateHuman, lockUser, deleteUser, User } from '../user';
|
||||
import { Config } from '../config';
|
||||
import { check } from 'k6';
|
||||
|
||||
export async function setup() {
|
||||
const tokens = loginByUsernamePassword(Config.admin as User);
|
||||
@@ -16,29 +15,22 @@ export async function setup() {
|
||||
|
||||
export default async function (data: any) {
|
||||
const human = await createHuman(`vu-${__VU}-${new Date(Date.now()).getTime()}`, data.org, data.tokens.accessToken);
|
||||
const updateRes = await updateHuman(
|
||||
// updateHuman/lockUser/deleteUser each register their own check; repeating them
|
||||
// here would double count, and k6 aggregates checks by name.
|
||||
await updateHuman(
|
||||
{
|
||||
profile: {
|
||||
nickName: `${new Date(Date.now()).toISOString()}`,
|
||||
human: {
|
||||
profile: {
|
||||
nickName: `${new Date(Date.now()).toISOString()}`,
|
||||
},
|
||||
},
|
||||
},
|
||||
human.userId,
|
||||
data.org,
|
||||
data.tokens.accessToken,
|
||||
);
|
||||
check(updateRes, {
|
||||
'update user is status ok': (r) => r.status >= 200 && r.status < 300,
|
||||
});
|
||||
|
||||
const lockRes = await lockUser(human.userId, data.org, data.tokens.accessToken);
|
||||
check(lockRes, {
|
||||
'lock user is status ok': (r) => r.status >= 200 && r.status < 300,
|
||||
});
|
||||
|
||||
const deleteRes = await deleteUser(human.userId, data.org, data.tokens.accessToken);
|
||||
check(deleteRes, {
|
||||
'delete user is status ok': (r) => r.status >= 200 && r.status < 300,
|
||||
});
|
||||
await lockUser(human.userId, data.org, data.tokens.accessToken);
|
||||
await deleteUser(human.userId, data.org, data.tokens.accessToken);
|
||||
}
|
||||
|
||||
export function teardown(data: any) {
|
||||
|
||||
+186
-95
@@ -3,6 +3,7 @@ import { Org } from './org';
|
||||
import http, { RefinedResponse } from 'k6/http';
|
||||
import url from './url';
|
||||
import { check } from 'k6';
|
||||
import { setTimeout } from 'k6/timers';
|
||||
import { Metadata } from './metadata';
|
||||
|
||||
export type User = {
|
||||
@@ -15,64 +16,149 @@ export interface Human extends User {
|
||||
loginNames: string[];
|
||||
}
|
||||
|
||||
const createHumanTrend = new Trend('user_create_human_duration', true);
|
||||
export function createHuman(username: string, org: Org, accessToken: string): Promise<Human> {
|
||||
return new Promise((resolve, reject) => {
|
||||
let response = http.asyncRequest(
|
||||
'POST',
|
||||
url('/v2/users/human'),
|
||||
JSON.stringify({
|
||||
username: username,
|
||||
organization: {
|
||||
orgId: org.organizationId,
|
||||
},
|
||||
profile: {
|
||||
givenName: 'Gigi',
|
||||
familyName: 'Zitizen',
|
||||
},
|
||||
email: {
|
||||
email: `${username}@zitadel.com`,
|
||||
isVerified: true,
|
||||
},
|
||||
password: {
|
||||
password: 'Password1!',
|
||||
changeRequired: false,
|
||||
},
|
||||
}),
|
||||
{
|
||||
headers: {
|
||||
authorization: `Bearer ${accessToken}`,
|
||||
'Content-Type': 'application/json',
|
||||
'x-zitadel-orgid': org.organizationId,
|
||||
},
|
||||
// A user is created by a write, but its login names can only be read back from the
|
||||
// query side. Under load that projection can lag behind the write, and the read
|
||||
// then returns no user at all. Retry briefly rather than resolving undefined, which
|
||||
// surfaces much later as "Cannot read property '0' of undefined" while mapping
|
||||
// login names, hiding the real cause.
|
||||
// Diagnostics from the last attempt, so an exhausted retry loop can say *why* it gave
|
||||
// up rather than only that it did. Two failures on 2026-08-27 burned 30 attempts each
|
||||
// and reported nothing more useful than "not readable", which cost a whole cycle.
|
||||
export type ReadUserResult = { user?: any; lastStatus: number; lastBody: string; attempts: number };
|
||||
|
||||
// Backoff that yields to the event loop. `sleep()` would not: k6 documents that it "blocks VU
|
||||
// execution and prevents promises from resolving and event handlers from running", and the same
|
||||
// is true of the synchronous `http.get`. That matters here and nowhere else in this file, because
|
||||
// this is the one helper called from inside a `.then()` callback while up to MaxVUs sibling
|
||||
// creations are still in flight -- blocking serialises every one of them behind this loop.
|
||||
function delay(ms: number): Promise<void> {
|
||||
return new Promise((resolve) => setTimeout(resolve, ms));
|
||||
}
|
||||
|
||||
async function readUserAfterCreate(path: string, userId: string, org: Org, accessToken: string): Promise<ReadUserResult> {
|
||||
let lastStatus = -1;
|
||||
let lastBody = '';
|
||||
let attempt = 0;
|
||||
for (attempt = 0; attempt < 60; attempt++) {
|
||||
const res = await http.asyncRequest('GET', url(`${path}/${userId}`), null, {
|
||||
tags: { name: `${path}/{userId}` },
|
||||
headers: {
|
||||
authorization: `Bearer ${accessToken}`,
|
||||
'Content-Type': 'application/json',
|
||||
'x-zitadel-orgid': org.organizationId,
|
||||
},
|
||||
});
|
||||
// A timed-out request has status 0 and a null body. Calling res.json() on that
|
||||
// throws "GoError: the body is null" and kills the iteration instead of consuming
|
||||
// a retry, which is exactly the failure this retry loop exists to absorb.
|
||||
if (res.status >= 200 && res.status < 300 && res.body) {
|
||||
const user = res.json('user') as any;
|
||||
// The user record can come back before its login names have been projected.
|
||||
// Accepting a merely-truthy user here returns a partial record, and every
|
||||
// caller immediately does loginNames[0] on it -- so this must wait for the
|
||||
// field itself, not just for the object. Observed at 600 VUs on
|
||||
// human_password_login, 2026-08-27: ten retries available, none consumed.
|
||||
if (user && Array.isArray(user.loginNames) && user.loginNames.length > 0) {
|
||||
// A loop that only reports itself when it gives up says nothing about how close it
|
||||
// came, which is how the budget below got guessed at three times before it was
|
||||
// measured once. Deliberately a log line and not a Trend: a Trend recorded here
|
||||
// reaches output.json for manipulate_user, where this runs inside the measured
|
||||
// iteration, and moving a published series to observe it would defeat the point.
|
||||
if (attempt > 0) {
|
||||
console.warn(
|
||||
`read-back retried: user ${userId} readable after ${attempt + 1} attempts, previous status ${lastStatus}`,
|
||||
);
|
||||
}
|
||||
return { user, lastStatus: res.status, lastBody: '', attempts: attempt + 1 };
|
||||
}
|
||||
}
|
||||
lastStatus = res.status;
|
||||
lastBody = (res.body ? String(res.body) : '<null body>').slice(0, 200);
|
||||
// ~100s of waiting in total (backoff rises to 2s and stays there). Sized against a
|
||||
// measured behaviour, not a guess: unloaded, login names are present on the *first*
|
||||
// GET, but a burst of MaxVUs concurrent creations queues them behind the projection
|
||||
// handler advisory lock -- the largest single database cost in the whole v4.17.1 sweep.
|
||||
//
|
||||
// Two things measured on 2026-08-31 that this budget cannot fix, recorded so the next
|
||||
// person does not widen it a fourth time:
|
||||
//
|
||||
// Retries are never consumed successfully. Across every run that day the warn above
|
||||
// fired zero times -- each user either had its login names on the *first* GET or never
|
||||
// had them at all. The read-back is bimodal, so the budget only sets how long we wait
|
||||
// before reporting a failure, not how many succeed.
|
||||
//
|
||||
// And some of those failures are permanent. In one 200-VU run all 200 creates returned
|
||||
// 2xx and, ten minutes later, the query side was still missing zitizen-6 entirely and
|
||||
// zitizen-7/-24/-25 had no login names. Widening the budget to ~220s changed nothing
|
||||
// except how long it took to give up, which is why it was reverted to this.
|
||||
//
|
||||
// Note also that serialising this loop used to grant an accidental grace period: user N
|
||||
// was not asked until users 0..N-1 had finished, so a late user got the queue's time for
|
||||
// free. Awaiting concurrently aligns every deadline on the same instant. That is correct
|
||||
// and strictly less forgiving, and it is a real effect -- it was simply not the cause of
|
||||
// the failures above, which were loss rather than lag.
|
||||
await delay(Math.min(100 * (attempt + 1), 2000));
|
||||
}
|
||||
return { user: undefined, lastStatus, lastBody, attempts: attempt };
|
||||
}
|
||||
|
||||
const createHumanTrend = new Trend('user_create_human_duration', true);
|
||||
export async function createHuman(username: string, org: Org, accessToken: string): Promise<Human> {
|
||||
const res = await http.asyncRequest(
|
||||
'POST',
|
||||
url('/v2/users/human'),
|
||||
JSON.stringify({
|
||||
username: username,
|
||||
organization: {
|
||||
orgId: org.organizationId,
|
||||
},
|
||||
profile: {
|
||||
givenName: 'Gigi',
|
||||
familyName: 'Zitizen',
|
||||
},
|
||||
email: {
|
||||
email: `${username}@zitadel.com`,
|
||||
isVerified: true,
|
||||
},
|
||||
password: {
|
||||
password: 'Password1!',
|
||||
changeRequired: false,
|
||||
},
|
||||
}),
|
||||
{
|
||||
headers: {
|
||||
authorization: `Bearer ${accessToken}`,
|
||||
'Content-Type': 'application/json',
|
||||
'x-zitadel-orgid': org.organizationId,
|
||||
},
|
||||
},
|
||||
);
|
||||
|
||||
if (
|
||||
!check(res, {
|
||||
'create user is status ok': (r) => r.status >= 200 && r.status < 300,
|
||||
})
|
||||
) {
|
||||
throw new Error(`unable to create user(username: ${username}) status: ${res.status} body: ${res.body}`);
|
||||
}
|
||||
createHumanTrend.add(res.timings.duration);
|
||||
|
||||
const userId = res.json('userId') as string;
|
||||
const read = await readUserAfterCreate('/v2/users', userId, org, accessToken);
|
||||
if (!read.user) {
|
||||
throw new Error(
|
||||
`user ${userId} not readable with login names after create ` +
|
||||
`(username: ${username}, attempts: ${read.attempts}, last status: ${read.lastStatus}, ` +
|
||||
`last body: ${read.lastBody})`,
|
||||
);
|
||||
|
||||
response
|
||||
.then((res) => {
|
||||
check(res, {
|
||||
'create user is status ok': (r) => r.status >= 200 && r.status < 300,
|
||||
}) || reject(`unable to create user(username: ${username}) status: ${res.status} body: ${res.body}`);
|
||||
createHumanTrend.add(res.timings.duration);
|
||||
|
||||
const user = http.get(url(`/v2/users/${res.json('userId')!}`), {
|
||||
headers: {
|
||||
authorization: `Bearer ${accessToken}`,
|
||||
'Content-Type': 'application/json',
|
||||
'x-zitadel-orgid': org.organizationId,
|
||||
},
|
||||
});
|
||||
resolve(user.json('user')! as unknown as Human);
|
||||
})
|
||||
.catch((reason) => {
|
||||
reject(reason);
|
||||
});
|
||||
});
|
||||
}
|
||||
return read.user as unknown as Human;
|
||||
}
|
||||
|
||||
const setEmailOTPOnHumanTrend = new Trend('set_human_email_otp_duration', true);
|
||||
export async function setEmailOTPOnHuman(user: User, org: Org, accessToken: string): Promise<void> {
|
||||
const response = await http.asyncRequest('POST', url(`/v2/users/${user.userId}/otp_email`), null, {
|
||||
tags: { name: '/v2/users/{userId}/otp_email' },
|
||||
headers: {
|
||||
authorization: `Bearer ${accessToken}`,
|
||||
'Content-Type': 'application/json',
|
||||
@@ -95,7 +181,8 @@ export function updateHuman(
|
||||
accessToken: string,
|
||||
): Promise<RefinedResponse<any>> {
|
||||
return new Promise((resolve, reject) => {
|
||||
let response = http.asyncRequest('PUT', url(`/v2beta/users/${userId}`), JSON.stringify(payload), {
|
||||
let response = http.asyncRequest('PATCH', url(`/v2/users/${userId}`), JSON.stringify(payload), {
|
||||
tags: { name: '/v2/users/{userId}' },
|
||||
headers: {
|
||||
authorization: `Bearer ${accessToken}`,
|
||||
'Content-Type': 'application/json',
|
||||
@@ -106,7 +193,7 @@ export function updateHuman(
|
||||
response
|
||||
.then((res) => {
|
||||
check(res, {
|
||||
'update user is status ok': (r) => r.status === 201,
|
||||
'update user is status ok': (r) => r.status >= 200 && r.status < 300,
|
||||
});
|
||||
updateHumanTrend.add(res.timings.duration);
|
||||
resolve(res);
|
||||
@@ -122,46 +209,44 @@ export interface Machine extends User {
|
||||
}
|
||||
|
||||
const createMachineTrend = new Trend('user_create_machine_duration', true);
|
||||
export function createMachine(username: string, org: Org, accessToken: string): Promise<Machine> {
|
||||
return new Promise((resolve, reject) => {
|
||||
let response = http.asyncRequest(
|
||||
'POST',
|
||||
url('/management/v1/users/machine'),
|
||||
JSON.stringify({
|
||||
userName: username,
|
||||
name: username,
|
||||
// bearer
|
||||
access_token_type: 0,
|
||||
}),
|
||||
{
|
||||
headers: {
|
||||
authorization: `Bearer ${accessToken}`,
|
||||
'Content-Type': 'application/json',
|
||||
'x-zitadel-orgid': org.organizationId,
|
||||
},
|
||||
export async function createMachine(username: string, org: Org, accessToken: string): Promise<Machine> {
|
||||
const res = await http.asyncRequest(
|
||||
'POST',
|
||||
url('/management/v1/users/machine'),
|
||||
JSON.stringify({
|
||||
userName: username,
|
||||
name: username,
|
||||
// bearer
|
||||
access_token_type: 0,
|
||||
}),
|
||||
{
|
||||
headers: {
|
||||
authorization: `Bearer ${accessToken}`,
|
||||
'Content-Type': 'application/json',
|
||||
'x-zitadel-orgid': org.organizationId,
|
||||
},
|
||||
},
|
||||
);
|
||||
|
||||
if (
|
||||
!check(res, {
|
||||
'create user is status ok': (r) => r.status >= 200 && r.status < 300,
|
||||
})
|
||||
) {
|
||||
throw new Error(`unable to create user(username: ${username}) status: ${res.status} body: ${res.body}`);
|
||||
}
|
||||
createMachineTrend.add(res.timings.duration);
|
||||
|
||||
const userId = res.json('userId') as string;
|
||||
const read = await readUserAfterCreate('/v2/users', userId, org, accessToken);
|
||||
if (!read.user) {
|
||||
throw new Error(
|
||||
`user ${userId} not readable with login names after create ` +
|
||||
`(username: ${username}, attempts: ${read.attempts}, last status: ${read.lastStatus}, ` +
|
||||
`last body: ${read.lastBody})`,
|
||||
);
|
||||
|
||||
response
|
||||
.then((res) => {
|
||||
check(res, {
|
||||
'create user is status ok': (r) => r.status >= 200 && r.status < 300,
|
||||
}) || reject(`unable to create user(username: ${username}) status: ${res.status} body: ${res.body}`);
|
||||
createMachineTrend.add(res.timings.duration);
|
||||
|
||||
const user = http.get(url(`/v2beta/users/${res.json('userId')!}`), {
|
||||
headers: {
|
||||
authorization: `Bearer ${accessToken}`,
|
||||
'Content-Type': 'application/json',
|
||||
'x-zitadel-orgid': org.organizationId,
|
||||
},
|
||||
});
|
||||
resolve(user.json('user')! as unknown as Machine);
|
||||
})
|
||||
.catch((reason) => {
|
||||
reject(reason);
|
||||
});
|
||||
});
|
||||
}
|
||||
return read.user as unknown as Machine;
|
||||
}
|
||||
|
||||
export type MachinePat = {
|
||||
@@ -172,6 +257,7 @@ const addMachinePatTrend = new Trend('user_add_machine_pat_duration', true);
|
||||
export function addMachinePat(userId: string, org: Org, accessToken: string): Promise<MachinePat> {
|
||||
return new Promise((resolve, reject) => {
|
||||
let response = http.asyncRequest('POST', url(`/management/v1/users/${userId}/pats`), null, {
|
||||
tags: { name: '/management/v1/users/{userId}/pats' },
|
||||
headers: {
|
||||
authorization: `Bearer ${accessToken}`,
|
||||
'Content-Type': 'application/json',
|
||||
@@ -198,6 +284,7 @@ const addMachineSecretTrend = new Trend('user_add_machine_secret_duration', true
|
||||
export function addMachineSecret(userId: string, org: Org, accessToken: string): Promise<MachineSecret> {
|
||||
return new Promise((resolve, reject) => {
|
||||
let response = http.asyncRequest('PUT', url(`/management/v1/users/${userId}/secret`), null, {
|
||||
tags: { name: '/management/v1/users/{userId}/secret' },
|
||||
headers: {
|
||||
authorization: `Bearer ${accessToken}`,
|
||||
'Content-Type': 'application/json',
|
||||
@@ -232,6 +319,7 @@ export function addMachineKey(userId: string, org: Org, accessToken: string, pub
|
||||
publicKey: publicKey,
|
||||
}),
|
||||
{
|
||||
tags: { name: '/management/v1/users/{userId}/keys' },
|
||||
headers: {
|
||||
authorization: `Bearer ${accessToken}`,
|
||||
'Content-Type': 'application/json',
|
||||
@@ -253,7 +341,8 @@ export function addMachineKey(userId: string, org: Org, accessToken: string, pub
|
||||
const lockUserTrend = new Trend('lock_user_duration', true);
|
||||
export function lockUser(userId: string, org: Org, accessToken: string): Promise<RefinedResponse<any>> {
|
||||
return new Promise((resolve, reject) => {
|
||||
let response = http.asyncRequest('POST', url(`/v2beta/users/${userId}/lock`), null, {
|
||||
let response = http.asyncRequest('POST', url(`/v2/users/${userId}/lock`), null, {
|
||||
tags: { name: '/v2/users/{userId}/lock' },
|
||||
headers: {
|
||||
authorization: `Bearer ${accessToken}`,
|
||||
'Content-Type': 'application/json',
|
||||
@@ -264,7 +353,7 @@ export function lockUser(userId: string, org: Org, accessToken: string): Promise
|
||||
response
|
||||
.then((res) => {
|
||||
check(res, {
|
||||
'update user is status ok': (r) => r.status >= 200 && r.status < 300,
|
||||
'lock user is status ok': (r) => r.status >= 200 && r.status < 300,
|
||||
});
|
||||
lockUserTrend.add(res.timings.duration);
|
||||
resolve(res);
|
||||
@@ -278,7 +367,8 @@ export function lockUser(userId: string, org: Org, accessToken: string): Promise
|
||||
const deleteUserTrend = new Trend('delete_user_duration', true);
|
||||
export function deleteUser(userId: string, org: Org, accessToken: string): Promise<RefinedResponse<any>> {
|
||||
return new Promise((resolve, reject) => {
|
||||
let response = http.asyncRequest('DELETE', url(`/v2beta/users/${userId}`), null, {
|
||||
let response = http.asyncRequest('DELETE', url(`/v2/users/${userId}`), null, {
|
||||
tags: { name: '/v2/users/{userId}' },
|
||||
headers: {
|
||||
authorization: `Bearer ${accessToken}`,
|
||||
'Content-Type': 'application/json',
|
||||
@@ -289,7 +379,7 @@ export function deleteUser(userId: string, org: Org, accessToken: string): Promi
|
||||
response
|
||||
.then((res) => {
|
||||
check(res, {
|
||||
'update user is status ok': (r) => r.status === 201,
|
||||
'delete user is status ok': (r) => r.status >= 200 && r.status < 300,
|
||||
});
|
||||
deleteUserTrend.add(res.timings.duration);
|
||||
resolve(res);
|
||||
@@ -304,6 +394,7 @@ const setUserMetadataTrend = new Trend('set_user_metadata_duration', true);
|
||||
export function setUserMetadata(metadata: Metadata[], userId: string, accessToken: string): Promise<RefinedResponse<any>> {
|
||||
return new Promise((resolve, reject) => {
|
||||
let response = http.asyncRequest('POST', url(`/v2/users/${userId}/metadata`), JSON.stringify({ metadata: metadata }), {
|
||||
tags: { name: '/v2/users/{userId}/metadata' },
|
||||
headers: {
|
||||
authorization: `Bearer ${accessToken}`,
|
||||
'Content-Type': 'application/json',
|
||||
|
||||
@@ -26,6 +26,7 @@ export async function addUserGrant(
|
||||
roleKeys: roles,
|
||||
}),
|
||||
{
|
||||
tags: { name: '/management/v1/users/{userId}/grants' },
|
||||
headers: {
|
||||
authorization: `Bearer ${accessToken}`,
|
||||
'Content-Type': 'application/json',
|
||||
|
||||
Reference in New Issue
Block a user