--- title: manipulate user benchmark of zitadel v4.17.1 sidebar_label: manipulate user --- import OutputSource from "./output.json"; import { BenchmarkChart } from "@/components/benchmark_chart"; Benchmark results of the v4.17.1 release of Zitadel. ## Performance test results | Metric | Value | | :------------------------------------ | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Baseline | none | | Purpose | Test current performance | | Test start | 2026-08-27 13:41:19 UTC | | Test end | 2026-08-27 14:11:30 UTC | | Test duration | 30min | | Executed test | manipulate_user | | k6 version | v2.1.0 | | VUs | 200 | | Client location | US1 | | ZITADEL location | US1 | | ZITADEL container specification | vCPU: 6
Memory: 6 Gi
Container min scale: 7
Container max scale: 7 | | ZITADEL Version | v4.17.1 | | ZITADEL Settings | Eventstore autovacuum (new in v4.17): Enabled: true
VacuumThreshold: 1000000
AnalyzeThreshold: 1000000
Tuned for the ~4,000 events/s expected during the benchmark, per the [tuning guide](https://zitadel.com/docs/self-hosting/manage/tuning#reading-events). | | ZITADEL feature flags | Enabled: loginDefaultOrg (instance), oidcTokenExchange (system), enableBackChannelLogout (system), consoleUseV2UserApi (instance)
improvedPerformance (instance): IMPROVED_PERFORMANCE_PROJECT_GRANT, IMPROVED_PERFORMANCE_PROJECT, IMPROVED_PERFORMANCE_USER_GRANT, IMPROVED_PERFORMANCE_ORG_DOMAIN_VERIFIED
Disabled: loginV2 (explicitly false at instance), userSchema, debugOidcParentError, oidcSingleV1SessionTermination, permissionCheckV2
Captured from the instance **before** the run (sequence 7, set 2026-08-27 13:35:31 UTC); the same capture covers `manipulate_user` at 13:41 and the rest of the sweep from 14:56, so all eleven targets ran the same configuration.
This restores like-for-like comparison with the v4 pages, which ran five improvedPerformance options. The fifth, IMPROVED_PERFORMANCE_ORG_BY_ID, is `reserved` in the proto and no longer exists. | | Database | type: psql
version: v17.4 | | Database location | US1 | | Database specification | vCPU: 8
memory: 32Gib | | ZITADEL metrics during test | CPU: p50 16%, p99 23% (peak 41%)
Memory: p50 7%, peak 12%
Instances: 7 (constant), no container startups recorded
Server-side: 89 req/s, latency p50 300ms / p95 12.9s / p99 36.6s | | Observed errors | 225 of 171,987 requests failed (0.13%). **Corrected by hand from the generated row, which reported these as a single population; they are two unrelated ones, 25 minutes apart.** (1) **150 platform errors, not ZITADEL errors**: HTTP 503 `Service error -27`, an HTML page from the Google frontend rather than a ZITADEL JSON error, confined entirely to a 55-second burst from 13:43:19 to 13:44:13 UTC, two minutes into the run, with no recurrence in the following 25 minutes. The shape is a container restart or replacement; a GCP post-mortem (restarts, panics, instance count) is pending and its findings so far are recorded in `benchmark/postmortems/2026-08-27-503-burst.md`. (2) **75 request timeouts** from 14:08:48 to 14:11:02 UTC, the documented degradation-over-sustained-churn behaviour of this target: the first arrives 1,658s (~28min) into the run and they cluster in the final minutes alongside a steady latency climb. The previous 200-VU run saw the equivalent 77 timeouts beginning at 1,255s. 127 of 137,522 checks failed (0.09%): create user (29), update user (41), lock user (25), delete user (32) — all of them within the two windows above. **The failed-check count is not comparable to the previous run's zero**: before commit `cdde3ab3e` a timeout killed the iteration before its check could be recorded (77 timeouts, 77 aborted iterations, 0 recorded check failures); with the response body now guarded, 75 timeouts aborted **0** iterations and the failures are recorded instead of being silently fatal. **This target ran at 200 VUs, not the 600 used for every other target**: at 600 VUs the same failure begins after 179s and produces a far higher error rate for identical throughput. | | Top 3 most expensive database queries | 1: Query events by a multi-aggregate / event_type filter set (13,352 s)
2: Projection handler advisory lock, one per projection per instance — **lock wait, not query work** (6,438 s)
3: Query events by instance_id, aggregate_types, event_types, position (paged) (3,970 s)
Ranked by total accumulated execution time summed across concurrent sessions (Cloud SQL Query Insights) — not CPU time; it can exceed the wall-clock window. | | Database metrics during test | CPU: 98% median, 100% peak
Memory: 40%
Connections: 46
Transactions: 1,487/s
IOPS: read 0, write 539
**CPU-saturated — the bottleneck for this target.** | | k6 Iterations per second | 19 | | k6 output | [output](#k6-output) | | flowchart outcome | **Potential issues.** Not resource-bound: database CPU 98%, ZITADEL CPU 16%, yet **35% of the top-5 query time is advisory lock wait** — predominantly the projection handler lock, which serialises this workload. Additional vCPU will not help. Logged as an issue rather than blocking further scale testing. This target also shows the documented degradation over sustained single-org churn, with per-operation latency climbing across the run. | ## Endpoint latencies ## k6 output ```bash █ TOTAL RESULTS checks_total.......: 137522 75.453808/s checks_succeeded...: 99.90% 137395 out of 137522 checks_failed......: 0.09% 127 out of 137522 ✓ user defined ✓ authorize status ok ✓ login name status ok ✓ login shows password page ✓ password status ok ✓ password callback ✓ code set ✓ token status ok ✓ access token created ✓ id token created ✓ info created ✓ org created ✗ create user is status ok ↳ 99% — ✓ 34370 / ✗ 29 ✗ update user is status ok ↳ 99% — ✓ 34329 / ✗ 41 ✗ lock user is status ok ↳ 99% — ✓ 34345 / ✗ 25 ✗ delete user is status ok ↳ 99% — ✓ 34338 / ✗ 32 ✓ org removed CUSTOM delete_user_duration.................: min=43.5ms avg=987.45ms max=17.06s p(50)=322.03ms p(95)=5.07s p(99)=8.54s lock_user_duration...................: min=24.92ms avg=138.46ms max=4.11s p(50)=108.29ms p(95)=329.45ms p(99)=502.43ms login_ui_enter_login_name_duration...: min=154.9ms avg=154.9ms max=154.9ms p(50)=154.9ms p(95)=154.9ms p(99)=154.9ms login_ui_enter_password_duration.....: min=14.56ms avg=14.56ms max=14.56ms p(50)=14.56ms p(95)=14.56ms p(99)=14.56ms login_ui_init_login_duration.........: min=109.41ms avg=109.41ms max=109.41ms p(50)=109.41ms p(95)=109.41ms p(99)=109.41ms login_ui_token_duration..............: min=85.43ms avg=85.43ms max=85.43ms p(50)=85.43ms p(95)=85.43ms p(99)=85.43ms org_create_org_duration..............: min=57.64ms avg=57.64ms max=57.64ms p(50)=57.64ms p(95)=57.64ms p(99)=57.64ms update_human_duration................: min=17.01ms avg=116.71ms max=2.02s p(50)=87.97ms p(95)=291.12ms p(99)=441.72ms user_create_human_duration...........: min=295.4ms avg=461.62ms max=2.28s p(50)=428.92ms p(95)=679.59ms p(99)=966.08ms HTTP http_req_duration....................: min=14.56ms avg=2.09s max=1m0s p(50)=304.58ms p(95)=10.23s p(99)=38.43s { expected_response:true }.........: min=14.56ms avg=2.07s max=59.92s p(50)=303.74ms p(95)=10.08s p(99)=38.08s http_req_failed......................: 0.13% 225 out of 171987 http_reqs............................: 171987 94.363623/s EXECUTION iteration_duration...................: min=497.6ms avg=10.5s max=1m57s p(50)=4.22s p(95)=40.23s p(99)=55.66s iterations...........................: 34399 18.873602/s vus..................................: 0 min=0 max=200 vus_max..............................: 200 min=200 max=200 NETWORK data_received........................: 98 MB 54 kB/s data_sent............................: 31 MB 17 kB/s running (30m22.6s), 000/200 VUs, 34399 complete and 0 interrupted iterations default ✓ [ 100% ] 200 VUs 30m0s ```