mirror of
https://github.com/andwati/zitadel.git
synced 2026-10-18 05:52:51 +00:00
main
10
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
   
|
fbe7b96138 |
fix(eventstore): prevent projections from skipping events (#12703)
<!-- CURSOR_AGENT_PR_BODY_BEGIN -->
# Which Problems Are Solved
Handler/v2 projections (`users14` and every other event projection) can
overlook committed events under load:
- `eventstore.commands_to_events` stamped `position` with `NOW()`
(transaction start). A slow create TX could commit after a faster TX
that started later, so the projector stored a later position and never
saw the earlier one.
- Resume used `OFFSET` on a **filtered** row count. That could skip
`in_tx_order = 1` at the next position (for example `user.human.added`).
- Two transactions in the same microsecond both start at `in_tx_order =
1`. A cursor of `in_tx_order > last` alone would drop the other TX’s
first event.
This does not repair rows that are already missing from projections.
# How the Problems Are Solved
- Setup step 77 replaces `eventstore.commands_to_events` so `created_at
= statement_timestamp()` and `position = EXTRACT(EPOCH FROM
clock_timestamp())` at INSERT time. Initialise `08_events_table.sql`
matches that for new databases. The function is `VOLATILE` to match
`clock_timestamp()`.
- The projector still drains in-flight writers with `pg_advisory_lock` /
`pg_advisory_unlock` in one statement, then caps with `position <=
EXTRACT(EPOCH FROM now())` (projector TX start). The exclusive lock is
**not** held across the event `SELECT`. The cap must stay `now()`, not
`clock_timestamp()`, or later inserts become a high-water mark.
- Resume is a lexicographic `EventSortKey` on `(position, in_tx_order,
instance_id, aggregate_type, aggregate_id, sequence)` via
`AfterEventSortKey`. SQL `OFFSET` is no longer used for projection
catch-up.
- Handler and FieldHandler persist the last event’s `in_tx_order` in
`current_states.filter_offset` (existing column, new meaning). Step 77
backfills matching rows from `events2`; unmatched rows keep their
previous value.
# Additional Changes
- `eventstore.Event` exposes `InTxOrder() uint32` (BaseEvent, repository
Event, v3 event, v1 models).
- v3 push SELECT/RETURNING now includes `in_tx_order` so newly written
events carry a real ordinal.
- `WithMinPosition` stays an inclusive `PositionAtLeast` floor and does
not smash the sort-key cursor.
- `SearchQuery.PositionAfter(decimal)` is unchanged (used by
access-token lookup). Do not confuse it with
`SearchQueryBuilder.AfterEventSortKey`.
- `commands_to_events` in setup 77 and initialise 08 is `VOLATILE
PARALLEL SAFE`. Shipped 40/64/70 stay `STABLE` because they still stamp
with `NOW()`.
- Drop unused `last_owner TEXT` DECLARE from setup 77 and initialise
`08_events_table.sql` (`i INTEGER` remains for the enforce-owner loop).
Shipped 40/64/70 keep the leftover.
- This PR’s migration is numbered 77 because `main` already has setup 76
(`76_users14_login_equality_indexes`). First-loop order is s70, then
main’s 76, then this 77.
# Additional Context
**Reviewer map**
| Start here | Why |
| --- | --- |
| `cmd/setup/77.sql` + `cmd/setup/77.go` | Position stamp + atomic
function replace/backfill after shipped `s70` and main’s s76 |
| `internal/eventstore/event_sort_key.go` + `search_query.go` |
Canonical cursor and `AfterEventSortKey` |
| `internal/eventstore/handler/v2/{state.go,handler.go}` |
Persist/resume cursor; `eventQuery` |
| `internal/eventstore/repository/sql/{query.go,postgres.go}` | Tuple
`>` filter, 6-column `ORDER BY`, await cap |
| `internal/eventstore/v3/push.go` / `push.sql` | Scan `in_tx_order` on
write |
Mechanical / low-risk: handler and query test fixtures, Event interface
stubs (`event_base.go`, repository Event, mocks, v1 models).
**Do not look for** a `70.sql` change — shipped migrations stay as on
`main`. Setup 40/64 still use `NOW()`; step 77 overwrites the function
afterward. Fresh `FirstInstance` (runs before `s70`) can still stamp
`NOW()` until 77; runtime pushes after 77 use `clock_timestamp()`.
Main’s `cmd/setup/76.go` is the users14 login-equality indexes, not this
stamp.
**Rollout**
1. Run setup so 77 replaces `commands_to_events` and backfills
`filter_offset`.
2. Restart ZITADEL so projectors load the new resume path.
3. Existing missing `users14` rows are not repaired by this PR.
**Tests**
- `go test ./internal/eventstore/... ./cmd/setup/`
- Sort-key SQL fixtures in `query_sort_key_test.go` (no `OFFSET`;
aggregate-id queries keep 6-column `ORDER BY`)
- `cmd/setup/77_test.go` asserts the embed has no unused `last_owner`
and still uses `clock_timestamp()` / `VOLATILE PARALLEL SAFE`
Related: #8509, #10480 (remove OFFSET), #10560 (advisory locks), #10710
(stable ORDER BY).
Out of scope: repairing already-missing projection rows, holding the
exclusive lock during SELECT, rewriting historical setup 40/64 SQL.
<!-- CURSOR_AGENT_PR_BODY_END -->
<div><a
href="https://cursor.com/agents/bc-3da6eb51-11d4-4a8a-8ba0-e190f116df05?cursor_ref=pr_footer&cursor_cta=open_in_web"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://cursor.com/assets/images/open-in-web-dark.png"><source
media="(prefers-color-scheme: light)"
srcset="https://cursor.com/assets/images/open-in-web-light.png"><img
alt="Open in Web" width="114" height="28"
src="https://cursor.com/assets/images/open-in-web-dark.png"></picture></a> <a
href="https://cursor.com/background-agent?bcId=bc-3da6eb51-11d4-4a8a-8ba0-e190f116df05&cursor_ref=pr_footer&cursor_cta=open_in_cursor"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://cursor.com/assets/images/open-in-cursor-dark.png"><source
media="(prefers-color-scheme: light)"
srcset="https://cursor.com/assets/images/open-in-cursor-light.png"><img
alt="Open in Cursor" width="131" height="28"
src="https://cursor.com/assets/images/open-in-cursor-dark.png"></picture></a> </div>
---------
Co-authored-by: Cursor Agent <[email protected]>
Co-authored-by: Silvan <[email protected]>
|
||
 
|
3be21ccbf0 |
docs(perf): v4.17.1 results and harness fixes (#12651)
Publishes the v4.17.1 benchmark sweep and the harness fixes the sweep uncovered. ## Results All 11 published targets ran for 30 minutes each against a fixed 7-container deployment (vCPU 6 / 6 Gi, psql v17.4, US1 client and server), at 600 VUs except where noted. Latencies are `http_req_duration` over the test phase. | target | VUs | iter/s | p50 | p95 | p99 | failed requests | |:---|---:|---:|---:|---:|---:|:---| | machine_pat_login | 600 | 3452 | 24.53ms | 872ms | 1.00s | 29 / 6,335,633 (0.00%) | | user_info | 600 | 3217 | 29.32ms | 943ms | 1.13s | 66 / 5,800,788 (0.00%) | | introspect | 600 | 2297 | 46.49ms | 1.16s | 1.36s | 1,400 / 4,159,863 (0.03%) | | machine_jwt_profile_grant | 600 | 1076 | 85.34ms | 2.57s | 2.93s | 97 / 1,986,156 (0.00%) | | machine_client_credentials_login | 600 | 834 | 56.78ms | 1.74s | 2.42s | 1,789 / 3,073,497 (0.05%) | | add_session | 600 | 767 | 95.25ms | 3.53s | 4.15s | 1 / 1,416,546 (0.00%) | | oidc_session | 600 | 326 | 77.47ms | 3.41s | 4.43s | 3 / 1,767,162 (0.00%) | | otp_session | 600 | 84 | 47.80ms | 13.83s | 20.22s | 182 / 466,756 (0.03%) | | password_session | 600 | 68 | 408.63ms | 20.00s | 24.05s | 1 / 253,280 (0.00%) | | human_password_login | 600 | 49 | 146.67ms | 4.08s | 1m0s | 9,227 / 683,256 (1.35%) | | manipulate_user | **200** | 19 | 304.58ms | 10.23s | 38.43s | 225 / 171,987 (0.13%) | Against the goals: - **#8352 is met.** `machine_jwt_profile_grant` reached **1076/s** against the 1000/s goal, with p50 at 85ms. - **#4424 is not met.** `human_password_login` runs at **49/s** against a 1200/s goal. The sweep ran with `improvedPerformance` enabled (`PROJECT_GRANT`, `PROJECT`, `USER_GRANT`, `ORG_DOMAIN_VERIFIED`), captured from the instance before the first target rather than assumed; the full flag set is on every page. ## What the observability rows show The containers were never the constraint. ZITADEL CPU sat at 9–50% (p50), memory at 6–7% (p50, peaking at 16%) of 6 GiB, and instance count was flat at 7 with no container startups in any window. Database CPU was at **86–98% median on nine of eleven targets**. Any "ZITADEL achieves N/s" figure that omits which resource ran out is true and misleading, so the pages say. The query breakdown then shows what the CPU numbers cannot: **advisory lock wait is the single largest category of database time across the sweep (91,857 s accumulated)**, ahead of event reads (64,805 s), event push (26,411 s) and projection reads (14,625 s). The three locks are the projection handler lock (`handler/v2/handler.go`), the eventstore read barrier (`repository/sql/query.go`) and the shared lock every push takes (`eventstore/v3/push.go`). That is what sets the `flowchart outcome` on each page, and it is not uniform: - **Scale** on the seven targets where a resource genuinely ran out — database CPU at 92–98% with 0% lock share. - **Potential issues** on `human_password_login`, `password_session`, `otp_session` and `manipulate_user`, where 45–74% of the top query time is lock wait. `password_session` runs at 51% database CPU and 50% container CPU and still only manages 68 iter/s; adding vCPU there buys nothing. Autovacuum is cleared as a contributor: 93.9% median DB CPU in buckets containing a vacuum against 93.1% without, within noise. The new v4.17 eventstore autovacuum is barely triggered — `events2` is not in the top twenty vacuumed tables. ## Harness fixes Defects found during the sweep, all of which corrupted published numbers or aborted runs: 1. **Colliding check names.** `updateHuman`, `lockUser` and `deleteUser` all registered checks as `update user is status ok`, two of them asserting `status === 201` against endpoints returning 200. k6 aggregates checks by name, so this produced a permanent, fictional 28% failure rate. Each check now has its own name and accepts 2xx. 2. **User-creation read-after-write race**, in three variants, each found after the previous fix. The read-back resolved `undefined` when the projection was behind, surfacing much later as `Cannot read property '0' of undefined`; then an unguarded `res.json()` turned every request timeout into a killed iteration; then a *partial* read — a valid user object with `loginNames` not yet projected — passed an `if (user)` guard and failed identically. `readUserAfterCreate()` now guards status and body before parsing, requires the field rather than the object, and retries with backoff, rejecting with the user id and username. 3. **Metric cardinality.** Id-bearing URLs produced 800k+ time series against k6's suggested 100k, driving k6 to 6.8 GB and an OOM mid-run. All 16 such call sites now set a grouped `name` tag. 4. **`password_session` wrote its CSV to `otp_session`'s filename** in the Makefile, silently overwriting the previous target's output. The read-back race is not only a harness defect: at 600 VUs it measures real load-induced projection lag, exceeding 25 s for login-name projection and 100 s for the user projection, against instant reads on an unloaded instance. ## Stable v2 user API Four calls in `benchmark/src/user.ts` still went to `/v2beta/users` — update, lock, delete and the machine read-back. They date from the original load-test bootstrap in April 2024 (#7635), when the user service was still v2beta, and were never moved when it was promoted; they surfaced in this diff only because the read-after-write fix rewrote the surrounding functions. All four are now on stable v2. `updateHuman` is the one that changed shape: `PUT /v2beta/users/{id}` is `UpdateHumanUser`, whose stable-v2 counterpart `PUT /v2/users/human/{id}` is deprecated in the proto, so it now calls `PATCH /v2/users/{id}` (`UpdateUser`) and `manipulate_user` nests the profile under `human`. Both RPCs land on `command.ChangeUserHuman`, so the measured work is unchanged. `lockUser`, `deleteUser` and the `createMachine` read-back move over unchanged in method and payload. No published result is affected: per-endpoint latencies come from custom Trends, and `run-bench.sh` drops the `name` system tag, so no artefact ever recorded a v2beta URL. The remaining `/management/v1` and `/admin/v1` calls are left alone — migrating those changes which API surface is benchmarked and would need a re-run. ## Timestamps Pages reported `Test start` as a bare `HH:MM`, so a run could not be correlated against monitoring the next day. Pages now carry the full UTC date and time and a `Test end` row, both derived from the first and last per-second row of `output.json`. `_template.mdx` is updated to match. ## Post-mortem `benchmark/postmortems/2026-08-27-503-burst.md` records two bursts of HTTP 503 `Service error -27` — an HTML page from the Google frontend, not a ZITADEL error — on `manipulate_user` and `human_password_login`, six hours apart. **The investigation is open**, and the note is explicit about what is established (no container startups; the two bursts have opposite server-side signatures), what is provisionally ruled out (restart, panic), and what is not yet known. It is included because both affected pages report the 503s as a separate population from the ZITADEL failures beside them, and that split needs its evidence somewhere. It also records two defects found by accident in the collected logs, unrelated to the 503s: `sql: transaction has already been committed or rolled back` (232 occurrences under load) and a missing i18n key, `"unable to get user by id" not found in language "en"`. ## Notes for review - **`manipulate_user` ran at 200 VUs, not 600, and the page says so.** Throughput saturates near 100 VUs; at 600 the same target produced 4,283 failed requests for identical throughput against 225 at 200 VUs. Its failure mode is degradation over sustained single-org churn — per-operation p50 climbs 1.7x–6x between the first and last five minutes of a run — not a concurrency threshold. - **Two `Observed errors` rows are hand-corrected** and deliberately differ from the generated text, on `manipulate_user` and `human_password_login`. Both cases are one generated number covering two unrelated failure populations; each page says it was corrected and why. - Org deletion failing with a 500 (`V3-C8l3V`) after heavy user churn is already tracked as #8510 and was hit repeatedly during cleanup. - **The run/publish tooling is no longer part of this PR.** It moved to a private repository, along with the cloud collection scripts, because deployment identity does not belong here. What remains is the published pages, the post-mortem findings and the harness fixes. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5 (1M context) <[email protected]> |
||
|
|
318acd4bc3 |
perf(query): speed up ListUsers login name equality filters (#12460)
<!-- CURSOR_AGENT_PR_BODY_BEGIN --> ## Summary `ListUsers` with `LoginNameQuery` + equals/equals-ignore-case was very slow on large orgs (e.g. ~330k users) because the generated SQL filtered the `projections.login_names3` **view** on the computed `login_name_lower` expression and correlated that subquery on `users14.id`. Postgres nested-looped every user and never used `login_names3_users_search (instance_id, user_name_lower)`. This change rewrites the **query planner** for that hot path: when an equals/equals-ignore-case login-name filter is present (and not under `OR`/`NOT`), the user list query **INNER JOINs** an indexed matches subquery instead of filtering via the view expression. The matches SQL mirrors `user_by_login_name.sql` (`user_name_lower` / domain paths + `preferred` / `is_primary`). Non-equals methods and OR combinations keep the previous view-based filter so semantics stay unchanged. Also adds a k6 use case that mirrors login v2 discovery (`loginNameQuery` EQUALS_IGNORE_CASE + `organizationIdQuery`, `limit: 2`): ```bash cd benchmark make users_by_login_name USER_AMOUNT=100000 VUS=10 DURATION=60s ``` ## Approach 1. `NewLoginNameSearchQuery` for equals / equals-ignore-case returns a marker `loginNameEqualsFilter` (other methods unchanged). 2. `prepareUsersQuery` extracts that marker when safe, then: - builds the usual `sq.SelectBuilder` **without** the login-name view predicate - adds `JoinClause` to `user_login_name_matches(.sql)` / `_case_sensitive.sql` as `login_name_matches` - keeps metadata JOIN/`DISTINCT` only when metadata filters are present (same as before) 3. Embedded SQL files under `internal/query/` for the matches subquery. Local smoke against ~330k synthetic users: baseline ~1392ms → rewritten path ~0.4ms for a single equals-ignore-case lookup. ## Test plan - [x] `go test ./internal/query/ -run 'TestLoginName|TestUsers|TestUserByLoginName'` (after generate-stubs) - [x] Existing `user_test` expected SQL updated (no always-on metadata join; login-name equals uses JOIN) - [ ] Run k6 before/after on a large `USER_AMOUNT` (e.g. 50k–100k+) and compare `list_users_duration` p50/p95/p99 - [x] Manual login v2 username discovery against a large org - [x] Confirm OR / NOT / CONTAINS login-name queries still return expected results <!-- CURSOR_AGENT_PR_BODY_END --> <div><a href="https://cursor.com/agents/bc-252883bd-48d9-492e-b619-5ccfa93cf9c3"><picture><source media="(prefers-color-scheme: dark)" srcset="https://cursor.com/assets/images/open-in-web-dark.png"><source media="(prefers-color-scheme: light)" srcset="https://cursor.com/assets/images/open-in-web-light.png"><img alt="Open in Web" width="114" height="28" src="https://cursor.com/assets/images/open-in-web-dark.png"></picture></a> <a href="https://cursor.com/background-agent?bcId=bc-252883bd-48d9-492e-b619-5ccfa93cf9c3"><picture><source media="(prefers-color-scheme: dark)" srcset="https://cursor.com/assets/images/open-in-cursor-dark.png"><source media="(prefers-color-scheme: light)" srcset="https://cursor.com/assets/images/open-in-cursor-light.png"><img alt="Open in Cursor" width="131" height="28" src="https://cursor.com/assets/images/open-in-cursor-dark.png"></picture></a> </div> --------- Co-authored-by: Cursor Agent <[email protected]> Co-authored-by: Silvan <[email protected]> |
||
|
93209bd348 |
chore(deps): update benchmark dependencies (#12442)
# Which Problems Are Solved Outdated deps on the benchmark typscript modules. # How the Problems Are Solved Upgrade all deps # Additional Changes - Add xk6 a prerequisite for running benchmarks - Update the moduleResolution setting in tsconfig to a non-deprecated value - Some automatic code formatting --------- Co-authored-by: Copilot Autofix powered by AI <[email protected]> |
||
 Max PeintnerandGitHub
|
b27fdfc28c |
chore: update dependencies (#12242)
# Which Problems Are Solved Fixes multiple high-severity Dependabot alerts for transitive dependencies (axios, protobufjs, systeminformation, tmp, fast-uri, @babel/plugin-transform-modules-systemjs). # How the Problems Are Solved **Direct dependency updates:** - `@opentelemetry/*` 0.217 → 0.218 in login (new otlp-transformer dropped protobufjs as direct dep) - `posthog-js` 1.374 → 1.380 in console (dropped the OpenTelemetry/protobufjs chain entirely) **pnpm overrides** for remaining transitive deps that can't be updated via their parents (all pulled by the gRPC stack, Cypress, karma, nx, or ajv): - `axios` → ≥1.17.0 (prototype pollution, proxy bypass) - `protobufjs` → ≥7.5.7 / ≥8.0.2 (code injection, prototype pollution, DoS) - `systeminformation` → ≥5.31.6 (command injection) - `tmp` → ≥0.2.6 (path traversal) - `fast-uri` → ≥3.1.2 (path traversal, host confusion) **Benchmark:** refreshed `package-lock.json` to resolve @babel and fast-uri alerts. |
||
|
|
96130b720c |
fix(benchmark): set proper audience to introspect token (#11586)
# Which Problems Are Solved Benchmark appears to run correctly, however the token used isn't valid. The backend (Zitadel 4.11-pre) prints a constant stream of info logs that the token isn't valid for the client. # How the Problems Are Solved Pass the generated project IDs as a scope for a new login. # Additional Changes - ensure keys for introspection and add session recipes. # Additional Context - Closes #11585 --------- Co-authored-by: Silvan <[email protected]> |
||
 Marco A.andGitHub
|
382aec2d61 |
chore: Service Account Naming Consistency (#11557)
# Which Problems Are Solved Inconsistent naming of service account, found in the following variations: - Machine User - machine user - Service User - Machine Account - Technical Account - User: Type Machine # How the Problems Are Solved Attentive search and replace. Localizations have been translated using Copilot # Additional Changes Some unused methods have been removed from the Go code. # Additional Context - Closes #11285 |
||
 
|
b99271755b |
chore: renamed one time password to OTP (#11447)
# Which Problems Are Solved naming consistency for TOTP/OTP # How the Problems Are Solved # Additional Changes # Additional Context - Closes #11299 - Closes #11302 - Closes #11304 --------- Co-authored-by: Marco A. <[email protected]> Co-authored-by: Florian Forster <[email protected]> Co-authored-by: Copilot <[email protected]> |
||
|
Marco A.andGitHub
|
96bbd6a65c |
chore: management console naming inconsistency (#11390)
# Which Problems Are Solved The following terms have all been renamed to management console: - Customer Portal (when used to mean the console) - Console - Admin Console # How the Problems Are Solved - Search & Replace smartly - Use Copilot for translation files Changes done to: backend + frontend codebase, docs, translations and protobufs (descriptions only) # Additional Context - Partially Closes #11279 |
||
|
f69a6ed4f3 |
chore: rehaul DevX (#10571)
# Which Problems Are Solved Replaces Turbo by Nx and lays the foundation for the next CI improvements. It enables using Nx Cloud to speed the up the pipelines that affect any node package. It streamlines the dev experience for frontend and backend developers by providing the following commands: | Task | Command | Notes | |------|---------|--------| | **Production** | `nx run PROJECT:prod` | Production server | | **Develop** | `nx run PROJECT:dev` | Hot reloading development server | | **Test** | `nx run PROJECT:test` | Run all tests | | **Lint** | `nx run PROJECT:lint` | Check code style | | **Lint Fix** | `nx run PROJECT:lint-fix` | Auto-fix style issues | The following values can be used for PROJECT: - @zitadel/zitadel (root commands) - @zitadel/api, - @zitadel/login, - @zitadel/console, - @zitadel/docs, - @zitadel/client - @zitadel/proto The project names and folders are streamlined: | Old Folder | New Folder | | --- | --- | | ./e2e | ./tests/functional-ui | | ./load-test | ./benchmark | | ./build/zitadel | ./apps/api | | ./console | ./apps/console (postponed so the PR is reviewable) | Also, all references to the TypeScript repo are removed so we can archive it. # How the Problems Are Solved - Ran `npx nx@latest init` - Replaced all turbo.json by project.json and fixed the target configs - Removed Turbo dependency - All JavaScript related code affected by a PRs changes is quality-checked using the `nx affected` command - We move PR checks that are runnable using Nx into the `check` workflow. For workflows where we don't use Nx, yet, we restore previously built dependency artifacts from Nx. - We only use a single and easy to understand dev container - The CONTRIBUTING.md is streamlined - The setup with a generated client pat is orchestrated with Nx - Everything related to the TypeScript repo is updated or removed. A **Deploy with Vercel** button is added to the docs and the CONTRIBUTING.md. # Additional Changes - NPM package names have a consistent pattern. - Docker bake is removed. The login container is built and released like the core container. - The integration tests build the login container before running, so they don't rely on the login container action anymore. This fixes consistently failing checks on PRs from forks. - The docs build in GitHub actions is removed, as we already build on Vercel. # Additional Context - Internal discussion: https://zitadel.slack.com/archives/C087ADF8LRX/p1756277884928169 - Workflow dispatch test: https://github.com/zitadel/zitadel/actions/runs/17760122959 --------- Co-authored-by: Florian Forster <[email protected]> Co-authored-by: Tim Möhlmann <[email protected]> Co-authored-by: Copilot <[email protected]> |
