<!-- CURSOR_AGENT_PR_BODY_BEGIN -->
# Which Problems Are Solved
Bulk unique-constraint deletes (org, IDP with many links, user, project)
reconstruct every uniqueness key in Go and emit N `DELETE`s. That can
OOM on large tenants.
A one-shot owner backfill is also not enough: during a rolling deploy,
older binaries still insert `owners = '{}'`, so a later owner-only
delete would miss those rows.
Owner-only org/user/grant delete only matches rows whose `owners` bag
contains that tag. Several grant and member types stamped the aggregate
resource owner, which is not always the user’s home org or the granted
org. Username-scope rewrites also dropped `user:` tags, so owner-only
user delete could miss rewritten username rows.
Command must not switch to owner-only delete on the first 79 of a
version, because leftover `'{}'` rows from rolling-update overlap are
only backfilled on a later 79. A failed 79 must not turn owner-delete
on, and backfill joins must not stamp the wrong org (unverified domains,
org-scoped usernames colliding with instance-scoped ones).
# How the Problems Are Solved
Uniqueness stays the existing primary key `(instance_id, unique_type,
unique_field)`. `owners` is only a lifecycle bag of tags (`org:<id>`,
`user:<id>`, `idp:<id>`, `project:<id>`, `grant:<id>`).
- Add `owners TEXT[] NOT NULL DEFAULT '{}'` on
`eventstore.unique_constraints` (init SQL for greenfield, setup 78 for
existing clusters).
- Create `unique_constraints_owners_gin` with `CREATE INDEX
CONCURRENTLY` **outside a transaction** (same pattern as step 76).
Built-in `GIN (owners)` only; no `btree_gin`.
- Stamp `WithOwners` on **add** events. Field-level removes ignore
owners. `RemoveByOwner` stores the tag in `Owners` and both v1/v3 and v2
pushers run `DELETE … WHERE instance_id = $1 AND owners @>
ARRAY[$tag]::text[]`.
- Setup 79 backfills `owners` from projections via numbered
`cmd/setup/79/*.sql` files. Table names are interpolated from
`internal/query/projection` constants. Matches use `unique_field IN (x,
lower(x))` so the primary key can be used. Missing projection tables
(`42P01`) are skipped.
- Org domains are tagged only when `is_verified`. Instance-scoped
usernames join only users that are not org-scoped (organization settings
or domain policy `user_login_must_be_domain`). Every username row still
gets `org:` + `user:` of that user.
- 79 is a **repeatable** migration. `Check` re-runs when
`lastRun["version"]` changed or when setup-only `ForceFinalize` is set
and `lastRun["finalized"]` is still false. `Finalized` is written only
after Execute succeeds.
- Failed setup events keep the last successful `lastRun`. Command reads
`finalized` only from `system.migration.repeatable.done`.
- Org / user / project remove skip N-key event replay **when that
`finalized` bit is true**. Otherwise they keep today’s N-key lists
**and** still emit `RemoveByOwner`. The “ready” result is cached in
memory once true.
Setup writes `finalized` on the 79 repeatable.done payload:
- First 79 of a version: backfill only, `finalized=false` (unless
`ForceFinalize` is already set).
- Later 79: version change or
`BackfillUniqueConstraintOwners.ForceFinalize` /
`ZITADEL_BACKFILLUNIQUECONSTRAINTOWNERS_FORCEFINALIZE` after the rolling
update is done. That run backfills leftovers and writes
`finalized=true`.
Member and grant owner bags now include every org that should delete the
row:
- Instance, org, and project members also stamp
`org:<user_resource_owner>`.
- User grants stamp grant, user, project, and granted orgs.
- Project grant members stamp the project org, user org, and granted
org.
- Project grants stamp both orgs plus `grant:<id>`. Grant remove emits
`RemoveByOwner(grant)`.
- Username-scope rewrites keep `user:<id>` as well as `org:`.
- User/grant orgs are carried as unexported add-event fields (not
persisted JSON). `WithOwners` skips empty and duplicate tags.
# Additional Changes
- Table-driven owner-tag tests, including project grant (two `org:` tags
plus `grant:`), cross-org members, user grants, and username-scope
rewrite `user:` tags.
- Table-driven 79 `Check` + `finalizeAfterSuccess` tests for version
change, sequential version bumps, ForceFinalize restart, and
already-finalized skip.
- Command tests: a failed 79 event with `finalized=true` does not enable
owner-delete.
- Integration fixtures (verified vs unverified org domain;
instance-scoped `bob` vs org-scoped `bob`).
- Instance-only and global unique types stay untagged (`'{}'`). Instance
wipe still uses `UniqueConstraintInstanceRemove`.
- Pusher owner cases live in `eventstore_pusher_owners_test.go` so
`eventstore_pusher_test.go` stays under 1k lines.
# Additional Context
A later cleanup can delete the N-key replay helpers once a release that
ran 79 a second time (or an operator who set `ForceFinalize`) is in
production. That is not required for correctness.
<!-- CURSOR_AGENT_PR_BODY_END -->
<div><a
href="https://cursor.com/agents/bc-44f9bc9b-9a93-4cd9-8dd1-663a7b9aad2a?cursor_ref=pr_footer&cursor_cta=open_in_web"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://cursor.com/assets/images/open-in-web-dark.png"><source
media="(prefers-color-scheme: light)"
srcset="https://cursor.com/assets/images/open-in-web-light.png"><img
alt="Open in Web" width="114" height="28"
src="https://cursor.com/assets/images/open-in-web-dark.png"></picture></a> <a
href="https://cursor.com/background-agent?bcId=bc-44f9bc9b-9a93-4cd9-8dd1-663a7b9aad2a&cursor_ref=pr_footer&cursor_cta=open_in_cursor"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://cursor.com/assets/images/open-in-cursor-dark.png"><source
media="(prefers-color-scheme: light)"
srcset="https://cursor.com/assets/images/open-in-cursor-light.png"><img
alt="Open in Cursor" width="131" height="28"
src="https://cursor.com/assets/images/open-in-cursor-dark.png"></picture></a> </div>
---------
Co-authored-by: Cursor Agent <[email protected]>
Co-authored-by: Silvan <[email protected]>
<!-- CURSOR_AGENT_PR_BODY_BEGIN -->
# Which Problems Are Solved
Handler/v2 projections (`users14` and every other event projection) can
overlook committed events under load:
- `eventstore.commands_to_events` stamped `position` with `NOW()`
(transaction start). A slow create TX could commit after a faster TX
that started later, so the projector stored a later position and never
saw the earlier one.
- Resume used `OFFSET` on a **filtered** row count. That could skip
`in_tx_order = 1` at the next position (for example `user.human.added`).
- Two transactions in the same microsecond both start at `in_tx_order =
1`. A cursor of `in_tx_order > last` alone would drop the other TX’s
first event.
This does not repair rows that are already missing from projections.
# How the Problems Are Solved
- Setup step 77 replaces `eventstore.commands_to_events` so `created_at
= statement_timestamp()` and `position = EXTRACT(EPOCH FROM
clock_timestamp())` at INSERT time. Initialise `08_events_table.sql`
matches that for new databases. The function is `VOLATILE` to match
`clock_timestamp()`.
- The projector still drains in-flight writers with `pg_advisory_lock` /
`pg_advisory_unlock` in one statement, then caps with `position <=
EXTRACT(EPOCH FROM now())` (projector TX start). The exclusive lock is
**not** held across the event `SELECT`. The cap must stay `now()`, not
`clock_timestamp()`, or later inserts become a high-water mark.
- Resume is a lexicographic `EventSortKey` on `(position, in_tx_order,
instance_id, aggregate_type, aggregate_id, sequence)` via
`AfterEventSortKey`. SQL `OFFSET` is no longer used for projection
catch-up.
- Handler and FieldHandler persist the last event’s `in_tx_order` in
`current_states.filter_offset` (existing column, new meaning). Step 77
backfills matching rows from `events2`; unmatched rows keep their
previous value.
# Additional Changes
- `eventstore.Event` exposes `InTxOrder() uint32` (BaseEvent, repository
Event, v3 event, v1 models).
- v3 push SELECT/RETURNING now includes `in_tx_order` so newly written
events carry a real ordinal.
- `WithMinPosition` stays an inclusive `PositionAtLeast` floor and does
not smash the sort-key cursor.
- `SearchQuery.PositionAfter(decimal)` is unchanged (used by
access-token lookup). Do not confuse it with
`SearchQueryBuilder.AfterEventSortKey`.
- `commands_to_events` in setup 77 and initialise 08 is `VOLATILE
PARALLEL SAFE`. Shipped 40/64/70 stay `STABLE` because they still stamp
with `NOW()`.
- Drop unused `last_owner TEXT` DECLARE from setup 77 and initialise
`08_events_table.sql` (`i INTEGER` remains for the enforce-owner loop).
Shipped 40/64/70 keep the leftover.
- This PR’s migration is numbered 77 because `main` already has setup 76
(`76_users14_login_equality_indexes`). First-loop order is s70, then
main’s 76, then this 77.
# Additional Context
**Reviewer map**
| Start here | Why |
| --- | --- |
| `cmd/setup/77.sql` + `cmd/setup/77.go` | Position stamp + atomic
function replace/backfill after shipped `s70` and main’s s76 |
| `internal/eventstore/event_sort_key.go` + `search_query.go` |
Canonical cursor and `AfterEventSortKey` |
| `internal/eventstore/handler/v2/{state.go,handler.go}` |
Persist/resume cursor; `eventQuery` |
| `internal/eventstore/repository/sql/{query.go,postgres.go}` | Tuple
`>` filter, 6-column `ORDER BY`, await cap |
| `internal/eventstore/v3/push.go` / `push.sql` | Scan `in_tx_order` on
write |
Mechanical / low-risk: handler and query test fixtures, Event interface
stubs (`event_base.go`, repository Event, mocks, v1 models).
**Do not look for** a `70.sql` change — shipped migrations stay as on
`main`. Setup 40/64 still use `NOW()`; step 77 overwrites the function
afterward. Fresh `FirstInstance` (runs before `s70`) can still stamp
`NOW()` until 77; runtime pushes after 77 use `clock_timestamp()`.
Main’s `cmd/setup/76.go` is the users14 login-equality indexes, not this
stamp.
**Rollout**
1. Run setup so 77 replaces `commands_to_events` and backfills
`filter_offset`.
2. Restart ZITADEL so projectors load the new resume path.
3. Existing missing `users14` rows are not repaired by this PR.
**Tests**
- `go test ./internal/eventstore/... ./cmd/setup/`
- Sort-key SQL fixtures in `query_sort_key_test.go` (no `OFFSET`;
aggregate-id queries keep 6-column `ORDER BY`)
- `cmd/setup/77_test.go` asserts the embed has no unused `last_owner`
and still uses `clock_timestamp()` / `VOLATILE PARALLEL SAFE`
Related: #8509, #10480 (remove OFFSET), #10560 (advisory locks), #10710
(stable ORDER BY).
Out of scope: repairing already-missing projection rows, holding the
exclusive lock during SELECT, rewriting historical setup 40/64 SQL.
<!-- CURSOR_AGENT_PR_BODY_END -->
<div><a
href="https://cursor.com/agents/bc-3da6eb51-11d4-4a8a-8ba0-e190f116df05?cursor_ref=pr_footer&cursor_cta=open_in_web"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://cursor.com/assets/images/open-in-web-dark.png"><source
media="(prefers-color-scheme: light)"
srcset="https://cursor.com/assets/images/open-in-web-light.png"><img
alt="Open in Web" width="114" height="28"
src="https://cursor.com/assets/images/open-in-web-dark.png"></picture></a> <a
href="https://cursor.com/background-agent?bcId=bc-3da6eb51-11d4-4a8a-8ba0-e190f116df05&cursor_ref=pr_footer&cursor_cta=open_in_cursor"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://cursor.com/assets/images/open-in-cursor-dark.png"><source
media="(prefers-color-scheme: light)"
srcset="https://cursor.com/assets/images/open-in-cursor-light.png"><img
alt="Open in Cursor" width="131" height="28"
src="https://cursor.com/assets/images/open-in-cursor-dark.png"></picture></a> </div>
---------
Co-authored-by: Cursor Agent <[email protected]>
Co-authored-by: Silvan <[email protected]>
<!-- CURSOR_AGENT_PR_BODY_BEGIN -->
# Which Problems Are Solved
Login V2 `ListUsers` lookups of the form username OR email (and email OR
phone) hash-join the full `users14` / `users14_humans` tables, then
filter. That cost grows with instance size (~40ms at 10k users, ~650ms
at 200k).
Top-level login-name equals already used an indexed join, but the same
login-name filter inside an `OrQuery` fell back to the slow view-based
exists query.
# How the Problems Are Solved
**TL;DR:** equality lookups on username, email, phone, or login name
become `INNER JOIN (UNION of indexed ID seeks) AS matches`. Mixed /
contains / name searches keep the old WHERE shape.
### Reviewer map
| Read first | Why |
| --- | --- |
| `internal/query/user_login_equality.go` | Typed filters + extractor +
UNION join |
| `internal/query/user.go` (`prepareUsersQuery`, constructors) | Wiring;
username/email/phone equals return `lowerEqualsFilter` |
| `cmd/setup/76.go` + `cmd/setup/76/*.sql` | Concurrent indexes for
existing deployments; drop invalid leftovers |
| `internal/query/projection/user.go` | Same indexes in `Init()` for new
databases |
| `internal/query/user_login_name_exists_test.go` | SQL-shape coverage |
| `internal/api/grpc/user/v2/integration_test/query_test.go` | Nested
username AND/OR user results |
### Query rewrite
- Extract one equals leaf, a flat `OrQuery` of those leaves, or the same
nested under `AndQuery`. Remaining AND conjuncts stay as WHERE. Equality
nested under `OrQuery` / `NotQuery` is left as WHERE.
- Login-name equals is a UNION arm that reuses `login_names3_users` SQL,
so it works inside OR as well as at the top level.
- EQUALS seeks use `LOWER(col)` for the index, then recheck `col =
value` so case-sensitive searches do not match extra rows.
EQUALS_IGNORE_CASE is unchanged.
- Empty OR arms are not skipped (that would change OR meaning); mixed
empty-OR falls back to WHERE. Login UI already omits empty arms.
### Indexes
- Setup 76: `CREATE INDEX CONCURRENTLY IF NOT EXISTS` on
`users14_username_lower_idx` `(instance_id, LOWER(username))` and
`users14_humans_phone_lower_idx` `(instance_id, LOWER(phone))`. Not
wrapped in a transaction.
- Before create, drop any leftover `pg_index.indisvalid = false` indexes
of those names (failed/cancelled `CREATE INDEX CONCURRENTLY`). `IF NOT
EXISTS` would otherwise skip rebuild.
- `Init()` creates the same indexes without `CONCURRENTLY` (empty
tables). Existing `users14_username` and email lower indexes are
unchanged.
# Additional Changes
- Unit tests assert UNION SQL for username OR email, email OR phone,
login name OR email, username OR username, nested `AndQuery(loginName,
org)`, and nested `AndQuery(Or(...), org)`; that contains / display-name
/ mixed OR / nested `OrQuery(AndQuery(...), ...)` do not UNION; that
email EQUALS rechecks the raw column; and that empty-OR is not
extracted.
- Setup 76 sqlmock coverage: missing `users14` skips work; valid/absent
indexes only create; invalid leftover username index is dropped then
both creates run.
- `TestServer_ListUsers_UsernameAndOr` checks live results: org AND
(alice OR bob) returns both users, sibling username AND is empty, nested
`AndQuery(Or(...), org)` returns both.
# Additional Context
- Base branch: `main`
- Extends the login-name equals rewrite from #12460 to
username/email/phone OR trees used by Login V2 search.
<!-- CURSOR_AGENT_PR_BODY_END -->
<div><a
href="https://cursor.com/agents/bc-1f043b1c-6168-45ca-9e9e-c18451994207?cursor_ref=pr_footer&cursor_cta=open_in_web"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://cursor.com/assets/images/open-in-web-dark.png"><source
media="(prefers-color-scheme: light)"
srcset="https://cursor.com/assets/images/open-in-web-light.png"><img
alt="Open in Web" width="114" height="28"
src="https://cursor.com/assets/images/open-in-web-dark.png"></picture></a> <a
href="https://cursor.com/background-agent?bcId=bc-1f043b1c-6168-45ca-9e9e-c18451994207&cursor_ref=pr_footer&cursor_cta=open_in_cursor"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://cursor.com/assets/images/open-in-cursor-dark.png"><source
media="(prefers-color-scheme: light)"
srcset="https://cursor.com/assets/images/open-in-cursor-light.png"><img
alt="Open in Cursor" width="131" height="28"
src="https://cursor.com/assets/images/open-in-cursor-dark.png"></picture></a> </div>
---------
Co-authored-by: Cursor Agent <[email protected]>
Co-authored-by: Silvan <[email protected]>
# Which Problems Are Solved
- Native iOS/Android passkeys need OS trust files on the ZITADEL domain;
without them, Associated Domains / App Links verification fails.
- Operators could not configure iOS Team ID + Bundle ID or Android
package name + SHA-256 fingerprints on OIDC apps.
- `/.well-known/apple-app-site-association` and
`/.well-known/assetlinks.json` were not served from application config.
# How the Problems Are Solved
- Add iOS/Android app-link fields on OIDC app create/update (Application
API v2 + Management), with validation.
- Persist and project those fields; query active app-link configs
instance-wide.
- Serve AASA (`webcredentials`) and Digital Asset Links
(`get_login_creds`) from well-known paths, with configurable
`Cache-Control` and fingerprint normalization at serve time.
- Console UI to edit the fields, with links to the well-known endpoints.
- Operator docs for configuration, endpoints, caching, and verification.
# Additional Changes
- Document on API fields that well-known responses may be HTTP-cached
and platform verifiers may delay propagation.
- Runtime config: `WellKnown.AppLinksCacheControlMaxAge` (default `5m`;
`0` → `no-store`).
# Additional Context
- Closes#12497
- Implemented and reviewed as stack:
- #12531 API contract
- #12532 storage wiring
- #12536 well-known endpoints
- #12537 console
- #12547 docs
---------
Co-authored-by: Cursor <[email protected]>
# Which Problems Are Solved
#12313 lets clients register themselves as OIDC applications (RFC 7591),
but a dynamically registered client has no way to manage itself
afterwards:
- The registration response returns no `registration_access_token` and
no `registration_client_uri`, which RFC 7591 and 7592 clients (including
the MCP SDKs) expect in order to manage their own registration.
- There is no way to read back, update or delete a registration without
an operator acting on the client's behalf through the Console or the
Management API. A client that needs to rotate a redirect URI has to ask
a human.
# How the Problems Are Solved
Implement the [OAuth 2.0 Dynamic Client Registration Management Protocol
(RFC 7592)](https://datatracker.ietf.org/doc/html/rfc7592): `GET`, `PUT`
and `DELETE` on `/oauth/v2/register/{client_id}`, wired through the same
`op.WithSetRouter` hook as the registration endpoint.
## Control plane: the same security settings, no new switch
The management endpoints are gated on the
`DynamicClientRegistrationSettings.enabled` introduced in #12313. No new
setting, no feature flag: a client that may register may manage what it
registered. When registration is disabled the endpoints answer `404`,
exactly as `POST /oauth/v2/register` does.
`allow_unauthenticated` selects the registration mode and does not apply
here, because the registration access token authorizes these endpoints
in both modes.
## Authorization: the registration access token, and nothing else
RFC 7592 §3 defines the credential as the registration access token
issued at registration and bound to a single client, not a user
credential. So a client manages only itself, proving it with the token
it was handed:
| | Registration (`POST`) | Management (`GET` / `PUT` / `DELETE`) |
| --- | --- | --- |
| Credential | Access token, or none in open mode | Registration access
token |
| Permission | `project.app.register_dynamic` in token mode | None |
Neither `project.app.register_dynamic`, which governs who may *create*
clients, nor `project.app.write` applies to the management endpoints.
One authorization path per endpoint is easier to reason about than two,
and there is no user identity to check against in the first place.
Operators keep managing dynamically registered clients through the
Management API and the Console, like any other application; they cannot
obtain a client's registration access token, by design.
Status codes: a missing, malformed or non matching token is `401`; a
client that no longer exists is `404`. The token's `client_id` is
checked against the path before any lookup, so a token bound to another
client is rejected without touching the database.
## The token itself
- **Stored like a client secret.** Only its passwap hash is persisted,
through a new
`project.application.config.oidc.registration_token.changed` event; the
plain token leaves the server exactly once, in the response that issues
it.
- **Bound to its client and organization** through authenticated
encryption with the instance key, the same mechanism as refresh tokens.
The opaque token handed to the client encrypts
`client_id:org_id:secret`, so the endpoint recovers the binding before
checking the secret against the stored hash.
- **Verified in O(1)** against the hash on the projection, folded into
the client lookup the handler already does, with a strongly consistent
fallback to the eventstore so a token that was just rotated is accepted
before it is projected.
- **Rotated on update** (RFC 7592 §4.4) and invalidated implicitly when
the client is deleted. A read does not rotate it. Every update rotates,
including one that changes no metadata, so a client always gets a usable
token back.
## Reusing the existing persistence
A small refactor extracts the shared OIDC application creation and
update logic (`pushOIDCApplication` gains a variadic `extraEvents`, and
`oidcApplicationChangeEvent` is lifted out of `UpdateOIDCApplication`),
so the dynamic commands reuse it without the permission check. The
registration token event is pushed atomically with the application, so a
registered client is never left unmanageable.
# Additional Changes
- Setup step **74** adds the `registration_token` column to the
`apps7_oidc_configs` projection, with a reducer for the new event
modelled on the client secret. Applications are not part of the
relational storage projection, so there is no second write path to keep
in sync.
- The registration response (`POST`) now also carries
`registration_access_token` and `registration_client_uri`.
- Docs: a "Manage a registration" section in the dynamic client
registration guide covering the three operations, the authorization
model, token rotation and the new limitations, plus the management
endpoints on the OIDC endpoints page.
- Unit tests for the command layer (issue, verify, rotate, update,
delete), the projection reducer, and the token binding and response
mapping. An integration test covers the full lifecycle (register, read,
update with rotation, delete, read again with `404`), a token bound to
another client, and the endpoints not being served when registration is
disabled. None of the integration calls carries a user access token,
which pins down that the management endpoints require no permission of
their own.
# Additional Context
- Part of the MCP authorization effort in #9810. Follow-up to #12313
(RFC 7591), which is now merged; this branch has been rebased on `main`
and carries the settings and authorization model agreed there.
- **Open question for the review**: the authorization model above
(registration access token only, no operator path through these
endpoints) is the one I proposed when #12313 was reviewed, but it was
never explicitly confirmed. If you would rather have `project.app.write`
also grant access to a client in the operator's own organization, that
is a contained change in `authorizeClientManagement`.
- Rotation is eventually consistent: right after an update the previous
token may keep working for a brief moment until the rotation is
projected, while the new token works immediately through the eventstore
fallback. This matches how ZITADEL validates access tokens, where
revocation is likewise eventually consistent. It is documented in the
guide and asserted as such in the integration test.
- A database migration is introduced for the new projection column. No
proto change and no new feature flag.
- Verified locally: full build, `go vet` including `-tags integration`,
`golangci-lint` with 0 issues, and the unit suites for every touched
package. As before, I cannot run the integration suite here, so CI
remains the check on it.
# Which Problems Are Solved
In Zitadel's append-only event-sourced architecture, the
`eventstore.events2` table grows indefinitely. PostgreSQL's default
autovacuum uses a percentage-based scale factor, so as the table grows,
the number of changed rows required to trigger a `VACUUM` or `ANALYZE`
drifts towards infinity. Without regular vacuums, the table's Visibility
Map becomes stale, disabling fast Index-Only Scans and forcing expensive
heap reads. Without regular analyzes, query planner statistics become
stale, leading to suboptimal execution plans.
This causes eventstore operations to progressively degrade as `events2`
grows, even without CPU, memory, or I/O saturation. A manual `VACUUM
ANALYZE` immediately restores performance, confirming the root cause.
- If `Eventstore.Autovacuum` is left at its default, `events2` keeps
using PostgreSQL's default, percentage-based autovacuum/autoanalyze
scale factors, which become impractically infrequent on large tables.
- There was previously no supported way to apply static, table-level
autovacuum tuning to `events2` through Zitadel's own configuration/setup
process.
# How the Problems Are Solved
- Added an `Eventstore.Autovacuum` runtime configuration block to
`cmd/defaults.yaml` (disabled by default):
```yaml
Eventstore:
Autovacuum:
Enabled: false # ZITADEL_EVENTSTORE_AUTOVACUUM_ENABLED
VacuumThreshold: 50000 # ZITADEL_EVENTSTORE_AUTOVACUUM_VACUUMTHRESHOLD
AnalyzeThreshold: 50000 # ZITADEL_EVENTSTORE_AUTOVACUUM_ANALYZETHRESHOLD
```
- Added a repeatable `zitadel setup` migration step
(`cmd/setup/eventstore_autovacuum.go`) that:
- When `Enabled: true`, disables the percentage-based
`autovacuum_vacuum_scale_factor`, `autovacuum_analyze_scale_factor`, and
`autovacuum_vacuum_insert_scale_factor` on `eventstore.events2`, and
applies static thresholds (`autovacuum_vacuum_insert_threshold`,
`autovacuum_vacuum_threshold`, `autovacuum_analyze_threshold`) from the
config instead.
- When `Enabled: false`, resets those storage parameters on
`eventstore.events2` back to the cluster defaults.
- Implements `Repeatable.Check()` so the step only re-runs when the
configuration actually changed since the last `zitadel setup` run.
- Added documentation in the new "Performance tuning" page.
# Additional Changes
- Move the projection documentation into performance tuning page
- Expand and update the projection documentation to the latest state in
zitadel. (contained some stale information)
# Additional Context
Several other issues describe read-performance symptoms consistent with
this same root cause (stale `events2` visibility map / planner
statistics at scale, without resource saturation). Since this PR
addresses the shared root cause:
- Closes#12448
- Closes#10754
- Closes#10260
- Closes#8585
- Closes#9239
---
_Generated by [Claude
Code](https://claude.ai/code/session_01HoKvEY7niCVgLCz7CajBwW)_
---------
Co-authored-by: Claude <[email protected]>
Co-authored-by: Copilot Autofix powered by AI <[email protected]>
Co-authored-by: Silvan <[email protected]>
* Use slices.Contains over custom function
* Correctly remove roles from granted roles
* fix(setup): repair user grants with stale roles (GHSA-v859-c572-qh5p)
Add setup step 73 that reconciles existing user grants whose roles were
left too broad by the buggy cascade removal in removeRoleFromUserGrant.
The corruption lives in the eventstore event payloads, so the step pushes
a corrective user.grant.cascade.changed event per affected grant (roles
intersected with the currently valid set) and re-triggers the user grant
projection. Runs in the second setup slice, after the projection tables
it reads have been created.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
* fix(setup): scope GHSA-v859-c572-qh5p repair to grant-based user grants
Direct user grants can never be hit by this bug (only ChangeProjectGrant's
multi-role cascade to grant-based grants can trigger it), so drop the
direct-grant branch from the finder query to avoid stripping unrelated,
legitimate roles that merely mismatch for other reasons (e.g. stale
role_key drift). Also exclude removed instances from the migration scope,
and log the number of grants fixed per instance.
Co-authored-by: Cursor <[email protected]>
---------
Co-authored-by: Claude Opus 4.8 <[email protected]>
Co-authored-by: Livio Spring <[email protected]>
Co-authored-by: Cursor <[email protected]>
# Which Problems Are Solved
Zitadel exposes the secrets generator configuration through its admin
api. This allows instance admins to manage them on their own and they
can create overwrite the system / runtime defaults (incl. expiration).
This very much needed in multi-instance scenarios such as zitadel.cloud.
Currently the invite code configuration was not manageable through the
API, but only runtime config.
# How the Problems Are Solved
- added the `invite_code` type to the API allowing it to be set and
retrieved.
- added the type to console's management list
- added the type to be stored on instance setup
- change the `GetSecretGenerator` endpoint to fall back to the runtime
config if no config is stored on the instance itself
- ensure the `length` and at least one charset is enabled, return an
error otherwise
- expiry is not enforced, so 0 allows codes with no expiry (current
state)
# Additional Changes
None
# Additional Context
- closes https://github.com/zitadel/zitadel/issues/10474
# Which Problems Are Solved
Login v2 uses the `users.v2.ListUsers`-endpoint to get a user by login
name. This query had an inefficient `WHERE`-clause.
# How the Problems Are Solved
Update the view and use a specific clause for this query.
# Additional information
Added in https://github.com/zitadel/zitadel/pull/10475
---------
Co-authored-by: Marco A. <[email protected]>
Co-authored-by: Livio Spring <[email protected]>
# Which Problems Are Solved
During the setup step we saw rare cases which caused setup to fail after
executing steps 40, 64 and 70.
# How the Problems Are Solved
Close currently open database connections so that they fetch the correct
type mapping for the `eventstore.command2` database type.
# Additional Changes
Ensure correct order of setup steps 64 and 70.
# Additional Context
None
---------
Co-authored-by: Livio Spring <[email protected]>
# Which Problems Are Solved
- The eventstore did not support intentionally overwriting the resource
owner when creating events for aggregates that may be reused across
owners.
- Resource owner handling was implicit and could not be controlled per
command/event type.
- We needed a safe way to distinguish between:
- keeping the existing aggregate owner, and
- explicitly setting a new owner for specific create-like events.
# How the Problems Are Solved
- Introduced a new eventstore command type with an explicit
enforce_owner flag.
- Updated eventstore.commands_to_events and eventstore.push so owner
assignment is now explicit:
- if enforce_owner is true, the command owner is written
- if enforce_owner is false, the existing aggregate owner is retained
when present
- Added EnforceResourceOwnerCommand and wiring so command types can opt
in to enforced owner behavior.
- Wired the new behavior through the v3 eventstore push path, including
compatibility fallback for older command type mapping.
- Added migration/setup changes to register and use the new command type
and SQL functions.
- Added and updated tests for owner overwrite and aggregate ID reuse
scenarios.
# Additional Changes
- Added small migration/setup robustness improvements related to
eventstore setup ordering and helper reuse.
- Added focused test coverage for enforced owner behavior and
sequencing.
- Events that currently allow owner changes (implement
EnforceResourceOwner) are:
- AddedEvent (action)
- GroupAddedEvent
- StartedEvent (idp intent)
- ProjectAddedEvent
- HumanAddedEvent
- HumanRegisteredEvent
- MachineAddedEvent
- CreatedEvent (schema user)
# Additional Context
- Follow-up for eventstore owner-handling correctness in create flows
and aggregate ID reuse cases.
- No additional issue link was attached for this change.
---------
Co-authored-by: abhishek kumar gupta <[email protected]>
# Which Problems Are Solved
Opaque tokens now use authenticated encryption.
# How the Problems Are Solved
- Upgrade zitadel/oidc to v3.47
- Copy crypto implementation for refresh and session tokens (internal to
zitadel)
- Added config that allows validating old tokens for gradual roll-out
# Additional Changes
- Set NX cache for `integration-test-build` to `false`, working on a
seperate fix.
# Additional Context
- closes#11315
---------
Co-authored-by: copilot-swe-agent[bot] <[email protected]>
Co-authored-by: muhlemmer <[email protected]>
## Problem description
- Setup currently creates/updates relational tables, but there is no
built-in way to fully reset the relational schema during iterative
development.
- Re-running setup after schema/projection changes can leave stale
relational objects and projection state behind, which makes local/dev
validation harder.
- There is no explicit, configurable switch in setup steps for
destructive schema recreation behavior.
## How the Problems Are Solved
- Adds a new setup step configuration section for relational tables with
a `ShouldRecreateSchema` flag (default `false`).
- Wires the new config flag into the transactional tables setup step.
- Extends the transactional tables execution logic to optionally:
- Drop the `zitadel` schema with `CASCADE`
- Clean related projection state entries for relational tables in
`projections.current_states`
- Recreate tables through the existing setup flow afterward
- Wraps the destructive operations in a transaction and keeps
error/rollback handling plus logging for visibility.
## Additional Changes
- Adds an explicit warning in step configuration that schema recreation
is intended for development and not production use.
- Keeps behavior fully backward-compatible by default
(`ShouldRecreateSchema: false`), so existing setups are unchanged unless
the flag is enabled.
## Additional Context
- Follow-up for relational setup/dev workflow improvements.
- PR: #11833
## How to use it
### Env `export ZITADEL_RELATIONALTABLES_SHOULDRECREATESCHEMA=true`
### Config
Add the following to your custom setup steps config:
```yaml
RelationalTables:
ShouldRecreateSchema: true
```
# Which Problems Are Solved
There were still some emails (passkey registration and domain claimed)
sent with links pointing to login v1 even when the login v2 was enabled
for the instance.
Also while looking into the issue, it was discovered that some links
pointing to login V2 were not correctly generated.
# How the Problems Are Solved
- Added default paths for passkey registration and domain claimed
notifications
- Fixed the existing paths to properly handle concatenation (resp. use
`url.ResolveReference`)
- Change their go types (from string) to `*url.URL`
- Added a mapstructure hook for string to url
- Removed unnecessary `InstanceSetupFeatures` and corresponding
conversions
- Refactored the methods on the `login.DefaultPaths` struct and added an
interface to the `Commands` to only need to pass a single config (and
not every method)
- Added an `OriginURL` method to the `DomainCtx` to prevent going from
url to string and back
- Added the use of the templates in case of enabled login v2 for passkey
registration and domain claimed)
# Additional Changes
None
# Additional Context
closes#10643
---------
Co-authored-by: Max Peintner <[email protected]>
Co-authored-by: Livio Spring <[email protected]>
Co-authored-by: Livio Spring <[email protected]>
Co-authored-by: Max Peintner <[email protected]>
Co-authored-by: Gayathri Vijayan <[email protected]>
# Which Problems Are Solved
Streams allow differentiating logs produced by different components of
Zitadel.
# How the Problems Are Solved
The `backend/v3/instrumentation/logging` package now exposes convenience
function for setting and getting a logger from the context. As well as
high-level functions to emit log records at various levels. When
constructing a new logger a "stream" needs to be specified:
- **runtime**: General runtime logs, such as startup and shutdown
messages. Default for logs that do not belong to the other categories.
- **request**: Logs for incoming API and HTTP requests.
- **event_handler**: Logs for event handling in projections.
- **queue**: Logs for the job queue processing.
- **event_pusher**: Logs for event pushing to the database. Disabled by
default, contains sensitive information.
Each line from the returned logger contains a `stream` field as well as
a `version` field with the current Zitadel version.
## Runtime config
Streams can be enabled by passing an array of stream names in the
runtime config. Because some log streams may contain sensitive data
(especially events), it is now also possible to mask values by their
key.
# Additional Changes
- Wrap `slogctx` in the `logging` package. (Except API error converter
packages, because of import cycle)
- Add some docs to `logging` package so other devs understand how to add
logging to Zitadel
- Add `logging.OnError` and `logging.WithError` helper functions with
`Panic()` and `Fatal()` methods, to preserve current calls in the `cmd`
packages.
- Add instance context extractor.
- Only output request details in the request info log. Request ID
remains propagated through context.
- Moved middleware functionality into protocol specific packages.
- Removed setting of URI to context in metric middleware. There were ony
setters and no getters. (Unused value)
- Reuse a single statusWriter in the middleware package for middlewares
that need to know the response status.
# Additional Context
- Closes#11333
- Closes#11331
- Partly #11330
---------
Co-authored-by: Silvan <[email protected]>
# Which Problems Are Solved
When starting Zitadel with Postgres version 18, setup fails with the
following error:
`level=error msg="migration failed" caller=".../cmd/setup/setup.go:373"
code=0A000 detail= error="ERROR: partitioned tables cannot be unlogged
(SQLSTATE 0A000)" hint= message="partitioned tables cannot be unlogged"
name=34_add_cache_schema severity=ERROR`
# How the Problems Are Solved
- Modify setup step 34 to ensure compatibility with PostgreSQL 18 by
changing the creation of the partitioned tables to`LOGGED` tables but
keep the partitions `UNLOGGED`.
- Added an additional setup step which alters the table persistence of
the partitioned tables to `LOGGED`.
# Additional Changes
- Bumped Postgres compatibility to version 18 in docs.
- Ensure default partitions for cache tables
## Additional Context
- closes https://github.com/zitadel/zitadel/issues/10712
- backport to v4
- migration from PostgreSQL version 17 to 18 was verified using
`pg_dumpall` and restoring the created backup file
- and new setups using PostgreSQL version 18 directly
---------
Co-authored-by: Copilot <[email protected]>
# Which Problems Are Solved
The current back-channel logout notification still used a single
projection handler for handling terminating sessions and notifying the
necessary clients. This can lead to back pressure and delayed
notifications.
# How the Problems Are Solved
The handler now only creates a job in the (river) worker queue to handle
the notification.
The worker will then search for all necessary clients to be informed and
create a job for each.
These jobs will then be picked up by the (same) worker(s) again, which
will create, sign and send the logout token to the client. If one
notification fails, this one job will be retired, but others can still
be processed in parallel and ore not affected.
Each successful job will still create an event on the session stating
the successful notification of the client.
The existing `OIDC.DefaultBackChannelLogoutLifetime` configuration has
been deprecated in favor of making it part of the new
`OIDC.BackChannelLogout` config.
# Additional Changes
None
# Additional Context
- feature check is still executed and will be removed seperately for
easier review: https://github.com/zitadel/zitadel/issues/11277
- closes https://github.com/zitadel/zitadel/issues/9279
- requires backport to v4.x
# Which Problems Are Solved
Zitadel did not provide easy correlation between errors, logs, traces
and metrics. The configuration for those instrumentations was also not
consistent, with some supporting different exporters then others.
Implementation and parsing of config was also spaghettified over
multiple packages, with awkward parsing and inconsistent naming of
options.
# How the Problems Are Solved
All telemetry is now merged under the name "instrumentation". Why?
1. We thought it was a good idea in the past to call the milestone
exporter `Telemtry` in the runtime config. Calling this `TelemetryV2`
looks weird.
2. Not everything is a meter and not everything is sent (tele...).
3. It's also
[defined](https://opentelemetry.io/docs/concepts/instrumentation/) as
such by the OTEL documentation.
## New features
- Adds structured, context based logging with trace-ID awareness
- Static log fields are added to the context, such as service and
request path
- Static log fields are injected in each logline emitted by the
application
- Structured logs can also be send to an otel exporter
- Structured logs can be printed to StdErr in text and JSON format
- Error sinks make sure every error is logged at the correct level:
- Warnings for client side errors (HTTP 400 range, Invalid request etc)
- Error for server side errors (Internal server errors)
- Metrics can now also be send to a OTEL collector. (previously they
could only be scraped from `/debug/metrics` with prometheus)
## Exporters
This change adds all the exporters supported by OTEL upstream and some
google specific exporters for our cloud deployment.
- StdOut / StdErr: all instrumentations
- OTEL gRPC / HTTP: all instrumentations
- Google: all instrumentations except logging
- Prometheus (pull-based): only metrics
The exception is profiling, which only supports the google exporting due
to lack of support by OTEL upstream.
## Configuration and structure
- All instrumentation is moved into the new `backend/v3/instrumentation`
package. It reuses configuration types, so both code and runtime
configuration are easier to understand.
- The `internal/telemetry` packages are removed.
- Instrumentation is started with a single function and a proper
shutdown function is now provided.
- Legacy configuration is still parsed from the runtime config, as long
as the new configuration is disabled. This allows backporting this
feature to v4 without breaking existing configurations.
# Additional Changes
- Devcontainer: set `$PATH` variable so installed go binaries can be run
individually, without NX.
- NX: install GCI tool to fix imports
# Additional Context
- Closes https://github.com/zitadel/zitadel/issues/8408
- Closes https://github.com/zitadel/zitadel/issues/6664
- Backport to v4
# Which Problems Are Solved
The payload in actions V2 is currently sent as JSON to the target
endpoint. It might get exposed to intermediary infrastructure or logging
systems.
For these scenarios there needs to be an application-layer encryption,
where the provider of the endpoint can define an key to be used for the
encryption.
# How the Problems Are Solved
- Added an additional option to the target to specify the payload type:
`JSON` (current and default), `JWT`, `JWE` (api and console)
- added endpoints to upload and manage public keys (to be used for
encryption) for a target
- updated all action v2 executions (interceptors, oidc, saml, ...) to
provide the `GetActiveSigningWebKey` from queries
- implemented jwt and jwe in the exections incl. refactoring of code and
tests
- changes to the authn_keys table:
- added a `fingerprint` column
- dropped not null constraint on expiration
- moved the `GetSignerOnce` into its own package to prevent circular
dependencies
# Additional Changes
None
# Additional Context
closes#11061
---------
Co-authored-by: Marco A. <[email protected]>
Co-authored-by: conblem <[email protected]>
Co-authored-by: Silvan <[email protected]>
The change was introduced in
https://github.com/zitadel/zitadel/pull/11178.
The fix is to prevent wiping the memberships because the projection did
not init yet.
### Changes
- Introduces a check to determine if the member role synchronization
should be executed based on the existence of a specific database table
(`projections.instance_members4`).
- Ensures that the synchronization process only runs if the required
table is present in the database.
# Which Problems Are Solved
Zitadel v4.7.2 fixed a security issue by switching to the permission v2
framework for user APIs. It appears that systems that are running since
before v2.68 that were affected by a precision bug in the eventstore,
which was fixed in that version. The precision bug results in certain
events being "skipped" while being projected into the fields table, used
by the new permission system. This caused certain membership roles to be
missing, resulting in empty user lists when executed by the affected
member. The permission system basically finds no matching memberships
and therefore returns no users at all.
# How the Problems Are Solved
After research we concluded that the legacy membership projections are
projected correctly. This PR synchronizes the projected state into the
fields table. As the membership roles are not marked unique, all rows
are first deleted and then the correct membership roles are then
inserted. The operation happens in a single transaction, during which
the fields table will remain locked for modifications. This to prevent
possible concurrent modifications to membership states.
# Additional Changes
- none
# Additional Context
- Introduced in
https://github.com/zitadel/zitadel/commit/0e17d0005a98ccbf92139961ef702ef03208ffd3
- Released in
[v4.7.2](https://github.com/zitadel/zitadel/releases/tag/v4.7.2)
- Related: https://github.com/zitadel/zitadel/issues/8863
---------
Co-authored-by: Silvan <[email protected]>
<!--
Please inform yourself about the contribution guidelines on submitting a
PR here:
https://github.com/zitadel/zitadel/blob/main/CONTRIBUTING.md#submit-a-pull-request-pr.
Take note of how PR/commit titles should be written and replace the
template texts in the sections below. Don't remove any of the sections.
It is important that the commit history clearly shows what is changed
and why.
Important: By submitting a contribution you agree to the terms from our
Licensing Policy as described here:
https://github.com/zitadel/zitadel/blob/main/LICENSING.md#community-contributions.
-->
One-time recovery codes are a common multi-factor authentication (MFA)
backup method, letting users access their account if they lose other MFA
devices. Support for recovery codes can also reduce support burden for
users locked out of their accounts and provide a more secure and
reliable form of verification than security questions.
# Which Problems Are Solved
Zitadel currently lacks support for recovery codes.
# How the Problems Are Solved
This PR partially addresses recovery code support in Zitadel.
Importantly, it adds recovery codes as a new 2FA `Factor` and an
additional `Check` type for the Session API.
```
Example recovery code flow:
1. User generates N new recovery codes using `POST /v2/users/{user_id}/recovery_codes`
2. Zitadel hashes and stores these codes and returns the un-hashed codes in the response
3. User creates new session with an additional check: `recoveryCode`
4. Code is checked against hash and, if valid, cannot be used again
5. User attempts to adds N more codes using the same endpoint
6. If `remaining_codes + N <= RecoveryCodes->MaxCount` config value, then recovery codes are added in addition to original codes
7. User can remove all recovery codes using `DEL /v2/users/:userId/recovery_codes`
```
This PR adds:
- [x] Session recovery_code check support on `POST+PATCH /v2/sessions`
endpoints
- [x] Adds `mfa_recovery_code_checked_at` column (default null) to
`projections.sessions8` table
- [x] Support for `SECOND_FACTOR_TYPE_RECOVERY_CODES` as available 2FA
method on login policy
- [x] Support for importing recovery codes in /import code
Missing, will _not_ implement in this PR:
- [ ] Admin console support for displaying Recovery Code settings for
user(s)
- [ ] Zitadel Typescript login support for recovery codes
TODO:
- [x] Additional unit and integration tests
- [x] Error translations
# Additional Changes
None
# Additional Context
- Closes#6898
---------
Co-authored-by: Livio Spring <[email protected]>
# Which Problems Are Solved
Depending on the metadata values (already existing), the newly created
index (#10415) cannot be created or error in the future.
# How the Problems Are Solved
- Create the index using `sha256` and change the query to use sha256 as
well when comparing bytes values such as user_metadata.
- Added a setup step to cleanup potentially created index on
`projections.user_metadata5`
# Additional Changes
None
# Additional Context
- relates to #10415
- requires backport to v4.x
A timing issue (a race condition) was identified in our event processing
system. Under specific circumstances, it was possible for the system to
skip processing certain events, leading to potential data
inconsistencies.
## Which problems are solved
The system tracks its progress through the event log using timestamps.
The issue occurred because we were using the timestamp from the start of
a database transaction. If a query to read new events began after the
transaction started but before the new event was committed, the query
would not see the new event and would fail to process it.
## How the problems are solved
The fix is to change which timestamp is used for tracking. We now use
the precise timestamp of when the event is actually written to the
database. This ensures that the event's timestamp is always correctly
ordered, closing the timing gap and preventing the race condition.
This change enhances the reliability and integrity of our event
processing pipeline. It guarantees that all events are processed in the
correct order and eliminates the risk of skipped events, ensuring data
is always consistent across the system.
## Additional information
original fix: https://github.com/zitadel/zitadel/pull/10560
# Which Problems Are Solved
Using the service ping, we want to have some additional insights to how
zitadel is configured. The current resource count report contains
already some amount of configured policies, such as the login_policy.
But we do not know if for example MFA is enforced.
# How the Problems Are Solved
- Added the following counts to the report:
- service users per organization
- MFA enforcements (though login policy)
- Notification policies with password change option enabled
- SCIM provisioned users (using user metadata)
- Since all of the above are conditional based on at least a column
inside a projection, a new `migration.CountTriggerConditional` has been
added, where a condition (column values) and an option to track updates
on that column should be considered for the count.
- For this to be possible, the following changes had to be made to the
existing sql resources:
- the `resource_name` has been added to unique constraint on the
`projection.resource_counts` table
- triggers have been added / changed to individually track `INSERT`,
`UPDATE`(s) and `DELETE` and be able to handle conditions
- an optional argument has been added to the
`projections.count_resource()` function to allow providing the
information to `UP` or `DOWN` count the resource on an update.
# Additional Changes
None
# Additional Context
- partially solves #10244 (reporting audit log retention limit will be
handled in #10245 directly)
- backport to v4.x
# Which Problems Are Solved
HTTP Request to HTTP providers for Email or SMS are not signed.
# How the Problems Are Solved
Add a Signing Key to the HTTP Provider resources, which is then used to
generate a header to sign the payload.
# Additional Changes
Additional tests for query side of the SMTP provider.
# Additional Context
Closes#10067
---------
Co-authored-by: Marco A. <[email protected]>
This PR initiates the rework of Zitadel's backend to state-persisted
objects. This change is a step towards a more scalable and maintainable
architecture.
## Changes
* **New `/backend/v3` package**: A new package structure has been
introduced to house the reworked backend logic. This includes:
* `domain`: Contains the core business logic, commands, and repository
interfaces.
* `storage`: Implements the repository interfaces for database
interactions with new transactional tables.
* `telemetry`: Provides logging and tracing capabilities.
* **Transactional Tables**: New database tables have been defined for
`instances`, `instance_domains`, `organizations`, and `org_domains`.
* **Projections**: New projections have been created to populate the new
relational tables from the existing event store, ensuring data
consistency during the migration.
* **Repositories**: New repositories provide an abstraction layer for
accessing and manipulating the data in the new tables.
* **Setup**: A new setup step for `TransactionalTables` has been added
to manage the database migrations for the new tables.
This PR lays the foundation for future work to fully transition to
state-persisted objects for these components, which will improve
performance and simplify data access patterns.
This PR initiates the rework of ZITADEL's backend to state-persisted
objects. This is a foundational step towards a new architecture that
will improve performance and maintainability.
The following objects are migrated from event-sourced aggregates to
state-persisted objects:
* Instances
* incl. Domains
* Orgs
* incl. Domains
The structure of the new backend implementation follows the software
architecture defined in this [wiki
page](https://github.com/zitadel/zitadel/wiki/Software-Architecturel).
This PR includes:
* The initial implementation of the new transactional repositories for
the objects listed above.
* Projections to populate the new relational tables from the existing
event store.
* Adjustments to the build and test process to accommodate the new
backend structure.
This is a work in progress and further changes will be made to complete
the migration.
---------
Co-authored-by: Copilot <[email protected]>
Co-authored-by: Iraq Jaber <[email protected]>
Co-authored-by: Iraq <[email protected]>
Co-authored-by: Tim Möhlmann <[email protected]>
# Which Problems Are Solved
I noticed some outdated / misleading logs when starting zitadel:
- The `init-projections` were no longer in beta for a long time.
- The LRU auth request cache is disabled by default, which results in
the following message, which has caused confusion by customers:
```level=info msg="auth request cache disabled" error="must provide a positive size"```
# How the Problems Are Solved
- Removed the beta info
- Disable cache initialization if possible
# Additional Changes
None
# Additional Context
- noticed internally
- backport to v4.x
# Which Problems Are Solved
When a SAML IDP is created, the signing algorithm defaults to
`RSA-SHA1`.
This PR adds the functionality to configure the signing algorithm while
creating or updating a SAML IDP. When nothing is specified, `RSA-SHA1`
is the default.
Available options:
* RSA_SHA1
* RSA_SHA256
* RSA_SHA512
# How the Problems Are Solved
By introducing a new optional config to specify the Signing Algorithm.
# Additional Changes
N/A
# Additional Context
- Closes#9842
An existing bug in the UpdateSAMLProvider API will be fixed as a
followup in a different
[PR](https://github.com/zitadel/zitadel/pull/10557).
---------
Co-authored-by: Stefan Benz <[email protected]>
# Which Problems Are Solved
Emails are still send only with URLs to login v1.
# How the Problems Are Solved
Add configuration for URLs as URL templates, so that links can point at
Login v2.
# Additional Changes
None
# Additional Context
Closes#10236
---------
Co-authored-by: Marco A. <[email protected]>
This PR is still WIP and needs changes to at least the tests.
# Which Problems Are Solved
To be able to report analytical / telemetry data from deployed Zitadel
systems back to a central endpoint, we designed a "service ping"
functionality. See also https://github.com/zitadel/zitadel/issues/9706.
This PR adds the first implementation to allow collection base data as
well as report amount of resources such as organizations, users per
organization and more.
# How the Problems Are Solved
- Added a worker to handle the different `ReportType` variations.
- Schedule a periodic job to start a `ServicePingReport`
- Configuration added to allow customization of what data will be
reported
- Setup step to generate and store a `systemID`
# Additional Changes
None
# Additional Context
relates to #9869
# Which Problems Are Solved
We are preparing to roll-out and stabilize webkeys in the next version
of Zitadel. Before removing legacy signing-key code, we must ensure all
existing instances have their webkeys generated.
# How the Problems Are Solved
Add a setup step which generate 2 webkeys for each existing instance
that didn't have webkeys yet.
# Additional Changes
Return an error from the config type-switch, when the type is unknown.
# Additional Context
- Part 1/2 of https://github.com/zitadel/zitadel/issues/10029
- Should be back-ported to v3
# Which Problems Are Solved
The resource usage to query user(s) on the database was high and
therefore could have performance impact.
# How the Problems Are Solved
Database queries involving the users and loginnames table were improved
and an index was added for user by email query.
# Additional Changes
- spellchecks
- updated apis on load tests
# additional info
needs cherry pick to v3
# Which Problems Are Solved
Add the ability to keep track of the current counts of projection
resources. We want to prevent calling `SELECT COUNT(*)` on tables, as
that forces a full scan and sudden spikes of DB resource uses.
# How the Problems Are Solved
- A resource_counts table is added
- Triggers that increment and decrement the counted values on inserts
and deletes
- Triggers that delete all counts of a table when the source table is
TRUNCATEd. This is not in the business logic, but prevents wrong counts
in case someone want to force a re-projection.
- Triggers that delete all counts if the parent resource is deleted
- Script to pre-populate the resource_counts table when a new source
table is added.
The triggers are reusable for any type of resource, in case we choose to
add more in the future.
Counts are aggregated by a given parent. Currently only `instance` and
`organization` are defined as possible parent. This can later be
extended to other types, such as `project`, should the need arise.
I deliberately chose to use `parent_id` to distinguish from the
de-factor `resource_owner` which is usually an organization ID. For
example:
- For users the parent is an organization and the `parent_id` matches
`resource_owner`.
- For organizations the parent is an instance, but the `resource_owner`
is the `org_id`. In this case the `parent_id` is the `instance_id`.
- Applications would have a similar problem, where the parent is a
project, but the `resource_owner` is the `org_id`
# Additional Context
Closes https://github.com/zitadel/zitadel/issues/9957
# Which Problems Are Solved
Currently if a user signs in using an IdP, once they sign out of
Zitadel, the corresponding IdP session is not terminated. This can be
the desired behavior. In some cases, e.g. when using a shared computer
it results in a potential security risk, since a follower user might be
able to sign in as the previous using the still open IdP session.
# How the Problems Are Solved
- Admins can enabled a federated logout option on SAML IdPs through the
Admin and Management APIs.
- During the termination of a login V1 session using OIDC end_session
endpoint, Zitadel will check if an IdP was used to authenticate that
session.
- In case there was a SAML IdP used with Federated Logout enabled, it
will intercept the logout process, store the information into the shared
cache and redirect to the federated logout endpoint in the V1 login.
- The V1 login federated logout endpoint checks every request on an
existing cache entry. On success it will create a SAML logout request
for the used IdP and either redirect or POST to the configured SLO
endpoint. The cache entry is updated with a `redirected` state.
- A SLO endpoint is added to the `/idp` handlers, which will handle the
SAML logout responses. At the moment it will check again for an existing
federated logout entry (with state `redirected`) in the cache. On
success, the user is redirected to the initially provided
`post_logout_redirect_uri` from the end_session request.
# Additional Changes
None
# Additional Context
- This PR merges the https://github.com/zitadel/zitadel/pull/9841 and
https://github.com/zitadel/zitadel/pull/9854 to main, additionally
updating the docs on Entra ID SAML.
- closes#9228
- backport to 3.x
---------
Co-authored-by: Silvan <[email protected]>
Co-authored-by: Zach Hirschtritt <[email protected]>
# Which Problems Are Solved
The execution handler projection handles all events to check if an
execution has to be provided to the worker to execute.
In this logic all events would be processed from the beginning which is
not necessary.
# How the Problems Are Solved
Add the current state to the execution handler projection, to avoid
processing all existing events.
# Additional Changes
Add custom configuration to the default, so that the transactions are
limited to some events.
# Additional Context
None
# Which Problems Are Solved
Step 54 was not executed during setup.
# How the Problems Are Solved
Added the step to setup jobs
# Additional Changes
none
# Additional Context
- the step was added in https://github.com/zitadel/zitadel/pull/9837
- thanks to @zhirschtritt for raising this.
# Which Problems Are Solved
When running a long-running Zitadel Setup, Kubernetes might decide to
move a pod to a new node automatically. Currently, this puts any
migrations into a broken state that an operator needs to manually run
the "cleanup" command on - assuming they catch the error.
The only super long running commands are typically projection pre-fill
operations, which depending on the size of the event table for that
projection, can take many hours - plenty of time for Kubernetes to make
unexpected decisions, especially in a busy cluster.
# How the Problems Are Solved
This change listens on `os.Interrupt` and `syscall.SIGTERM`, cancels the
current Setup context, and runs the `Cleanup` command. The logs then
look something like this:
```shell
...
INFO[0000] verify migration caller="/Users/zach/src/zitadel/internal/migration/migration.go:43" name=repeatable_delete_stale_org_fields
INFO[0000] starting migration caller="/Users/zach/src/zitadel/internal/migration/migration.go:66" name=repeatable_delete_stale_org_fields
INFO[0000] execute delete query caller="/Users/zach/src/zitadel/cmd/setup/39.go:37" instance_id=281297936179003398 migration=repeatable_delete_stale_org_fields progress=1/1
INFO[0000] verify migration caller="/Users/zach/src/zitadel/internal/migration/migration.go:43" name=repeatable_fill_fields_for_instance_domains
INFO[0000] starting migration caller="/Users/zach/src/zitadel/internal/migration/migration.go:66" name=repeatable_fill_fields_for_instance_domains
----- SIGTERM signal issued -----
INFO[0000] received interrupt signal, shutting down: interrupt caller="/Users/zach/src/zitadel/cmd/setup/setup.go:121"
INFO[0000] query failed caller="/Users/zach/src/zitadel/internal/eventstore/repository/sql/query.go:135" error="timeout: context already done: context canceled"
DEBU[0000] filter eventstore failed caller="/Users/zach/src/zitadel/internal/eventstore/handler/v2/field_handler.go:155" error="ID=SQL-KyeAx Message=unable to filter events Parent=(timeout: context already done: context canceled)" projection=instance_domain_fields
DEBU[0000] unable to rollback tx caller="/Users/zach/src/zitadel/internal/eventstore/handler/v2/field_handler.go:110" error="sql: transaction has already been committed or rolled back" projection=instance_domain_fields
INFO[0000] process events failed caller="/Users/zach/src/zitadel/internal/eventstore/handler/v2/field_handler.go:72" error="ID=SQL-KyeAx Message=unable to filter events Parent=(timeout: context already done: context canceled)" projection=instance_domain_fields
DEBU[0000] trigger iteration caller="/Users/zach/src/zitadel/internal/eventstore/handler/v2/field_handler.go:73" iteration=0 projection=instance_domain_fields
ERRO[0000] migration failed caller="/Users/zach/src/zitadel/internal/migration/migration.go:68" error="ID=SQL-KyeAx Message=unable to filter events Parent=(timeout: context already done: context canceled)" name=repeatable_fill_fields_for_instance_domains
ERRO[0000] migration finish failed caller="/Users/zach/src/zitadel/internal/migration/migration.go:71" error="context canceled" name=repeatable_fill_fields_for_instance_domains
----- Cleanup before exiting -----
INFO[0000] cleanup started caller="/Users/zach/src/zitadel/cmd/setup/cleanup.go:30"
INFO[0000] cleanup migration caller="/Users/zach/src/zitadel/cmd/setup/cleanup.go:47" name=repeatable_fill_fields_for_instance_domains
```
# Additional Changes
* `mustExecuteMigration` -> `executeMigration`: **must**Execute logged a
Fatal error previously which calls os.Exit so no cleanup was possible.
Instead, this PR returns an error and assigns it to a shared error in
the Setup closure that defer can check.
* `initProjections` now returns an error instead of exiting
# Additional Context
This behavior might be unwelcome or at least unexpected in some cases.
Putting it behind a feature flag or config setting is likely a good
followup.
---------
Co-authored-by: Silvan <[email protected]>
# Which Problems Are Solved
Zitadel setup with v2.71.0 could result in errors regarding the
idp_templates6_ldap3 subtable.
# How the Problems Are Solved
Rename the subtable idp_templates6_ldap3 to idp_templates6_ldap2 if no
idp_templates6_ldap2 is existing and rename column `rootCA` to
`root_ca`.
# Additional Changes
None
# Additional Context
Related PR #9292
---------
Co-authored-by: Silvan <[email protected]>
# Which Problems Are Solved
If configuration `notifications.LegacyEnabled` is set to false when
using cockroachdb as a database Zitadel start does not work and prints
the following error: `level=fatal msg="unable to start zitadel"
caller="github.com/zitadel/zitadel/cmd/start/start_from_init.go:44"
error="can't scan into dest[0]: cannot scan NULL into *string"`
# How the Problems Are Solved
The combination of the setting and cockraochdb are checked and a better
error is provided to the user.
# Additional Context
- introduced with https://github.com/zitadel/zitadel/pull/9321
# Which Problems Are Solved
The recently introduced notification queue have potential race conditions.
# How the Problems Are Solved
Current code is refactored to use the queue package, which is safe in
regards of concurrency.
# Additional Changes
- the queue is included in startup
- improved code quality of queue
# Additional Context
- closes https://github.com/zitadel/zitadel/issues/9278
# Which Problems Are Solved
Setup fails to push all role permission events when running Zitadel with
CockroachDB. `TransactionRetryError`s were visible in logs which finally
times out the setup job with `timeout: context deadline exceeded`
# How the Problems Are Solved
As suggested in the [Cockroach documentation](timeout: context deadline
exceeded), _"break down larger transactions"_. The commands to be pushed
for the role permissions are chunked in 50 events per push. This
chunking is only done with CockroachDB.
# Additional Changes
- gci run fixed some unrelated imports
- access to `command.Commands` for the setup job, so we can reuse the
sync logic.
# Additional Context
Closes#9293
---------
Co-authored-by: Silvan <[email protected]>
# Which Problems Are Solved
Some OAuth2 and OIDC providers require the use of PKCE for all their
clients. While ZITADEL already recommended the same for its clients, it
did not yet support the option on the IdP configuration.
# How the Problems Are Solved
- A new boolean `use_pkce` is added to the add/update generic OAuth/OIDC
endpoints.
- A new checkbox is added to the generic OAuth and OIDC provider
templates.
- The `rp.WithPKCE` option is added to the provider if the use of PKCE
has been set.
- The `rp.WithCodeChallenge` and `rp.WithCodeVerifier` options are added
to the OIDC/Auth BeginAuth and CodeExchange function.
- Store verifier or any other persistent argument in the intent or auth
request.
- Create corresponding session object before creating the intent, to be
able to store the information.
- (refactored session structs to use a constructor for unified creation
and better overview of actual usage)
Here's a screenshot showing the URI including the PKCE params:
# Additional Changes
None.
# Additional Context
- Closes#6449
- This PR replaces the existing PR (#8228) of @doncicuto. The base he
did was cherry picked. Thank you very much for that!
---------
Co-authored-by: Miguel Cabrerizo <[email protected]>
Co-authored-by: Stefan Benz <[email protected]>
# Which Problems Are Solved
Performance issue for GRPC call `zitadel.user.v2.UserService.ListUsers`
due to lack of org filtering on `ListUsers`
# Additional Context
Replace this example with links to related issues, discussions, discord
threads, or other sources with more context.
Use the Closing #issue syntax for issues that are resolved with this PR.
- Closes https://github.com/zitadel/zitadel/issues/9191
---------
Co-authored-by: Iraq Jaber <[email protected]>
Co-authored-by: Tim Möhlmann <[email protected]>
# Which Problems Are Solved
OIDC applications can configure the used login version, which is
currently not possible for SAML applications.
# How the Problems Are Solved
Add the same functionality dependent on the feature-flag for SAML
applications.
# Additional Changes
None
# Additional Context
Closes#9267
Follow up issue for frontend changes #9354
---------
Co-authored-by: Livio Spring <[email protected]>
jmaretteandGitHub
