<!-- CURSOR_AGENT_PR_BODY_BEGIN -->
# Which Problems Are Solved
Bulk unique-constraint deletes (org, IDP with many links, user, project)
reconstruct every uniqueness key in Go and emit N `DELETE`s. That can
OOM on large tenants.
A one-shot owner backfill is also not enough: during a rolling deploy,
older binaries still insert `owners = '{}'`, so a later owner-only
delete would miss those rows.
Owner-only org/user/grant delete only matches rows whose `owners` bag
contains that tag. Several grant and member types stamped the aggregate
resource owner, which is not always the user’s home org or the granted
org. Username-scope rewrites also dropped `user:` tags, so owner-only
user delete could miss rewritten username rows.
Command must not switch to owner-only delete on the first 79 of a
version, because leftover `'{}'` rows from rolling-update overlap are
only backfilled on a later 79. A failed 79 must not turn owner-delete
on, and backfill joins must not stamp the wrong org (unverified domains,
org-scoped usernames colliding with instance-scoped ones).
# How the Problems Are Solved
Uniqueness stays the existing primary key `(instance_id, unique_type,
unique_field)`. `owners` is only a lifecycle bag of tags (`org:<id>`,
`user:<id>`, `idp:<id>`, `project:<id>`, `grant:<id>`).
- Add `owners TEXT[] NOT NULL DEFAULT '{}'` on
`eventstore.unique_constraints` (init SQL for greenfield, setup 78 for
existing clusters).
- Create `unique_constraints_owners_gin` with `CREATE INDEX
CONCURRENTLY` **outside a transaction** (same pattern as step 76).
Built-in `GIN (owners)` only; no `btree_gin`.
- Stamp `WithOwners` on **add** events. Field-level removes ignore
owners. `RemoveByOwner` stores the tag in `Owners` and both v1/v3 and v2
pushers run `DELETE … WHERE instance_id = $1 AND owners @>
ARRAY[$tag]::text[]`.
- Setup 79 backfills `owners` from projections via numbered
`cmd/setup/79/*.sql` files. Table names are interpolated from
`internal/query/projection` constants. Matches use `unique_field IN (x,
lower(x))` so the primary key can be used. Missing projection tables
(`42P01`) are skipped.
- Org domains are tagged only when `is_verified`. Instance-scoped
usernames join only users that are not org-scoped (organization settings
or domain policy `user_login_must_be_domain`). Every username row still
gets `org:` + `user:` of that user.
- 79 is a **repeatable** migration. `Check` re-runs when
`lastRun["version"]` changed or when setup-only `ForceFinalize` is set
and `lastRun["finalized"]` is still false. `Finalized` is written only
after Execute succeeds.
- Failed setup events keep the last successful `lastRun`. Command reads
`finalized` only from `system.migration.repeatable.done`.
- Org / user / project remove skip N-key event replay **when that
`finalized` bit is true**. Otherwise they keep today’s N-key lists
**and** still emit `RemoveByOwner`. The “ready” result is cached in
memory once true.
Setup writes `finalized` on the 79 repeatable.done payload:
- First 79 of a version: backfill only, `finalized=false` (unless
`ForceFinalize` is already set).
- Later 79: version change or
`BackfillUniqueConstraintOwners.ForceFinalize` /
`ZITADEL_BACKFILLUNIQUECONSTRAINTOWNERS_FORCEFINALIZE` after the rolling
update is done. That run backfills leftovers and writes
`finalized=true`.
Member and grant owner bags now include every org that should delete the
row:
- Instance, org, and project members also stamp
`org:<user_resource_owner>`.
- User grants stamp grant, user, project, and granted orgs.
- Project grant members stamp the project org, user org, and granted
org.
- Project grants stamp both orgs plus `grant:<id>`. Grant remove emits
`RemoveByOwner(grant)`.
- Username-scope rewrites keep `user:<id>` as well as `org:`.
- User/grant orgs are carried as unexported add-event fields (not
persisted JSON). `WithOwners` skips empty and duplicate tags.
# Additional Changes
- Table-driven owner-tag tests, including project grant (two `org:` tags
plus `grant:`), cross-org members, user grants, and username-scope
rewrite `user:` tags.
- Table-driven 79 `Check` + `finalizeAfterSuccess` tests for version
change, sequential version bumps, ForceFinalize restart, and
already-finalized skip.
- Command tests: a failed 79 event with `finalized=true` does not enable
owner-delete.
- Integration fixtures (verified vs unverified org domain;
instance-scoped `bob` vs org-scoped `bob`).
- Instance-only and global unique types stay untagged (`'{}'`). Instance
wipe still uses `UniqueConstraintInstanceRemove`.
- Pusher owner cases live in `eventstore_pusher_owners_test.go` so
`eventstore_pusher_test.go` stays under 1k lines.
# Additional Context
A later cleanup can delete the N-key replay helpers once a release that
ran 79 a second time (or an operator who set `ForceFinalize`) is in
production. That is not required for correctness.
<!-- CURSOR_AGENT_PR_BODY_END -->
<div><a
href="https://cursor.com/agents/bc-44f9bc9b-9a93-4cd9-8dd1-663a7b9aad2a?cursor_ref=pr_footer&cursor_cta=open_in_web"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://cursor.com/assets/images/open-in-web-dark.png"><source
media="(prefers-color-scheme: light)"
srcset="https://cursor.com/assets/images/open-in-web-light.png"><img
alt="Open in Web" width="114" height="28"
src="https://cursor.com/assets/images/open-in-web-dark.png"></picture></a> <a
href="https://cursor.com/background-agent?bcId=bc-44f9bc9b-9a93-4cd9-8dd1-663a7b9aad2a&cursor_ref=pr_footer&cursor_cta=open_in_cursor"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://cursor.com/assets/images/open-in-cursor-dark.png"><source
media="(prefers-color-scheme: light)"
srcset="https://cursor.com/assets/images/open-in-cursor-light.png"><img
alt="Open in Cursor" width="131" height="28"
src="https://cursor.com/assets/images/open-in-cursor-dark.png"></picture></a> </div>
---------
Co-authored-by: Cursor Agent <[email protected]>
Co-authored-by: Silvan <[email protected]>
# Which Problems Are Solved
ZITADEL opens its Postgres connection pool via `pgxpool.NewWithConfig`
(`internal/database/postgres/pg.go`) but never reads
`pgxpool.Pool.Stat()`,
so operators have no way to observe pool saturation (acquired/idle/total
conns vs. `MaxConns`) or connection churn (acquire waits, canceled/empty
acquires, idle/lifetime destroys, new connections) through ZITADEL's
existing metrics pipeline.
# How the Problems Are Solved
- Adds `RegisterCounterObserver` (backed by an OTel
`Int64ObservableCounter`)
to the `Metrics` interface, its implementation, and the test mock
(`backend/v3/instrumentation/metric.go`,
`backend/v3/instrumentation/metrics/metric.go`,
`backend/v3/instrumentation/metrics/mock.go`), mirroring the existing
`RegisterValueObserver`/`RegisterUpDownSumObserver` — there was
previously
no observable instrument for cumulative counters.
- Adds `database.RegisterPoolMetrics(ctx, pool)`
(`internal/database/metrics.go`), which registers 5 observable gauges
(`acquired_conns`, `constructing_conns`, `idle_conns`, `max_conns`,
`total_conns`) and 7 observable counters (`acquire_count`,
`acquire_duration_milliseconds`, `canceled_acquire_count`,
`empty_acquire_count`, `max_idle_destroy_count`,
`max_lifetime_destroy_count`, `new_conns_count`) sourced directly from
`pgxpool.Pool.Stat()`, published under `zitadel.database.pool.*` metric
names.
- Wires the registration call into `cmd/start/start.go` right after the
pool is created, so the running server process exposes these series on
`/metrics`.
# Additional Changes
- Adds unit tests covering the new `RegisterCounterObserver` method
(`backend/v3/instrumentation/metric_test.go`) and the pool metrics
registration (`internal/database/metrics_test.go`).
# Additional Context
- Scope is intentionally limited to `cmd/start` (the long-running
process
that serves `/metrics`); one-shot commands (`setup`, `mirror`,
`initialise`, `key`) are left untouched since an exporter running for a
single command invocation wouldn't produce useful series.
- https://pkg.go.dev/github.com/jackc/pgx/v5/pgxpool#Stat
<!-- CURSOR_AGENT_PR_BODY_BEGIN -->
# Which Problems Are Solved
Handler/v2 projections (`users14` and every other event projection) can
overlook committed events under load:
- `eventstore.commands_to_events` stamped `position` with `NOW()`
(transaction start). A slow create TX could commit after a faster TX
that started later, so the projector stored a later position and never
saw the earlier one.
- Resume used `OFFSET` on a **filtered** row count. That could skip
`in_tx_order = 1` at the next position (for example `user.human.added`).
- Two transactions in the same microsecond both start at `in_tx_order =
1`. A cursor of `in_tx_order > last` alone would drop the other TX’s
first event.
This does not repair rows that are already missing from projections.
# How the Problems Are Solved
- Setup step 77 replaces `eventstore.commands_to_events` so `created_at
= statement_timestamp()` and `position = EXTRACT(EPOCH FROM
clock_timestamp())` at INSERT time. Initialise `08_events_table.sql`
matches that for new databases. The function is `VOLATILE` to match
`clock_timestamp()`.
- The projector still drains in-flight writers with `pg_advisory_lock` /
`pg_advisory_unlock` in one statement, then caps with `position <=
EXTRACT(EPOCH FROM now())` (projector TX start). The exclusive lock is
**not** held across the event `SELECT`. The cap must stay `now()`, not
`clock_timestamp()`, or later inserts become a high-water mark.
- Resume is a lexicographic `EventSortKey` on `(position, in_tx_order,
instance_id, aggregate_type, aggregate_id, sequence)` via
`AfterEventSortKey`. SQL `OFFSET` is no longer used for projection
catch-up.
- Handler and FieldHandler persist the last event’s `in_tx_order` in
`current_states.filter_offset` (existing column, new meaning). Step 77
backfills matching rows from `events2`; unmatched rows keep their
previous value.
# Additional Changes
- `eventstore.Event` exposes `InTxOrder() uint32` (BaseEvent, repository
Event, v3 event, v1 models).
- v3 push SELECT/RETURNING now includes `in_tx_order` so newly written
events carry a real ordinal.
- `WithMinPosition` stays an inclusive `PositionAtLeast` floor and does
not smash the sort-key cursor.
- `SearchQuery.PositionAfter(decimal)` is unchanged (used by
access-token lookup). Do not confuse it with
`SearchQueryBuilder.AfterEventSortKey`.
- `commands_to_events` in setup 77 and initialise 08 is `VOLATILE
PARALLEL SAFE`. Shipped 40/64/70 stay `STABLE` because they still stamp
with `NOW()`.
- Drop unused `last_owner TEXT` DECLARE from setup 77 and initialise
`08_events_table.sql` (`i INTEGER` remains for the enforce-owner loop).
Shipped 40/64/70 keep the leftover.
- This PR’s migration is numbered 77 because `main` already has setup 76
(`76_users14_login_equality_indexes`). First-loop order is s70, then
main’s 76, then this 77.
# Additional Context
**Reviewer map**
| Start here | Why |
| --- | --- |
| `cmd/setup/77.sql` + `cmd/setup/77.go` | Position stamp + atomic
function replace/backfill after shipped `s70` and main’s s76 |
| `internal/eventstore/event_sort_key.go` + `search_query.go` |
Canonical cursor and `AfterEventSortKey` |
| `internal/eventstore/handler/v2/{state.go,handler.go}` |
Persist/resume cursor; `eventQuery` |
| `internal/eventstore/repository/sql/{query.go,postgres.go}` | Tuple
`>` filter, 6-column `ORDER BY`, await cap |
| `internal/eventstore/v3/push.go` / `push.sql` | Scan `in_tx_order` on
write |
Mechanical / low-risk: handler and query test fixtures, Event interface
stubs (`event_base.go`, repository Event, mocks, v1 models).
**Do not look for** a `70.sql` change — shipped migrations stay as on
`main`. Setup 40/64 still use `NOW()`; step 77 overwrites the function
afterward. Fresh `FirstInstance` (runs before `s70`) can still stamp
`NOW()` until 77; runtime pushes after 77 use `clock_timestamp()`.
Main’s `cmd/setup/76.go` is the users14 login-equality indexes, not this
stamp.
**Rollout**
1. Run setup so 77 replaces `commands_to_events` and backfills
`filter_offset`.
2. Restart ZITADEL so projectors load the new resume path.
3. Existing missing `users14` rows are not repaired by this PR.
**Tests**
- `go test ./internal/eventstore/... ./cmd/setup/`
- Sort-key SQL fixtures in `query_sort_key_test.go` (no `OFFSET`;
aggregate-id queries keep 6-column `ORDER BY`)
- `cmd/setup/77_test.go` asserts the embed has no unused `last_owner`
and still uses `clock_timestamp()` / `VOLATILE PARALLEL SAFE`
Related: #8509, #10480 (remove OFFSET), #10560 (advisory locks), #10710
(stable ORDER BY).
Out of scope: repairing already-missing projection rows, holding the
exclusive lock during SELECT, rewriting historical setup 40/64 SQL.
<!-- CURSOR_AGENT_PR_BODY_END -->
<div><a
href="https://cursor.com/agents/bc-3da6eb51-11d4-4a8a-8ba0-e190f116df05?cursor_ref=pr_footer&cursor_cta=open_in_web"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://cursor.com/assets/images/open-in-web-dark.png"><source
media="(prefers-color-scheme: light)"
srcset="https://cursor.com/assets/images/open-in-web-light.png"><img
alt="Open in Web" width="114" height="28"
src="https://cursor.com/assets/images/open-in-web-dark.png"></picture></a> <a
href="https://cursor.com/background-agent?bcId=bc-3da6eb51-11d4-4a8a-8ba0-e190f116df05&cursor_ref=pr_footer&cursor_cta=open_in_cursor"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://cursor.com/assets/images/open-in-cursor-dark.png"><source
media="(prefers-color-scheme: light)"
srcset="https://cursor.com/assets/images/open-in-cursor-light.png"><img
alt="Open in Cursor" width="131" height="28"
src="https://cursor.com/assets/images/open-in-cursor-dark.png"></picture></a> </div>
---------
Co-authored-by: Cursor Agent <[email protected]>
Co-authored-by: Silvan <[email protected]>
<!-- CURSOR_AGENT_PR_BODY_BEGIN -->
# Which Problems Are Solved
Login V2 `ListUsers` lookups of the form username OR email (and email OR
phone) hash-join the full `users14` / `users14_humans` tables, then
filter. That cost grows with instance size (~40ms at 10k users, ~650ms
at 200k).
Top-level login-name equals already used an indexed join, but the same
login-name filter inside an `OrQuery` fell back to the slow view-based
exists query.
# How the Problems Are Solved
**TL;DR:** equality lookups on username, email, phone, or login name
become `INNER JOIN (UNION of indexed ID seeks) AS matches`. Mixed /
contains / name searches keep the old WHERE shape.
### Reviewer map
| Read first | Why |
| --- | --- |
| `internal/query/user_login_equality.go` | Typed filters + extractor +
UNION join |
| `internal/query/user.go` (`prepareUsersQuery`, constructors) | Wiring;
username/email/phone equals return `lowerEqualsFilter` |
| `cmd/setup/76.go` + `cmd/setup/76/*.sql` | Concurrent indexes for
existing deployments; drop invalid leftovers |
| `internal/query/projection/user.go` | Same indexes in `Init()` for new
databases |
| `internal/query/user_login_name_exists_test.go` | SQL-shape coverage |
| `internal/api/grpc/user/v2/integration_test/query_test.go` | Nested
username AND/OR user results |
### Query rewrite
- Extract one equals leaf, a flat `OrQuery` of those leaves, or the same
nested under `AndQuery`. Remaining AND conjuncts stay as WHERE. Equality
nested under `OrQuery` / `NotQuery` is left as WHERE.
- Login-name equals is a UNION arm that reuses `login_names3_users` SQL,
so it works inside OR as well as at the top level.
- EQUALS seeks use `LOWER(col)` for the index, then recheck `col =
value` so case-sensitive searches do not match extra rows.
EQUALS_IGNORE_CASE is unchanged.
- Empty OR arms are not skipped (that would change OR meaning); mixed
empty-OR falls back to WHERE. Login UI already omits empty arms.
### Indexes
- Setup 76: `CREATE INDEX CONCURRENTLY IF NOT EXISTS` on
`users14_username_lower_idx` `(instance_id, LOWER(username))` and
`users14_humans_phone_lower_idx` `(instance_id, LOWER(phone))`. Not
wrapped in a transaction.
- Before create, drop any leftover `pg_index.indisvalid = false` indexes
of those names (failed/cancelled `CREATE INDEX CONCURRENTLY`). `IF NOT
EXISTS` would otherwise skip rebuild.
- `Init()` creates the same indexes without `CONCURRENTLY` (empty
tables). Existing `users14_username` and email lower indexes are
unchanged.
# Additional Changes
- Unit tests assert UNION SQL for username OR email, email OR phone,
login name OR email, username OR username, nested `AndQuery(loginName,
org)`, and nested `AndQuery(Or(...), org)`; that contains / display-name
/ mixed OR / nested `OrQuery(AndQuery(...), ...)` do not UNION; that
email EQUALS rechecks the raw column; and that empty-OR is not
extracted.
- Setup 76 sqlmock coverage: missing `users14` skips work; valid/absent
indexes only create; invalid leftover username index is dropped then
both creates run.
- `TestServer_ListUsers_UsernameAndOr` checks live results: org AND
(alice OR bob) returns both users, sibling username AND is empty, nested
`AndQuery(Or(...), org)` returns both.
# Additional Context
- Base branch: `main`
- Extends the login-name equals rewrite from #12460 to
username/email/phone OR trees used by Login V2 search.
<!-- CURSOR_AGENT_PR_BODY_END -->
<div><a
href="https://cursor.com/agents/bc-1f043b1c-6168-45ca-9e9e-c18451994207?cursor_ref=pr_footer&cursor_cta=open_in_web"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://cursor.com/assets/images/open-in-web-dark.png"><source
media="(prefers-color-scheme: light)"
srcset="https://cursor.com/assets/images/open-in-web-light.png"><img
alt="Open in Web" width="114" height="28"
src="https://cursor.com/assets/images/open-in-web-dark.png"></picture></a> <a
href="https://cursor.com/background-agent?bcId=bc-1f043b1c-6168-45ca-9e9e-c18451994207&cursor_ref=pr_footer&cursor_cta=open_in_cursor"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://cursor.com/assets/images/open-in-cursor-dark.png"><source
media="(prefers-color-scheme: light)"
srcset="https://cursor.com/assets/images/open-in-cursor-light.png"><img
alt="Open in Cursor" width="131" height="28"
src="https://cursor.com/assets/images/open-in-cursor-dark.png"></picture></a> </div>
---------
Co-authored-by: Cursor Agent <[email protected]>
Co-authored-by: Silvan <[email protected]>
# Which Problems Are Solved
We run multiple deployments (regions) under a single GCP project. Within
these deployments we scale containers. Currently all metrics target the
same time series in GCP, regardless or region or container. All metics
ended up in `generic_node` and Cloud Monitoring rejects the duplicate
and out-of-order points.
This can be fixed by setting vendor specific attributes such as
`faas.name` (K_SERVICE), `faas.version` (K_REVISION), `faas.instance`
(instance id from the metadata server), and `cloud.region` (metadata
instance/region).
# How the Problems Are Solved
Use `go.opentelemetry.io/contrib/detectors/gcp` to auto-detect values as
mentioned above. A new `Instrumentation.Detectors` config entry is
added, which accepts and array of enum for detecotrs. At the moment only
Google is accepted, other entries from
[contrib](https://github.com/open-telemetry/opentelemetry-go-contrib/tree/main/detectors)
can be added if deemed usefull.
I've considered using `autodetect` contrib package, identifiers and
constructor, but this pulls in the full AWS SDK, increasing zitadel's
binary size by 50MiB.
By default detection is switched off, as it relies on unauthenticated
link-local calls that are insecure anywhere outside of the
infrastructure it targets. It also prevents breaking existing
deployments that rely on detection being off already. (May break
existing trace and logging pipelines).
# Additional Changes
- none
# Additional Context
- (internal only): https://github.com/caos/infra/pull/1813
---------
Co-authored-by: Copilot Autofix powered by AI <[email protected]>
Co-authored-by: Claude Opus 5 <[email protected]>
* fix(crypto): use authenticated encryption for IDP intent tokens
IDP intent tokens were encrypted with AES-CFB which provides no
integrity protection, making them malleable. Switch to A256GCM via
the existing AuthAlgorithm interface (EncryptToken/DecryptToken),
the same approach already used for OIDC and session tokens.
The idpConfigEncryption key type is widened from EncryptionAlgorithm
to AuthEncryptionAlgorithm so that data-at-rest encryption (IDP
client secrets, LDAP bind passwords, SAML keys) continues to use
the CFB path unmodified.
Co-authored-by: Cursor <[email protected]>
* fix(proto): raise IDP intent token max length to 500 for authenticated tokens
The switch to authenticated A256GCM (JWE compact serialization) makes IDP
intent tokens longer (~270 chars) than the previous AES-CFB base64 output.
Raise the max_len/max_length validation and OpenAPI constraints on every
idp_intent_token field from 200 to 500 so the new tokens are accepted:
user v2/v2beta (idp + user_service), session v2/v2beta, and resources
user v3alpha.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
---------
Co-authored-by: Cursor <[email protected]>
Co-authored-by: Claude Opus 4.8 <[email protected]>
# Which Problems Are Solved
The `/.well-known/apple-app-site-association` and
`/.well-known/assetlinks.json` responses are instance specific: the body
depends on the requested host. The responses did not declare this, so a
shared cache whose cache key does not include the host could store one
instance's file and serve it on another instance's domain. This could
result in a wrong or empty file being returned intermittently, breaking
native passkey verification.
Dropping the `public` directive (#12634) was not sufficient: depending
on its cache mode, a shared cache may store any response carrying a
`max-age`.
The docs page for native app links also still documented the old
`Cache-Control: public` header and only the single `get_login_creds`
relation (missing `handle_all_urls`, added in #12634).
# How the Problems Are Solved
- Send **`Vary: Host`** on both well-known responses, declaring that the
body depends on the host: a spec-compliant shared cache must store
per-host variants or skip caching. Clients addressing a single host (the
Apple and Google verifiers, browsers) are unaffected and keep honoring
the configured `max-age`.
- Correct the outdated caching reasoning in the code comments.
- Guard the header with unit and integration tests.
- Update the native app links docs page: both Digital Asset Links
relations and the current response headers.
# Additional Changes
None. `WellKnown.AppLinksCacheControlMaxAge` behavior is unchanged (`0`
still results in `no-store`).
# Additional Context
- Follow-up to #12634; relates to #12497.
- Verified with interleaved requests against two instances: responses
are no longer reused across domains, and each domain consistently serves
its own configuration.
- To be cherry-picked to `v4.x` from `main` once merged.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Fable 5 <[email protected]>
# Which Problems Are Solved
- Native iOS/Android passkeys need OS trust files on the ZITADEL domain;
without them, Associated Domains / App Links verification fails.
- Operators could not configure iOS Team ID + Bundle ID or Android
package name + SHA-256 fingerprints on OIDC apps.
- `/.well-known/apple-app-site-association` and
`/.well-known/assetlinks.json` were not served from application config.
# How the Problems Are Solved
- Add iOS/Android app-link fields on OIDC app create/update (Application
API v2 + Management), with validation.
- Persist and project those fields; query active app-link configs
instance-wide.
- Serve AASA (`webcredentials`) and Digital Asset Links
(`get_login_creds`) from well-known paths, with configurable
`Cache-Control` and fingerprint normalization at serve time.
- Console UI to edit the fields, with links to the well-known endpoints.
- Operator docs for configuration, endpoints, caching, and verification.
# Additional Changes
- Document on API fields that well-known responses may be HTTP-cached
and platform verifiers may delay propagation.
- Runtime config: `WellKnown.AppLinksCacheControlMaxAge` (default `5m`;
`0` → `no-store`).
# Additional Context
- Closes#12497
- Implemented and reviewed as stack:
- #12531 API contract
- #12532 storage wiring
- #12536 well-known endpoints
- #12537 console
- #12547 docs
---------
Co-authored-by: Cursor <[email protected]>
# Which Problems Are Solved
#12313 lets clients register themselves as OIDC applications (RFC 7591),
but a dynamically registered client has no way to manage itself
afterwards:
- The registration response returns no `registration_access_token` and
no `registration_client_uri`, which RFC 7591 and 7592 clients (including
the MCP SDKs) expect in order to manage their own registration.
- There is no way to read back, update or delete a registration without
an operator acting on the client's behalf through the Console or the
Management API. A client that needs to rotate a redirect URI has to ask
a human.
# How the Problems Are Solved
Implement the [OAuth 2.0 Dynamic Client Registration Management Protocol
(RFC 7592)](https://datatracker.ietf.org/doc/html/rfc7592): `GET`, `PUT`
and `DELETE` on `/oauth/v2/register/{client_id}`, wired through the same
`op.WithSetRouter` hook as the registration endpoint.
## Control plane: the same security settings, no new switch
The management endpoints are gated on the
`DynamicClientRegistrationSettings.enabled` introduced in #12313. No new
setting, no feature flag: a client that may register may manage what it
registered. When registration is disabled the endpoints answer `404`,
exactly as `POST /oauth/v2/register` does.
`allow_unauthenticated` selects the registration mode and does not apply
here, because the registration access token authorizes these endpoints
in both modes.
## Authorization: the registration access token, and nothing else
RFC 7592 §3 defines the credential as the registration access token
issued at registration and bound to a single client, not a user
credential. So a client manages only itself, proving it with the token
it was handed:
| | Registration (`POST`) | Management (`GET` / `PUT` / `DELETE`) |
| --- | --- | --- |
| Credential | Access token, or none in open mode | Registration access
token |
| Permission | `project.app.register_dynamic` in token mode | None |
Neither `project.app.register_dynamic`, which governs who may *create*
clients, nor `project.app.write` applies to the management endpoints.
One authorization path per endpoint is easier to reason about than two,
and there is no user identity to check against in the first place.
Operators keep managing dynamically registered clients through the
Management API and the Console, like any other application; they cannot
obtain a client's registration access token, by design.
Status codes: a missing, malformed or non matching token is `401`; a
client that no longer exists is `404`. The token's `client_id` is
checked against the path before any lookup, so a token bound to another
client is rejected without touching the database.
## The token itself
- **Stored like a client secret.** Only its passwap hash is persisted,
through a new
`project.application.config.oidc.registration_token.changed` event; the
plain token leaves the server exactly once, in the response that issues
it.
- **Bound to its client and organization** through authenticated
encryption with the instance key, the same mechanism as refresh tokens.
The opaque token handed to the client encrypts
`client_id:org_id:secret`, so the endpoint recovers the binding before
checking the secret against the stored hash.
- **Verified in O(1)** against the hash on the projection, folded into
the client lookup the handler already does, with a strongly consistent
fallback to the eventstore so a token that was just rotated is accepted
before it is projected.
- **Rotated on update** (RFC 7592 §4.4) and invalidated implicitly when
the client is deleted. A read does not rotate it. Every update rotates,
including one that changes no metadata, so a client always gets a usable
token back.
## Reusing the existing persistence
A small refactor extracts the shared OIDC application creation and
update logic (`pushOIDCApplication` gains a variadic `extraEvents`, and
`oidcApplicationChangeEvent` is lifted out of `UpdateOIDCApplication`),
so the dynamic commands reuse it without the permission check. The
registration token event is pushed atomically with the application, so a
registered client is never left unmanageable.
# Additional Changes
- Setup step **74** adds the `registration_token` column to the
`apps7_oidc_configs` projection, with a reducer for the new event
modelled on the client secret. Applications are not part of the
relational storage projection, so there is no second write path to keep
in sync.
- The registration response (`POST`) now also carries
`registration_access_token` and `registration_client_uri`.
- Docs: a "Manage a registration" section in the dynamic client
registration guide covering the three operations, the authorization
model, token rotation and the new limitations, plus the management
endpoints on the OIDC endpoints page.
- Unit tests for the command layer (issue, verify, rotate, update,
delete), the projection reducer, and the token binding and response
mapping. An integration test covers the full lifecycle (register, read,
update with rotation, delete, read again with `404`), a token bound to
another client, and the endpoints not being served when registration is
disabled. None of the integration calls carries a user access token,
which pins down that the management endpoints require no permission of
their own.
# Additional Context
- Part of the MCP authorization effort in #9810. Follow-up to #12313
(RFC 7591), which is now merged; this branch has been rebased on `main`
and carries the settings and authorization model agreed there.
- **Open question for the review**: the authorization model above
(registration access token only, no operator path through these
endpoints) is the one I proposed when #12313 was reviewed, but it was
never explicitly confirmed. If you would rather have `project.app.write`
also grant access to a client in the operator's own organization, that
is a contained change in `authorizeClientManagement`.
- Rotation is eventually consistent: right after an update the previous
token may keep working for a brief moment until the rotation is
projected, while the new token works immediately through the eventstore
fallback. This matches how ZITADEL validates access tokens, where
revocation is likewise eventually consistent. It is documented in the
guide and asserted as such in the integration test.
- A database migration is introduced for the new projection column. No
proto change and no new feature flag.
- Verified locally: full build, `go vet` including `-tags integration`,
`golangci-lint` with 0 issues, and the unit suites for every touched
package. As before, I cannot run the integration suite here, so CI
remains the check on it.
# Which Problems Are Solved
ZITADEL cannot currently be used as an OAuth 2.0 Authorization Server by
clients that need to register themselves at runtime:
- There is no [OAuth 2.0 Dynamic Client Registration (RFC
7591)](https://datatracker.ietf.org/doc/html/rfc7591) endpoint, and no
`registration_endpoint` is advertised in the discovery document.
- This blocks [Model Context Protocol
(MCP)](https://modelcontextprotocol.io) clients (Claude Desktop,
claude.ai, Cursor, the MCP SDKs), whose OAuth 2.1 authorization profile
expects an Authorization Server that supports dynamic client
registration. They self-register before any user context exists and only
then start the authorization code + PKCE flow.
# How the Problems Are Solved
## Control plane: instance security settings
No feature flag and no runtime config. A
`DynamicClientRegistrationSettings` message is nested under the instance
`SecuritySettings` in `settings/v2`, next to `embedded_iframe` and
`enable_impersonation`:
| `enabled` | `allow_unauthenticated` | Behaviour |
| --- | --- | --- |
| `false` (default) | — | `POST /oauth/v2/register` returns `404`,
`registration_endpoint` absent from discovery |
| `true` | `false` (default) | Token-gated registration, client homed in
the token's organization |
| `true` | `true` | Open registration for MCP hosts, client homed in the
instance default organization |
Both values are carried on `authz.Instance` and loaded with the
instance, exactly like `EnableImpersonation`, so the discovery document
and the endpoint read them per request without an extra query.
`allow_unauthenticated` implies `enabled` in both implementations, so a
single setting closes the endpoint.
Adding the two columns bumps the projection table to
`projections.security_policies3`, following what the impersonation
setting did for `security_policies2`.
## Authorization: a dedicated permission, not `project.app.write`
- New org-scoped permission **`project.app.register_dynamic`**, granted
by default to `ORG_OWNER`, `IAM_OWNER` and `IAM_ORG_MANAGER` — in both
`InternalAuthZ` and `SystemAuthZ`, mirroring where `project.app.write`
already lives.
- New built-in role **`ORG_DYNAMIC_CLIENT_REGISTRAR`** carrying only
that permission, so a service user can self-register clients without
gaining write access to the organization's existing applications.
- Open mode (`allow_unauthenticated = true`) requires no token and no
permission: that is the MCP self-register path.
- A valid token whose user lacks the permission is answered with **`403
insufficient_scope`** (RFC 6750 §3.1) rather than `401 invalid_token`,
so a caller can tell "your token is bad" from "your token may not do
this". `verifyAccessToken` also reports invalid tokens as permission
denied, so the authorization failure is marked explicitly to keep the
two apart.
The check runs once, at the endpoint, right after the token is verified
and before any state is created, so it also gates the auto-provisioning
of the organization's DCR project. The permission itself lives in the
command layer next to the others
(`Commands.CheckPermissionRegisterDynamicClient`). As the OIDC endpoints
are not behind the authorization interceptor, the caller identity is
derived from the verified token, the way token exchange does for the
impersonating actor.
## Registration itself
- **Registered clients are ordinary OIDC applications.** The endpoint
reuses the existing application-creation events
(`NewApplicationAddedEvent` and `NewOIDCConfigAddedEvent`), so the whole
token, authorization and introspection flow keeps working unchanged.
They are stored in a dedicated, auto-provisioned project named `ZITADEL
DCR` per organization (name lookup), so they do not pollute the
`IAMProject` or other projects.
- Input is validated with the existing `GetOIDCV1Compliance` rules;
`private_key_jwt` and `jwks`/`jwks_uri` are rejected with
`invalid_client_metadata`.
- **Endpoint wiring without forking the library.** `NewServer` calls
`op.RegisterServer` instead of `op.RegisterLegacyServer`, so the new
route can be added through the same `op.WithSetRouter` hook as the
authorize callback. `op.RegisterLegacyServer` appends a middleware after
the caller options, which chi forbids once a route has been registered;
`op.NewIssuerInterceptor` reproduces the issuer middleware it would
otherwise add. The `/oauth/v2` routing prefix is unchanged.
# Additional Changes
- **`application_type` inference for custom-scheme redirects** (last
commit, isolated). `application_type` is an OpenID Connect member; RFC
7591 does not define it, so clients that only implement RFC 7591 omit
it. Native MCP hosts doing so register custom-scheme redirect URIs,
which ZITADEL reserves for native applications, and the OIDC default of
`web` rejected them with `invalid_redirect_uri`. Inference is limited to
custom schemes and to an absent `application_type`, so every request
accepted today keeps its current application type and auth method — an
`http` loopback redirect still yields a web application. Happy to drop
this commit if you would rather keep it out of this PR.
- **Docs**: a Dynamic Client Registration integration guide, a
`registration_endpoint` section on the OpenID Connect endpoints page,
and the new role in the administrators table (plus its description in
the Console translations).
# Additional Context
- Part of #9810. This is **Phase 1 (DCR)**; #12315 adds RFC 7592 and
#12316 adds CIMD, both stacked on this branch and rebased once this
lands.
- Implements the control-plane and authorization model requested in the
review of this PR, itself a refinement of the direction in
https://github.com/zitadel/zitadel/issues/9810#issuecomment-4891176997.
- **Out of scope (deliberately)**: RFC 7592, CIMD, persisting free-form
metadata (`logo_uri`, `contacts`, and similar), and `private_key_jwt` /
`jwks_uri` (rejected cleanly). No new event types.
- Follow-ups explicitly left out, per the review: DCR client lifecycle
and richer audit, consent and custom scopes, JWT / `aud` (RFC 8707), and
RFC 8414 AS metadata (currently absent from both ZITADEL and
`zitadel/oidc`).
# Which Problems Are Solved
In Zitadel's append-only event-sourced architecture, the
`eventstore.events2` table grows indefinitely. PostgreSQL's default
autovacuum uses a percentage-based scale factor, so as the table grows,
the number of changed rows required to trigger a `VACUUM` or `ANALYZE`
drifts towards infinity. Without regular vacuums, the table's Visibility
Map becomes stale, disabling fast Index-Only Scans and forcing expensive
heap reads. Without regular analyzes, query planner statistics become
stale, leading to suboptimal execution plans.
This causes eventstore operations to progressively degrade as `events2`
grows, even without CPU, memory, or I/O saturation. A manual `VACUUM
ANALYZE` immediately restores performance, confirming the root cause.
- If `Eventstore.Autovacuum` is left at its default, `events2` keeps
using PostgreSQL's default, percentage-based autovacuum/autoanalyze
scale factors, which become impractically infrequent on large tables.
- There was previously no supported way to apply static, table-level
autovacuum tuning to `events2` through Zitadel's own configuration/setup
process.
# How the Problems Are Solved
- Added an `Eventstore.Autovacuum` runtime configuration block to
`cmd/defaults.yaml` (disabled by default):
```yaml
Eventstore:
Autovacuum:
Enabled: false # ZITADEL_EVENTSTORE_AUTOVACUUM_ENABLED
VacuumThreshold: 50000 # ZITADEL_EVENTSTORE_AUTOVACUUM_VACUUMTHRESHOLD
AnalyzeThreshold: 50000 # ZITADEL_EVENTSTORE_AUTOVACUUM_ANALYZETHRESHOLD
```
- Added a repeatable `zitadel setup` migration step
(`cmd/setup/eventstore_autovacuum.go`) that:
- When `Enabled: true`, disables the percentage-based
`autovacuum_vacuum_scale_factor`, `autovacuum_analyze_scale_factor`, and
`autovacuum_vacuum_insert_scale_factor` on `eventstore.events2`, and
applies static thresholds (`autovacuum_vacuum_insert_threshold`,
`autovacuum_vacuum_threshold`, `autovacuum_analyze_threshold`) from the
config instead.
- When `Enabled: false`, resets those storage parameters on
`eventstore.events2` back to the cluster defaults.
- Implements `Repeatable.Check()` so the step only re-runs when the
configuration actually changed since the last `zitadel setup` run.
- Added documentation in the new "Performance tuning" page.
# Additional Changes
- Move the projection documentation into performance tuning page
- Expand and update the projection documentation to the latest state in
zitadel. (contained some stale information)
# Additional Context
Several other issues describe read-performance symptoms consistent with
this same root cause (stale `events2` visibility map / planner
statistics at scale, without resource saturation). Since this PR
addresses the shared root cause:
- Closes#12448
- Closes#10754
- Closes#10260
- Closes#8585
- Closes#9239
---
_Generated by [Claude
Code](https://claude.ai/code/session_01HoKvEY7niCVgLCz7CajBwW)_
---------
Co-authored-by: Claude <[email protected]>
Co-authored-by: Copilot Autofix powered by AI <[email protected]>
Co-authored-by: Silvan <[email protected]>
* Use slices.Contains over custom function
* Correctly remove roles from granted roles
* fix(setup): repair user grants with stale roles (GHSA-v859-c572-qh5p)
Add setup step 73 that reconciles existing user grants whose roles were
left too broad by the buggy cascade removal in removeRoleFromUserGrant.
The corruption lives in the eventstore event payloads, so the step pushes
a corrective user.grant.cascade.changed event per affected grant (roles
intersected with the currently valid set) and re-triggers the user grant
projection. Runs in the second setup slice, after the projection tables
it reads have been created.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
* fix(setup): scope GHSA-v859-c572-qh5p repair to grant-based user grants
Direct user grants can never be hit by this bug (only ChangeProjectGrant's
multi-role cascade to grant-based grants can trigger it), so drop the
direct-grant branch from the finder query to avoid stripping unrelated,
legitimate roles that merely mismatch for other reasons (e.g. stale
role_key drift). Also exclude removed instances from the migration scope,
and log the number of grants fixed per instance.
Co-authored-by: Cursor <[email protected]>
---------
Co-authored-by: Claude Opus 4.8 <[email protected]>
Co-authored-by: Livio Spring <[email protected]>
Co-authored-by: Cursor <[email protected]>
# Which Problems Are Solved
Zitadel exposes the secrets generator configuration through its admin
api. This allows instance admins to manage them on their own and they
can create overwrite the system / runtime defaults (incl. expiration).
This very much needed in multi-instance scenarios such as zitadel.cloud.
Currently the invite code configuration was not manageable through the
API, but only runtime config.
# How the Problems Are Solved
- added the `invite_code` type to the API allowing it to be set and
retrieved.
- added the type to console's management list
- added the type to be stored on instance setup
- change the `GetSecretGenerator` endpoint to fall back to the runtime
config if no config is stored on the instance itself
- ensure the `length` and at least one charset is enabled, return an
error otherwise
- expiry is not enforced, so 0 allows codes with no expiry (current
state)
# Additional Changes
None
# Additional Context
- closes https://github.com/zitadel/zitadel/issues/10474
# Which Problems Are Solved
Login v2 uses the `users.v2.ListUsers`-endpoint to get a user by login
name. This query had an inefficient `WHERE`-clause.
# How the Problems Are Solved
Update the view and use a specific clause for this query.
# Additional information
Added in https://github.com/zitadel/zitadel/pull/10475
---------
Co-authored-by: Marco A. <[email protected]>
Co-authored-by: Livio Spring <[email protected]>
# Which Problems Are Solved
During the setup step we saw rare cases which caused setup to fail after
executing steps 40, 64 and 70.
# How the Problems Are Solved
Close currently open database connections so that they fetch the correct
type mapping for the `eventstore.command2` database type.
# Additional Changes
Ensure correct order of setup steps 64 and 70.
# Additional Context
None
---------
Co-authored-by: Livio Spring <[email protected]>
# Which Problems Are Solved
The updated default denylist added an entry for IPv4-mapped IPv6
addresses to prevent IPv6 encapsulation bypasses.
This is not necessary since the IP already gets resolved into v4 and now
blocks them all.
# How the Problems Are Solved
Removed the entry.
# Additional Changes
None
# Additional Context
None
# Which Problems Are Solved
Enable FIPS 140-3 compliant build.
# How the Problems Are Solved
- Add runtime config validation, if the FIPS flag is enabled fail the
application when a non-compliant hasher is used, or throw a warning when
a legacy verifier is used
- Add a build matrix for FIPS certified build:
- Go binary is built with `GOFIPS140=certified`
- Login container uses a separate base:
[ubi9](https://catalog.redhat.com/en/software/containers/ubi9/ubi/615bcf606feffc5384e8452e)
from redhat which provides a FIPS certified OpenSSL (used by NodeJS TLS
stack)
- Non-FIPS images where already pushed to both Github Container Registry
and Google Artifact Repository (GAR). Fips images are only pushed to the
GAR.
- Tag versions are suffixed `-fips`. So on release the following images
will be additionally available:
```
europe-docker.pkg.dev/zitadel-common/zitadel-repo/zitadel-login:vX.Y.Z-fips
europe-docker.pkg.dev/zitadel-common/zitadel-repo/zitadel:vX.Y.Z-fips-debug
europe-docker.pkg.dev/zitadel-common/zitadel-repo/zitadel:vX.Y.Z-fips
```
# Other changes
- Bumb Go toolchain. At least v1.25.10 is required for a
GOFIPS140=certified setting.
# Additional Context
- Closes https://github.com/zitadel/zitadel/issues/4335
- Build [test
run](https://github.com/zitadel/zitadel/actions/runs/27253916052)
pushing FIPS and non-FIPS images
# Which Problems Are Solved
- The eventstore did not support intentionally overwriting the resource
owner when creating events for aggregates that may be reused across
owners.
- Resource owner handling was implicit and could not be controlled per
command/event type.
- We needed a safe way to distinguish between:
- keeping the existing aggregate owner, and
- explicitly setting a new owner for specific create-like events.
# How the Problems Are Solved
- Introduced a new eventstore command type with an explicit
enforce_owner flag.
- Updated eventstore.commands_to_events and eventstore.push so owner
assignment is now explicit:
- if enforce_owner is true, the command owner is written
- if enforce_owner is false, the existing aggregate owner is retained
when present
- Added EnforceResourceOwnerCommand and wiring so command types can opt
in to enforced owner behavior.
- Wired the new behavior through the v3 eventstore push path, including
compatibility fallback for older command type mapping.
- Added migration/setup changes to register and use the new command type
and SQL functions.
- Added and updated tests for owner overwrite and aggregate ID reuse
scenarios.
# Additional Changes
- Added small migration/setup robustness improvements related to
eventstore setup ordering and helper reuse.
- Added focused test coverage for enforced owner behavior and
sequencing.
- Events that currently allow owner changes (implement
EnforceResourceOwner) are:
- AddedEvent (action)
- GroupAddedEvent
- StartedEvent (idp intent)
- ProjectAddedEvent
- HumanAddedEvent
- HumanRegisteredEvent
- MachineAddedEvent
- CreatedEvent (schema user)
# Additional Context
- Follow-up for eventstore owner-handling correctness in create flows
and aggregate ID reuse cases.
- No additional issue link was attached for this change.
---------
Co-authored-by: abhishek kumar gupta <[email protected]>
# Which Problems Are Solved
- Upgrading to `zitadel/passwap` v0.12.1 introduced new encoded-hash
validation paths that still had review feedback open.
- Secret hasher defaults were internally inconsistent (`Hasher.Cost: 4`
vs `Limits.Bcrypt.MinCost: 10`), which could reject hashes created by
the configured hasher.
- New validation error IDs/messages and test coverage needed to be
aligned with project conventions and expected behavior branches.
# How the Problems Are Solved
- Kept the dependency upgrade to `zitadel/passwap` v0.12.1 and completed
the validation integration.
- Updated `ValidateEncodedHash` error handling in
`internal/crypto/passwap.go` to:
- use unique random-style error IDs,
- return `Errors.Hash.NotSupported` for no-verifier cases,
- keep invalid-hash branches mapped to invalid argument errors.
- Expanded `TestHasher_ValidateEncodedHash` in
`internal/crypto/passwap_test.go` to cover and assert:
- bounds error branch,
- no-verifier branch,
- generic invalid-hash branch,
- expected ZITADEL error IDs/messages.
- Restored lost inline verifier-context comments for argon2 and md5plain
verifier entries.
# Additional Changes
- Added the missing explanatory `Limits` comment for `SecretHasher` in
`cmd/defaults.yaml`.
- Corrected `SecretHasher.Limits.Bcrypt.MinCost` from `10` to `4` to
match the configured default bcrypt cost and avoid configuration
footguns.
# Additional Context
- Follow-up for PR review feedback in
https://github.com/zitadel/zitadel/pull/12179#pullrequestreview-4313121965
---------
Co-authored-by: copilot-swe-agent[bot] <[email protected]>
Co-authored-by: Livio Spring <[email protected]>
# Which Problems Are Solved
Opaque tokens now use authenticated encryption.
# How the Problems Are Solved
- Upgrade zitadel/oidc to v3.47
- Copy crypto implementation for refresh and session tokens (internal to
zitadel)
- Added config that allows validating old tokens for gradual roll-out
# Additional Changes
- Set NX cache for `integration-test-build` to `false`, working on a
seperate fix.
# Additional Context
- closes#11315
---------
Co-authored-by: copilot-swe-agent[bot] <[email protected]>
Co-authored-by: muhlemmer <[email protected]>
# Which Problems Are Solved
Add OTP Email challenge needed for Create/Set session in the RT model.
# How the Problems Are Solved
Introduce `OTPEmailChallengeCommand` (validate/execute/events) for
session creation checks
Add unit tests
# Additional Changes
Remove unused fields from `session_challenge_otp_sms.go`
# Additional Context
- Related to https://github.com/zitadel/zitadel/issues/11035
---------
Co-authored-by: Marco A. <[email protected]>
# Which Problems Are Solved
https://github.com/zitadel/zitadel/pull/11390 renamed "Console" to
"Management Console". While
https://github.com/zitadel/zitadel/pull/11706 already reverted an
unintended rename of the feature key to enable the management console to
use the V2 API for user creation. It was now also discovered that the
rename of the feature itself also broke existing (default)
configurations.
# How the Problems Are Solved
Added a `mapstructure` tag on the instance feature to handle existing
configs.
# Additional Changes
Removed unused `TokenExchange` from the default configuration.
# Additional Context
- relates to #11390
- relates to #11706
- requires backport to v4.x
---------
Co-authored-by: Copilot <[email protected]>
Co-authored-by: Marco A. <[email protected]>
# Which Problems Are Solved
As part of #11035 , this PR implements the Passkey check logic for
session validation
# How the Problems Are Solved
- Refactor webauth FinishLogin to support new domain model
- Add webauth config to defaults
- Implement passkey check logic and tests
- Manual transaction management to avoid stalling the DB while
FinishLogin callback is executed
- Update passkey Type condition to allow passing a text operation
(equal, contains, etc..)
# Additional Context
This is a cherry-picked PR + minor changes, coming from
https://github.com/zitadel/zitadel/pull/11164
- Relates to #11035
---------
Co-authored-by: Fabienne Bühler <[email protected]>
Co-authored-by: Gayathri Vijayan <[email protected]>
# Which Problems Are Solved
This PR adds the OTP SMS Challenge needed for CreateSession API
https://github.com/zitadel/zitadel/issues/11035
# How the Problems Are Solved
- Introduce `OTPSMSChallengeCommand` (validate/execute/events) for
session creation checks
- Introduce `GetOTPCryptoGeneratorConfigWithDefault` to fetch
per-instance secret generator settings
- Add unit tests
# Additional Changes
Refactor tests in `session_check_recovery_code_test.go` to make the
helper functions reusable in `session_challenge_otp_sms_test.go`
# Additional Context
Related to https://github.com/zitadel/zitadel/issues/11035
# Which Problems Are Solved
System API users currently authenticate using raw RSA public keys
configured via Path or KeyData. This approach doesn't integrate well
with Kubernetes tooling.
# How the Problems Are Solved
Allow for the `path`/`keyData` to be an X.509 certificate.
The `NotBefore` and `NotAfter` fields of the certificate are beeing
respected when validating the JWT.
# Additional Changes
# Additional Context
- Closes#11442
---------
Co-authored-by: Livio Spring <[email protected]>
# Which Problems Are Solved
As part of moving the session API to the relational database, this PR
adds the functionality to delete sessions through the session API v2 and
the OIDC end_session endpoint.
# How the Problems Are Solved
- added `SessionDeleteCommand` in domain package
- added possibility to pass the session token verifier to the domain
package
- use feature flag to switch API and OIDC usage from CQRS to relation
table
- added a `permissionCheck` condition to let permissions be checked
directly in the repository / sql
- part of it is still a placeholder until actual permission tables are
done
- added a `session_deleted` table to store (recently) deleted sessions
to handle necessary checks / cases like delete my own session, which
needs to be idempotent
# Additional Changes
Fixed `Matches` function of `existsCondition` (as missing an
implementation)
# Additional Context
- closes#11037
---------
Co-authored-by: Stefan Benz <[email protected]>
## Problem description
- Setup currently creates/updates relational tables, but there is no
built-in way to fully reset the relational schema during iterative
development.
- Re-running setup after schema/projection changes can leave stale
relational objects and projection state behind, which makes local/dev
validation harder.
- There is no explicit, configurable switch in setup steps for
destructive schema recreation behavior.
## How the Problems Are Solved
- Adds a new setup step configuration section for relational tables with
a `ShouldRecreateSchema` flag (default `false`).
- Wires the new config flag into the transactional tables setup step.
- Extends the transactional tables execution logic to optionally:
- Drop the `zitadel` schema with `CASCADE`
- Clean related projection state entries for relational tables in
`projections.current_states`
- Recreate tables through the existing setup flow afterward
- Wraps the destructive operations in a transaction and keeps
error/rollback handling plus logging for visibility.
## Additional Changes
- Adds an explicit warning in step configuration that schema recreation
is intended for development and not production use.
- Keeps behavior fully backward-compatible by default
(`ShouldRecreateSchema: false`), so existing setups are unchanged unless
the flag is enabled.
## Additional Context
- Follow-up for relational setup/dev workflow improvements.
- PR: #11833
## How to use it
### Env `export ZITADEL_RELATIONALTABLES_SHOULDRECREATESCHEMA=true`
### Config
Add the following to your custom setup steps config:
```yaml
RelationalTables:
ShouldRecreateSchema: true
```
# Which Problems Are Solved
As part of #11035 , this PR implements the IDP intent check logic for
session validation
# How the Problems Are Solved
- Inject encryption algorithm for IDP intent tokens at DB initialization
- Add the IDP intent repository as an invoke option
- Rework crypto mocks using gomock
- Implement IDP intent check logic
- minor - DB: expand IDP Intent deletion testing table with a test case
where IDP intent is not found
# Noteworthy
I did not implement `domain.Transactional` given the doubts that
@adlerhurst expressed in https://github.com/zitadel/zitadel/pull/11804 .
So, there is no transaction AT ALL in here because the `Execute()`
method contains one single DB call (Delete) and `Validate()` just does
Get calls.
# Additional Context
This is a cherry-picked PR + minor changes, coming from
https://github.com/zitadel/zitadel/pull/11164
- Relates to #11035
- Depends on https://github.com/zitadel/zitadel/pull/11804
# Which Problems Are Solved
The total number of configured IDPs seemed of.
We were counting only the IDP table for non-templated IDPs.
# How the Problems Are Solved
Add a count trigger migration for the IDP template table.
Entries will be counted under the existing
`ResourceCountIdentityProvider`
# Additional Changes
- none
# Additional Context
- Reported internally
- Implemented in #9979
- Related https://github.com/zitadel/zitadel/issues/9957
Co-authored-by: Livio Spring <[email protected]>
# Which Problems Are Solved
As part of #11035 , this PR implements the password check logic for
session validation
# How the Problems Are Solved
- Add system config to default configuration of `domain` package for
easy initialization. Intialize the system settings when Zitadel starts
up
- Add password hasher verify logic to the default configuration of
`domain` package. Initialize it when Zitadel starts up.
- Add settings repositories with their mocks
- Implement the logic for doing a password check
# Additional Context
This is a cherry-picked PR + minor changes, coming from
https://github.com/zitadel/zitadel/pull/11164
- Relates to #11035
- Depends on https://github.com/zitadel/zitadel/pull/11777
# Which Problems Are Solved
Panics may cause a service discruption by unexpectedly closing an
request's connection. Or in the case of gRPC completely killing the
service.
Allthough panics are still individual bugs that need to be solved, this
PR makes sure a panic is gracefully handled and an understandable error
is returned to the client.
# How the Problems Are Solved
- Recover in the middleware interceptors for the 3 API protocols (HTTP,
gRPC, connect).
- HTTP middleware uses formatted responses for:
- OIDC errors (JSON formatted response)
- UI (error page rendering)
- SCIM
- Upon recovery an alert level log is printed (ERROR+4 for stdlib log
handlers)
# Additional Context
- internal observation
---------
Co-authored-by: Livio Spring <[email protected]>
Co-authored-by: Livio Spring <[email protected]>
Users deploying ZITADEL against a managed PostgreSQL service (RDS, Cloud
SQL, Azure Database, etc.) often do not have superuser access and cannot
provide `Admin.*` credentials. The documented workaround — provisioning
the user and database manually and then running `start-from-setup` —
silently skips schema bootstrapping, causing `relation
"eventstore.events" does not exist` errors with no clear recovery path.
The root cause is that `zitadel init` conflates two steps that require
different privileges without exposing them separately:
- **Provisioning step** (`CREATE ROLE`, `CREATE DATABASE`, `GRANT`) —
requires superuser.
- **Schema bootstrapping step** (create
`eventstore`/`projections`/`system` schemas and base tables) — requires
only DB owner.
Users who handle the provisioning step externally have no supported way
to run schema bootstrapping alone.
## Changes
- **`cmd/initialise/verify_schema.go`** (renamed from
`verify_zitadel.go`): Rename `newZitadel()` → `newSchema()` (internal);
rename the `init zitadel` sub-command to `zitadel init schema`
(backwards-compatible alias kept) with a clear description that it
bootstraps the ZITADEL database schema without admin/superuser
privileges. Fix stale error log message to reference `init schema`.
- **`cmd/initialise/verify_schema_test.go`** (renamed from
`verify_zitadel_test.go`): Test file renamed to match source file.
- **`cmd/initialise/init.go`**: Update call site to `newSchema()`. Add
guidance in the `init` command's Long description about using `zitadel
init schema` for users without admin credentials.
- **`cmd/initialise/verify_database.go`**: Add a `pg_database` catalog
pre-check before attempting `CREATE DATABASE`, so `zitadel init` with
`ADMIN=service_user` no longer fails with `permission denied to create
database` when the database was already provisioned externally.
- **`cmd/initialise/verify_database_test.go`**: Add test cases covering
the new catalog-check skip path, the existing error-skip path, and the
error-propagation path when the `pg_database` query itself fails.
- **`apps/docs/content/self-hosting/manage/database/index.mdx`**: Inline
the `_postgres.mdx` partial (now only PostgreSQL is supported), add a
top-level callout, and add a **Managed PostgreSQL / No Admin Access**
section with explicit 3-step instructions and security guidance (strong
passwords, SSL, TLS). Additional improvements: add inline `# Use
'require' or 'verify-full' for production` comments on `Mode: disable`
lines in the YAML example; add clarifying comment to the redundant
`GRANT` in the SQL snippet; replace admonition syntax with proper
Fumadocs `<Callout>` components; add full database connection env vars
to the `start-from-setup` example.
- **`apps/docs/content/self-hosting/manage/updating_scaling.mdx`**:
Rewrite the init phase description to clearly distinguish the
provisioning and schema bootstrapping steps, and document `zitadel init
schema` as the path for manual provisioning. Replace admonition syntax
with proper Fumadocs `<Callout>` components.
- **`apps/docs/content/self-hosting/manage/database/_postgres.mdx`**:
Deleted (content merged into `index.mdx`).
## Problem
This relates to https://github.com/zitadel/zitadel/discussions/9363
I think we can improve our UX in cases where a user wants to use an
external DB and/or does not want to share too broad permissions with
zitadel
## Related problems
* https://github.com/zitadel/zitadel/issues/10432
* https://github.com/zitadel/zitadel/discussions/8583
* https://github.com/zitadel/zitadel/issues/7903
* https://github.com/zitadel/zitadel/issues/9718
* https://github.com/zitadel/zitadel/issues/8012
* https://github.com/zitadel/zitadel/issues/8558
## Related PRs
https://github.com/zitadel/zitadel/pull/11021
# Which Problems Are Solved
If the `cache.objects`-table does not exist setup failes with the
following error:
`ERROR: relation "cache.objects" does not exist at character 44`
# How the Problems Are Solved
Changed statement to prevent immediately throw an error if the table is
missing.
# Additional Context
- reported in
https://discord.com/channels/927474939156643850/1473769282880934089
- backport to v4
# Which Problems Are Solved
When running Zitadel behind a reverse proxy and especially when the API
and the login UI don't run on the same domain, Zitadel needs to be
configured to trust the corresponding domains and use them in public
responses, like email links and more.
This can be done by adding a trusted domain. However it's currently only
possible through the API and not in the instance setup process.
# How the Problems Are Solved
Added a possibility to configure multiple trusted domains in the first
instance setup process.
# Additional Changes
None
# Additional Context
- closes#11153
# Which Problems Are Solved
As part of the consistent naming effort, this PR focuses on
"Organization domain".
# How the Problems Are Solved
- All terms referring referring to Organization Domains where changed to
be Organization Domain
# Additional Changes
None
# Additional Context
- closes [#11283](https://github.com/zitadel/zitadel/issues/11283)
---------
Co-authored-by: Copilot <[email protected]>
# Which Problems Are Solved
naming inconsistencies: Configuration <-> Settings
# How the Problems Are Solved
# Additional Changes
# Additional Context
- Closes#11282
# Which Problems Are Solved
Group attributes weren't masked if their corresponding key was
configured. For example setting `data` as a masked key entry in the
runtime config would still print unmasked event data. This was because
ReplaceAttr does not receive group attributes, only the flattened
attributes with group information.
# How the Problems Are Solved
Check the current groups stack in the replacer. When a configured key is
found in the group, mask the current attribute. This means that the
attribute structure is preserved and all sub-keys of a masked group are
still printed.
# Additional Changes
- Use a binary search for key matching. As we are now comparing slice
against slice, a binary search should make the process a little faster
if many keys are configured.
# Additional Context
- Follow-up for PR #11435
---------
Co-authored-by: Copilot <[email protected]>
Co-authored-by: Copilot <[email protected]>
Co-authored-by: muhlemmer <[email protected]>
# Which Problems Are Solved
There were still some emails (passkey registration and domain claimed)
sent with links pointing to login v1 even when the login v2 was enabled
for the instance.
Also while looking into the issue, it was discovered that some links
pointing to login V2 were not correctly generated.
# How the Problems Are Solved
- Added default paths for passkey registration and domain claimed
notifications
- Fixed the existing paths to properly handle concatenation (resp. use
`url.ResolveReference`)
- Change their go types (from string) to `*url.URL`
- Added a mapstructure hook for string to url
- Removed unnecessary `InstanceSetupFeatures` and corresponding
conversions
- Refactored the methods on the `login.DefaultPaths` struct and added an
interface to the `Commands` to only need to pass a single config (and
not every method)
- Added an `OriginURL` method to the `DomainCtx` to prevent going from
url to string and back
- Added the use of the templates in case of enabled login v2 for passkey
registration and domain claimed)
# Additional Changes
None
# Additional Context
closes#10643
---------
Co-authored-by: Max Peintner <[email protected]>
Co-authored-by: Livio Spring <[email protected]>
Co-authored-by: Livio Spring <[email protected]>
Co-authored-by: Max Peintner <[email protected]>
Co-authored-by: Gayathri Vijayan <[email protected]>
# Which Problems Are Solved
Inconsistent naming of service account, found in the following
variations:
- Machine User
- machine user
- Service User
- Machine Account
- Technical Account
- User: Type Machine
# How the Problems Are Solved
Attentive search and replace.
Localizations have been translated using Copilot
# Additional Changes
Some unused methods have been removed from the Go code.
# Additional Context
- Closes#11285
# Which Problems Are Solved
Naming inconsistencies - User (Human)
# How the Problems Are Solved
Most of the occurrences have not been changed. `User (Human)` was mostly
changed when talking about code objects and where I felt it was
necessary to distinguish them from machine users.
When both human and machine user occurrences were found, the machine
user has been changed to service account (see
https://github.com/zitadel/zitadel/issues/11285)
# Additional Context
- Closes#11284
---------
Co-authored-by: Wim Van Laer <[email protected]>
# Which Problems Are Solved
This PR marks the OIDC Back-Channel Logout as general available (GA).
# How the Problems Are Solved
- API:
- deprecated the feature toggle (to prevent breaking changes) in v2beta
and v2 API
- removed all related event logic and updated the projection iteration
- removed the toggle usage for back-channel logout
- removed related translations
- Management Console:
- removed the feature toggle and its translations
- Docs:
- removed all beta labels and feature toggle notes
# Additional Changes
none
# Additional Context
- closes#11277
- requires backport to v4.x
## Which problems are solved
https://github.com/zitadel/zitadel/pull/11484 introduced a regression
that causes the setup process to fail on existing Zitadel deployments.
This prevents users from upgrading to recent versions without
encountering setup failures.
## How the problems are solved
Setup step 69 has been corrected to properly handle existing deployment
configurations and prevent setup failures during initialization and
upgrades.
## Additional Context
introduced by
[7a41fe968b](https://github.com/zitadel/zitadel/commit/7a41fe968b9fcb69b378336740b74e5448c1ff81)
### Testing
- [x] Verified setup succeeds on fresh deployments
- [x] Verified setup succeeds on existing deployments
- [x] Verified migration from PostgreSQL 17 to 18 after running setup of
this version
Tim MöhlmannandGitHub
jmaretteandGitHub
Mridang AgarwallaandGitHub
Florian ForsterandGitHub
