# Which Problems Are Solved
This PR adds implementation to add a Zitadel IdP at the
organization-level.
# How the Problems Are Solved
- Added handling/converters for the `AddZitadelProvider` endpoint in
`ManagementService` in the server layer
- Registered a new `org.idp.zitadel.added` event for org-level Zitadel
providers
- Added `AddOrgZitadelProvider` command to validate the request and push
`org.idp.zitadel.added` event to the eventstore
- Added the `org.idp.zitadel.added` event to the projection reducer
- Added unit and integration tests
# Additional Changes
added more tests for the ZitadelProvider in AdminService
# Additional Context
- Closes https://github.com/zitadel/zitadel/issues/11823
- Follow-up for PRs https://github.com/zitadel/zitadel/pull/12018,
https://github.com/zitadel/zitadel/pull/12020,
https://github.com/zitadel/zitadel/pull/12055
# Which Problems Are Solved
Persisting instance-level Zitadel provider in the projections.
# How the Problems Are Solved
- Defining a new projection table `projections.idp_templates6_zitadel`
and columns for the Zitadel provider
- Adding a reducer `reduceZitadelIDPAdded` to process instance-level
`ZitadelIDPAddedEvent`
- Unit tests
# Additional Changes
N/A
# Additional Context
- Related to https://github.com/zitadel/zitadel/issues/11823
- Follow-up for PRs https://github.com/zitadel/zitadel/pull/12018,
https://github.com/zitadel/zitadel/pull/12020
This update introduces a new table for managing administrator role
permissions, along with the necessary repository and projection logic to
handle adding and removing permissions. It also includes tests to ensure
the correct functionality of the permission management system.
closes https://github.com/zitadel/zitadel/issues/10219
---------
Co-authored-by: Marco Ardizzone <[email protected]>
# Which Problems Are Solved
A test was failing because of timezone flakiness
# How the Problems Are Solved
The actual problem was an `updatedAt` field was not filled in because of
the `createdAt` was missing on an event. This caused a zero value to be
filled in into the `updatedAt` field. Postgres then stored this value
with seconds in it's timezone info, which go could not store.
By adding filling in the `createdAt` field, this does not occur.
# Which Problems Are Solved
In https://github.com/zitadel/zitadel/pull/11858#discussion_r2993742914
we discussed on how to translate the ES passkey state to a relational
one. That is used to filter out passkeys during convertion to webauthn
credentials.
It was decided to consider passkeys in ready state those that have a
verified at `time.Time` set.
The change required fixing the reducer for the eventstore event.
# How the Problems Are Solved
- Set `VerifiedAt` once `HumanWebAuthNVerifiedEvent` is received
- Also set the verification to `nil`
# Additional Context
- Discussion #11858
- Follow-up for PR #11858
# Which Problems Are Solved
As part of #11035 , this PR implements the Passkey check logic for
session validation
# How the Problems Are Solved
- Refactor webauth FinishLogin to support new domain model
- Add webauth config to defaults
- Implement passkey check logic and tests
- Manual transaction management to avoid stalling the DB while
FinishLogin callback is executed
- Update passkey Type condition to allow passing a text operation
(equal, contains, etc..)
# Additional Context
This is a cherry-picked PR + minor changes, coming from
https://github.com/zitadel/zitadel/pull/11164
- Relates to #11035
---------
Co-authored-by: Fabienne Bühler <[email protected]>
Co-authored-by: Gayathri Vijayan <[email protected]>
# Which Problems Are Solved
- Recovery codes added via `GenerateRecoveryCodes` were never listed as
active authentication methods by `ListAuthenticationMethodTypes`. The
`user_auth_method` projection reducer (`reduceInitAuthMethod`) set the
state to `MFAStateNotReady`, expecting a subsequent verification event
to transition it to `MFAStateReady`. Unlike TOTP, U2F, and Passwordless
— which all have distinct verification events — recovery codes have no
such event, so they were permanently stuck in `NotReady`.
# How the Problems Are Solved
- Introduced a dedicated `reduceAddRecoveryCodes` reducer for
`HumanRecoveryCodesAddedEvent` that sets `MFAStateReady` immediately,
since recovery codes are usable as soon as they are generated.
- Uses `NewUpsertStatement` (instead of `NewCreateStatement`) so that
regenerating recovery codes for the same user updates the existing row
rather than failing on a duplicate key conflict.
- Removed the `HumanRecoveryCodesAddedEvent` case from
`reduceInitAuthMethod`.
# Additional Changes
- Added `UserAuthMethodTypeRecoveryCode` mapping in `authMethodTypeToPb`
unit test.
- Added integration tests for `ListAuthenticationMethodTypes` covering
both the "recovery codes present" and "recovery codes removed" cases.
# Additional Context
* https://github.com/zitadel/zitadel/issues/11317
---------
Co-authored-by: Marco A. <[email protected]>
Co-authored-by: Cursor <[email protected]>
# Which Problems Are Solved
As part of moving the session API to the relational database, this PR
adds the functionality to delete sessions through the session API v2 and
the OIDC end_session endpoint.
# How the Problems Are Solved
- added `SessionDeleteCommand` in domain package
- added possibility to pass the session token verifier to the domain
package
- use feature flag to switch API and OIDC usage from CQRS to relation
table
- added a `permissionCheck` condition to let permissions be checked
directly in the repository / sql
- part of it is still a placeholder until actual permission tables are
done
- added a `session_deleted` table to store (recently) deleted sessions
to handle necessary checks / cases like delete my own session, which
needs to be idempotent
# Additional Changes
Fixed `Matches` function of `existsCondition` (as missing an
implementation)
# Additional Context
- closes#11037
---------
Co-authored-by: Stefan Benz <[email protected]>
Introduces the `administrators` relational table and repository layer
for the v3 backend, unifying instance, organization, project, and
project-grant membership into a single scoped model.
Closes#10220
## Changes
- **Domain**: `Administrator` struct, `AdministratorScope` enum,
`AdministratorRepository` interface with typed CRUD, column, condition,
and change helpers. Generated enumer and GoMock.
- **Migration** (`016`): `zitadel.administrators` table with a generated
`id`, scope-alignment check constraint, cascading foreign keys, and a
child `administrator_roles` table.
- **Repository**: CTE-based role mutations (`AddRole`, `RemoveRole`,
`SetRoles`) for atomicity. Roles aggregated via `ARRAY_AGG`.
- **Projection**: Event reducers for
`member.added/changed/removed/cascade_removed` across all four admin
scopes.
- **Fixes**: `Rows.Close()` error wrapping; `TextArray` nil-safe scan +
full `pgtype` array interface; `authorization` and `projectGrant` scan
helpers fixed for `ARRAY_AGG` under `database/sql`; removed unused `ctx`
parameter from `callReduce`.
---------
Co-authored-by: Marco Ardizzone <[email protected]>
## Problem description
During tests we found concurrency issues in the relational table
projections. Tests got flaky if a table (e.g. organization domains) had
too many foreign keys to tables which were projected by another handler.
## Resolution
The projections of the relational tables got merged into one handler to
enforce sequential reduction of events.
## Additional info
- closes https://github.com/zitadel/zitadel/issues/11703
- closes https://github.com/zitadel/zitadel/issues/10810
---------
Co-authored-by: Marco Ardizzone <[email protected]>
# Which Problems Are Solved
Add integration tests for recovery codes event reducers in the RT model.
# How the Problems Are Solved
Added integration tests for recovery codes event reducers
# Additional Changes
N/A
# Additional Context
- Closes https://github.com/zitadel/zitadel/issues/11707
- Todo: add tests for other reducers in the `users_relational`
projection.
---------
Co-authored-by: Livio Spring <[email protected]>
- Transitioned from using `handler.Statements` to a repository pattern
for managing organization data in migrations.
- Introduced a new method to set the updated timestamp for
organizations.
- Updated the projection logic to utilize the repository for creating,
updating, and deleting organization records.
# Which Problems Are Solved
This PR provides an example for testing DB reduction to relational
tables without exploiting integration tests.
# How the Problems Are Solved
- Set up a disposable embedded Postgres DB used to store the results of
the calls to the reducers and verify such results
- Add utility method to return transactions from the existing DB pool.
These transactions can be used by the single test cases to insert and
read data from DB
- Add a test to verify that the handler contains the expected reducers
(checking also that they are on the right aggregate type)
- Add tests to verify the single reduce functions. A parent test creates
the required objects to satisfy the foreign key constraints.
# Additional Changes
- Typo fixed in `postgres.DecodeConfig()`
- Discard unneeded `projectionName` parameter for `Executor` function
# Additional Context
- Closes#11430
# Which Problems Are Solved
Adds a repository implementation to add/remove recovery codes to the
users relational table.
# How the Problems Are Solved
This is achieved by:
* adding the following columns to the users table: `recovery_codes`,
`recovery_code_last_successful_check`, `recovery_code_failed_attempts`
* adding the `RecoveryCodes` field to the `HumanUser` domain with fields
to set recovery `codes`, `lastSuccessfullyCheckedAt` timestamp, and
`failedAttempts`
* setting `recoveryCodes` in the `Get` user query statement to return a
json object with details related to the recovery codes
* adding the repository-layer implementation to add/remove recovery
codes and set fields related to recovery code checks.
* adding projection reducers to handle the following events:
`HumanRecoveryCodesAddedEvent`, `HumanRecoveryCodesRemovedEvent`,
`HumanRecoveryCodeCheckSucceededEvent`, and
`HumanRecoveryCodeCheckFailedEvent`
* adding unit tests
# Additional Changes
* fix the error message when the recovery is empty in
`internal/command/user_human_recovery_codes.go`
* add a new error message for empty recovery code during checks in
`en.yaml`
# Additional Context
- Closes https://github.com/zitadel/zitadel/issues/11212
- Follow-up: integration tests for the reducers will be added in a
different PR after this
[PR](https://github.com/zitadel/zitadel/pull/11478) is merged
## Summary
This PR introduces the relational user repository for backend v3,
including schema, repository APIs, reducers/projections, and tests for
both human and machine users.
## What changed
* Added user-focused relational migrations (users, verifications,
metadata, PATs, machine keys, passkeys, identity provider links) and
related FK wiring.
* Implemented/expanded repository domain contracts and storage
implementations for human and machine user flows.
* Added support for machine secret handling and aligned access token
type modeling.
* Extended user domain models (verifications, password/email/phone/TOTP
state, passkeys, metadata, invite handling, identity provider links).
* Updated projection/reduction coverage and integration tests for key
user lifecycle events.
* Regenerated and updated mocks/enums plus supporting database
change/column utilities.
## Notes for reviewers
* This PR is large because it establishes the foundational relational
user storage layer and related test scaffolding. Sorry for that.
## Closes
Closes#10218Closes#10224Closes#10225Closes#10209Closes#10802
---------
Co-authored-by: Iraq Jaber <[email protected]>
Co-authored-by: Copilot <[email protected]>
Co-authored-by: Marco A. <[email protected]>
Co-authored-by: Tim Möhlmann <[email protected]>
# Which Problems Are Solved
After a user is deactivated, their sessions are still valid. When the
user is redirected to login page, they're stuck in a loop on a blank
page on the `users/me` endpoint as the user's session is still valid.
# How the Problems Are Solved
By terminating a deactivated user's session. This is done by:
- Adding a new reducer in session projection to delete sessions for
`UserDeactivatedEvent`, `UserLockedEvent` and `UserRemovedEvent` events
- Adding a new reducer in the relation session projection to delete
sessions based on `UserDeactivatedEvent`, `UserLockedEvent` events
# Additional Changes
N/A
# Additional Context
- Closes https://github.com/zitadel/zitadel/issues/10307
- todo: add integration test cases for relational session reducers based
on user deactivated, locked, deleted events
# Which Problems Are Solved
In many cases IAM was used where instance was meant.
# How the Problems Are Solved
- Rename wrong usage of IAM to instance all over the code base.
- Add an agents.md which explains the terms used in Zitadel's
architecture.
# Additional Context
- closes#11280
---------
Co-authored-by: Copilot <[email protected]>
Co-authored-by: muhlemmer <[email protected]>
Co-authored-by: Copilot <[email protected]>
Co-authored-by: Marco A. <[email protected]>
# Which Problems Are Solved
This PR marks the OIDC Back-Channel Logout as general available (GA).
# How the Problems Are Solved
- API:
- deprecated the feature toggle (to prevent breaking changes) in v2beta
and v2 API
- removed all related event logic and updated the projection iteration
- removed the toggle usage for back-channel logout
- removed related translations
- Management Console:
- removed the feature toggle and its translations
- Docs:
- removed all beta labels and feature toggle notes
# Additional Changes
none
# Additional Context
- closes#11277
- requires backport to v4.x
# Which Problems Are Solved
In the context of moving our datastore to relational tables, Identity
Provider Intents have no definition of a relational table nor any DB API
that can be used to store the IDP intents as relational objects.
# How the Problems Are Solved
The PR follows the guidelines left in #11190
- A table `zitadel.identity_provider_intents` is created: it defines its
relational constraints (foreign keys) with the other tables, as well as
an ENUM to represent the state of the intent.
- An IDP Intent repository has been created, which offers CRUD
operations implemented as `Create`, `Get`, `Update` and `Delete` methods
for the IDP intent.
- The repository also defines all columns and conditions useful for
performing DB operations on the IDP intents.
- DB mocks have been added as well to ease testing on the `domain`
package.
- Reduction functions have been implemented for the existing IDP events
so to keep the relational table in sync with the event store.
# Additional Changes
To allow testing LDAP intent flow, the `Session.FetchUser()` method has
been extracted to a separate file. A corresponding mock has been created
as well in `fetch_user_mock.go`. The files are mutually excluded and
added at build time through go build tags: the mock is added when the
`integration` tag is specified, while the real implementation is added
when the same tag IS NOT added to the build tags.
# Additional Context
- Closes#11190
---------
Co-authored-by: Livio Spring <[email protected]>
Co-authored-by: Silvan <[email protected]>
# Which Problems Are Solved
This PR marks the OAuth2 Token Exchange as general available (GA).
# How the Problems Are Solved
- API:
- deprecated the feature toggle (to prevent breaking changes) in v2beta
and v2 API
- removed all related event logic and updated the projection interation
- removed the toggle usage for token exchange
- removed related translations
- Management Console:
- removed the feature toggle and its translations
- Docs:
- removed all beta labels and feature toggle notes
# Additional Changes
- removed unused `Errors.WebKey.FeatureDisabled` translations
# Additional Context
- closes#11114
---------
Co-authored-by: Marco A. <[email protected]>
# Which Problems Are Solved
- Some mail providers (e.g.: MS Exchange) have deprecated plain auth
# How the Problems Are Solved
Adds XOauth2 auth option to SMPT
- SMTP config now can contain XOAuth2 auth config
- Proto files are updated to have a oneof for configuration selection.
Old fields for configuring plain auth are still available not to create
a breaking change. These can be removed in the future.
- Columns are added to the database to persist the config
# Additional Changes
none
# Additional Context
Replace this example with links to related issues, discussions, discord
threads, or other sources with more context.
Use the Closing #issue syntax for issues that are resolved with this PR.
- Closes#8042
---------
Co-authored-by: Copilot <[email protected]>
Co-authored-by: Tim Möhlmann <[email protected]>
# Which Problems Are Solved
The following terms have all been renamed to management console:
- Customer Portal (when used to mean the console)
- Console
- Admin Console
# How the Problems Are Solved
- Search & Replace smartly
- Use Copilot for translation files
Changes done to: backend + frontend codebase, docs, translations and
protobufs (descriptions only)
# Additional Context
- Partially Closes#11279
# Which Problems Are Solved
User with older instances, created at version 3.3.3 or earlier may have
been affected with feature flags no longer applying. Older version of
Zitadel had a bug in console where toggling a single feature would apply
a state to all features, creating around 12 events each request. Then 4
feature flags and their event types were removed in v4, meaning that the
`offset` value in the projection's current state could be wrong by 4 if
the instance was affected by the above bug.
# How the Problems Are Solved
Increment the feature projection versions, so that the tables get
recomputed and the current states entries becomes accurate again.
# Additional Changes
- none
# Additional Context
- Original bug: https://github.com/zitadel/zitadel/issues/10459
- Reported through support
# Which Problems Are Solved
As part of going to a relation storage model, we need to implement the
repository for handling sessions and their events.
# How the Problems Are Solved
- Adds a migration step to create the necessary tables and types in the
database.
- Adds the repository implementation for sessions.
- Implements a projection handler for session events to handle the
events written by the old command handlers.
- Updates the `Session` type and `SessionRepository` interface:
- added `CreatorID`
- Adjusted `Delete` method to return amount of affected columns
- Removed sub-repository clumns from the main interface
- Added `CreatorIDCondition`, `ExpirationCondition`, `ExistsFactor` and
`ExistsMetadata`
- Removed `SetUserAgent`
- Methods of `SessionFactor` and `SessionChallenge` have been made
public
- fixed enumer on `SessionFactorType`
- Adds a `JSON` type to the repository package to easy can JSON columns,
resp. jsonb object inside a property of a `JSONArray` object.
- Adds a `NoChange` interface, which can be implemented to signal that a
`Change` interface implementation is not writing any changes to the
statement builder, e.g. `CTEChange`
# Additional Changes
- Fixes the operators enums. `TextOperations` had wrong comments and
`NumberOperations` additionally also had from comparisons.
- Fixes downgrade of migration step 008.
# Additional Context
- closes https://github.com/zitadel/zitadel/issues/10212
# Which Problems Are Solved
The payload in actions V2 is currently sent as JSON to the target
endpoint. It might get exposed to intermediary infrastructure or logging
systems.
For these scenarios there needs to be an application-layer encryption,
where the provider of the endpoint can define an key to be used for the
encryption.
# How the Problems Are Solved
- Added an additional option to the target to specify the payload type:
`JSON` (current and default), `JWT`, `JWE` (api and console)
- added endpoints to upload and manage public keys (to be used for
encryption) for a target
- updated all action v2 executions (interceptors, oidc, saml, ...) to
provide the `GetActiveSigningWebKey` from queries
- implemented jwt and jwe in the exections incl. refactoring of code and
tests
- changes to the authn_keys table:
- added a `fingerprint` column
- dropped not null constraint on expiration
- moved the `GetSignerOnce` into its own package to prevent circular
dependencies
# Additional Changes
None
# Additional Context
closes#11061
---------
Co-authored-by: Marco A. <[email protected]>
Co-authored-by: conblem <[email protected]>
Co-authored-by: Silvan <[email protected]>
# Which Problems Are Solved
This PR implements relational table and repository interface for the
authorizations feature as described in
https://github.com/zitadel/zitadel/issues/10767
# How the Problems Are Solved
By
* adding new `zitadel.authorizations` and `zitadel.authorization_roles`
tables
* implementing the `AuthorizationRepository` interface to create,
update, query, and delete user authorizations and roles
* implementing `authorization_relational.go` projection to update
`zitadel.authorizations` and `zitadel.authorization_roles` tables based
on events
# Additional Changes
N/A
# Additional Context
- Closes#10767
---------
Co-authored-by: Livio Spring <[email protected]>
# Which Problems Are Solved
The `SecretGenerators` setting is missing among the instance settings in
the new relational model.
# How the Problems Are Solved
This PR adds:
* `SettingTypeSecretGenerator` type to the settings model
* `SecretGeneratorSettingsRepository` and
`secretGeneratorSettingsJSONChanges` interfaces to set, retrieve, and
delete `secret_generator` settings, and corresponding unit tests.
* projection reducers to reduce the following events:
* `SecretGeneratorAddedEvent`
* `SecretGeneratorChangedEvent`
* `SecretGeneratorRemovedEvent`
* integration tests to test that the events get reduced correctly
# Additional Changes
N/A
# Additional Context
- Part of https://github.com/zitadel/zitadel/issues/10207
---------
Co-authored-by: Copilot <[email protected]>
Co-authored-by: Marco A. <[email protected]>
# Which Problems Are Solved
`IS NULL` and `IS NOT NULL` conditions were incorrectly marked as
restricting columns,
and currently fails unit tests. The restricting columns are mostly used
to ensure single row selection during DELETE and UPDATE . When a `IS NOT
NULL` condition is passed, the filter may (unintentionally) apply to all
rows, while still allowing the query to execute.
Settings and other repositories (still to be done) have unified tables
for instance-level settings and organization-level settings. In those
cases the organization ID may be null or have a value. The unique
condition is a compound of multiple columns, in which case `IS NULL` is
valid as part of the other restricting columns.
# How the Problems Are Solved
In order to prevent unsafe usage of null related filters and not break
settings:
- null conditions are no longer considered restricting columns
- A function to force the restricting column state on a condition is
provided
- The settings unique condition function forces restricting column state
on the returned condition.
# Additional Changes
- add `backend/v3` to the NX build.
# Additional Context
- [pipeline
failures](https://github.com/zitadel/zitadel/actions/runs/19852295780/job/56881654340?pr=11115#step:13:3197)
---------
Co-authored-by: Elio Bischof <[email protected]>
# Which Problems Are Solved
As part of https://github.com/zitadel/zitadel/issues/9599 to push to
move to a relational db table model, this is the migration of the
settings
# Additional Changes
I had to add 2 non source code files
`backend/v3/storage/database/events_testing/font.otf` and
`backend/v3/storage/database/events_testing/picture.png`, these are used
as part of the reduce tests, hence why they're sourced inside
`backend/v3/storage/database/events_testing/settings_instance_test.go`
- Closes https://github.com/zitadel/zitadel/issues/10207
---------
Co-authored-by: adlerhurst <[email protected]>
Co-authored-by: Copilot <[email protected]>
Co-authored-by: Tim Möhlmann <[email protected]>
Co-authored-by: Stefan Benz <[email protected]>
<!--
Please inform yourself about the contribution guidelines on submitting a
PR here:
https://github.com/zitadel/zitadel/blob/main/CONTRIBUTING.md#submit-a-pull-request-pr.
Take note of how PR/commit titles should be written and replace the
template texts in the sections below. Don't remove any of the sections.
It is important that the commit history clearly shows what is changed
and why.
Important: By submitting a contribution you agree to the terms from our
Licensing Policy as described here:
https://github.com/zitadel/zitadel/blob/main/LICENSING.md#community-contributions.
-->
One-time recovery codes are a common multi-factor authentication (MFA)
backup method, letting users access their account if they lose other MFA
devices. Support for recovery codes can also reduce support burden for
users locked out of their accounts and provide a more secure and
reliable form of verification than security questions.
# Which Problems Are Solved
Zitadel currently lacks support for recovery codes.
# How the Problems Are Solved
This PR partially addresses recovery code support in Zitadel.
Importantly, it adds recovery codes as a new 2FA `Factor` and an
additional `Check` type for the Session API.
```
Example recovery code flow:
1. User generates N new recovery codes using `POST /v2/users/{user_id}/recovery_codes`
2. Zitadel hashes and stores these codes and returns the un-hashed codes in the response
3. User creates new session with an additional check: `recoveryCode`
4. Code is checked against hash and, if valid, cannot be used again
5. User attempts to adds N more codes using the same endpoint
6. If `remaining_codes + N <= RecoveryCodes->MaxCount` config value, then recovery codes are added in addition to original codes
7. User can remove all recovery codes using `DEL /v2/users/:userId/recovery_codes`
```
This PR adds:
- [x] Session recovery_code check support on `POST+PATCH /v2/sessions`
endpoints
- [x] Adds `mfa_recovery_code_checked_at` column (default null) to
`projections.sessions8` table
- [x] Support for `SECOND_FACTOR_TYPE_RECOVERY_CODES` as available 2FA
method on login policy
- [x] Support for importing recovery codes in /import code
Missing, will _not_ implement in this PR:
- [ ] Admin console support for displaying Recovery Code settings for
user(s)
- [ ] Zitadel Typescript login support for recovery codes
TODO:
- [x] Additional unit and integration tests
- [x] Error translations
# Additional Changes
None
# Additional Context
- Closes#6898
---------
Co-authored-by: Livio Spring <[email protected]>
This pull request introduces a new feature that allows adding, updating,
and querying project grants The changes are primarily in the backend and
include new database tables, repositories, and domain logic to support
project grants.
## Changes
* New `project_grants` table: A new table `zitadel.project_grants` is
introduced to store project grants It includes columns for
`instance_id`, `id`, `state`, `project_id`, `granted_org_id` and
`granting_org_id`.
* New `ProjectGrantRepository`: A new repository
`ProjectGrantRepository` is created to handle all database operations
for project grants. It provides methods to `Get`, `List`, `Create`,
`Update`, and `Delete`.
* New `project_grant_roles` table: A new table
`zitadel.project_grant_roles` is introduced to store project grant roles
It includes columns for `instance_id`, `grant_id`, `key`, `project_id`
and `project_org_id`.
* New `ProjectGrantRoleRepository`: A new repository
`ProjectGrantRoleRepository` is created to handle all database
operations for project grant roles. It provides methods to `Get`,
`List`, `Add`, and `Delete`.
* New `project_grant_relational_projection`: A new projection
`project_grant_relational_projection` is added to update the
`zitadel.project_grants` and `zitadel.project_grant_roles` tables based
on events. It handles all project grant events.
* Updated `ProjectRepository`: The `ProjectRepository` is updated to
list granted projects.
# Additional Context
Closes https://github.com/zitadel/zitadel/issues/10766
---------
Co-authored-by: Stefan Benz <[email protected]>
Co-authored-by: Marco A. <[email protected]>
Co-authored-by: Silvan <[email protected]>
# Which Problems Are Solved
If the groups of a user is requested through a scope, details related to
the user's groups are added in the token claims.
# How the Problems Are Solved
By querying the user groups that the user is a part of, and setting them
in the token claims.
This PR introduces two scopes to request user groups: `groups` and
`urn:zitadel:iam:user:groups`
* `groups` returns an array of group names in the token claims with the
same name
* `urn:zitadel:iam:user:groups` returns an array of JSON objects with
information about the user groups in the token claims with the same name
# Additional Changes
N/A
# Additional Context
- Closes#9702
- Follow-up for PRs:
* #10455
* #10853
* #10758
* #10940
---------
Co-authored-by: Livio Spring <[email protected]>
# Which Problems Are Solved
1. Adding users to user groups and removing users from user groups.
2. Searching for users in user groups by group IDs or user IDs
# How the Problems Are Solved
By adding:
1. The API definitions to manage users in users groups
3. The command-layer implementation of adding users/removing users
to/from user groups.
4. The projection table group_users1
5. Query-side implementation to search for users in user groups
# Additional Changes
1. Remove debug statements from unit tests.
2. Fix removal of groups when orgs are removed
3. Add unit tests for groups projection
# Additional Context
* Related to #9702
* Follow-up for PRs
* https://github.com/zitadel/zitadel/pull/10455
* https://github.com/zitadel/zitadel/pull/10758
* https://github.com/zitadel/zitadel/pull/10853
# Which Problems Are Solved
Relational table's primary keys are part of the SQL table definition.
However, by abstraction through the repository interfaces callers
(service layer) have no clear understanding of the primary key definion.
This might lead to confusion during implementation:
- The repository makes a sanity check on required conditions before
query execution and returns an error if the condition was not met.
- When a Get method is executed, an error is returned if not exactly one
row is found. Get replaces the legacy GetXxxByID methods we have on
projections. However, a list of flexible conditions can be passed.
- Operations like UPDATE and DELETE should often only affect a single
row, identified by the primary key. If the primary key conditions are
missing an error is returned.
Because there is no "type safety" enforced here, the errors are returned
during runtime. When all possible combinations of API arguments are not
properly tested such errors are actually bugs in the experience of the
end-user. (The API should do it's own input validation, but a
programmer's error is easily made).
# How the Problems Are Solved
Add `PrimaryKeyColumns` and `PrimaryKeyCondition` to existing repository
intefaces.
# Additional Changes
- Where reducers already used repositories, the PrimaryKeyConditions are
used where possible.
# Additional Context
- Part of
https://github.com/zitadel/zitadel/wiki/Decision-Log#expose-primary-key-definition-in-repository-interfaces
---------
Co-authored-by: Marco A. <[email protected]>
This pull request introduces a new feature that allows adding, updating,
and querying metadata for organizations. The changes are primarily in
the backend and include new database tables, repositories, and domain
logic to support organization metadata.
## Changes
* New `org_metadata` table: A new table `zitadel.org_metadata` is
introduced to store metadata for organizations. It includes columns for
`instance_id`, `org_id`, `key`, and `value`. The `value` is stored as a
`BYTEA` type to allow for flexible data storage.
* New `OrganizationMetadataRepository`: A new repository
`OrganizationMetadataRepository` is created to handle all database
operations for organization metadata. It provides methods to `Get`,
`List`, `Set`, and `Remove` metadata.
* New `org_metadata_relational_projection`: A new projection
`org_metadata_relational_projection` is added to update the
`zitadel.org_metadata` table based on events. It handles `MetadataSet`,
`MetadataRemoved`, and `MetadataRemovedAll` events.
* Updated `OrganizationRepository`: The `OrganizationRepository` is
updated to support loading organization metadata. A new method
`LoadMetadata` is added to enable joining the `org_metadata` table when
querying for organizations.
* Updated Organization domain: The Organization domain model is updated
to include a new field `Metadata` of type `[]*OrganizationMetadata`.
## Additional Info
* Extensible Design: The new metadata feature is designed to be
extensible, allowing for future enhancements such as indexing on
specific JSON fields within the `value` column.
* closes https://github.com/zitadel/zitadel/issues/10206
* closes https://github.com/zitadel/zitadel/issues/10214
# Which Problems Are Solved
This is the second PR related to the backend implementation of
GroupService to manage user groups.
The first [PR](https://github.com/zitadel/zitadel/pull/10455) implements
the Command-side.
This PR implements the query side.
# How the Problems Are Solved
* Query-side implementation to search/list groups by
* a list of Group IDs
* by the Group name
* by the Organization ID
# Additional Changes
N/A
# Additional Context
- Follow-up for PR #10455
---------
Co-authored-by: Livio Spring <[email protected]>
# Which Problems Are Solved
Add projects to the relational tables
# How the Problems Are Solved
- Define table migrations
- Define and implement Project and Project Role repositories.
- Provide projection handlers to populate the relational tables.
# Additional Changes
- Statement Builder now has a constructor which allows setting of a base
query with arguments.
- Certain operations, like Get, Update and Delete require the Primary
Key to be set as conditions. However, this requires knowledge of the
implementation and table definition. This PR proposes an additional
condition for repositories: `PrimaryKeyCondition`. This gives clarity on
the required IDs for these operations.
- Added couple of helpers to the repository package, to allow more DRY
code.
- getOne / getMany: generic functions for query execution and scanning.
- checkRestrictingColumns, checkPkCondition: simplify condition
checking, instead of using ladders of conditionals.
- Added a couple of helpers to the repository test package:
- Transaction, savepoint and rollback helpers.
- Create instance and organization helpers for objects that depend on
them (like projects).
# Additional Context
- after https://github.com/zitadel/zitadel/pull/10809
- closes#10765
This pull request introduces a significant refactoring of the database
interaction layer, focusing on improving explicitness, transactional
control, and error handling. The core change is the removal of the
stateful `QueryExecutor` from repository instances. Instead, it is now
passed as an argument to each method that interacts with the database.
This change makes transaction management more explicit and flexible, as
the same repository instance can be used with a database pool or a
specific transaction without needing to be re-instantiated.
### Key Changes
- **Explicit `QueryExecutor` Passing:**
- All repository methods (`Get`, `List`, `Create`, `Update`, `Delete`,
etc.) in `InstanceRepository`, `OrganizationRepository`,
`UserRepository`, and their sub-repositories now require a
`database.QueryExecutor` (e.g., a `*pgxpool.Pool` or `pgx.Tx`) as the
first argument.
- Repository constructors no longer accept a `QueryExecutor`. For
example, `repository.InstanceRepository(pool)` is now
`repository.InstanceRepository()`.
- **Enhanced Error Handling:**
- A new `database.MissingConditionError` is introduced to enforce
required query conditions, such as ensuring an `instance_id` is always
present in `UPDATE` and `DELETE` operations.
- The database error wrapper in the `postgres` package now correctly
identifies and wraps `pgx.ErrTooManyRows` and similar errors from the
`scany` library into a `database.MultipleRowsFoundError`.
- **Improved Database Conditions:**
- The `database.Condition` interface now includes a
`ContainsColumn(Column) bool` method. This allows for runtime checks to
ensure that critical filters (like `instance_id`) are included in a
query, preventing accidental cross-tenant data modification.
- A new `database.Exists()` condition has been added to support `EXISTS`
subqueries, enabling more complex filtering logic, such as finding an
organization that has a specific domain.
- **Repository and Interface Refactoring:**
- The method for loading related entities (e.g., domains for an
organization) has been changed from a boolean flag (`Domains(true)`) to
a more explicit, chainable method (`LoadDomains()`). This returns a new
repository instance configured to load the sub-resource, promoting
immutability.
- The custom `OrgIdentifierCondition` has been removed in favor of using
the standard `database.Condition` interface, simplifying the API.
- **Code Cleanup and Test Updates:**
- Unnecessary struct embeddings and metadata have been removed.
- All integration and repository tests have been updated to reflect the
new method signatures, passing the database pool or transaction object
explicitly.
- New tests have been added to cover the new `ExistsDomain`
functionality and other enhancements.
These changes make the data access layer more robust, predictable, and
easier to work with, especially in the context of database transactions.
# Which Problems Are Solved
Depending on the metadata values (already existing), the newly created
index (#10415) cannot be created or error in the future.
# How the Problems Are Solved
- Create the index using `sha256` and change the query to use sha256 as
well when comparing bytes values such as user_metadata.
- Added a setup step to cleanup potentially created index on
`projections.user_metadata5`
# Additional Changes
None
# Additional Context
- relates to #10415
- requires backport to v4.x
# Which Problems Are Solved
HTTP Request to HTTP providers for Email or SMS are not signed.
# How the Problems Are Solved
Add a Signing Key to the HTTP Provider resources, which is then used to
generate a header to sign the payload.
# Additional Changes
Additional tests for query side of the SMTP provider.
# Additional Context
Closes#10067
---------
Co-authored-by: Marco A. <[email protected]>
This PR initiates the rework of Zitadel's backend to state-persisted
objects. This change is a step towards a more scalable and maintainable
architecture.
## Changes
* **New `/backend/v3` package**: A new package structure has been
introduced to house the reworked backend logic. This includes:
* `domain`: Contains the core business logic, commands, and repository
interfaces.
* `storage`: Implements the repository interfaces for database
interactions with new transactional tables.
* `telemetry`: Provides logging and tracing capabilities.
* **Transactional Tables**: New database tables have been defined for
`instances`, `instance_domains`, `organizations`, and `org_domains`.
* **Projections**: New projections have been created to populate the new
relational tables from the existing event store, ensuring data
consistency during the migration.
* **Repositories**: New repositories provide an abstraction layer for
accessing and manipulating the data in the new tables.
* **Setup**: A new setup step for `TransactionalTables` has been added
to manage the database migrations for the new tables.
This PR lays the foundation for future work to fully transition to
state-persisted objects for these components, which will improve
performance and simplify data access patterns.
This PR initiates the rework of ZITADEL's backend to state-persisted
objects. This is a foundational step towards a new architecture that
will improve performance and maintainability.
The following objects are migrated from event-sourced aggregates to
state-persisted objects:
* Instances
* incl. Domains
* Orgs
* incl. Domains
The structure of the new backend implementation follows the software
architecture defined in this [wiki
page](https://github.com/zitadel/zitadel/wiki/Software-Architecturel).
This PR includes:
* The initial implementation of the new transactional repositories for
the objects listed above.
* Projections to populate the new relational tables from the existing
event store.
* Adjustments to the build and test process to accommodate the new
backend structure.
This is a work in progress and further changes will be made to complete
the migration.
---------
Co-authored-by: Copilot <[email protected]>
Co-authored-by: Iraq Jaber <[email protected]>
Co-authored-by: Iraq <[email protected]>
Co-authored-by: Tim Möhlmann <[email protected]>
This PR overhauls our event projection system to make it more robust and
prevent skipped events under high load. The core change replaces our
custom, transaction-based locking with standard PostgreSQL advisory
locks. We also introduce a worker pool to manage concurrency and prevent
database connection exhaustion.
### Key Changes
* **Advisory Locks for Projections:** Replaces exclusive row locks and
inspection of `pg_stat_activity` with PostgreSQL advisory locks for
managing projection state. This is a more reliable and standard approach
to distributed locking.
* **Simplified Await Logic:** Removes the complex logic for awaiting
open transactions, simplifying it to a more straightforward time-based
filtering of events.
* **Projection Worker Pool:** Implements a worker pool to limit
concurrent projection triggers, preventing connection exhaustion and
improving stability under load. A new `MaxParallelTriggers`
configuration option is introduced.
### Problem Solved
Under high throughput, a race condition could cause projections to miss
events from the eventstore. This led to inconsistent data in projection
tables (e.g., a user grant might be missing). This PR fixes the
underlying locking and concurrency issues to ensure all events are
processed reliably.
### How it Works
1. **Event Writing:** When writing events, a *shared* advisory lock is
taken. This signals that a write is in progress.
2. **Event Handling (Projections):**
* A projection worker attempts to acquire an *exclusive* advisory lock
for that specific projection. If the lock is already held, it means
another worker is on the job, so the current one backs off.
* Once the lock is acquired, the worker briefly acquires and releases
the same *shared* lock used by event writers. This acts as a barrier,
ensuring it waits for any in-flight writes to complete.
* Finally, it processes all events that occurred before its transaction
began.
### Additional Information
* ZITADEL no longer modifies the `application_name` PostgreSQL variable
during event writes.
* The lock on the `current_states` table is now `FOR NO KEY UPDATE`.
* Fixes https://github.com/zitadel/zitadel/issues/8509
---------
Co-authored-by: Copilot <[email protected]>
Co-authored-by: Tim Möhlmann <[email protected]>
# Which Problems Are Solved
This PR introduces a new feature flag `EnableRelationalTables` that will
be used in following implementations to decide whether Zitadel should
use the relational model or the event sourcing one.
# TODO
- [x] Implement flag at system level
- [x] Display the flag on console:
https://github.com/zitadel/zitadel/pull/10615
# How the Problems Are Solved
- Implement loading the flag from config
- Add persistence of the flag through gRPC endpoint
(SetInstanceFeatures)
- Implement reading of the flag through gRPC endpoint
(GetInstanceFeatures)
# Additional Changes
Some minor refactoring to remove un-needed generics annotations
# Additional Context
- Closes#10574
---------
Co-authored-by: Silvan <[email protected]>
# Which Problems Are Solved
Some users have reported the need of retrieving users given a metadata
key, metadata value or both. This change introduces metadata search
filter on the `ListUsers()` endpoint to allow Zitadel users to search
for user records by metadata.
The changes affect only v2 APIs.
# How the Problems Are Solved
- Add new search filter to `ListUserRequest`: `MetaKey` and `MetaValue`
- Add SQL indices on metadata key and metadata value
- Update query to left join `user_metadata` table
# Additional Context
- Closes#9053
- Depends on https://github.com/zitadel/zitadel/pull/10567
---------
Co-authored-by: Silvan <[email protected]>
Co-authored-by: Stefan Benz <[email protected]>
# Which Problems Are Solved
The event execution system currently uses a projection handler that
subscribes to and processes all events for all instances. This creates a
high static cost because the system over-fetches event data, handling
many events that are not needed by most instances. This inefficiency is
also reflected in high "rows returned" metrics in the database.
# How the Problems Are Solved
Eliminate the use of a project handler. Instead, events for which
"execution targets" are defined, are directly pushed to the queue by the
eventstore. A Router is populated in the Instance object in the authz
middleware.
- By joining the execution targets to the instance, no additional
queries are needed anymore.
- As part of the instance object, execution targets are now cached as
well.
- Events are queued within the same transaction, giving transactional
guarantees on delivery.
- Uses the "insert many fast` variant of River. Multiple jobs are queued
in a single round-trip to the database.
- Fix compatibility with PostgreSQL 15
# Additional Changes
- The signing key was stored as plain-text in the river job payload in
the DB. This violated our [Secrets
Storage](https://zitadel.com/docs/concepts/architecture/secrets#secrets-storage)
principle. This change removed the field and only uses the encrypted
version of the signing key.
- Fixed the target ordering from descending to ascending.
- Some minor linter warnings on the use of `io.WriteString()`.
# Additional Context
- Introduced in https://github.com/zitadel/zitadel/pull/9249
- Closes https://github.com/zitadel/zitadel/issues/10553
- Closes https://github.com/zitadel/zitadel/issues/9832
- Closes https://github.com/zitadel/zitadel/issues/10372
- Closes https://github.com/zitadel/zitadel/issues/10492
---------
Co-authored-by: Stefan Benz <[email protected]>
# Which Problems Are Solved
When a SAML IDP is created, the signing algorithm defaults to
`RSA-SHA1`.
This PR adds the functionality to configure the signing algorithm while
creating or updating a SAML IDP. When nothing is specified, `RSA-SHA1`
is the default.
Available options:
* RSA_SHA1
* RSA_SHA256
* RSA_SHA512
# How the Problems Are Solved
By introducing a new optional config to specify the Signing Algorithm.
# Additional Changes
N/A
# Additional Context
- Closes#9842
An existing bug in the UpdateSAMLProvider API will be fixed as a
followup in a different
[PR](https://github.com/zitadel/zitadel/pull/10557).
---------
Co-authored-by: Stefan Benz <[email protected]>
Livio SpringandGitHub
Gayathri VijayanandGitHub
Wim Van LaerandGitHub
