# Which Problems Are Solved
In many cases IAM was used where instance was meant.
# How the Problems Are Solved
- Rename wrong usage of IAM to instance all over the code base.
- Add an agents.md which explains the terms used in Zitadel's
architecture.
# Additional Context
- closes#11280
---------
Co-authored-by: Copilot <[email protected]>
Co-authored-by: muhlemmer <[email protected]>
Co-authored-by: Copilot <[email protected]>
Co-authored-by: Marco A. <[email protected]>
# Which Problems Are Solved
This PR marks the OIDC Back-Channel Logout as general available (GA).
# How the Problems Are Solved
- API:
- deprecated the feature toggle (to prevent breaking changes) in v2beta
and v2 API
- removed all related event logic and updated the projection iteration
- removed the toggle usage for back-channel logout
- removed related translations
- Management Console:
- removed the feature toggle and its translations
- Docs:
- removed all beta labels and feature toggle notes
# Additional Changes
none
# Additional Context
- closes#11277
- requires backport to v4.x
# Which Problems Are Solved
In the context of moving our datastore to relational tables, Identity
Provider Intents have no definition of a relational table nor any DB API
that can be used to store the IDP intents as relational objects.
# How the Problems Are Solved
The PR follows the guidelines left in #11190
- A table `zitadel.identity_provider_intents` is created: it defines its
relational constraints (foreign keys) with the other tables, as well as
an ENUM to represent the state of the intent.
- An IDP Intent repository has been created, which offers CRUD
operations implemented as `Create`, `Get`, `Update` and `Delete` methods
for the IDP intent.
- The repository also defines all columns and conditions useful for
performing DB operations on the IDP intents.
- DB mocks have been added as well to ease testing on the `domain`
package.
- Reduction functions have been implemented for the existing IDP events
so to keep the relational table in sync with the event store.
# Additional Changes
To allow testing LDAP intent flow, the `Session.FetchUser()` method has
been extracted to a separate file. A corresponding mock has been created
as well in `fetch_user_mock.go`. The files are mutually excluded and
added at build time through go build tags: the mock is added when the
`integration` tag is specified, while the real implementation is added
when the same tag IS NOT added to the build tags.
# Additional Context
- Closes#11190
---------
Co-authored-by: Livio Spring <[email protected]>
Co-authored-by: Silvan <[email protected]>
# Which Problems Are Solved
This PR marks the OAuth2 Token Exchange as general available (GA).
# How the Problems Are Solved
- API:
- deprecated the feature toggle (to prevent breaking changes) in v2beta
and v2 API
- removed all related event logic and updated the projection interation
- removed the toggle usage for token exchange
- removed related translations
- Management Console:
- removed the feature toggle and its translations
- Docs:
- removed all beta labels and feature toggle notes
# Additional Changes
- removed unused `Errors.WebKey.FeatureDisabled` translations
# Additional Context
- closes#11114
---------
Co-authored-by: Marco A. <[email protected]>
# Which Problems Are Solved
- Some mail providers (e.g.: MS Exchange) have deprecated plain auth
# How the Problems Are Solved
Adds XOauth2 auth option to SMPT
- SMTP config now can contain XOAuth2 auth config
- Proto files are updated to have a oneof for configuration selection.
Old fields for configuring plain auth are still available not to create
a breaking change. These can be removed in the future.
- Columns are added to the database to persist the config
# Additional Changes
none
# Additional Context
Replace this example with links to related issues, discussions, discord
threads, or other sources with more context.
Use the Closing #issue syntax for issues that are resolved with this PR.
- Closes#8042
---------
Co-authored-by: Copilot <[email protected]>
Co-authored-by: Tim Möhlmann <[email protected]>
# Which Problems Are Solved
The following terms have all been renamed to management console:
- Customer Portal (when used to mean the console)
- Console
- Admin Console
# How the Problems Are Solved
- Search & Replace smartly
- Use Copilot for translation files
Changes done to: backend + frontend codebase, docs, translations and
protobufs (descriptions only)
# Additional Context
- Partially Closes#11279
# Which Problems Are Solved
User with older instances, created at version 3.3.3 or earlier may have
been affected with feature flags no longer applying. Older version of
Zitadel had a bug in console where toggling a single feature would apply
a state to all features, creating around 12 events each request. Then 4
feature flags and their event types were removed in v4, meaning that the
`offset` value in the projection's current state could be wrong by 4 if
the instance was affected by the above bug.
# How the Problems Are Solved
Increment the feature projection versions, so that the tables get
recomputed and the current states entries becomes accurate again.
# Additional Changes
- none
# Additional Context
- Original bug: https://github.com/zitadel/zitadel/issues/10459
- Reported through support
# Which Problems Are Solved
As part of going to a relation storage model, we need to implement the
repository for handling sessions and their events.
# How the Problems Are Solved
- Adds a migration step to create the necessary tables and types in the
database.
- Adds the repository implementation for sessions.
- Implements a projection handler for session events to handle the
events written by the old command handlers.
- Updates the `Session` type and `SessionRepository` interface:
- added `CreatorID`
- Adjusted `Delete` method to return amount of affected columns
- Removed sub-repository clumns from the main interface
- Added `CreatorIDCondition`, `ExpirationCondition`, `ExistsFactor` and
`ExistsMetadata`
- Removed `SetUserAgent`
- Methods of `SessionFactor` and `SessionChallenge` have been made
public
- fixed enumer on `SessionFactorType`
- Adds a `JSON` type to the repository package to easy can JSON columns,
resp. jsonb object inside a property of a `JSONArray` object.
- Adds a `NoChange` interface, which can be implemented to signal that a
`Change` interface implementation is not writing any changes to the
statement builder, e.g. `CTEChange`
# Additional Changes
- Fixes the operators enums. `TextOperations` had wrong comments and
`NumberOperations` additionally also had from comparisons.
- Fixes downgrade of migration step 008.
# Additional Context
- closes https://github.com/zitadel/zitadel/issues/10212
# Which Problems Are Solved
The payload in actions V2 is currently sent as JSON to the target
endpoint. It might get exposed to intermediary infrastructure or logging
systems.
For these scenarios there needs to be an application-layer encryption,
where the provider of the endpoint can define an key to be used for the
encryption.
# How the Problems Are Solved
- Added an additional option to the target to specify the payload type:
`JSON` (current and default), `JWT`, `JWE` (api and console)
- added endpoints to upload and manage public keys (to be used for
encryption) for a target
- updated all action v2 executions (interceptors, oidc, saml, ...) to
provide the `GetActiveSigningWebKey` from queries
- implemented jwt and jwe in the exections incl. refactoring of code and
tests
- changes to the authn_keys table:
- added a `fingerprint` column
- dropped not null constraint on expiration
- moved the `GetSignerOnce` into its own package to prevent circular
dependencies
# Additional Changes
None
# Additional Context
closes#11061
---------
Co-authored-by: Marco A. <[email protected]>
Co-authored-by: conblem <[email protected]>
Co-authored-by: Silvan <[email protected]>
# Which Problems Are Solved
This PR implements relational table and repository interface for the
authorizations feature as described in
https://github.com/zitadel/zitadel/issues/10767
# How the Problems Are Solved
By
* adding new `zitadel.authorizations` and `zitadel.authorization_roles`
tables
* implementing the `AuthorizationRepository` interface to create,
update, query, and delete user authorizations and roles
* implementing `authorization_relational.go` projection to update
`zitadel.authorizations` and `zitadel.authorization_roles` tables based
on events
# Additional Changes
N/A
# Additional Context
- Closes#10767
---------
Co-authored-by: Livio Spring <[email protected]>
# Which Problems Are Solved
The `SecretGenerators` setting is missing among the instance settings in
the new relational model.
# How the Problems Are Solved
This PR adds:
* `SettingTypeSecretGenerator` type to the settings model
* `SecretGeneratorSettingsRepository` and
`secretGeneratorSettingsJSONChanges` interfaces to set, retrieve, and
delete `secret_generator` settings, and corresponding unit tests.
* projection reducers to reduce the following events:
* `SecretGeneratorAddedEvent`
* `SecretGeneratorChangedEvent`
* `SecretGeneratorRemovedEvent`
* integration tests to test that the events get reduced correctly
# Additional Changes
N/A
# Additional Context
- Part of https://github.com/zitadel/zitadel/issues/10207
---------
Co-authored-by: Copilot <[email protected]>
Co-authored-by: Marco A. <[email protected]>
# Which Problems Are Solved
`IS NULL` and `IS NOT NULL` conditions were incorrectly marked as
restricting columns,
and currently fails unit tests. The restricting columns are mostly used
to ensure single row selection during DELETE and UPDATE . When a `IS NOT
NULL` condition is passed, the filter may (unintentionally) apply to all
rows, while still allowing the query to execute.
Settings and other repositories (still to be done) have unified tables
for instance-level settings and organization-level settings. In those
cases the organization ID may be null or have a value. The unique
condition is a compound of multiple columns, in which case `IS NULL` is
valid as part of the other restricting columns.
# How the Problems Are Solved
In order to prevent unsafe usage of null related filters and not break
settings:
- null conditions are no longer considered restricting columns
- A function to force the restricting column state on a condition is
provided
- The settings unique condition function forces restricting column state
on the returned condition.
# Additional Changes
- add `backend/v3` to the NX build.
# Additional Context
- [pipeline
failures](https://github.com/zitadel/zitadel/actions/runs/19852295780/job/56881654340?pr=11115#step:13:3197)
---------
Co-authored-by: Elio Bischof <[email protected]>
# Which Problems Are Solved
As part of https://github.com/zitadel/zitadel/issues/9599 to push to
move to a relational db table model, this is the migration of the
settings
# Additional Changes
I had to add 2 non source code files
`backend/v3/storage/database/events_testing/font.otf` and
`backend/v3/storage/database/events_testing/picture.png`, these are used
as part of the reduce tests, hence why they're sourced inside
`backend/v3/storage/database/events_testing/settings_instance_test.go`
- Closes https://github.com/zitadel/zitadel/issues/10207
---------
Co-authored-by: adlerhurst <[email protected]>
Co-authored-by: Copilot <[email protected]>
Co-authored-by: Tim Möhlmann <[email protected]>
Co-authored-by: Stefan Benz <[email protected]>
<!--
Please inform yourself about the contribution guidelines on submitting a
PR here:
https://github.com/zitadel/zitadel/blob/main/CONTRIBUTING.md#submit-a-pull-request-pr.
Take note of how PR/commit titles should be written and replace the
template texts in the sections below. Don't remove any of the sections.
It is important that the commit history clearly shows what is changed
and why.
Important: By submitting a contribution you agree to the terms from our
Licensing Policy as described here:
https://github.com/zitadel/zitadel/blob/main/LICENSING.md#community-contributions.
-->
One-time recovery codes are a common multi-factor authentication (MFA)
backup method, letting users access their account if they lose other MFA
devices. Support for recovery codes can also reduce support burden for
users locked out of their accounts and provide a more secure and
reliable form of verification than security questions.
# Which Problems Are Solved
Zitadel currently lacks support for recovery codes.
# How the Problems Are Solved
This PR partially addresses recovery code support in Zitadel.
Importantly, it adds recovery codes as a new 2FA `Factor` and an
additional `Check` type for the Session API.
```
Example recovery code flow:
1. User generates N new recovery codes using `POST /v2/users/{user_id}/recovery_codes`
2. Zitadel hashes and stores these codes and returns the un-hashed codes in the response
3. User creates new session with an additional check: `recoveryCode`
4. Code is checked against hash and, if valid, cannot be used again
5. User attempts to adds N more codes using the same endpoint
6. If `remaining_codes + N <= RecoveryCodes->MaxCount` config value, then recovery codes are added in addition to original codes
7. User can remove all recovery codes using `DEL /v2/users/:userId/recovery_codes`
```
This PR adds:
- [x] Session recovery_code check support on `POST+PATCH /v2/sessions`
endpoints
- [x] Adds `mfa_recovery_code_checked_at` column (default null) to
`projections.sessions8` table
- [x] Support for `SECOND_FACTOR_TYPE_RECOVERY_CODES` as available 2FA
method on login policy
- [x] Support for importing recovery codes in /import code
Missing, will _not_ implement in this PR:
- [ ] Admin console support for displaying Recovery Code settings for
user(s)
- [ ] Zitadel Typescript login support for recovery codes
TODO:
- [x] Additional unit and integration tests
- [x] Error translations
# Additional Changes
None
# Additional Context
- Closes#6898
---------
Co-authored-by: Livio Spring <[email protected]>
This pull request introduces a new feature that allows adding, updating,
and querying project grants The changes are primarily in the backend and
include new database tables, repositories, and domain logic to support
project grants.
## Changes
* New `project_grants` table: A new table `zitadel.project_grants` is
introduced to store project grants It includes columns for
`instance_id`, `id`, `state`, `project_id`, `granted_org_id` and
`granting_org_id`.
* New `ProjectGrantRepository`: A new repository
`ProjectGrantRepository` is created to handle all database operations
for project grants. It provides methods to `Get`, `List`, `Create`,
`Update`, and `Delete`.
* New `project_grant_roles` table: A new table
`zitadel.project_grant_roles` is introduced to store project grant roles
It includes columns for `instance_id`, `grant_id`, `key`, `project_id`
and `project_org_id`.
* New `ProjectGrantRoleRepository`: A new repository
`ProjectGrantRoleRepository` is created to handle all database
operations for project grant roles. It provides methods to `Get`,
`List`, `Add`, and `Delete`.
* New `project_grant_relational_projection`: A new projection
`project_grant_relational_projection` is added to update the
`zitadel.project_grants` and `zitadel.project_grant_roles` tables based
on events. It handles all project grant events.
* Updated `ProjectRepository`: The `ProjectRepository` is updated to
list granted projects.
# Additional Context
Closes https://github.com/zitadel/zitadel/issues/10766
---------
Co-authored-by: Stefan Benz <[email protected]>
Co-authored-by: Marco A. <[email protected]>
Co-authored-by: Silvan <[email protected]>
# Which Problems Are Solved
If the groups of a user is requested through a scope, details related to
the user's groups are added in the token claims.
# How the Problems Are Solved
By querying the user groups that the user is a part of, and setting them
in the token claims.
This PR introduces two scopes to request user groups: `groups` and
`urn:zitadel:iam:user:groups`
* `groups` returns an array of group names in the token claims with the
same name
* `urn:zitadel:iam:user:groups` returns an array of JSON objects with
information about the user groups in the token claims with the same name
# Additional Changes
N/A
# Additional Context
- Closes#9702
- Follow-up for PRs:
* #10455
* #10853
* #10758
* #10940
---------
Co-authored-by: Livio Spring <[email protected]>
# Which Problems Are Solved
1. Adding users to user groups and removing users from user groups.
2. Searching for users in user groups by group IDs or user IDs
# How the Problems Are Solved
By adding:
1. The API definitions to manage users in users groups
3. The command-layer implementation of adding users/removing users
to/from user groups.
4. The projection table group_users1
5. Query-side implementation to search for users in user groups
# Additional Changes
1. Remove debug statements from unit tests.
2. Fix removal of groups when orgs are removed
3. Add unit tests for groups projection
# Additional Context
* Related to #9702
* Follow-up for PRs
* https://github.com/zitadel/zitadel/pull/10455
* https://github.com/zitadel/zitadel/pull/10758
* https://github.com/zitadel/zitadel/pull/10853
# Which Problems Are Solved
Relational table's primary keys are part of the SQL table definition.
However, by abstraction through the repository interfaces callers
(service layer) have no clear understanding of the primary key definion.
This might lead to confusion during implementation:
- The repository makes a sanity check on required conditions before
query execution and returns an error if the condition was not met.
- When a Get method is executed, an error is returned if not exactly one
row is found. Get replaces the legacy GetXxxByID methods we have on
projections. However, a list of flexible conditions can be passed.
- Operations like UPDATE and DELETE should often only affect a single
row, identified by the primary key. If the primary key conditions are
missing an error is returned.
Because there is no "type safety" enforced here, the errors are returned
during runtime. When all possible combinations of API arguments are not
properly tested such errors are actually bugs in the experience of the
end-user. (The API should do it's own input validation, but a
programmer's error is easily made).
# How the Problems Are Solved
Add `PrimaryKeyColumns` and `PrimaryKeyCondition` to existing repository
intefaces.
# Additional Changes
- Where reducers already used repositories, the PrimaryKeyConditions are
used where possible.
# Additional Context
- Part of
https://github.com/zitadel/zitadel/wiki/Decision-Log#expose-primary-key-definition-in-repository-interfaces
---------
Co-authored-by: Marco A. <[email protected]>
This pull request introduces a new feature that allows adding, updating,
and querying metadata for organizations. The changes are primarily in
the backend and include new database tables, repositories, and domain
logic to support organization metadata.
## Changes
* New `org_metadata` table: A new table `zitadel.org_metadata` is
introduced to store metadata for organizations. It includes columns for
`instance_id`, `org_id`, `key`, and `value`. The `value` is stored as a
`BYTEA` type to allow for flexible data storage.
* New `OrganizationMetadataRepository`: A new repository
`OrganizationMetadataRepository` is created to handle all database
operations for organization metadata. It provides methods to `Get`,
`List`, `Set`, and `Remove` metadata.
* New `org_metadata_relational_projection`: A new projection
`org_metadata_relational_projection` is added to update the
`zitadel.org_metadata` table based on events. It handles `MetadataSet`,
`MetadataRemoved`, and `MetadataRemovedAll` events.
* Updated `OrganizationRepository`: The `OrganizationRepository` is
updated to support loading organization metadata. A new method
`LoadMetadata` is added to enable joining the `org_metadata` table when
querying for organizations.
* Updated Organization domain: The Organization domain model is updated
to include a new field `Metadata` of type `[]*OrganizationMetadata`.
## Additional Info
* Extensible Design: The new metadata feature is designed to be
extensible, allowing for future enhancements such as indexing on
specific JSON fields within the `value` column.
* closes https://github.com/zitadel/zitadel/issues/10206
* closes https://github.com/zitadel/zitadel/issues/10214
# Which Problems Are Solved
This is the second PR related to the backend implementation of
GroupService to manage user groups.
The first [PR](https://github.com/zitadel/zitadel/pull/10455) implements
the Command-side.
This PR implements the query side.
# How the Problems Are Solved
* Query-side implementation to search/list groups by
* a list of Group IDs
* by the Group name
* by the Organization ID
# Additional Changes
N/A
# Additional Context
- Follow-up for PR #10455
---------
Co-authored-by: Livio Spring <[email protected]>
# Which Problems Are Solved
Add projects to the relational tables
# How the Problems Are Solved
- Define table migrations
- Define and implement Project and Project Role repositories.
- Provide projection handlers to populate the relational tables.
# Additional Changes
- Statement Builder now has a constructor which allows setting of a base
query with arguments.
- Certain operations, like Get, Update and Delete require the Primary
Key to be set as conditions. However, this requires knowledge of the
implementation and table definition. This PR proposes an additional
condition for repositories: `PrimaryKeyCondition`. This gives clarity on
the required IDs for these operations.
- Added couple of helpers to the repository package, to allow more DRY
code.
- getOne / getMany: generic functions for query execution and scanning.
- checkRestrictingColumns, checkPkCondition: simplify condition
checking, instead of using ladders of conditionals.
- Added a couple of helpers to the repository test package:
- Transaction, savepoint and rollback helpers.
- Create instance and organization helpers for objects that depend on
them (like projects).
# Additional Context
- after https://github.com/zitadel/zitadel/pull/10809
- closes#10765
This pull request introduces a significant refactoring of the database
interaction layer, focusing on improving explicitness, transactional
control, and error handling. The core change is the removal of the
stateful `QueryExecutor` from repository instances. Instead, it is now
passed as an argument to each method that interacts with the database.
This change makes transaction management more explicit and flexible, as
the same repository instance can be used with a database pool or a
specific transaction without needing to be re-instantiated.
### Key Changes
- **Explicit `QueryExecutor` Passing:**
- All repository methods (`Get`, `List`, `Create`, `Update`, `Delete`,
etc.) in `InstanceRepository`, `OrganizationRepository`,
`UserRepository`, and their sub-repositories now require a
`database.QueryExecutor` (e.g., a `*pgxpool.Pool` or `pgx.Tx`) as the
first argument.
- Repository constructors no longer accept a `QueryExecutor`. For
example, `repository.InstanceRepository(pool)` is now
`repository.InstanceRepository()`.
- **Enhanced Error Handling:**
- A new `database.MissingConditionError` is introduced to enforce
required query conditions, such as ensuring an `instance_id` is always
present in `UPDATE` and `DELETE` operations.
- The database error wrapper in the `postgres` package now correctly
identifies and wraps `pgx.ErrTooManyRows` and similar errors from the
`scany` library into a `database.MultipleRowsFoundError`.
- **Improved Database Conditions:**
- The `database.Condition` interface now includes a
`ContainsColumn(Column) bool` method. This allows for runtime checks to
ensure that critical filters (like `instance_id`) are included in a
query, preventing accidental cross-tenant data modification.
- A new `database.Exists()` condition has been added to support `EXISTS`
subqueries, enabling more complex filtering logic, such as finding an
organization that has a specific domain.
- **Repository and Interface Refactoring:**
- The method for loading related entities (e.g., domains for an
organization) has been changed from a boolean flag (`Domains(true)`) to
a more explicit, chainable method (`LoadDomains()`). This returns a new
repository instance configured to load the sub-resource, promoting
immutability.
- The custom `OrgIdentifierCondition` has been removed in favor of using
the standard `database.Condition` interface, simplifying the API.
- **Code Cleanup and Test Updates:**
- Unnecessary struct embeddings and metadata have been removed.
- All integration and repository tests have been updated to reflect the
new method signatures, passing the database pool or transaction object
explicitly.
- New tests have been added to cover the new `ExistsDomain`
functionality and other enhancements.
These changes make the data access layer more robust, predictable, and
easier to work with, especially in the context of database transactions.
# Which Problems Are Solved
Depending on the metadata values (already existing), the newly created
index (#10415) cannot be created or error in the future.
# How the Problems Are Solved
- Create the index using `sha256` and change the query to use sha256 as
well when comparing bytes values such as user_metadata.
- Added a setup step to cleanup potentially created index on
`projections.user_metadata5`
# Additional Changes
None
# Additional Context
- relates to #10415
- requires backport to v4.x
# Which Problems Are Solved
HTTP Request to HTTP providers for Email or SMS are not signed.
# How the Problems Are Solved
Add a Signing Key to the HTTP Provider resources, which is then used to
generate a header to sign the payload.
# Additional Changes
Additional tests for query side of the SMTP provider.
# Additional Context
Closes#10067
---------
Co-authored-by: Marco A. <[email protected]>
This PR initiates the rework of Zitadel's backend to state-persisted
objects. This change is a step towards a more scalable and maintainable
architecture.
## Changes
* **New `/backend/v3` package**: A new package structure has been
introduced to house the reworked backend logic. This includes:
* `domain`: Contains the core business logic, commands, and repository
interfaces.
* `storage`: Implements the repository interfaces for database
interactions with new transactional tables.
* `telemetry`: Provides logging and tracing capabilities.
* **Transactional Tables**: New database tables have been defined for
`instances`, `instance_domains`, `organizations`, and `org_domains`.
* **Projections**: New projections have been created to populate the new
relational tables from the existing event store, ensuring data
consistency during the migration.
* **Repositories**: New repositories provide an abstraction layer for
accessing and manipulating the data in the new tables.
* **Setup**: A new setup step for `TransactionalTables` has been added
to manage the database migrations for the new tables.
This PR lays the foundation for future work to fully transition to
state-persisted objects for these components, which will improve
performance and simplify data access patterns.
This PR initiates the rework of ZITADEL's backend to state-persisted
objects. This is a foundational step towards a new architecture that
will improve performance and maintainability.
The following objects are migrated from event-sourced aggregates to
state-persisted objects:
* Instances
* incl. Domains
* Orgs
* incl. Domains
The structure of the new backend implementation follows the software
architecture defined in this [wiki
page](https://github.com/zitadel/zitadel/wiki/Software-Architecturel).
This PR includes:
* The initial implementation of the new transactional repositories for
the objects listed above.
* Projections to populate the new relational tables from the existing
event store.
* Adjustments to the build and test process to accommodate the new
backend structure.
This is a work in progress and further changes will be made to complete
the migration.
---------
Co-authored-by: Copilot <[email protected]>
Co-authored-by: Iraq Jaber <[email protected]>
Co-authored-by: Iraq <[email protected]>
Co-authored-by: Tim Möhlmann <[email protected]>
This PR overhauls our event projection system to make it more robust and
prevent skipped events under high load. The core change replaces our
custom, transaction-based locking with standard PostgreSQL advisory
locks. We also introduce a worker pool to manage concurrency and prevent
database connection exhaustion.
### Key Changes
* **Advisory Locks for Projections:** Replaces exclusive row locks and
inspection of `pg_stat_activity` with PostgreSQL advisory locks for
managing projection state. This is a more reliable and standard approach
to distributed locking.
* **Simplified Await Logic:** Removes the complex logic for awaiting
open transactions, simplifying it to a more straightforward time-based
filtering of events.
* **Projection Worker Pool:** Implements a worker pool to limit
concurrent projection triggers, preventing connection exhaustion and
improving stability under load. A new `MaxParallelTriggers`
configuration option is introduced.
### Problem Solved
Under high throughput, a race condition could cause projections to miss
events from the eventstore. This led to inconsistent data in projection
tables (e.g., a user grant might be missing). This PR fixes the
underlying locking and concurrency issues to ensure all events are
processed reliably.
### How it Works
1. **Event Writing:** When writing events, a *shared* advisory lock is
taken. This signals that a write is in progress.
2. **Event Handling (Projections):**
* A projection worker attempts to acquire an *exclusive* advisory lock
for that specific projection. If the lock is already held, it means
another worker is on the job, so the current one backs off.
* Once the lock is acquired, the worker briefly acquires and releases
the same *shared* lock used by event writers. This acts as a barrier,
ensuring it waits for any in-flight writes to complete.
* Finally, it processes all events that occurred before its transaction
began.
### Additional Information
* ZITADEL no longer modifies the `application_name` PostgreSQL variable
during event writes.
* The lock on the `current_states` table is now `FOR NO KEY UPDATE`.
* Fixes https://github.com/zitadel/zitadel/issues/8509
---------
Co-authored-by: Copilot <[email protected]>
Co-authored-by: Tim Möhlmann <[email protected]>
# Which Problems Are Solved
This PR introduces a new feature flag `EnableRelationalTables` that will
be used in following implementations to decide whether Zitadel should
use the relational model or the event sourcing one.
# TODO
- [x] Implement flag at system level
- [x] Display the flag on console:
https://github.com/zitadel/zitadel/pull/10615
# How the Problems Are Solved
- Implement loading the flag from config
- Add persistence of the flag through gRPC endpoint
(SetInstanceFeatures)
- Implement reading of the flag through gRPC endpoint
(GetInstanceFeatures)
# Additional Changes
Some minor refactoring to remove un-needed generics annotations
# Additional Context
- Closes#10574
---------
Co-authored-by: Silvan <[email protected]>
# Which Problems Are Solved
Some users have reported the need of retrieving users given a metadata
key, metadata value or both. This change introduces metadata search
filter on the `ListUsers()` endpoint to allow Zitadel users to search
for user records by metadata.
The changes affect only v2 APIs.
# How the Problems Are Solved
- Add new search filter to `ListUserRequest`: `MetaKey` and `MetaValue`
- Add SQL indices on metadata key and metadata value
- Update query to left join `user_metadata` table
# Additional Context
- Closes#9053
- Depends on https://github.com/zitadel/zitadel/pull/10567
---------
Co-authored-by: Silvan <[email protected]>
Co-authored-by: Stefan Benz <[email protected]>
# Which Problems Are Solved
The event execution system currently uses a projection handler that
subscribes to and processes all events for all instances. This creates a
high static cost because the system over-fetches event data, handling
many events that are not needed by most instances. This inefficiency is
also reflected in high "rows returned" metrics in the database.
# How the Problems Are Solved
Eliminate the use of a project handler. Instead, events for which
"execution targets" are defined, are directly pushed to the queue by the
eventstore. A Router is populated in the Instance object in the authz
middleware.
- By joining the execution targets to the instance, no additional
queries are needed anymore.
- As part of the instance object, execution targets are now cached as
well.
- Events are queued within the same transaction, giving transactional
guarantees on delivery.
- Uses the "insert many fast` variant of River. Multiple jobs are queued
in a single round-trip to the database.
- Fix compatibility with PostgreSQL 15
# Additional Changes
- The signing key was stored as plain-text in the river job payload in
the DB. This violated our [Secrets
Storage](https://zitadel.com/docs/concepts/architecture/secrets#secrets-storage)
principle. This change removed the field and only uses the encrypted
version of the signing key.
- Fixed the target ordering from descending to ascending.
- Some minor linter warnings on the use of `io.WriteString()`.
# Additional Context
- Introduced in https://github.com/zitadel/zitadel/pull/9249
- Closes https://github.com/zitadel/zitadel/issues/10553
- Closes https://github.com/zitadel/zitadel/issues/9832
- Closes https://github.com/zitadel/zitadel/issues/10372
- Closes https://github.com/zitadel/zitadel/issues/10492
---------
Co-authored-by: Stefan Benz <[email protected]>
# Which Problems Are Solved
When a SAML IDP is created, the signing algorithm defaults to
`RSA-SHA1`.
This PR adds the functionality to configure the signing algorithm while
creating or updating a SAML IDP. When nothing is specified, `RSA-SHA1`
is the default.
Available options:
* RSA_SHA1
* RSA_SHA256
* RSA_SHA512
# How the Problems Are Solved
By introducing a new optional config to specify the Signing Algorithm.
# Additional Changes
N/A
# Additional Context
- Closes#9842
An existing bug in the UpdateSAMLProvider API will be fixed as a
followup in a different
[PR](https://github.com/zitadel/zitadel/pull/10557).
---------
Co-authored-by: Stefan Benz <[email protected]>
# Which Problems Are Solved
Some events that are now unused are clogging the event queue from time
to time.
# How the Problems Are Solved
Remove the events described in #10458
# Additional Changes
- Updated `stringer` and `enumer` in Makefile target `core_generate_all`
to resolve generated files compilation issues
# Notes
It looks like there are a lot of changes, but most of it is fixing
translation files. I suggest doing a review per-commit
# Additional Context
- Closes#10458
- Depends on https://github.com/zitadel/zitadel/pull/10513
# Which Problems Are Solved
It should not be possible to start 2 projections with the same name.
If this happens, it can cause issues with the event store such as events
being skipped/unprocessed and can be very hard/time-consuming to
diagnose.
# How the Problems Are Solved
A check was added to make sure no 2 projections have the same table
Closes https://github.com/zitadel/zitadel/issues/10453
---------
Co-authored-by: Silvan <[email protected]>
# Which Problems Are Solved
The current handling of event subscriptions for actions is bad, esp. on
instances with a lot of events
(https://github.com/zitadel/zitadel/issues/9832#issuecomment-2866236414).
This led to severe problems on zitadel.cloud for such instances.
# How the Problems Are Solved
As a workaround until the handling can be improved, we introduce an
option for projections to be disabled completely for specific instances:
`SkipInstanceIDs`
# Additional Changes
None
# Additional Context
- relates to https://github.com/zitadel/zitadel/issues/9832
# Which Problems Are Solved
Currently the username uniqueness is on instance level, we want to
achieve a way to set it at organization level.
# How the Problems Are Solved
Addition of endpoints and a resource on organization level, where this
setting can be managed. If nothing it set, the uniqueness is expected to
be at instance level, where only users with instance permissions should
be able to change this setting.
# Additional Changes
None
# Additional Context
Includes #10086Closes#9964
---------
Co-authored-by: Marco A. <[email protected]>
## Which problems are solved
The execution of statements of projections did not have the context
present.
## How the problems were solved
Pass the context to the execute function
## Additional info
This change is required to use the repositories of the relational tables
in projections.
# Which Problems Are Solved
Remove the feature flag that allowed triggers in introspection. This
option was a fallback in case introspection would not function properly
without triggers. The API documentation asked for anyone using this flag
to raise an issue. No such issue was received, hence we concluded it is
safe to remove it.
# How the Problems Are Solved
- Remove flags from the system and instance level feature APIs.
- Remove trigger functions that are no longer used
- Adjust tests that used the flag.
# Additional Changes
- none
# Additional Context
- Closes#10026
- Flag was introduced in #7356
---------
Co-authored-by: Silvan <[email protected]>
# Which Problems Are Solved
Stabilize the usage of webkeys.
# How the Problems Are Solved
- Remove all legacy signing key code from the OIDC API
- Remove the webkey feature flag from proto
- Remove the webkey feature flag from console
- Cleanup documentation
# Additional Changes
- Resolved some canonical header linter errors in OIDC
- Use the constant for `projections.lock` in the saml package.
# Additional Context
- Closes#10029
- After #10105
- After #10061
# Which Problems Are Solved
Stabilize the optimized introspection code and cleanup unused code.
# How the Problems Are Solved
- `oidc_legacy_introspection` feature flag is removed and reserved.
- `OPStorage` which are no longer needed have their bodies removed.
- The method definitions need to remain in place so the interface
remains implemented.
- A panic is thrown in case any such method is still called
# Additional Changes
- A number of `OPStorage` methods related to token creation were already
unused. These are also cleaned up.
# Additional Context
- Closes#10027
- #7822
---------
Co-authored-by: Livio Spring <[email protected]>
# Which Problems Are Solved
This PR implements https://github.com/zitadel/zitadel/issues/9850
# How the Problems Are Solved
- New protobuf definition
- Implementation of retrieval of system translations
- Implementation of retrieval and persistence of organization and
instance level translations
# Additional Context
- Closes#9850
# TODO
- [x] Integration tests for Get and Set hosted login translation
endpoints
- [x] DB migration test
- [x] Command function tests
- [x] Command util functions tests
- [x] Query function test
- [x] Query util functions tests
# Which Problems Are Solved
The resource usage to query user(s) on the database was high and
therefore could have performance impact.
# How the Problems Are Solved
Database queries involving the users and loginnames table were improved
and an index was added for user by email query.
# Additional Changes
- spellchecks
- updated apis on load tests
# additional info
needs cherry pick to v3
# Which Problems Are Solved
This pull request addresses a significant gap in the user service v2
API, which currently lacks methods for managing machine users.
# How the Problems Are Solved
This PR adds new API endpoints to the user service v2 to manage machine
users including their secret, keys and personal access tokens.
Additionally, there's now a CreateUser and UpdateUser endpoints which
allow to create either a human or machine user and update them. The
existing `CreateHumanUser` endpoint has been deprecated along the
corresponding management service endpoints. For details check the
additional context section.
# Additional Context
- Closes https://github.com/zitadel/zitadel/issues/9349
## More details
- API changes: https://github.com/zitadel/zitadel/pull/9680
- Implementation: https://github.com/zitadel/zitadel/pull/9763
- Tests: https://github.com/zitadel/zitadel/pull/9771
## Follow-ups
- Metadata: support managing user metadata using resource API
https://github.com/zitadel/zitadel/pull/10005
- Machine token type: support managing the machine token type (migrate
to new enum with zero value unspecified?)
---------
Co-authored-by: Copilot <[email protected]>
Co-authored-by: Livio Spring <[email protected]>
# Eventstore fixes
- `event.Position` used float64 before which can lead to [precision
loss](https://github.com/golang/go/issues/47300). The type got replaced
by [a type without precision
loss](https://github.com/jackc/pgx-shopspring-decimal)
- the handler reported the wrong error if the current state was updated
and therefore took longer to retry failed events.
# Mirror fixes
- max age of auth requests can be configured to speed up copying data
from `auth.auth_requests` table. Auth requests last updated before the
set age will be ignored. Default is 1 month
- notification projections are skipped because notifications should be
sent by the source system. The projections are set to the latest
position
- ensure that mirror can be executed multiple times
---------
Co-authored-by: Livio Spring <[email protected]>
# Which Problems Are Solved
Currently if a user signs in using an IdP, once they sign out of
Zitadel, the corresponding IdP session is not terminated. This can be
the desired behavior. In some cases, e.g. when using a shared computer
it results in a potential security risk, since a follower user might be
able to sign in as the previous using the still open IdP session.
# How the Problems Are Solved
- Admins can enabled a federated logout option on SAML IdPs through the
Admin and Management APIs.
- During the termination of a login V1 session using OIDC end_session
endpoint, Zitadel will check if an IdP was used to authenticate that
session.
- In case there was a SAML IdP used with Federated Logout enabled, it
will intercept the logout process, store the information into the shared
cache and redirect to the federated logout endpoint in the V1 login.
- The V1 login federated logout endpoint checks every request on an
existing cache entry. On success it will create a SAML logout request
for the used IdP and either redirect or POST to the configured SLO
endpoint. The cache entry is updated with a `redirected` state.
- A SLO endpoint is added to the `/idp` handlers, which will handle the
SAML logout responses. At the moment it will check again for an existing
federated logout entry (with state `redirected`) in the cache. On
success, the user is redirected to the initially provided
`post_logout_redirect_uri` from the end_session request.
# Additional Changes
None
# Additional Context
- This PR merges the https://github.com/zitadel/zitadel/pull/9841 and
https://github.com/zitadel/zitadel/pull/9854 to main, additionally
updating the docs on Entra ID SAML.
- closes#9228
- backport to 3.x
---------
Co-authored-by: Silvan <[email protected]>
Co-authored-by: Zach Hirschtritt <[email protected]>
# Which Problems Are Solved
When users are removed, their auth factors stay in the projection. This
data inconsistency is visible if a removed user is recreated with the
same ID. In such a case, the login UI and the query API methods show the
removed users auth methods. This is unexpected behavior.
The old users auth methods are not usable to log in and they are not
found by the command side. This is expected behavior.
# How the Problems Are Solved
The auth factors projection reduces the user removed event by deleting
all factors.
# Additional Context
- Reported by support request
- requires backport to 2.x and 3.x
# Which Problems Are Solved
When a user changes their password, Zitadel needs to terminate all of
that user's active sessions. This query can take many seconds on
deployments with large session and user tables. This happens as part of
session projection handling, so doesn't directly impact user experience,
but potentially bogs down the projection handler which isn't great. In
the future, this index could be used to power a "see all of my current
sessions" feature in Zitadel.
# How the Problems Are Solved
Adds new index on `user_id` column on `projections.sessions8` table.
Alternatively, we can index on `(instance_id, user_id)` instead but
opted for keeping the index smaller as we already index on `instance_id`
separately.
# Additional Changes
None
# Additional Context
None
---------
Co-authored-by: Silvan <[email protected]>
# Which Problems Are Solved
In Actions v2, if a target is removed, which is still used in an
execution, the target is still listed when list executions.
# How the Problems Are Solved
Removed targets are now also removed from the executions.
# Additional Changes
To be sure the list executions include a check if the target is still
existing.
# Additional Context
None
Co-authored-by: Livio Spring <[email protected]>
# Which Problems Are Solved
The `auth.auth_requests` table is not cleaned up so long running Zitadel
installations can contain many rows.
The mirror command can take long because a the data are first copied
into memory (or disk) on cockroach and users do not get any output from
mirror. This is unfortunate because people don't know if Zitadel got
stuck.
# How the Problems Are Solved
Enhance logging throughout the projection processes and introduce a
configuration option for the maximum age of authentication requests.
# Additional Changes
None
# Additional Context
closes https://github.com/zitadel/zitadel/issues/9764
---------
Co-authored-by: Livio Spring <[email protected]>
# Which Problems Are Solved
Actions v2 is not a feature flag anymore, include functionality on
executions is not used and json tags of proto messages are handled
incorrectly.
# How the Problems Are Solved
- Remove actions from the feature flags on system and instance level
- Remove include type on executions, only in the API, later maybe in the
handling logic as well
- Use protojson in request and response handling of actions v2
# Additional Changes
- Correct integration tests for request and response handling
- Use json.RawMessage for events, so that the event payload is not
base64 encoded
- Added separate context for async webhook calls, that executions are
not cancelled when called async
# Additional Context
Related to #9759Closes#9710
---------
Co-authored-by: Livio Spring <[email protected]>
Livio SpringandGitHub
