# Which Problems Are Solved
As part of https://github.com/zitadel/zitadel/issues/11917 we want to
introduce error slugs so (API) clients can rely on stable,
machine-readable errors and act accordingly.
# How the Problems Are Solved
- Added a `NewSlug` helper function in the domain package.
- Added `ErrorDetails` to the `ZitadelError`
- Added an `zitadel.error.v2.ErrorDetail` proto message
- Updated the connectRPC error interceptor to map new slug based errors
to the new `ErrorDetail`
- Defined some common slugs and error functions like internal errors
- Defined (session) specific slugs used in the `DeleteSession` and
`CheckUser` functions and replaced old implementations
- Updated integration tests to check specific errors if the relation
database feature is enabled
- Updated doc and guideline to reflect the latest changes and decisions
- Updated DeleteSession endpoint API to list possible slugs
# Additional Changes
None
# Additional Context
- closes#11957
---------
Co-authored-by: Wim Van Laer <[email protected]>
# Which Problems Are Solved
`@zitadel/api:generate-go` failed due to duplicate generated files.
https://github.com/zitadel/zitadel/pull/11820 removed the old generated
file and added a generate command to the `internal/crypto/crypto.go`
file. However, there was already a `internal/crypto/generate.go` with a
different output file name (the old file).
# How the Problems Are Solved
Removed the `internal/crypto/generate.go` file and moved the second
generate into `internal/crypto/code.go`.
# Additional Changes
Noticed that mockgen is an old version and updated it. Also then checked
all other tools and updated them.
# Additional Context
- relates to #11820
- noted internally
# Which Problems Are Solved
Add Passkey challenge needed for Create/Set session in the RT model.
# How the Problems Are Solved
Introduce `PasskeyChallengeCommand` (validate/execute/events) for
session creation checks
Add a generic `BeginWebAuthNLogin` helper that returns session data +
assertion JSON bytes.
Add unit tests
# Additional Changes
N/A
# Additional Context
- Related to https://github.com/zitadel/zitadel/issues/11035
---------
Co-authored-by: Copilot <[email protected]>
Co-authored-by: Marco A. <[email protected]>
# Which Problems Are Solved
As part of #11035 , this PR implements the TOTP check logic for session
validation
# How the Problems Are Solved
- Implement TOTP check logic and tests
- The tarpit function has been moved to a common file so that it can be
used by both TOTP and password checks
- Manual transaction management to avoid stalling the DB while verifier
function is executed
- Update `database.Change` `Matches()` function to allow comparison of
`time.Time` values
- A converter package from GRPC to Domain model has been added
# Additional Context
This is a cherry-picked PR + changes, coming from
https://github.com/zitadel/zitadel/pull/11164
- Relates to #11035
# Which Problems Are Solved
Add OTP Email challenge needed for Create/Set session in the RT model.
# How the Problems Are Solved
Introduce `OTPEmailChallengeCommand` (validate/execute/events) for
session creation checks
Add unit tests
# Additional Changes
Remove unused fields from `session_challenge_otp_sms.go`
# Additional Context
- Related to https://github.com/zitadel/zitadel/issues/11035
---------
Co-authored-by: Marco A. <[email protected]>
# Which Problems Are Solved
As part of #11035 , this PR implements the Passkey check logic for
session validation
# How the Problems Are Solved
- Refactor webauth FinishLogin to support new domain model
- Add webauth config to defaults
- Implement passkey check logic and tests
- Manual transaction management to avoid stalling the DB while
FinishLogin callback is executed
- Update passkey Type condition to allow passing a text operation
(equal, contains, etc..)
# Additional Context
This is a cherry-picked PR + minor changes, coming from
https://github.com/zitadel/zitadel/pull/11164
- Relates to #11035
---------
Co-authored-by: Fabienne Bühler <[email protected]>
Co-authored-by: Gayathri Vijayan <[email protected]>
# Which Problems Are Solved
This PR adds the OTP SMS Challenge needed for CreateSession API
https://github.com/zitadel/zitadel/issues/11035
# How the Problems Are Solved
- Introduce `OTPSMSChallengeCommand` (validate/execute/events) for
session creation checks
- Introduce `GetOTPCryptoGeneratorConfigWithDefault` to fetch
per-instance secret generator settings
- Add unit tests
# Additional Changes
Refactor tests in `session_check_recovery_code_test.go` to make the
helper functions reusable in `session_challenge_otp_sms_test.go`
# Additional Context
Related to https://github.com/zitadel/zitadel/issues/11035
# Which Problems Are Solved
Adding recovery code check step required for the CreateSession RT
implementation.
# How the Problems Are Solved
- Introduce `RecoveryCodeCheckCommand` (validate/execute/events) for
session creation checks
- Add unit tests
# Additional Changes
- refactor password-check code to reuse a shared lockout-policy helper
- add some missing error translations in `internal/static/i18n/en.yaml`
# Additional Context
- Related to https://github.com/zitadel/zitadel/issues/11035
---------
Co-authored-by: Silvan <[email protected]>
Co-authored-by: Copilot Autofix powered by AI <[email protected]>
# Which Problems Are Solved
2 tests were failing on the new session checks:
- Password check (events) fails due to a race condition
- IDP intent check (events) (also password test) fails due to a flaky
time comparison
# How the Problems Are Solved
- Race condition: remove setting the password hasher in the test. It's
already globally set in `domain_test.go`
- Time comparison: use `assert.WithinRange` instead of `assert.InDelta`
and use 2 second range to make sure you don't get resolution issues
# Additional Context
- Closes#11878
# Which Problems Are Solved
As part of moving the session API to the relational database, this PR
adds the functionality to delete sessions through the session API v2 and
the OIDC end_session endpoint.
# How the Problems Are Solved
- added `SessionDeleteCommand` in domain package
- added possibility to pass the session token verifier to the domain
package
- use feature flag to switch API and OIDC usage from CQRS to relation
table
- added a `permissionCheck` condition to let permissions be checked
directly in the repository / sql
- part of it is still a placeholder until actual permission tables are
done
- added a `session_deleted` table to store (recently) deleted sessions
to handle necessary checks / cases like delete my own session, which
needs to be idempotent
# Additional Changes
Fixed `Matches` function of `existsCondition` (as missing an
implementation)
# Additional Context
- closes#11037
---------
Co-authored-by: Stefan Benz <[email protected]>
Introduces the `administrators` relational table and repository layer
for the v3 backend, unifying instance, organization, project, and
project-grant membership into a single scoped model.
Closes#10220
## Changes
- **Domain**: `Administrator` struct, `AdministratorScope` enum,
`AdministratorRepository` interface with typed CRUD, column, condition,
and change helpers. Generated enumer and GoMock.
- **Migration** (`016`): `zitadel.administrators` table with a generated
`id`, scope-alignment check constraint, cascading foreign keys, and a
child `administrator_roles` table.
- **Repository**: CTE-based role mutations (`AddRole`, `RemoveRole`,
`SetRoles`) for atomicity. Roles aggregated via `ARRAY_AGG`.
- **Projection**: Event reducers for
`member.added/changed/removed/cascade_removed` across all four admin
scopes.
- **Fixes**: `Rows.Close()` error wrapping; `TextArray` nil-safe scan +
full `pgtype` array interface; `authorization` and `projectGrant` scan
helpers fixed for `ARRAY_AGG` under `database/sql`; removed unused `ctx`
parameter from `callReduce`.
---------
Co-authored-by: Marco Ardizzone <[email protected]>
# Which Problems Are Solved
As part of #11035 , this PR implements the IDP intent check logic for
session validation
# How the Problems Are Solved
- Inject encryption algorithm for IDP intent tokens at DB initialization
- Add the IDP intent repository as an invoke option
- Rework crypto mocks using gomock
- Implement IDP intent check logic
- minor - DB: expand IDP Intent deletion testing table with a test case
where IDP intent is not found
# Noteworthy
I did not implement `domain.Transactional` given the doubts that
@adlerhurst expressed in https://github.com/zitadel/zitadel/pull/11804 .
So, there is no transaction AT ALL in here because the `Execute()`
method contains one single DB call (Delete) and `Validate()` just does
Get calls.
# Additional Context
This is a cherry-picked PR + minor changes, coming from
https://github.com/zitadel/zitadel/pull/11164
- Relates to #11035
- Depends on https://github.com/zitadel/zitadel/pull/11804
# Which Problems Are Solved
As part of #11035 , this PR implements the password check logic for
session validation
# How the Problems Are Solved
- Add system config to default configuration of `domain` package for
easy initialization. Intialize the system settings when Zitadel starts
up
- Add password hasher verify logic to the default configuration of
`domain` package. Initialize it when Zitadel starts up.
- Add settings repositories with their mocks
- Implement the logic for doing a password check
# Additional Context
This is a cherry-picked PR + minor changes, coming from
https://github.com/zitadel/zitadel/pull/11164
- Relates to #11035
- Depends on https://github.com/zitadel/zitadel/pull/11777
# Which Problems Are Solved
As part of #11035 , this PR implements the user check logic for session
validation
# How the Problems Are Solved
- Add user and session repositories as invoke options, as well as their
mocks to prepare the ground for testing
- Implement the logic for the UserCheck command, following what has been
done on eventstore side
- Add a session permission chek interface to be used during the
`Validate()` part of the Commander
# Additional Context
This is a cherry-picked PR + minor changes, coming from
https://github.com/zitadel/zitadel/pull/11164
- Relates to #11035
---------
Co-authored-by: Silvan <[email protected]>
# Which Problems Are Solved
This is a small refactoring for tests in `backend/v3/domain` package to
use `assert.ErrorIs` instead of `assert.Equal` for error assertion.
# How the Problems Are Solved
- Add custom `Is()` function to `MultipleObjectsUpdatedError` error
- Substitute `assert.Equal` with `assert.ErrorIs` in instance and
organization tests
# Additional Context
This was part of #11035
- Transitioned from using `handler.Statements` to a repository pattern
for managing organization data in migrations.
- Introduced a new method to set the updated timestamp for
organizations.
- Updated the projection logic to utilize the repository for creating,
updating, and deleting organization records.
# Which Problems Are Solved
Renaming of:
- Resource Owner
- OrgID
- OrganizationID
- Organization Id
# How the Problems Are Solved
Find & Replace, evalue entries, is resource owner referring to an
organization ? If so, do change
# Additional Context
- Closes#11305
Introduce a new SQL migration that creates the `login_names` table and
associated triggers/functions for managing login names based on domain
policies and user updates. Manual tests confirm the functionality works
as intended.
# Which Problems Are Solved
Adds a repository implementation to add/remove recovery codes to the
users relational table.
# How the Problems Are Solved
This is achieved by:
* adding the following columns to the users table: `recovery_codes`,
`recovery_code_last_successful_check`, `recovery_code_failed_attempts`
* adding the `RecoveryCodes` field to the `HumanUser` domain with fields
to set recovery `codes`, `lastSuccessfullyCheckedAt` timestamp, and
`failedAttempts`
* setting `recoveryCodes` in the `Get` user query statement to return a
json object with details related to the recovery codes
* adding the repository-layer implementation to add/remove recovery
codes and set fields related to recovery code checks.
* adding projection reducers to handle the following events:
`HumanRecoveryCodesAddedEvent`, `HumanRecoveryCodesRemovedEvent`,
`HumanRecoveryCodeCheckSucceededEvent`, and
`HumanRecoveryCodeCheckFailedEvent`
* adding unit tests
# Additional Changes
* fix the error message when the recovery is empty in
`internal/command/user_human_recovery_codes.go`
* add a new error message for empty recovery code during checks in
`en.yaml`
# Additional Context
- Closes https://github.com/zitadel/zitadel/issues/11212
- Follow-up: integration tests for the reducers will be added in a
different PR after this
[PR](https://github.com/zitadel/zitadel/pull/11478) is merged
## Summary
This PR introduces the relational user repository for backend v3,
including schema, repository APIs, reducers/projections, and tests for
both human and machine users.
## What changed
* Added user-focused relational migrations (users, verifications,
metadata, PATs, machine keys, passkeys, identity provider links) and
related FK wiring.
* Implemented/expanded repository domain contracts and storage
implementations for human and machine user flows.
* Added support for machine secret handling and aligned access token
type modeling.
* Extended user domain models (verifications, password/email/phone/TOTP
state, passkeys, metadata, invite handling, identity provider links).
* Updated projection/reduction coverage and integration tests for key
user lifecycle events.
* Regenerated and updated mocks/enums plus supporting database
change/column utilities.
## Notes for reviewers
* This PR is large because it establishes the foundational relational
user storage layer and related test scaffolding. Sorry for that.
## Closes
Closes#10218Closes#10224Closes#10225Closes#10209Closes#10802
---------
Co-authored-by: Iraq Jaber <[email protected]>
Co-authored-by: Copilot <[email protected]>
Co-authored-by: Marco A. <[email protected]>
Co-authored-by: Tim Möhlmann <[email protected]>
# Which Problems Are Solved
Inconsistent naming of service account, found in the following
variations:
- Machine User
- machine user
- Service User
- Machine Account
- Technical Account
- User: Type Machine
# How the Problems Are Solved
Attentive search and replace.
Localizations have been translated using Copilot
# Additional Changes
Some unused methods have been removed from the Go code.
# Additional Context
- Closes#11285
# Which Problems Are Solved
Naming inconsistencies - User (Human)
# How the Problems Are Solved
Most of the occurrences have not been changed. `User (Human)` was mostly
changed when talking about code objects and where I felt it was
necessary to distinguish them from machine users.
When both human and machine user occurrences were found, the machine
user has been changed to service account (see
https://github.com/zitadel/zitadel/issues/11285)
# Additional Context
- Closes#11284
---------
Co-authored-by: Wim Van Laer <[email protected]>
# Which Problems Are Solved
In the context of moving our datastore to relational tables, Identity
Provider Intents have no definition of a relational table nor any DB API
that can be used to store the IDP intents as relational objects.
# How the Problems Are Solved
The PR follows the guidelines left in #11190
- A table `zitadel.identity_provider_intents` is created: it defines its
relational constraints (foreign keys) with the other tables, as well as
an ENUM to represent the state of the intent.
- An IDP Intent repository has been created, which offers CRUD
operations implemented as `Create`, `Get`, `Update` and `Delete` methods
for the IDP intent.
- The repository also defines all columns and conditions useful for
performing DB operations on the IDP intents.
- DB mocks have been added as well to ease testing on the `domain`
package.
- Reduction functions have been implemented for the existing IDP events
so to keep the relational table in sync with the event store.
# Additional Changes
To allow testing LDAP intent flow, the `Session.FetchUser()` method has
been extracted to a separate file. A corresponding mock has been created
as well in `fetch_user_mock.go`. The files are mutually excluded and
added at build time through go build tags: the mock is added when the
`integration` tag is specified, while the real implementation is added
when the same tag IS NOT added to the build tags.
# Additional Context
- Closes#11190
---------
Co-authored-by: Livio Spring <[email protected]>
Co-authored-by: Silvan <[email protected]>
# Which Problems Are Solved
Streams allow differentiating logs produced by different components of
Zitadel.
# How the Problems Are Solved
The `backend/v3/instrumentation/logging` package now exposes convenience
function for setting and getting a logger from the context. As well as
high-level functions to emit log records at various levels. When
constructing a new logger a "stream" needs to be specified:
- **runtime**: General runtime logs, such as startup and shutdown
messages. Default for logs that do not belong to the other categories.
- **request**: Logs for incoming API and HTTP requests.
- **event_handler**: Logs for event handling in projections.
- **queue**: Logs for the job queue processing.
- **event_pusher**: Logs for event pushing to the database. Disabled by
default, contains sensitive information.
Each line from the returned logger contains a `stream` field as well as
a `version` field with the current Zitadel version.
## Runtime config
Streams can be enabled by passing an array of stream names in the
runtime config. Because some log streams may contain sensitive data
(especially events), it is now also possible to mask values by their
key.
# Additional Changes
- Wrap `slogctx` in the `logging` package. (Except API error converter
packages, because of import cycle)
- Add some docs to `logging` package so other devs understand how to add
logging to Zitadel
- Add `logging.OnError` and `logging.WithError` helper functions with
`Panic()` and `Fatal()` methods, to preserve current calls in the `cmd`
packages.
- Add instance context extractor.
- Only output request details in the request info log. Request ID
remains propagated through context.
- Moved middleware functionality into protocol specific packages.
- Removed setting of URI to context in metric middleware. There were ony
setters and no getters. (Unused value)
- Reuse a single statusWriter in the middleware package for middlewares
that need to know the response status.
# Additional Context
- Closes#11333
- Closes#11331
- Partly #11330
---------
Co-authored-by: Silvan <[email protected]>
# Which Problems Are Solved
The following terms have all been renamed to management console:
- Customer Portal (when used to mean the console)
- Console
- Admin Console
# How the Problems Are Solved
- Search & Replace smartly
- Use Copilot for translation files
Changes done to: backend + frontend codebase, docs, translations and
protobufs (descriptions only)
# Additional Context
- Partially Closes#11279
# Which Problems Are Solved
Zitadel did not provide easy correlation between errors, logs, traces
and metrics. The configuration for those instrumentations was also not
consistent, with some supporting different exporters then others.
Implementation and parsing of config was also spaghettified over
multiple packages, with awkward parsing and inconsistent naming of
options.
# How the Problems Are Solved
All telemetry is now merged under the name "instrumentation". Why?
1. We thought it was a good idea in the past to call the milestone
exporter `Telemtry` in the runtime config. Calling this `TelemetryV2`
looks weird.
2. Not everything is a meter and not everything is sent (tele...).
3. It's also
[defined](https://opentelemetry.io/docs/concepts/instrumentation/) as
such by the OTEL documentation.
## New features
- Adds structured, context based logging with trace-ID awareness
- Static log fields are added to the context, such as service and
request path
- Static log fields are injected in each logline emitted by the
application
- Structured logs can also be send to an otel exporter
- Structured logs can be printed to StdErr in text and JSON format
- Error sinks make sure every error is logged at the correct level:
- Warnings for client side errors (HTTP 400 range, Invalid request etc)
- Error for server side errors (Internal server errors)
- Metrics can now also be send to a OTEL collector. (previously they
could only be scraped from `/debug/metrics` with prometheus)
## Exporters
This change adds all the exporters supported by OTEL upstream and some
google specific exporters for our cloud deployment.
- StdOut / StdErr: all instrumentations
- OTEL gRPC / HTTP: all instrumentations
- Google: all instrumentations except logging
- Prometheus (pull-based): only metrics
The exception is profiling, which only supports the google exporting due
to lack of support by OTEL upstream.
## Configuration and structure
- All instrumentation is moved into the new `backend/v3/instrumentation`
package. It reuses configuration types, so both code and runtime
configuration are easier to understand.
- The `internal/telemetry` packages are removed.
- Instrumentation is started with a single function and a proper
shutdown function is now provided.
- Legacy configuration is still parsed from the runtime config, as long
as the new configuration is disabled. This allows backporting this
feature to v4 without breaking existing configurations.
# Additional Changes
- Devcontainer: set `$PATH` variable so installed go binaries can be run
individually, without NX.
- NX: install GCI tool to fix imports
# Additional Context
- Closes https://github.com/zitadel/zitadel/issues/8408
- Closes https://github.com/zitadel/zitadel/issues/6664
- Backport to v4
# Which Problems Are Solved
As part of going to a relation storage model, we need to implement the
repository for handling sessions and their events.
# How the Problems Are Solved
- Adds a migration step to create the necessary tables and types in the
database.
- Adds the repository implementation for sessions.
- Implements a projection handler for session events to handle the
events written by the old command handlers.
- Updates the `Session` type and `SessionRepository` interface:
- added `CreatorID`
- Adjusted `Delete` method to return amount of affected columns
- Removed sub-repository clumns from the main interface
- Added `CreatorIDCondition`, `ExpirationCondition`, `ExistsFactor` and
`ExistsMetadata`
- Removed `SetUserAgent`
- Methods of `SessionFactor` and `SessionChallenge` have been made
public
- fixed enumer on `SessionFactorType`
- Adds a `JSON` type to the repository package to easy can JSON columns,
resp. jsonb object inside a property of a `JSONArray` object.
- Adds a `NoChange` interface, which can be implemented to signal that a
`Change` interface implementation is not writing any changes to the
statement builder, e.g. `CTEChange`
# Additional Changes
- Fixes the operators enums. `TextOperations` had wrong comments and
`NumberOperations` additionally also had from comparisons.
- Fixes downgrade of migration step 008.
# Additional Context
- closes https://github.com/zitadel/zitadel/issues/10212
# Which Problems Are Solved
This PR implements relational table and repository interface for the
authorizations feature as described in
https://github.com/zitadel/zitadel/issues/10767
# How the Problems Are Solved
By
* adding new `zitadel.authorizations` and `zitadel.authorization_roles`
tables
* implementing the `AuthorizationRepository` interface to create,
update, query, and delete user authorizations and roles
* implementing `authorization_relational.go` projection to update
`zitadel.authorizations` and `zitadel.authorization_roles` tables based
on events
# Additional Changes
N/A
# Additional Context
- Closes#10767
---------
Co-authored-by: Livio Spring <[email protected]>
# Which Problems Are Solved
This PR implements the endpoints listed in
https://github.com/zitadel/zitadel/issues/10444 .
# How the Problems Are Solved
The implementation follows the same pattern as the Instance one. The
`Add` and `Remove` are implementing the `Commander` interface, while
`List` Domains endpoints are implementing the `Querier` interace.
The PR adds also some forgotten calls to the relational logic for the
Instance API. The commits are identifiable by the label `instance:`
prepended to them. This is the complete list: bafce01b0 , 6fe7257ab ,
e27bc9c34 , 8133ede73 , fde667953 , a3e1135ed , ee1a6f9c1
# Additional Context
- Closes#10444
---------
Co-authored-by: Elio Bischof <[email protected]>
Co-authored-by: Lebogang Phoshoko <[email protected]>
Co-authored-by: Wim Van Laer <[email protected]>
Co-authored-by: Max Peintner <[email protected]>
Co-authored-by: Gayathri Vijayan <[email protected]>
# Which Problems Are Solved
The `SecretGenerators` setting is missing among the instance settings in
the new relational model.
# How the Problems Are Solved
This PR adds:
* `SettingTypeSecretGenerator` type to the settings model
* `SecretGeneratorSettingsRepository` and
`secretGeneratorSettingsJSONChanges` interfaces to set, retrieve, and
delete `secret_generator` settings, and corresponding unit tests.
* projection reducers to reduce the following events:
* `SecretGeneratorAddedEvent`
* `SecretGeneratorChangedEvent`
* `SecretGeneratorRemovedEvent`
* integration tests to test that the events get reduced correctly
# Additional Changes
N/A
# Additional Context
- Part of https://github.com/zitadel/zitadel/issues/10207
---------
Co-authored-by: Copilot <[email protected]>
Co-authored-by: Marco A. <[email protected]>
Update the `Check` struct to utilize `crypto.CryptoValue` for the `Code`
field, enhancing security and type safety. Adjust the `CheckTypeInit`
struct accordingly to maintain consistency.
# Which Problems Are Solved
As part of https://github.com/zitadel/zitadel/issues/9599 to push to
move to a relational db table model, this is the migration of the
settings
# Additional Changes
I had to add 2 non source code files
`backend/v3/storage/database/events_testing/font.otf` and
`backend/v3/storage/database/events_testing/picture.png`, these are used
as part of the reduce tests, hence why they're sourced inside
`backend/v3/storage/database/events_testing/settings_instance_test.go`
- Closes https://github.com/zitadel/zitadel/issues/10207
---------
Co-authored-by: adlerhurst <[email protected]>
Co-authored-by: Copilot <[email protected]>
Co-authored-by: Tim Möhlmann <[email protected]>
Co-authored-by: Stefan Benz <[email protected]>
This pull request introduces a new set of domain models and repositories
for managing users within the new v3 structure. The goal is to establish
a clean, robust, and extensible foundation for user management.
### Summary
This PR accomplishes the following:
- **Polymorphic User Model**: Introduces a `User` model in user.go that
can represent either a `Human` or a `Machine` user. This provides a
unified way to handle different user types while allowing for specific
attributes and logic for each.
- **Repository Pattern**: Implements the repository pattern for
accessing and manipulating user data. This decouples the business logic
from the data storage implementation. The main repository interfaces
(`UserRepository`, `HumanUserRepository`, `MachineUserRepository`) are
now defined in `backend/v3/domain/user_repository.go`.
- **Comprehensive User Models**: The user.go file now also defines the
domain models for associated user data, including `UserMetadata`,
`IdentityProviderLink`, `Passkey`, `PersonalAccessToken`, and
`MachineKey`.
- **Generic Verification and Check System**: Introduces verification.go
and check.go to provide a standardized way of handling various
verification flows (e.g., email verification, OTP checks).
- **Generated Code**: Adds new generated files for enums
(`humangender_enumer.go`, `machinekeytype_enumer.go`,
`passkeytype_enumer.go`, `personalaccesstokentype_enumer.go`,
`userstate_enumer.go`, `usertype_enumer.go`) and mocks for the new
repositories.
- **Session Domain Refactoring**: Updates the session domain
(`backend/v3/domain/session*.go`) to align with the new user model
structure.
This foundational work paves the way for migrating user management logic
to the new `v3` backend, with a more maintainable and testable
architecture.
part of #10224
---------
Co-authored-by: Copilot <[email protected]>
Co-authored-by: Marco A. <[email protected]>
This pull request introduces a new feature that allows adding, updating,
and querying project grants The changes are primarily in the backend and
include new database tables, repositories, and domain logic to support
project grants.
## Changes
* New `project_grants` table: A new table `zitadel.project_grants` is
introduced to store project grants It includes columns for
`instance_id`, `id`, `state`, `project_id`, `granted_org_id` and
`granting_org_id`.
* New `ProjectGrantRepository`: A new repository
`ProjectGrantRepository` is created to handle all database operations
for project grants. It provides methods to `Get`, `List`, `Create`,
`Update`, and `Delete`.
* New `project_grant_roles` table: A new table
`zitadel.project_grant_roles` is introduced to store project grant roles
It includes columns for `instance_id`, `grant_id`, `key`, `project_id`
and `project_org_id`.
* New `ProjectGrantRoleRepository`: A new repository
`ProjectGrantRoleRepository` is created to handle all database
operations for project grant roles. It provides methods to `Get`,
`List`, `Add`, and `Delete`.
* New `project_grant_relational_projection`: A new projection
`project_grant_relational_projection` is added to update the
`zitadel.project_grants` and `zitadel.project_grant_roles` tables based
on events. It handles all project grant events.
* Updated `ProjectRepository`: The `ProjectRepository` is updated to
list granted projects.
# Additional Context
Closes https://github.com/zitadel/zitadel/issues/10766
---------
Co-authored-by: Stefan Benz <[email protected]>
Co-authored-by: Marco A. <[email protected]>
Co-authored-by: Silvan <[email protected]>
# Which Problems Are Solved
As part moving sessions to the relational table, the repository to be
able to manage session needs to be defined. This allows the business
logic / service-layer to implement the necessary commands and the
storage-layer to implement the underlying repository.
# How the Problems Are Solved
This PR defines the `SessionRepository` with all underlying object like
the user agent, metadata, factors (including. challenges).
# Additional Changes
None
# Additional Context
- relates to #10212
---------
Co-authored-by: Copilot <[email protected]>
# Which Problems Are Solved
This PR implements the endpoints listed in
https://github.com/zitadel/zitadel/issues/10443 .
# How the Problems Are Solved
The implementation follows the same pattern as the Organization one.
There are no peculiarities to this PR. The `Update`, `Delete` are
implementing the `Commander` interface, while `Get` and `List`
instance(s) endpoints are implementing the `Querier` interace.
# Additional Context
- Closes#10443
- Depends on #10445
This pull request refactors the command execution and transaction
handling logic within the `domain` package. The primary goal is to
simplify the invoker chain, clarify transaction management, and improve
the overall structure for better maintainability and readability.
### Key Changes:
- **Simplified Invoker Chain**: The previous chain of invokers has been
refactored. A new `transactionInvoker` is introduced to explicitly
manage database transactions for commands that require them. This
responsibility is no longer spread across different parts of the
invocation flow.
- **`Transactional` Interface**: A new `Transactional` interface has
been introduced. Commands that need to run within a database transaction
now implement this interface. This provides a clear and explicit
contract for transactional operations.
- **Consolidated Transaction Logic**: The logic for starting,
committing, and rolling back transactions is now centralized within the
`transactionInvoker` and the `eventStoreInvoker`. The `EnsureTx` method
on `InvokeOpts` has been removed, and transaction handling is now
managed automatically by the invocation framework.
- **Improved Event Handling**: The `eventStoreInvoker` and its
`eventCollector` have been rewritten to be more robust and to correctly
handle nested and batched commands, ensuring that all events are
collected and published within the same transaction.
- **Code Cleanup**: A significant amount of now-unused code, including
the old `invoker_test.go`, has been removed. Various comments and
variable declarations have also been cleaned up.
- **Updated Command Implementations**: All commands in the `domain`
package (e.g., `ActivateOrgCommand`, `DeactivateOrgCommand`,
`DeleteOrgCommand`, `UpdateOrgCommand`) have been updated to use the new
`Transactional` interface where appropriate, removing manual transaction
management.
- **Enhanced Testing**: Tests have been added and updated to cover the
new transaction and event handling logic, ensuring the refactoring is
sound.
- **Use `PrimaryKeyCondition`**: Use the
`repository.PrimaryKeyCondition(...)` method instead of using
`database.And(...)` where possible.
These changes lead to a cleaner, more decoupled architecture for command
handling in ZITADEL.
---------
Co-authored-by: Marco A. <[email protected]>
# Which Problems Are Solved
Relational table's primary keys are part of the SQL table definition.
However, by abstraction through the repository interfaces callers
(service layer) have no clear understanding of the primary key definion.
This might lead to confusion during implementation:
- The repository makes a sanity check on required conditions before
query execution and returns an error if the condition was not met.
- When a Get method is executed, an error is returned if not exactly one
row is found. Get replaces the legacy GetXxxByID methods we have on
projections. However, a list of flexible conditions can be passed.
- Operations like UPDATE and DELETE should often only affect a single
row, identified by the primary key. If the primary key conditions are
missing an error is returned.
Because there is no "type safety" enforced here, the errors are returned
during runtime. When all possible combinations of API arguments are not
properly tested such errors are actually bugs in the experience of the
end-user. (The API should do it's own input validation, but a
programmer's error is easily made).
# How the Problems Are Solved
Add `PrimaryKeyColumns` and `PrimaryKeyCondition` to existing repository
intefaces.
# Additional Changes
- Where reducers already used repositories, the PrimaryKeyConditions are
used where possible.
# Additional Context
- Part of
https://github.com/zitadel/zitadel/wiki/Decision-Log#expose-primary-key-definition-in-repository-interfaces
---------
Co-authored-by: Marco A. <[email protected]>
# Which Problems Are Solved
This PR implements the endpoints listed in #10445 using relational
tables.
- [x] UpdateOrganization
- [x] ListOrganizations
- [x] DeleteOrganization
- [x] DeactivateOrganization
- [x] ActivateOrganization
# How the Problems Are Solved
- **UpdateOrganization:** Implemented logic for `Validate()`,
`Execute()` and `Events()`. On CQRS side, updating an organization emits
events related to the organization and its domains. Separate commanders
have been made for the domain events so that their logic is run
accordingly and we keep separation of concerns. Implementation of the
domains is left to its own ticket.
Because some domain-related data is changed during the organization
update, its original values are saved as pointers inside
[UpdateOrgCommand](https://github.com/zitadel/zitadel/blob/4f87cd8d0cfd75f290aa13c342e4939cecebef73/backend/v3/domain/org_update.go#L20).
These variables will be then used as input for the domain commanders
(that will run after the update organization one).
- **ListOrganizations:** Implemented logic for `Validate()`,
`Execute()`. Utility methods have been made to parse the input queries.
This commander is used both by the v2 and v2beta endpoint, so a
translation layer has been put in place to translate all v2beta requests
into v2. The commander strictly handles v2 request only.
The translation layer between v2beta and v2 APIs was made so to make it
simple to remove it once the v2beta APIs will be removed (a simple
deletion should suffice). A `converter` package [is
shipped](https://github.com/zitadel/zitadel/blob/4f87cd8d0cfd75f290aa13c342e4939cecebef73/backend/v3/api/org/v2/convert/README.md#L5)
as well with instruction for removal. TODOs have been put in place all
over the code to indicate that the related piece of code needs to be
removed once v2beta -> v2 transition is complete.
- **DeleteOrganization:** Implemented logic for `Validate()`,
`Execute()` and `Events()`. The `Events()` method is mostly incomplete
as it requires a lot of data that is currently not retrievable due to
the lack of their relational tables.
- **DeactivateOrganization** and **ActivateOrganization**: Implemented
logic for `Validate()`, `Execute()` and `Events()`. No notable remarks.
👉 All endpoints return matching errors to the CQRS counterpart: this way
allows re-using the integration tests and
👉 All endpoints have been unit-tested through DB mocking
👉 A bunch of TODOs have been put in place for future reworks that are
needed (e.g. `Update()` method on repository should also return a
timestamp)
# Additional Changes
- A test utility method to allow mocking query options have been made.
This was necessary because `database.QueryOption` is a function and the
only comparison possible with functions is `nil` check. So, to mock
this, `database.QueryOption` is converted first to `database.QueryOpts`.
See 2276742adaef1998ea4844e1f0dca2079e45ccfb
- A `BaseCommand` has been created with the idea to collect methods that
are going to be needed by other commands. For the moment, it only offers
a method to convert query text operations from gRPC to `domain` model.
See bb85456dd101282fa76d2e2ad165eed4c8ef61b9
- SQL operations have been reworked to allow for queries using
`ContainsIgnoreCase`, `EndsWithIgnoreCase` and other `IgnoreCase`
operations. This change is pending approval and might be reworked. See
https://github.com/zitadel/zitadel/pull/10704#discussion_r2392886177
# Additional Context
`UpdateOrganization` is partially working because the logic for handling
domain updates is not done (there is only a draft of the commanders to
showcase the call chain). See
https://github.com/zitadel/zitadel/pull/10704#discussion_r2356143178
Permissions are not implemented as they are missing. See
https://github.com/zitadel/zitadel/pull/10771
- Closes: #10445
---------
Co-authored-by: Silvan <[email protected]>
This pull request introduces a new feature that allows adding, updating,
and querying metadata for organizations. The changes are primarily in
the backend and include new database tables, repositories, and domain
logic to support organization metadata.
## Changes
* New `org_metadata` table: A new table `zitadel.org_metadata` is
introduced to store metadata for organizations. It includes columns for
`instance_id`, `org_id`, `key`, and `value`. The `value` is stored as a
`BYTEA` type to allow for flexible data storage.
* New `OrganizationMetadataRepository`: A new repository
`OrganizationMetadataRepository` is created to handle all database
operations for organization metadata. It provides methods to `Get`,
`List`, `Set`, and `Remove` metadata.
* New `org_metadata_relational_projection`: A new projection
`org_metadata_relational_projection` is added to update the
`zitadel.org_metadata` table based on events. It handles `MetadataSet`,
`MetadataRemoved`, and `MetadataRemovedAll` events.
* Updated `OrganizationRepository`: The `OrganizationRepository` is
updated to support loading organization metadata. A new method
`LoadMetadata` is added to enable joining the `org_metadata` table when
querying for organizations.
* Updated Organization domain: The Organization domain model is updated
to include a new field `Metadata` of type `[]*OrganizationMetadata`.
## Additional Info
* Extensible Design: The new metadata feature is designed to be
extensible, allowing for future enhancements such as indexing on
specific JSON fields within the `value` column.
* closes https://github.com/zitadel/zitadel/issues/10206
* closes https://github.com/zitadel/zitadel/issues/10214
# Which Problems Are Solved
There is no implementation of the permission framework in the new
command pattern. We risk forgetting permission options as we are already
migrating the service layer to the new pattern.
# How the Problems Are Solved
Use an permission checker interface, which will be a sub-interface of
the Permission Repository. The permission checker is passed in the
command executor and invoker as part of the command options. From here
the implementation can call the correct permission method.
This PR proposes a check method for each
[level](https://github.com/zitadel/zitadel/wiki/Internal-AuthZ#levels).
# Additional Changes
- none
# Additional Context
- Draft design:
https://github.com/zitadel/zitadel/wiki/Internal-AuthZ#enforcement
- Related for queries: https://github.com/zitadel/zitadel/pull/10756
# Which Problems Are Solved
Add projects to the relational tables
# How the Problems Are Solved
- Define table migrations
- Define and implement Project and Project Role repositories.
- Provide projection handlers to populate the relational tables.
# Additional Changes
- Statement Builder now has a constructor which allows setting of a base
query with arguments.
- Certain operations, like Get, Update and Delete require the Primary
Key to be set as conditions. However, this requires knowledge of the
implementation and table definition. This PR proposes an additional
condition for repositories: `PrimaryKeyCondition`. This gives clarity on
the required IDs for these operations.
- Added couple of helpers to the repository package, to allow more DRY
code.
- getOne / getMany: generic functions for query execution and scanning.
- checkRestrictingColumns, checkPkCondition: simplify condition
checking, instead of using ladders of conditionals.
- Added a couple of helpers to the repository test package:
- Transaction, savepoint and rollback helpers.
- Create instance and organization helpers for objects that depend on
them (like projects).
# Additional Context
- after https://github.com/zitadel/zitadel/pull/10809
- closes#10765
This pull request introduces a significant refactoring of the database
interaction layer, focusing on improving explicitness, transactional
control, and error handling. The core change is the removal of the
stateful `QueryExecutor` from repository instances. Instead, it is now
passed as an argument to each method that interacts with the database.
This change makes transaction management more explicit and flexible, as
the same repository instance can be used with a database pool or a
specific transaction without needing to be re-instantiated.
### Key Changes
- **Explicit `QueryExecutor` Passing:**
- All repository methods (`Get`, `List`, `Create`, `Update`, `Delete`,
etc.) in `InstanceRepository`, `OrganizationRepository`,
`UserRepository`, and their sub-repositories now require a
`database.QueryExecutor` (e.g., a `*pgxpool.Pool` or `pgx.Tx`) as the
first argument.
- Repository constructors no longer accept a `QueryExecutor`. For
example, `repository.InstanceRepository(pool)` is now
`repository.InstanceRepository()`.
- **Enhanced Error Handling:**
- A new `database.MissingConditionError` is introduced to enforce
required query conditions, such as ensuring an `instance_id` is always
present in `UPDATE` and `DELETE` operations.
- The database error wrapper in the `postgres` package now correctly
identifies and wraps `pgx.ErrTooManyRows` and similar errors from the
`scany` library into a `database.MultipleRowsFoundError`.
- **Improved Database Conditions:**
- The `database.Condition` interface now includes a
`ContainsColumn(Column) bool` method. This allows for runtime checks to
ensure that critical filters (like `instance_id`) are included in a
query, preventing accidental cross-tenant data modification.
- A new `database.Exists()` condition has been added to support `EXISTS`
subqueries, enabling more complex filtering logic, such as finding an
organization that has a specific domain.
- **Repository and Interface Refactoring:**
- The method for loading related entities (e.g., domains for an
organization) has been changed from a boolean flag (`Domains(true)`) to
a more explicit, chainable method (`LoadDomains()`). This returns a new
repository instance configured to load the sub-resource, promoting
immutability.
- The custom `OrgIdentifierCondition` has been removed in favor of using
the standard `database.Condition` interface, simplifying the API.
- **Code Cleanup and Test Updates:**
- Unnecessary struct embeddings and metadata have been removed.
- All integration and repository tests have been updated to reflect the
new method signatures, passing the database pool or transaction object
explicitly.
- New tests have been added to cover the new `ExistsDomain`
functionality and other enhancements.
These changes make the data access layer more robust, predictable, and
easier to work with, especially in the context of database transactions.
Improves compatibility of eventstore and related database components
with the new relational table package.
## Which problems are solved
1. **Incompatible Database Interfaces**: The existing eventstore was
tightly coupled to the database package, which is incompatible with the
new, more abstract relational table package in v3. This prevented the
new command-side logic from pushing events to the legacy eventstore.
2. **Missing Health Checks**: The database interfaces in the new package
lacked a Ping method, making it impossible to perform health checks on
database connections.
3. **Event Publishing Logic**: The command handling logic in domain
needed a way to collect and push events to the legacy eventstore after a
command was successfully executed.
## How the problems are solved
1. **`LegacyEventstore` Interface**:
* A new `LegacyEventstore` interface is introduced in the new
`database/eventstore` . This interface exposes a `PushWithNewClient`
method that accepts the new `database.QueryExecutor` interface,
decoupling the v3 domain from the legacy implementation.
* The `internal/eventstore.Eventstore` now implements this interface. A
wrapper, PushWithClient, is added to convert the old database client
types (`*sql.DB`, `*sql.Tx`) into the new `QueryExecutor` types before
calling `PushWithNewClient`.
2. **Database Interface Updates**:
* The `database.Pool` and `database.Client` interfaces in
`storage/eventstore` have been updated to include a Ping method,
allowing for consistent health checks across different database
dialects.
* The `postgres` and `sql` dialect implementations have been updated to
support this new method.
3. **Command and Invoker Refactoring**:
* The `Commander` interface in domain now includes an `Events()
[]legacy_es.Command` method. This allows commands to declare which
events they will generate.
* The `eventCollector` in the invoker logic has been redesigned. It now
ensures a database transaction is started before executing a command.
After successful execution, it calls the `Events()` method on the
command to collect the generated events and appends them to a list.
* The `eventStoreInvoker` then pushes all collected events to the legacy
eventstore using the new `LegacyEventstore` interface, ensuring that
events are only pushed if the entire command (and any sub-commands)
executes successfully within the transaction.
4. **Testing**:
* New unit tests have been added for the invoker to verify that events
are correctly collected from single commands, batched commands, and
nested commands.
These changes create a clean bridge between the new v3 command-side
logic and the existing v1 eventstore, allowing for incremental adoption
of the new architecture while maintaining full functionality.
## Additional Information
closes https://github.com/zitadel/zitadel/issues/10442
This PR initiates the rework of Zitadel's backend to state-persisted
objects. This change is a step towards a more scalable and maintainable
architecture.
## Changes
* **New `/backend/v3` package**: A new package structure has been
introduced to house the reworked backend logic. This includes:
* `domain`: Contains the core business logic, commands, and repository
interfaces.
* `storage`: Implements the repository interfaces for database
interactions with new transactional tables.
* `telemetry`: Provides logging and tracing capabilities.
* **Transactional Tables**: New database tables have been defined for
`instances`, `instance_domains`, `organizations`, and `org_domains`.
* **Projections**: New projections have been created to populate the new
relational tables from the existing event store, ensuring data
consistency during the migration.
* **Repositories**: New repositories provide an abstraction layer for
accessing and manipulating the data in the new tables.
* **Setup**: A new setup step for `TransactionalTables` has been added
to manage the database migrations for the new tables.
This PR lays the foundation for future work to fully transition to
state-persisted objects for these components, which will improve
performance and simplify data access patterns.
This PR initiates the rework of ZITADEL's backend to state-persisted
objects. This is a foundational step towards a new architecture that
will improve performance and maintainability.
The following objects are migrated from event-sourced aggregates to
state-persisted objects:
* Instances
* incl. Domains
* Orgs
* incl. Domains
The structure of the new backend implementation follows the software
architecture defined in this [wiki
page](https://github.com/zitadel/zitadel/wiki/Software-Architecturel).
This PR includes:
* The initial implementation of the new transactional repositories for
the objects listed above.
* Projections to populate the new relational tables from the existing
event store.
* Adjustments to the build and test process to accommodate the new
backend structure.
This is a work in progress and further changes will be made to complete
the migration.
---------
Co-authored-by: Copilot <[email protected]>
Co-authored-by: Iraq Jaber <[email protected]>
Co-authored-by: Iraq <[email protected]>
Co-authored-by: Tim Möhlmann <[email protected]>
