# Which Problems Are Solved
As part of our efforts to simplify the structure and versions of our
APIs, were moving all existing v2beta endpoints to v2 and deprecate
them. They will be removed in Zitadel V5.
# How the Problems Are Solved
- This PR moves the authorization v2beta service and its endpoints to a
corresponding v2 version. The v2beta service and endpoints are
deprecated.
- The docs are moved to the new GA service and its endpoints. The v2beta
is not displayed anymore.
- The comments and have been improved and, where not already done, moved
from swagger annotations to proto.
- All required fields have been marked with (google.api.field_behavior)
= REQUIRED and validation rules have been added where missing.
- The `organization_id` to create an authorization is now required to be
always passed. There's no implicit fallback to the project's
organization anymore.
- The `user_id` filter has been removed in favor of the recently added
`in_user_ids` filter.
- The returned `Authorization` object has been reworked to return
`project`, `organization` and `roles` as objects like the granted `user`
already was.
- Additionally the `roles` now not only contain the granted `role_keys`,
but also the `display_name` and `group`. To implement this the query has
been updated internally. Existing APIs are unchanged and still return
just the keys.
# Additional Changes
None
# Additional Context
- part of https://github.com/zitadel/zitadel/issues/10772
- closes#10746
- requires backport to v4.x
# Which Problems Are Solved
This PR implements the endpoints listed in #10445 using relational
tables.
- [x] UpdateOrganization
- [x] ListOrganizations
- [x] DeleteOrganization
- [x] DeactivateOrganization
- [x] ActivateOrganization
# How the Problems Are Solved
- **UpdateOrganization:** Implemented logic for `Validate()`,
`Execute()` and `Events()`. On CQRS side, updating an organization emits
events related to the organization and its domains. Separate commanders
have been made for the domain events so that their logic is run
accordingly and we keep separation of concerns. Implementation of the
domains is left to its own ticket.
Because some domain-related data is changed during the organization
update, its original values are saved as pointers inside
[UpdateOrgCommand](https://github.com/zitadel/zitadel/blob/4f87cd8d0cfd75f290aa13c342e4939cecebef73/backend/v3/domain/org_update.go#L20).
These variables will be then used as input for the domain commanders
(that will run after the update organization one).
- **ListOrganizations:** Implemented logic for `Validate()`,
`Execute()`. Utility methods have been made to parse the input queries.
This commander is used both by the v2 and v2beta endpoint, so a
translation layer has been put in place to translate all v2beta requests
into v2. The commander strictly handles v2 request only.
The translation layer between v2beta and v2 APIs was made so to make it
simple to remove it once the v2beta APIs will be removed (a simple
deletion should suffice). A `converter` package [is
shipped](https://github.com/zitadel/zitadel/blob/4f87cd8d0cfd75f290aa13c342e4939cecebef73/backend/v3/api/org/v2/convert/README.md#L5)
as well with instruction for removal. TODOs have been put in place all
over the code to indicate that the related piece of code needs to be
removed once v2beta -> v2 transition is complete.
- **DeleteOrganization:** Implemented logic for `Validate()`,
`Execute()` and `Events()`. The `Events()` method is mostly incomplete
as it requires a lot of data that is currently not retrievable due to
the lack of their relational tables.
- **DeactivateOrganization** and **ActivateOrganization**: Implemented
logic for `Validate()`, `Execute()` and `Events()`. No notable remarks.
👉 All endpoints return matching errors to the CQRS counterpart: this way
allows re-using the integration tests and
👉 All endpoints have been unit-tested through DB mocking
👉 A bunch of TODOs have been put in place for future reworks that are
needed (e.g. `Update()` method on repository should also return a
timestamp)
# Additional Changes
- A test utility method to allow mocking query options have been made.
This was necessary because `database.QueryOption` is a function and the
only comparison possible with functions is `nil` check. So, to mock
this, `database.QueryOption` is converted first to `database.QueryOpts`.
See 2276742adaef1998ea4844e1f0dca2079e45ccfb
- A `BaseCommand` has been created with the idea to collect methods that
are going to be needed by other commands. For the moment, it only offers
a method to convert query text operations from gRPC to `domain` model.
See bb85456dd101282fa76d2e2ad165eed4c8ef61b9
- SQL operations have been reworked to allow for queries using
`ContainsIgnoreCase`, `EndsWithIgnoreCase` and other `IgnoreCase`
operations. This change is pending approval and might be reworked. See
https://github.com/zitadel/zitadel/pull/10704#discussion_r2392886177
# Additional Context
`UpdateOrganization` is partially working because the logic for handling
domain updates is not done (there is only a draft of the commanders to
showcase the call chain). See
https://github.com/zitadel/zitadel/pull/10704#discussion_r2356143178
Permissions are not implemented as they are missing. See
https://github.com/zitadel/zitadel/pull/10771
- Closes: #10445
---------
Co-authored-by: Silvan <[email protected]>
# Which Problems Are Solved
As part of our efforts to simplify the structure and versions of our
APIs, were moving all existing v2beta endpoints to v2 and deprecate
them. They will be removed in Zitadel V5.
# How the Problems Are Solved
- This PR moves app v2beta service and its endpoints to a corresponding
to application v2 version. The v2beta service and endpoints are
deprecated.
- The comments and have been improved and, where not already done, moved
from swagger annotations to proto.
- All required fields have been marked with (google.api.field_behavior)
= REQUIRED and validation rules have been added where missing.
- Name ID of the application always `application_id`, previously was
also `id` and `app_id`.
- Get rid of all `app` abbreviations and name it `application` including
the service name, `AppState` -> `ApplicationState` and `AppSorting` ->
`ApplicationSorting`
- Updated `CreateApplicationRequest`:
- renamed `creation_request_type` to `application_type` and all its
options to `XY_configuration` instead of `XY_request`
- `RegenerateClientSecret`
- renamed method to `GenerateClientSecret`
- removed `app_type` from request
- `ListApplicationRequest`:
- removed required `project_id` and provided it as a filter
- Type `ApplicationNameQuery` has been renamed to
`ApplicationNameFilter` as its usage in the request
- Renamed all fields and types from `config` to `configuration`
- Updated `DeleteApplicationKeyRequest`
- removed `organization_id`
- Updated `GetApplicationKeyRequest`:
- removed `project_id`, `application_id` and `organization_id``
- Updated `ListApplicationKeysRequest`:
- removed oneOf `resource_id` and moved the options into filters
- Name ID of the application key always `key_id`.
- removed unnecessary package prefixed (`zitadel.application.v2`)
- formatted using `buf`
# Additional Changes
None
# Additional Context
- part of https://github.com/zitadel/zitadel/issues/10772
- requires backport to v4.x
# Which Problems Are Solved
As part of our efforts to simplify the structure and versions of our
APIs, were moving all existing v2beta endpoints to v2 and deprecate
them. They will be removed in Zitadel V5.
# How the Problems Are Solved
- This PR moves the internal permission v2beta service and its endpoints
to a corresponding v2 version. The v2beta service and endpoints are
deprecated.
- The docs are moved to the new GA service and its endpoints. The v2beta
is not displayed anymore.
- The comments and have been improved and, where not already done, moved
from swagger annotations to proto.
- All required fields have been marked with (google.api.field_behavior)
= REQUIRED and validation rules have been added where missing.
- Listing administrators of a project grant can now be done with the
`ProjectGrant` (`project_id` and `organization_id`) instead of a
`project_id`, which corresponds to creation of the administrator ship of
such grant.
- formatted using `buf`
# Additional Changes
None
# Additional Context
- part of https://github.com/zitadel/zitadel/issues/10772
- requires backport to v4.x
---------
Co-authored-by: Gayathri Vijayan <[email protected]>
# Which Problems Are Solved
Replaces Turbo by Nx and lays the foundation for the next CI
improvements. It enables using Nx Cloud to speed the up the pipelines
that affect any node package.
It streamlines the dev experience for frontend and backend developers by
providing the following commands:
| Task | Command | Notes |
|------|---------|--------|
| **Production** | `nx run PROJECT:prod` | Production server |
| **Develop** | `nx run PROJECT:dev` | Hot reloading development server
|
| **Test** | `nx run PROJECT:test` | Run all tests |
| **Lint** | `nx run PROJECT:lint` | Check code style |
| **Lint Fix** | `nx run PROJECT:lint-fix` | Auto-fix style issues |
The following values can be used for PROJECT:
- @zitadel/zitadel (root commands)
- @zitadel/api,
- @zitadel/login,
- @zitadel/console,
- @zitadel/docs,
- @zitadel/client
- @zitadel/proto
The project names and folders are streamlined:
| Old Folder | New Folder |
| --- | --- |
| ./e2e | ./tests/functional-ui |
| ./load-test | ./benchmark |
| ./build/zitadel | ./apps/api |
| ./console | ./apps/console (postponed so the PR is reviewable) |
Also, all references to the TypeScript repo are removed so we can
archive it.
# How the Problems Are Solved
- Ran `npx nx@latest init`
- Replaced all turbo.json by project.json and fixed the target configs
- Removed Turbo dependency
- All JavaScript related code affected by a PRs changes is
quality-checked using the `nx affected` command
- We move PR checks that are runnable using Nx into the `check`
workflow. For workflows where we don't use Nx, yet, we restore
previously built dependency artifacts from Nx.
- We only use a single and easy to understand dev container
- The CONTRIBUTING.md is streamlined
- The setup with a generated client pat is orchestrated with Nx
- Everything related to the TypeScript repo is updated or removed. A
**Deploy with Vercel** button is added to the docs and the
CONTRIBUTING.md.
# Additional Changes
- NPM package names have a consistent pattern.
- Docker bake is removed. The login container is built and released like
the core container.
- The integration tests build the login container before running, so
they don't rely on the login container action anymore. This fixes
consistently failing checks on PRs from forks.
- The docs build in GitHub actions is removed, as we already build on
Vercel.
# Additional Context
- Internal discussion:
https://zitadel.slack.com/archives/C087ADF8LRX/p1756277884928169
- Workflow dispatch test:
https://github.com/zitadel/zitadel/actions/runs/17760122959
---------
Co-authored-by: Florian Forster <[email protected]>
Co-authored-by: Tim Möhlmann <[email protected]>
Co-authored-by: Copilot <[email protected]>
# Which Problems Are Solved
This PR adds API definition and backend implementation for GroupService
to manage user groups.
# How the Problems Are Solved
* API definition to create, update, retrieve, and delete groups is added
* Command-side implementation to create, update, and delete user groups
as part of the GroupV2 API is added
# Additional Changes
N/A
# Additional Context
- Related to #10089, #9702 (parent ticket)
- User contribution: https://github.com/zitadel/zitadel/pull/9428/files
- Additional functionalities to list/search user groups, add
permissions, manage users in groups, group scopes will be added in
subsequent PRs.
- Also needs documentation, which will be added once the entire feature
is available
---------
Co-authored-by: Livio Spring <[email protected]>
# Which Problems Are Solved
The current cache interface implementation for postgres is not
compatible with Postgres18, since we rely on partitioned unlogged
tables, which are no longer supported.
# How the Problems Are Solved
Use postgres 17 and update compatibility in the docs.
# Additional Changes
None
# Additional Context
- requires backport to v3.x, v4.x
# Which Problems Are Solved
Project Grant ID would have needed to be unique to be handled properly
on the projections, but was defined as the organization ID the project
was granted to, so could be non-unique.
# How the Problems Are Solved
Generate the Project Grant ID even in the v2 APIs, which also includes
fixes in the integration tests.
Additionally to that, the logic for some functionality had to be
extended as the Project Grant ID is not provided anymore in the API, so
had to be queried before creating events for Project Grants.
# Additional Changes
Included fix for authorizations, when an authorization was intended to
be created for a project, without providing any organization
information, which also showed some faulty integration tests.
# Additional Context
Partially closes#10745
---------
Co-authored-by: Livio Spring <[email protected]>
Co-authored-by: Silvan <[email protected]>
# Which Problems Are Solved
This PR fixes the self-management of users for metadata and own removal
and improves the corresponding permission checks.
While looking into the problems, I also noticed that there's a bug in
the metadata mapping when using `api.metadata.push` in actions v1 and
that re-adding a previously existing key after its removal was not
possible.
# How the Problems Are Solved
- Added a parameter `allowSelfManagement` to checkPermissionOnUser to
not require a permission if a user is changing its own data.
- Updated use of `NewPermissionCheckUserWrite` including prevention of
self-management for metadata.
- Pass permission check to the command side (for metadata functions) to
allow it implicitly for login v1 and actions v1.
- Use of json.Marshal for the metadata mapping (as with
`AppendMetadata`)
- Check the metadata state when comparing the value.
# Additional Changes
- added a variadic `roles` parameter to the `CreateOrgMembership`
integration test helper function to allow defining specific roles.
# Additional Context
- noted internally while testing v4.1.x
- requires backport to v4.x
- closes https://github.com/zitadel/zitadel/issues/10470
- relates to https://github.com/zitadel/zitadel/pull/10426
# Which Problems Are Solved
Flakiness and conflicts in value from gofakeit.
# How the Problems Are Solved
Move Gofakeit calls to the integration package, to guarantee proper
usage and values for integration testing.
# Additional Changes
None
# Additional Context
None
This PR initiates the rework of Zitadel's backend to state-persisted
objects. This change is a step towards a more scalable and maintainable
architecture.
## Changes
* **New `/backend/v3` package**: A new package structure has been
introduced to house the reworked backend logic. This includes:
* `domain`: Contains the core business logic, commands, and repository
interfaces.
* `storage`: Implements the repository interfaces for database
interactions with new transactional tables.
* `telemetry`: Provides logging and tracing capabilities.
* **Transactional Tables**: New database tables have been defined for
`instances`, `instance_domains`, `organizations`, and `org_domains`.
* **Projections**: New projections have been created to populate the new
relational tables from the existing event store, ensuring data
consistency during the migration.
* **Repositories**: New repositories provide an abstraction layer for
accessing and manipulating the data in the new tables.
* **Setup**: A new setup step for `TransactionalTables` has been added
to manage the database migrations for the new tables.
This PR lays the foundation for future work to fully transition to
state-persisted objects for these components, which will improve
performance and simplify data access patterns.
This PR initiates the rework of ZITADEL's backend to state-persisted
objects. This is a foundational step towards a new architecture that
will improve performance and maintainability.
The following objects are migrated from event-sourced aggregates to
state-persisted objects:
* Instances
* incl. Domains
* Orgs
* incl. Domains
The structure of the new backend implementation follows the software
architecture defined in this [wiki
page](https://github.com/zitadel/zitadel/wiki/Software-Architecturel).
This PR includes:
* The initial implementation of the new transactional repositories for
the objects listed above.
* Projections to populate the new relational tables from the existing
event store.
* Adjustments to the build and test process to accommodate the new
backend structure.
This is a work in progress and further changes will be made to complete
the migration.
---------
Co-authored-by: Copilot <[email protected]>
Co-authored-by: Iraq Jaber <[email protected]>
Co-authored-by: Iraq <[email protected]>
Co-authored-by: Tim Möhlmann <[email protected]>
This PR overhauls our event projection system to make it more robust and
prevent skipped events under high load. The core change replaces our
custom, transaction-based locking with standard PostgreSQL advisory
locks. We also introduce a worker pool to manage concurrency and prevent
database connection exhaustion.
### Key Changes
* **Advisory Locks for Projections:** Replaces exclusive row locks and
inspection of `pg_stat_activity` with PostgreSQL advisory locks for
managing projection state. This is a more reliable and standard approach
to distributed locking.
* **Simplified Await Logic:** Removes the complex logic for awaiting
open transactions, simplifying it to a more straightforward time-based
filtering of events.
* **Projection Worker Pool:** Implements a worker pool to limit
concurrent projection triggers, preventing connection exhaustion and
improving stability under load. A new `MaxParallelTriggers`
configuration option is introduced.
### Problem Solved
Under high throughput, a race condition could cause projections to miss
events from the eventstore. This led to inconsistent data in projection
tables (e.g., a user grant might be missing). This PR fixes the
underlying locking and concurrency issues to ensure all events are
processed reliably.
### How it Works
1. **Event Writing:** When writing events, a *shared* advisory lock is
taken. This signals that a write is in progress.
2. **Event Handling (Projections):**
* A projection worker attempts to acquire an *exclusive* advisory lock
for that specific projection. If the lock is already held, it means
another worker is on the job, so the current one backs off.
* Once the lock is acquired, the worker briefly acquires and releases
the same *shared* lock used by event writers. This acts as a barrier,
ensuring it waits for any in-flight writes to complete.
* Finally, it processes all events that occurred before its transaction
began.
### Additional Information
* ZITADEL no longer modifies the `application_name` PostgreSQL variable
during event writes.
* The lock on the `current_states` table is now `FOR NO KEY UPDATE`.
* Fixes https://github.com/zitadel/zitadel/issues/8509
---------
Co-authored-by: Copilot <[email protected]>
Co-authored-by: Tim Möhlmann <[email protected]>
# Which Problems Are Solved
Some users have reported the need of retrieving users given a metadata
key, metadata value or both. This change introduces metadata search
filter on the `ListUsers()` endpoint to allow Zitadel users to search
for user records by metadata.
The changes affect only v2 APIs.
# How the Problems Are Solved
- Add new search filter to `ListUserRequest`: `MetaKey` and `MetaValue`
- Add SQL indices on metadata key and metadata value
- Update query to left join `user_metadata` table
# Additional Context
- Closes#9053
- Depends on https://github.com/zitadel/zitadel/pull/10567
---------
Co-authored-by: Silvan <[email protected]>
Co-authored-by: Stefan Benz <[email protected]>
# Which Problems Are Solved
The event execution system currently uses a projection handler that
subscribes to and processes all events for all instances. This creates a
high static cost because the system over-fetches event data, handling
many events that are not needed by most instances. This inefficiency is
also reflected in high "rows returned" metrics in the database.
# How the Problems Are Solved
Eliminate the use of a project handler. Instead, events for which
"execution targets" are defined, are directly pushed to the queue by the
eventstore. A Router is populated in the Instance object in the authz
middleware.
- By joining the execution targets to the instance, no additional
queries are needed anymore.
- As part of the instance object, execution targets are now cached as
well.
- Events are queued within the same transaction, giving transactional
guarantees on delivery.
- Uses the "insert many fast` variant of River. Multiple jobs are queued
in a single round-trip to the database.
- Fix compatibility with PostgreSQL 15
# Additional Changes
- The signing key was stored as plain-text in the river job payload in
the DB. This violated our [Secrets
Storage](https://zitadel.com/docs/concepts/architecture/secrets#secrets-storage)
principle. This change removed the field and only uses the encrypted
version of the signing key.
- Fixed the target ordering from descending to ascending.
- Some minor linter warnings on the use of `io.WriteString()`.
# Additional Context
- Introduced in https://github.com/zitadel/zitadel/pull/9249
- Closes https://github.com/zitadel/zitadel/issues/10553
- Closes https://github.com/zitadel/zitadel/issues/9832
- Closes https://github.com/zitadel/zitadel/issues/10372
- Closes https://github.com/zitadel/zitadel/issues/10492
---------
Co-authored-by: Stefan Benz <[email protected]>
# Which Problems Are Solved
This change makes it easier to delete the integration database
# How the Problems Are Solved
Gives the integration database a volume you can address via name
`docker volume rm config_zitadel_integration_db`
# Which Problems Are Solved
EntraID userinfo gets incorrectly unmarshalled again in the
`RetrieveIdentityProviderIntent` endpoint.
# How the Problems Are Solved
Correctly use the already available information and not try to marshall
it into a `RawInformation` struct again.
# Additional Changes
None
# Additional Context
Closes https://github.com/zitadel/typescript/issues/578
---------
Co-authored-by: Livio Spring <[email protected]>
# Which Problems Are Solved
When a user with an `ORG_PROJECT_CREATOR` role tries to create a
project, the request fails with `No matching permissions found
(AUTH-AWfge)` error. This is because `project.write` was set as the
required permission instead of `project.create` during project creation.
# How the Problems Are Solved
By setting the right required permission (`project.create`) while
creating new projects.
# Additional Changes
N/A
# Additional Context
- Closes#10399
# Which Problems Are Solved
Flakiness in integration tests regarding gofakeit functions, which
provided the same names on 2 different occasions.
# How the Problems Are Solved
Attach a random string to the provided names, so that they are not
dependent on the gofakeit code.
# Additional Changes
None
# Additional Context
None
---------
Co-authored-by: Marco A. <[email protected]>
# Which Problems Are Solved
The Actions v2beta API is not yet promoted to GA.
# How the Problems Are Solved
Promote Actions v2Beta API to Actions v2 API.
# Additional Changes
None
# Additional Context
None
# Which Problems Are Solved
Currently the username uniqueness is on instance level, we want to
achieve a way to set it at organization level.
# How the Problems Are Solved
Addition of endpoints and a resource on organization level, where this
setting can be managed. If nothing it set, the uniqueness is expected to
be at instance level, where only users with instance permissions should
be able to change this setting.
# Additional Changes
None
# Additional Context
Includes #10086Closes#9964
---------
Co-authored-by: Marco A. <[email protected]>
# Which Problems Are Solved
The commands for the resource based v2beta AuthorizationService API are
added.
Authorizations, previously knows as user grants, give a user in a
specific organization and project context roles.
The project can be owned or granted.
The given roles can be used to restrict access within the projects
applications.
The commands for the resource based v2beta InteralPermissionService API
are added.
Administrators, previously knows as memberships, give a user in a
specific organization and project context roles.
The project can be owned or granted.
The give roles give the user permissions to manage different resources
in Zitadel.
API definitions from https://github.com/zitadel/zitadel/issues/9165 are
implemented.
Contains endpoints for user metadata.
# How the Problems Are Solved
### New Methods
- CreateAuthorization
- UpdateAuthorization
- DeleteAuthorization
- ActivateAuthorization
- DeactivateAuthorization
- ListAuthorizations
- CreateAdministrator
- UpdateAdministrator
- DeleteAdministrator
- ListAdministrators
- SetUserMetadata to set metadata on a user
- DeleteUserMetadata to delete metadata on a user
- ListUserMetadata to query for metadata of a user
## Deprecated Methods
### v1.ManagementService
- GetUserGrantByID
- ListUserGrants
- AddUserGrant
- UpdateUserGrant
- DeactivateUserGrant
- ReactivateUserGrant
- RemoveUserGrant
- BulkRemoveUserGrant
### v1.AuthService
- ListMyUserGrants
- ListMyProjectPermissions
# Additional Changes
- Permission checks for metadata functionality on query and command side
- correct existence checks for resources, for example you can only be an
administrator on an existing project
- combined all member tables to singular query for the administrators
- add permission checks for command an query side functionality
- combined functions on command side where necessary for easier
maintainability
# Additional Context
Closes#9165
---------
Co-authored-by: Elio Bischof <[email protected]>
Co-authored-by: Copilot <[email protected]>
Co-authored-by: Livio Spring <[email protected]>
# Which Problems Are Solved
The current maintained gRPC server in combination with a REST (grpc)
gateway is getting harder and harder to maintain. Additionally, there
have been and still are issues with supporting / displaying `oneOf`s
correctly.
We therefore decided to exchange the server implementation to
connectRPC, which apart from supporting connect as protocol, also also
"standard" gRCP clients as well as HTTP/1.1 / rest like clients, e.g.
curl directly call the server without any additional gateway.
# How the Problems Are Solved
- All v2 services are moved to connectRPC implementation. (v1 services
are still served as pure grpc servers)
- All gRPC server interceptors were migrated / copied to a corresponding
connectRPC interceptor.
- API.ListGrpcServices and API. ListGrpcMethods were changed to include
the connect services and endpoints.
- gRPC server reflection was changed to a `StaticReflector` using the
`ListGrpcServices` list.
- The `grpc.Server` interfaces was split into different combinations to
be able to handle the different cases (grpc server and prefixed gateway,
connect server with grpc gateway, connect server only, ...)
- Docs of services serving connectRPC only with no additional gateway
(instance, webkey, project, app, org v2 beta) are changed to expose that
- since the plugin is not yet available on buf, we download it using
`postinstall` hook of the docs
# Additional Changes
- WebKey service is added as v2 service (in addition to the current
v2beta)
# Additional Context
closes#9483
---------
Co-authored-by: Elio Bischof <[email protected]>
# Which Problems Are Solved
This PR *partially* addresses #9450 . Specifically, it implements the
resource based API for the apps. APIs for app keys ARE not part of this
PR.
# How the Problems Are Solved
- `CreateApplication`, `PatchApplication` (update) and
`RegenerateClientSecret` endpoints are now unique for all app types:
API, SAML and OIDC apps.
- All new endpoints have integration tests
- All new endpoints are using permission checks V2
# Additional Changes
- The `ListApplications` endpoint allows to do sorting (see protobuf for
details) and filtering by app type (see protobuf).
- SAML and OIDC update endpoint can now receive requests for partial
updates
# Additional Context
Partially addresses #9450
# Which Problems Are Solved
IDs filter definition was changed in another PR and not changed in the
Project service.
# How the Problems Are Solved
Correctly use the IDs filter.
# Additional Changes
Add timeout to the integration tests.
# Additional Context
None
# Which Problems Are Solved
Permission checks in project v2beta API did not cover projects and
granted projects correctly.
# How the Problems Are Solved
Add permission checks v1 correctly to the list queries, add correct
permission checks v2 for projects.
# Additional Changes
Correct Pre-Checks for project grants that the right resource owner is
used.
# Additional Context
Permission checks v2 for project grants is still outstanding under
#9972.
# Which Problems Are Solved
This pull request addresses a significant gap in the user service v2
API, which currently lacks methods for managing machine users.
# How the Problems Are Solved
This PR adds new API endpoints to the user service v2 to manage machine
users including their secret, keys and personal access tokens.
Additionally, there's now a CreateUser and UpdateUser endpoints which
allow to create either a human or machine user and update them. The
existing `CreateHumanUser` endpoint has been deprecated along the
corresponding management service endpoints. For details check the
additional context section.
# Additional Context
- Closes https://github.com/zitadel/zitadel/issues/9349
## More details
- API changes: https://github.com/zitadel/zitadel/pull/9680
- Implementation: https://github.com/zitadel/zitadel/pull/9763
- Tests: https://github.com/zitadel/zitadel/pull/9771
## Follow-ups
- Metadata: support managing user metadata using resource API
https://github.com/zitadel/zitadel/pull/10005
- Machine token type: support managing the machine token type (migrate
to new enum with zero value unspecified?)
---------
Co-authored-by: Copilot <[email protected]>
Co-authored-by: Livio Spring <[email protected]>
# Which Problems Are Solved
The login v1 allowed to use JWTs as IdP using the JWT IDP. The login V2
uses idp intents for such cases, which were not yet able to handle JWT
IdPs.
# How the Problems Are Solved
- Added handling of JWT IdPs in `StartIdPIntent` and `RetrieveIdPIntent`
- The redirect returned by the start, uses the existing `authRequestID`
and `userAgentID` parameter names for compatibility reasons.
- Added `/idps/jwt` endpoint to handle the proxied (callback) endpoint ,
which extracts and validates the JWT against the configured endpoint.
# Additional Changes
None
# Additional Context
- closes#9758
# Which Problems Are Solved
Resource management of projects and sub-resources was before limited by
the context provided by the management API, which would mean you could
only manage resources belonging to a specific organization.
# How the Problems Are Solved
With the addition of a resource-based API, it is now possible to manage
projects and sub-resources on the basis of the resources themselves,
which means that as long as you have the permission for the resource,
you can create, read, update and delete it.
- CreateProject to create a project under an organization
- UpdateProject to update an existing project
- DeleteProject to delete an existing project
- DeactivateProject and ActivateProject to change the status of a
project
- GetProject to query for a specific project with an identifier
- ListProject to query for projects and granted projects
- CreateProjectGrant to create a project grant with project and granted
organization
- UpdateProjectGrant to update the roles of a project grant
- DeactivateProjectGrant and ActivateProjectGrant to change the status
of a project grant
- DeleteProjectGrant to delete an existing project grant
- ListProjectGrants to query for project grants
- AddProjectRole to add a role to an existing project
- UpdateProjectRole to change texts of an existing role
- RemoveProjectRole to remove an existing role
- ListProjectRoles to query for project roles
# Additional Changes
- Changes to ListProjects, which now contains granted projects as well
- Changes to messages as defined in the
[API_DESIGN](https://github.com/zitadel/zitadel/blob/main/API_DESIGN.md)
- Permission checks for project functionality on query and command side
- Added testing to unit tests on command side
- Change update endpoints to no error returns if nothing changes in the
resource
- Changed all integration test utility to the new service
- ListProjects now also correctly lists `granted projects`
- Permission checks for project grant and project role functionality on
query and command side
- Change existing pre checks so that they also work resource specific
without resourceowner
- Added the resourceowner to the grant and role if no resourceowner is
provided
- Corrected import tests with project grants and roles
- Added testing to unit tests on command side
- Change update endpoints to no error returns if nothing changes in the
resource
- Changed all integration test utility to the new service
- Corrected some naming in the proto files to adhere to the API_DESIGN
# Additional Context
Closes#9177
---------
Co-authored-by: Livio Spring <[email protected]>
# Which Problems Are Solved
- It is not possible to specify a custom organization ID when creating
an organization. According to
https://github.com/zitadel/zitadel/discussions/9202#discussioncomment-11929464
this is "an inconsistency in the V2 API".
# How the Problems Are Solved
- Adds the `org_id` as an optional parameter to the
`AddOrganizationRequest` in the `v2beta` API.
# Additional Changes
None.
# Additional Context
- Discussion
[#9202](https://github.com/zitadel/zitadel/discussions/9202)
- I was mostly interested in how much work it'd be to add this field.
Then after completing this, I thought I'd submit this PR. I won't be
angry if you just close this PR with the reasoning "we didn't ask for
it". 😄
- Even though I don't think this is a breaking change, I didn't add this
to the `v2` API yet (don't know what the process for this is TBH). The
changes should be analogous, so if you want me to, just request it.
---------
Co-authored-by: Stefan Benz <[email protected]>
<!--
Please inform yourself about the contribution guidelines on submitting a
PR here:
https://github.com/zitadel/zitadel/blob/main/CONTRIBUTING.md#submit-a-pull-request-pr.
Take note of how PR/commit titles should be written and replace the
template texts in the sections below. Don't remove any of the sections.
It is important that the commit history clearly shows what is changed
and why.
Important: By submitting a contribution you agree to the terms from our
Licensing Policy as described here:
https://github.com/zitadel/zitadel/blob/main/LICENSING.md#community-contributions.
-->
# Which Problems Are Solved
These changes introduce resource-based API endpoints for managing
instances and custom domains.
There are 4 types of changes:
- Endpoint implementation: consisting of the protobuf interface and the
implementation of the endpoint. E.g:
606439a17227b629c1d018842dc3f1c569e4627a
- (Integration) Tests: testing the implemented endpoint. E.g:
cdfe1f0372b30cb74e34f0f23c6ada776e4477e9
- Fixes: Bugs found during development that are being fixed. E.g:
acbbeedd3259b785948c1d702eb98f5810b3e60a
- Miscellaneous: code needed to put everything together or that doesn't
fit any of the above categories. E.g:
529df92abce1ffd69c0b3214bd835be404fd0de0 or
6802cb5468fbe24664ae6639fd3a40679222a2fd
# How the Problems Are Solved
_Ticked checkboxes indicate that the functionality is complete_
- [x] Instance
- [x] Create endpoint
- [x] Create endpoint tests
- [x] Update endpoint
- [x] Update endpoint tests
- [x] Get endpoint
- [x] Get endpoint tests
- [x] Delete endpoint
- [x] Delete endpoint tests
- [x] Custom Domains
- [x] Add custom domain
- [x] Add custom domain tests
- [x] Remove custom domain
- [x] Remove custom domain tests
- [x] List custom domains
- [x] List custom domains tests
- [x] Trusted Domains
- [x] Add trusted domain
- [x] Add trusted domain tests
- [x] Remove trusted domain
- [x] Remove trusted domain tests
- [x] List trusted domains
- [x] List trusted domains tests
# Additional Changes
When looking for instances (through the `ListInstances` endpoint)
matching a given query, if you ask for the results to be order by a
specific column, the query will fail due to a syntax error. This is
fixed in acbbeedd3259b785948c1d702eb98f5810b3e60a . Further explanation
can be found in the commit message
# Additional Context
- Relates to #9452
- CreateInstance has been excluded:
https://github.com/zitadel/zitadel/issues/9930
- Permission checks / instance retrieval (middleware) needs to be
changed to allow context based permission checks
(https://github.com/zitadel/zitadel/issues/9929), required for
ListInstances
---------
Co-authored-by: Livio Spring <[email protected]>
# Which Problems Are Solved
Actions v2 is not a feature flag anymore, include functionality on
executions is not used and json tags of proto messages are handled
incorrectly.
# How the Problems Are Solved
- Remove actions from the feature flags on system and instance level
- Remove include type on executions, only in the API, later maybe in the
handling logic as well
- Use protojson in request and response handling of actions v2
# Additional Changes
- Correct integration tests for request and response handling
- Use json.RawMessage for events, so that the event payload is not
base64 encoded
- Added separate context for async webhook calls, that executions are
not cancelled when called async
# Additional Context
Related to #9759Closes#9710
---------
Co-authored-by: Livio Spring <[email protected]>
This pull request improves the scalability of the session API by
enhancing middleware tracing and refining SQL query behavior for user
authentication methods.
# Which Problems Are Solved
- Eventstore subscriptions locked each other during they wrote the
events to the event channels of the subscribers in push.
- `ListUserAuthMethodTypesRequired` query used `Bitmap heap scan` to
join the tables needed.
- The auth and oidc package triggered projections often when data were
read.
- The session API triggered the user projection each time a user was
searched to write the user check command.
# How the Problems Are Solved
- the `sync.Mutex` was replaced with `sync.RWMutex` to allow parallel
read of the map
- The query was refactored to use index scans only
- if the data should already be up-to-date `shouldTriggerBulk` is set to
false
- as the user should already exist for some time the trigger was
removed.
# Additional Changes
- refactoring of `tracing#Span.End` calls
# Additional Context
- part of https://github.com/zitadel/zitadel/issues/9239
---------
Co-authored-by: Tim Möhlmann <[email protected]>
# Which Problems Are Solved
Actions v2 are not executed in different functions, as provided by the
actions v1.
# How the Problems Are Solved
Add functionality to call actions v2 through OIDC and SAML logic to
complement tokens and SAMLResponses.
# Additional Changes
- Corrected testing for retrieved intent information
- Added testing for IDP types
- Corrected handling of context for issuer in SAML logic
# Additional Context
- Closes#7247
- Dependent on https://github.com/zitadel/saml/pull/97
- docs for migration are done in separate issue:
https://github.com/zitadel/zitadel/issues/9456
---------
Co-authored-by: Silvan <[email protected]>
# Which Problems Are Solved
The recently introduced notification queue have potential race conditions.
# How the Problems Are Solved
Current code is refactored to use the queue package, which is safe in
regards of concurrency.
# Additional Changes
- the queue is included in startup
- improved code quality of queue
# Additional Context
- closes https://github.com/zitadel/zitadel/issues/9278
# Which Problems Are Solved
Setup fails to push all role permission events when running Zitadel with
CockroachDB. `TransactionRetryError`s were visible in logs which finally
times out the setup job with `timeout: context deadline exceeded`
# How the Problems Are Solved
As suggested in the [Cockroach documentation](timeout: context deadline
exceeded), _"break down larger transactions"_. The commands to be pushed
for the role permissions are chunked in 50 events per push. This
chunking is only done with CockroachDB.
# Additional Changes
- gci run fixed some unrelated imports
- access to `command.Commands` for the setup job, so we can reuse the
sync logic.
# Additional Context
Closes#9293
---------
Co-authored-by: Silvan <[email protected]>
# Which Problems Are Solved
The `creationDate` property on user search V2 endpoint was missing
# How the Problems Are Solved
Added property in v2 `object.proto` and in the function creating the
details on each call
# Additional Changes
- none
# Additional Context
closes#8552
---------
Co-authored-by: Stefan Benz <[email protected]>
# Which Problems Are Solved
Some OAuth2 and OIDC providers require the use of PKCE for all their
clients. While ZITADEL already recommended the same for its clients, it
did not yet support the option on the IdP configuration.
# How the Problems Are Solved
- A new boolean `use_pkce` is added to the add/update generic OAuth/OIDC
endpoints.
- A new checkbox is added to the generic OAuth and OIDC provider
templates.
- The `rp.WithPKCE` option is added to the provider if the use of PKCE
has been set.
- The `rp.WithCodeChallenge` and `rp.WithCodeVerifier` options are added
to the OIDC/Auth BeginAuth and CodeExchange function.
- Store verifier or any other persistent argument in the intent or auth
request.
- Create corresponding session object before creating the intent, to be
able to store the information.
- (refactored session structs to use a constructor for unified creation
and better overview of actual usage)
Here's a screenshot showing the URI including the PKCE params:
# Additional Changes
None.
# Additional Context
- Closes#6449
- This PR replaces the existing PR (#8228) of @doncicuto. The base he
did was cherry picked. Thank you very much for that!
---------
Co-authored-by: Miguel Cabrerizo <[email protected]>
Co-authored-by: Stefan Benz <[email protected]>
# Which Problems Are Solved
The OAuth2 Device Authorization Grant could not yet been handled through
the new login UI, resp. using the session API.
This PR adds the ability for the login UI to get the required
information to display the user and handle their decision (approve with
authorization or deny) using the OIDC Service API.
# How the Problems Are Solved
- Added a `GetDeviceAuthorizationRequest` endpoint, which allows getting
the `id`, `client_id`, `scope`, `app_name` and `project_name` of the
device authorization request
- Added a `AuthorizeOrDenyDeviceAuthorization` endpoint, which allows to
approve/authorize with the session information or deny the request. The
identification of the request is done by the `device_authorization_id` /
`id` returned in the previous request.
- To prevent leaking the `device_code` to the UI, but still having an
easy reference, it's encrypted and returned as `id`, resp. decrypted
when used.
- Fixed returned error types for device token responses on token
endpoint:
- Explicitly return `access_denied` (without internal error) when user
denied the request
- Default to `invalid_grant` instead of `access_denied`
- Explicitly check on initial state when approving the reqeust
- Properly handle done case (also relates to initial check)
- Documented the flow and handling in custom UIs (according to OIDC /
SAML)
# Additional Changes
- fixed some typos and punctuation in the corresponding OIDC / SAML
guides.
- added some missing translations for auth and saml request
# Additional Context
- closes#6239
---------
Co-authored-by: Tim Möhlmann <[email protected]>
# Which Problems Are Solved
New integration tests can't use command side to simulate successful
intents.
# How the Problems Are Solved
Add endpoints to only in integration tests available sink to create
already successful intents.
# Additional Changes
None
# Additional Context
Closes#8557
---------
Co-authored-by: Livio Spring <[email protected]>
# Which Problems Are Solved
OIDC applications can configure the used login version, which is
currently not possible for SAML applications.
# How the Problems Are Solved
Add the same functionality dependent on the feature-flag for SAML
applications.
# Additional Changes
None
# Additional Context
Closes#9267
Follow up issue for frontend changes #9354
---------
Co-authored-by: Livio Spring <[email protected]>
# Which Problems Are Solved
Through configuration on projects, there can be additional permission
checks enabled through an OIDC or SAML flow, which were not included in
the OIDC and SAML services.
# How the Problems Are Solved
Add permission check through the query-side of Zitadel in a singular SQL
query, when an OIDC or SAML flow should be linked to a SSO session. That
way it is eventual consistent, but will not impact the performance on
the eventstore. The permission check is defined in the API, which
provides the necessary function to the command side.
# Additional Changes
Added integration tests for the permission check on OIDC and SAML
service for every combination.
Corrected session list integration test, to content checks without
ordering.
Corrected get auth and saml request integration tests, to check for
timestamp of creation, not start of test.
# Additional Context
Closes#9265
---------
Co-authored-by: Livio Spring <[email protected]>
# Which Problems Are Solved
* Adds support for the service provider configuration SCIM v2 endpoints
# How the Problems Are Solved
* Adds support for the service provider configuration SCIM v2 endpoints
* `GET /scim/v2/{orgId}/ServiceProviderConfig`
* `GET /scim/v2/{orgId}/ResourceTypes`
* `GET /scim/v2/{orgId}/ResourceTypes/{name}`
* `GET /scim/v2/{orgId}/Schemas`
* `GET /scim/v2/{orgId}/Schemas/{id}`
# Additional Context
Part of #8140
Co-authored-by: Stefan Benz <[email protected]>
# Which Problems Are Solved
* Adds support for the bulk SCIM v2 endpoint
# How the Problems Are Solved
* Adds support for the bulk SCIM v2 endpoint under `POST
/scim/v2/{orgID}/Bulk`
# Additional Context
Part of #8140
Co-authored-by: Stefan Benz <[email protected]>
# Which Problems Are Solved
* Adds support for the patch user SCIM v2 endpoint
# How the Problems Are Solved
* Adds support for the patch user SCIM v2 endpoint under `PATCH
/scim/v2/{orgID}/Users/{id}`
# Additional Context
Part of #8140
# Which Problems Are Solved
- Adds support for the list users SCIM v2 endpoint
# How the Problems Are Solved
- Adds support for the list users SCIM v2 endpoints under `GET
/scim/v2/{orgID}/Users` and `POST /scim/v2/{orgID}/Users/.search`
# Additional Changes
- adds a new function `SearchUserMetadataForUsers` to the query layer to
query a metadata keyset for given user ids
- adds a new function `NewUserMetadataExistsQuery` to the query layer to
query a given metadata key value pair exists
- adds a new function `CountUsers` to the query layer to count users
without reading any rows
- handle `ErrorAlreadyExists` as scim errors `uniqueness`
- adds `NumberLessOrEqual` and `NumberGreaterOrEqual` query comparison
methods
- adds `BytesQuery` with `BytesEquals` and `BytesNotEquals` query
comparison methods
# Additional Context
Part of #8140
Supported fields for scim filters:
* `meta.created`
* `meta.lastModified`
* `id`
* `username`
* `name.familyName`
* `name.givenName`
* `emails` and `emails.value`
* `active` only eq and ne
* `externalId` only eq and ne
Livio SpringandGitHub
