docs: introduce versioned docs and migrate to fuma (#11166)

## Todos for release

- [x] Configure Env in docs project on vercel
- [x] Configure Root Path in the docs project on vercel
- [ ] Remove old CSP https://github.com/zitadel/website/pull/1592

## What we did

This pull request migrates the project documentation from the old
`docs/` directory to the new `apps/docs/` directory, introduces a new
documentation system built with Next.js and Fumadocs, and updates all
relevant references, configuration files, and documentation to reflect
this change. It also adds new configuration and ignore files for the new
documentation app, updates CI and linting to exclude the new docs from
certain checks, and revises the contributing guidelines accordingly.

**Documentation System Migration and New Docs App**

* Migrated all documentation from `docs/` to `apps/docs/`, and updated
all references in `README.md`, `CONTRIBUTING.md`, and other files to
point to the new location.
[[1]](diffhunk://#diff-eca12c0a30e25b4b46522ebf89465a03ba72a03f540796c979137931d8f92055L585-L640)
[[2]](diffhunk://#diff-eca12c0a30e25b4b46522ebf89465a03ba72a03f540796c979137931d8f92055L660-R607)
[[3]](diffhunk://#diff-eca12c0a30e25b4b46522ebf89465a03ba72a03f540796c979137931d8f92055L740-R687)
[[4]](diffhunk://#diff-b335630551682c19a781afebcf4d07bf978fb1f8ac04c6bf87428ed5106870f5L2-R3)
[[5]](diffhunk://#diff-b335630551682c19a781afebcf4d07bf978fb1f8ac04c6bf87428ed5106870f5L30-R30)
* Added a new Next.js/Fumadocs-based documentation app under
`apps/docs/`, including core app files, layouts, routing, search API,
and a comprehensive `README.md` with development and contribution
instructions.
[[1]](diffhunk://#diff-5a1b07344a2c1b4d3f37b23ff1388b62cd9f57dea3c1cd23d8a0412b7602b132R1-R74)
[[2]](diffhunk://#diff-462b9ad1eabbb7d1c29bb9c36e4931eb180fd7190900e6d2babf8f4d66ad1c28R1-R39)
[[3]](diffhunk://#diff-e16ae25660ded787b10ac35dea96d5ecaacf895dae0afc8a9bd4382dc79a8c87R1-R7)
[apps/docs/app/[[...slug]]/layout.tsxR1-R81](diffhunk://#diff-59e08acde4e805b7aeccef1dcf98f1d71dfc550777e6b9402085cee0e9fa4e0aR1-R81),
[apps/docs/app/[[...slug]]/page.tsxR1-R79](diffhunk://#diff-e5df3f80d0fa01e12d63d81f29c57a9d14e846c78fd3beabb2ef768e38fd9580R1-R79),
[[4]](diffhunk://#diff-389b34918e040cacaa87cd7201ffa462cc2b0b716736f537e3d3c660ac69353bR1-R7)
[[5]](diffhunk://#diff-d3b03416d1c457b19f1c27b26f2db741412df841b875c26ccadc36e4522247f4R1-R29)
[[6]](diffhunk://#diff-c8fb8339570a5305809be7c618e14705fd86390dc278e6ce8ba224a7bc8b0c3cR1-R25)

**Configuration and Tooling Updates**

* Updated `.github/workflows/codeql.yml`, `.golangci.yaml`, and
`.github/dependabot.yml` to properly handle the new docs app: excluded
`apps/docs` from certain checks, added npm dependency updates for the
docs app, and excluded generated content.
[[1]](diffhunk://#diff-12783128521e452af0cfac94b99b8d250413c516ec71fe6d97dbea666ff7ba27L8-R14)
[[2]](diffhunk://#diff-9917ddc9f1c3304218f7269265b746d997c5c0615478177b5fceecd33ef47cb5R5-R6)
[[3]](diffhunk://#diff-9917ddc9f1c3304218f7269265b746d997c5c0615478177b5fceecd33ef47cb5R126-R129)
[[4]](diffhunk://#diff-dd4fbda47e51f1e35defb9275a9cd9c212ecde0b870cba89ddaaae65c5f3cd28R89-R106)
* Updated `.devcontainer/devcontainer.json` to use the latest Go 1.25.3
version for consistency.

**Licensing and Miscellaneous**

* Added `apps/docs/` to the list of licensed directories in
`LICENSING.md`.

These changes ensure the documentation is now maintained in a modern,
scalable system and all project tooling is updated to support the new
structure.

---------

Co-authored-by: Federico Coppede <[email protected]>
This commit is contained in:
Florian Forster
2026-01-29 08:38:36 -08:00
committed by GitHub
co-authored by Federico Coppede
parent 7b6395c348
commit d8ea041c56
2826 changed files with 78739 additions and 23571 deletions
@@ -0,0 +1,32 @@
---
title: Technical Advisory 10000
---
## Date and Version
Version: 2.32.0
Date: Calendar Week 32
## Description
Currently, by default, users are directed to the "Select Account Page" on the ZITADEL login.
However, this can be modified by including a [prompt or a login hint](/apis/openidoauth/endpoints#additional-parameters) in the authentication request.
As a result of this default behavior, users who already have an active session in one application and wish to log in to a second one will need to select their user account, even if no other session is active.
To address this, we are going to change this behavior so that users will be automatically authenticated when logging into a second application, as long as they only have one active session.
## Statement
This behavior change was tracked in the following issue: [Reuse current session if no prompt is selected](https://github.com/zitadel/zitadel/issues/4841)
and released in version [v2.32.0](https://github.com/zitadel/zitadel/releases/tag/v2.32.0)
## Mitigation
If you want to prompt users to always select their account on purpose, please make sure to include the `select_account` [prompt](/apis/openidoauth/endpoints#additional-parameters) in your authentication request.
## Impact
Once this update has been released and deployed, your users will be automatically authenticated
No action will be required on your part if this is the intended behavior.
@@ -0,0 +1,32 @@
---
title: Technical Advisory 10001
---
## Date and Version
Version: 2.35.0
Date: Calendar Week 34
## Description
Currently, disabling the `Allow Register` setting in the Login Policy, will disable any registration - local and through External Identity Providers (IDP).
This might be a good solution, if you manage all users yourself and do not want them to create any new account.
If you on the other hand want users to be able to federate their accounts from another IDP and only want to disable local registration, there's currently no option to do so.
Further ZITADEL provided the possibility to disable registration on each IDP with the introduction of IDP Templates.
To address this, we are going to change the behavior of the setting mentioned above, so that if disable, it will only prevent local registration. Registration of a federated user will still be possible - if not disabled by the corresponding IDP Template.
## Statement
This behavior change was tracked in the following PR: [Restrict AllowRegistration check to local registration](https://github.com/zitadel/zitadel/pull/5939).
The change was part of version [v2.35.0](https://github.com/zitadel/zitadel/releases/tag/v2.35.0)
## Mitigation
If you want to prevent user creation / registration through an IDP, be sure to disable the `isCreationAllowed` option on the desired IDP Templates.
## Impact
Once this update has been released and deployed, the `Allow Register` setting in the Login Policy will only affect local registrations and users might be able to create a ZITADEL account through an IDP, depending on your IDP provider options.
@@ -0,0 +1,35 @@
---
title: Technical Advisory 10002
---
## Date and Version
Version: 2.40.0
Date: Calendar week 44
## Description
Since Angular Material v15 many of the UI components have been refactored
to be based on the official Material Design Components for Web (MDC).
These refactored components do not support dynamic styling, so in order to keep the library up-to-date,
the management console UI will lose its dynamic theming capability.
## Statement
This design change is tracked in the following PR: [feat(console): MDC components](https://github.com/zitadel/zitadel/pull/6482).
As soon as the release version is published, we will include the version here.
## Mitigation
If you need users to have your branding settings
(background-, button-, link and text coloring), you should implement your
own user facing UI yourself and not use ZITADELs management console UI. Assets like your logo and icons will still be used.
## Impact
Once this update has been released and deployed, the management console UI won't apply any coloring of your branding settings to the UI components.
<Callout>
ZITADEL hosted Login-UI is not affected by this change.
</Callout>
@@ -0,0 +1,46 @@
---
title: Technical Advisory 10003
---
## Date and Version
Version: 2.38.0
Date: Calendar week 41
## Description
When users are redirected to the ZITADEL Login-UI without any organizational context, they're currently presented a login screen,
based on the instance settings, e.g. available IDPs and possible login mechanisms. If the user will then register themselves,
by the registration form or through an IDP, the user will always be created on the default organization.
This behavior led to confusion, e.g. when activating IDPs on default org would not show up in the Login-UI, because they would still be loaded from the instance settings.
To improve this, we're introducing the following change:
If users are redirected to the Login-UI without any organizational context, they will be presented a login screen based on the settings of the default organization (incl. IDPs).
<Callout>
If the registration (and also authentication) needs to occur on a specified organization, apps can already
specify this by providing [an organization scope](/apis/openidoauth/scopes#reserved-scopes).
</Callout>
## Statement
This change was tracked in the following PR:
[feat(login): use default org for login without provided org context](https://github.com/zitadel/zitadel/pull/6625), which was released in Version [2.38.0](https://github.com/zitadel/zitadel/releases/tag/v2.38.0)
## Mitigation
There's no action needed on your side currently as existing instances are not affected directly and IAM_OWNER can activate the flag at their own pace.
## Impact
Once this update has been released and deployed, newly created instances will always use the default organization and its settings as default context for the login.
Already existing instances will still use the instance settings by default and can switch to the new default by ["Activating the 'LoginDefaultOrg' feature"](/reference/api/admin/zitadel.admin.v1.AdminService.ActivateFeatureLoginDefaultOrg) through the Admin API.
**This change is irreversible!**
<Callout>
Regardless of the change:
If a known username is entered on the first screen, the login switches its context to the organization of that user and settings will be updated to that organization as well.
</Callout>
@@ -0,0 +1,37 @@
---
title: Technical Advisory 10004
---
## Date and Version
Version: 2.39.0
Date: 2023-10-14
## Description
Due to storage optimizations ZITADEL changes the behavior of sequences.
This change improves command (create, update, delete) performance of ZITADEL.
Sequences are no longer unique inside an instance.
From now on sequences are upcounting per aggregate id.
For example sequences of newly created users begin at 1.
Existing sequences remain untouched.
## Statement
This change is tracked in the following PR: [new eventstore framework](https://github.com/zitadel/zitadel/issues/5358).
As soon as the release version is published, we will include the version here.
## Mitigation
If you use the ListEvents API to scrape events use the creation date instead of the sequence.
If you use sequences on a list of objects it's no longer guaranteed to have unique sequences across the list.
Therefore it's recommended to use the change data of the objects instead.
## Impact
Once this update has been released and deployed, sequences are no longer unique inside an instance.
ZITADEL will increase parallel write capabilities, because there is no global sequence to track anymore.
Editor service does not respond the different services of ZITADEL anymore, it returns zitadel.
As we are switching to resource based API's there is no need for this field anymore.
@@ -0,0 +1,33 @@
---
title: Technical Advisory 10005
---
## Date and Version
Version: 2.39.0
Date: Calendar week 41/42 2023
## Description
Migrating to version >= 2.39 from < 2.39 will cause down time during setup starts and the new version is started.
This is caused by storage optimizations which replace the `eventstore.events` database table with the new `eventstore.events2` table.
All existing events are migrated during the execution of the `zitadel setup` command.
New events will be inserted into the new `eventstore.events2` table. The old table `evetstore.events` is renamed to `eventstore.events_old` and will be dropped in a future release of ZITADEL.
## Statement
This change is tracked in the following PR: [new eventstore framework](https://github.com/zitadel/zitadel/issues/5358).
As soon as the release version is published, we will include the version here.
## Mitigation
If you use this table for TRIGGERS or Change data capture please check the new table definition and change your code to use `eventstore.events2`.
## Impact
Once the setup step renamed the table. Old versions of ZITADEL are not able to read/write events.
<Callout>
If the upgrade fails make sure to rename `eventstore.events_old` to `eventstore.events`. This change enables older ZITADEL versions to work properly.
</Callout>
@@ -0,0 +1,29 @@
---
title: Technical Advisory 10006
---
## Date and Version
Version: 2.39.0
Date: Calendar week 41/42 2023
## Description
Versions >= 2.39.0 require the cockroach database user of ZITADEL to be granted to the `VIEWACTIVITY` grant. This can either be reached by grant the role manually or execute the `zitadel init` command.
Cockroach versions 22.2 < 22.2.11 and 23.1 < 23.1.4 will fail the migration. Please make sure to upgrade to more recent versions first. ZITADEL recommends to use the latest stable version of Cockroachdb.
## Statement
To query correct order of events the cockroach database user of ZITADEL needs additional privileges to query the `crdb_internal.cluster_transactions`-table
## Mitigation
Before migrating to versions >= 2.39.0 make sure the cockroach database user has sufficient grants.
Cockroachdb version is up to date.
## Impact
If the user doesn't have sufficient grants, events won't be updated.
@@ -0,0 +1,63 @@
---
title: Technical Advisory 10007
---
## Date and Version
Version: Upcoming
Date: Upcoming
## Affected Users
This advisory applies to self-hosted ZITADEL installations with custom roles to permissions mappings in the *InternalAuthZ.RolePermissionMappings* configuration section.
## Description
In upcoming ZITADEL versions, RBAC also applies to [system users defined in the ZITADEL runtime configuration](/guides/integrate/zitadel-apis/access-zitadel-system-api#runtime-configuration).
This enables fine grained access control to the system API as well as other APIs for system users.
ZITADEL defines the new default roles *SYSTEM_OWNER* and *SYSTEM_OWNER_VIEWER*.
System users without any memberships defined in the configuration will be assigned the *SYSTEM_OWNER* role.
**Self-hosting users who define their own custom mapping at the *InternalAuthZ.RolePermissionMappings* configuration section**, have to define the *SYSTEM_OWNER* role in their configuration too to be able to access the system API with the default system user membership.
## Statement
This change is tracked in the following PR: [feat: add SYSTEM_OWNER role](https://github.com/zitadel/zitadel/pull/6765).
As soon as the release version is published, we will include the version here.
## Mitigation
If you have a custom role mapping configured, make sure you configure the new role *SYSTEM_OWNER* before migrating to upcoming ZITADEL versions.
As a reference, these are the default mappings:
```yaml
InternalAuthZ:
RolePermissionMappings:
- Role: "SYSTEM_OWNER"
Permissions:
- "system.instance.read"
- "system.instance.write"
- "system.instance.delete"
- "system.domain.read"
- "system.domain.write"
- "system.domain.delete"
- "system.debug.read"
- "system.debug.write"
- "system.debug.delete"
- "system.feature.write"
- "system.limits.write"
- "system.limits.delete"
- "system.quota.write"
- "system.quota.delete"
- "system.iam.member.read"
- Role: "SYSTEM_OWNER_VIEWER"
Permissions:
- "system.instance.read"
- "system.domain.read"
- "system.debug.read"
...
```
## Impact
If the system users don't have the correct memberships and roles which resolve to permissions, the system users lose access to the system API.
@@ -0,0 +1,50 @@
---
title: Technical Advisory 10008
---
## Date and Version
Versions: 2.44.0, 2.43 >= 2.43.6, 2.42 > 2.42.12
Date: 2024-01-25
## Description
The versions mentioned above introduce a new flag `--init-projections` to `zitadel setup` commands (`setup`, `start-from-setup`, `start-from-init`)
This flag enables prefilling of newly added or changed projections (database tables) during setup phase instead of start phase which are used to query data, for example users.
This new feature adds the following fields to the `setup` configuration, previously only used in `start`-command:
- `AssetStore`: Storage for assets like user avatar, organization logo, icon, font, etc.
- `OIDC`
- `Login`
- `WebAuthNName`
- `Telemetry`
- `SystemAPIUsers`
If you use different configurations on `setup` and `start` and have overwritten previously mentioned configurations please make sure to also add them to the configuration provided to ZITADEL `setup` command.
## Statement
Filling of projections can get time-consuming as your system grows and this can cause downtime of self-hosted installations of ZITADEL because queries first need to ensure data consistency.
Before this release, this step was executed after the start of ZITADEL and therefore lead to inconsistent retrieval of data until the projections were up-to-date.
## Mitigation
Enable the flag (`--init-projections=true`) in setup phase and make sure the previous deployment of ZITADEL remains active until the new revision started properly.
Note that the flag is enabled by default starting with the following versions:
- \>=2.49.0
- 2.48.x: >=2.48.2
- 2.47.x: >=2.47.7
- 2.46.x: >=2.46.4
- 2.45.x: >=2.45.4
- 2.44.x: >=2.44.6
- 2.43.x: >=2.43.10
- 2.42.x: >=2.42.16
## Impact
Decreases downtime of starting new revisions with new or changed projections dramatically.
@@ -0,0 +1,27 @@
---
title: Technical Advisory 10009
---
## Date and Version
Version: 2.53.0
Date: 2024-05-28
## Description
There were rare cases where Cockroachdb got blocked during runtime of ZITADEL and returned `WRITE_TOO_OLD`-errors to ZITADEL. The root cause of the problem is described in [this github issue of the database](https://github.com/cockroachdb/cockroach/issues/77119). The workaround provided by the database is enabling the `enable_durable_locking_for_serializable`-flag as described [here](https://github.com/cockroachdb/cockroach/issues/75456#issuecomment-1936277716).
Because enabling flags requires admin privileges the statement must be executed manually or by executing `zitadel init` command.
## Statement
Ensure lock distribution for `FOR UPDATE`-statements on Cockroachdb.
## Mitigation
Cockroachdb version >= 23.2.
## Impact
Adding additional raft queries to `FOR UPDATE`-statements can impact performance slightly but ensures availability of the system.
@@ -0,0 +1,30 @@
---
title: Technical Advisory 10010
---
## Date and Version
Version: 2.53.0
Date: 2024-05-28
## Description
Version 2.53.0 optimizes the way tokens are created and migrates them to the v2 implementation already used by OAuth / OIDC tokens created through the session API.
Because of this tokens events are no longer created on the user itself. To be as backwards compatible as possible a separate event is created on the user for the audit log.
## Statement
This change was tracked in the following PR:
[perf(oidc): optimize token creation](https://github.com/zitadel/zitadel/pull/7822), which was released in Version [2.53.0](https://github.com/zitadel/zitadel/releases/tag/v2.53.0)
## Mitigation
If you use the ListEvents API to check the audit trail of a user or being able to compute Daily or Monthly Active Users, be sure to also include the `user.token.v2.added` event type in your search
if you already query for the `user.token.added` event type.
## Impact
Once this update has been released and deployed, the `user.token.added` event will no longer be created when a user access token is created, but instead a `user.token.v2.added`.
Existing `user.token.added` events will be untouched.
@@ -0,0 +1,29 @@
---
title: Technical Advisory 10011
---
## Date and Version
Version: 2.59.0
Date: 2024-08-19
## Description
Version 2.59.0 allows more combinations in the identity provider options. As of now, **automatic creation** and **automatic linking options** were only considered if the corresponding **allowed option** (account creation / linking allowed) was enabled.
Starting with this release, this is no longer needed and allows administrators to address cases, where only an **automatic creation** is allowed, but users themselves should not be allowed to **manually** create new accounts using an identity provider or edit the information during the process.
Also, allowing users to only link to the proposed existing account is now possible with an enabled **automatic linking option**, while disabling **account linking allowed**.
## Statement
This change was tracked in the following PR:
[feat(idp): provide auto only options](https://github.com/zitadel/zitadel/pull/8420), which was released in Version [2.59.0](https://github.com/zitadel/zitadel/releases/tag/v2.59.0)
## Mitigation
If you previously enabled one of the **automatic** options with the corresponding **allowed** option, be sure that this is the intended behavior.
## Impact
Once this update has been released and deployed, the **automatic** options can be activated with the corresponding **allowed** option.
@@ -0,0 +1,60 @@
---
title: Technical Advisory 10012
---
## Date and Version
Version: 2.63.0
Date: 2024-09-26
## Description
In version 2.63.0 we've increased the transaction duration for projections.
ZITADEL has an event driven architecture. After events are pushed to the eventstore,
they are reduced into projections in bulk batches. Projections allow for efficient lookup of data through normalized SQL tables.
We've investigated multiple reports of outdated projections.
For example created users missing in get requests, or missing data after a ZITADEL upgrade[^1].
The conclusion is that the transaction in which we perform a bulk of queries can timeout.
The old setting defined a transaction duration of 500ms for a bulk of 200 events.
A single event may create multiple statements in a single projection.
A timeout may occur even if the actual bulk size is less than 200,
which then results in more back-pressure on a busy system, leading to more timeouts and effectively dead-locking a projection.
Increasing or disabling the projection transaction duration solved dead-locks in all reported cases.
We've decided to increase the transaction duration to 1 minute.
Due to the high value it is functionally similar to disabling,
however it still provides a safety net for transaction that do freeze,
perhaps due to connection issues with the database.
[^1]: Changes written to the eventstore are the main source of truth. When a projection is out of date, some request may serve incomplete or no data. The data itself is however not lost.
## Statement
A summary of bug reports can be found in the following issue: [Missing data due to outdated projections](https://github.com/zitadel/zitadel/issues/8517).
This change was submitted in the following PR:
[fix(projection): increase transaction duration](https://github.com/zitadel/zitadel/pull/8632), which will be released in Version [2.63.0](https://github.com/zitadel/zitadel/releases/tag/v2.63.0)
## Mitigation
If you have a custom configuration for projections, this update will not apply to your system or some projections. When encountering projection dead-lock consider increasing the timeout to the new default value.
Note that entries under `Customizations` overwrite the global settings for a single projection.
```yaml
Projections:
TransactionDuration: 1m # ZITADEL_PROJECTIONS_TRANSACTIONDURATION
BulkLimit: 200 # ZITADEL_PROJECTIONS_BULKLIMIT
Customizations:
custom_texts:
BulkLimit: 400
project_grant_fields:
TransactionDuration: 0s
BulkLimit: 2000
```
## Impact
Once this update has been released and deployed, transactions are allowed to run longer. No other functional impact is expected.
@@ -0,0 +1,25 @@
---
title: Technical Advisory 10013
---
## Date
Date: 2024-12-09
## Description
ZITADEL currently provides a "latest" and "stable" release tags as well as maintenance branches, where bug fixes are backported.
We also publish release candidates regularly.
The "stable" release channel was introduced for users seeking a more reliable and production-ready version of the software.
However, most customers have their own deployment policies and cycles.
Backports and security fixes are currently done as needed or required by customers.
zitadel.cloud follows a similar approach, where the latest release is deployed a few days after its creation.
## Mitigation
If you used the "stable" Docker release, please consider switching to a specific version tag and follow the [release notes on GitHub](https://github.com/zitadel/zitadel/releases) for latest changes.
## Impact
The "stable" version will no longer be published or updated, and the corresponding Docker image tag will not be maintained anymore.
@@ -0,0 +1,26 @@
---
title: Technical Advisory 10014
---
## Date
Versions: >= v2.67.3, v2.66 >= v2.66.6
Date: 2025-01-17
## Description
Prior to version [v2.66.0](https://github.com/zitadel/zitadel/releases/tag/v2.66.0), some project grants were incorrectly created under the granted organization instead of the project owner's organization. To find these grants, users had to set the `x-zitadel-orgid` header to the granted organization ID when using the [`ListAllProjectGrants`](/reference/api/management/zitadel.management.v1.ManagementService.ListAllProjectGrants) gRPC method.
Zitadel [v2.66.0](https://github.com/zitadel/zitadel/releases/tag/v2.66.0) corrected this behavior for new grants. However, existing grants were not automatically updated. Version v2.66.6 corrects the owner of these existing grants.
## Impact
After the release of v2.66.6, if your application uses the [`ListAllProjectGrants`](/reference/api/management/zitadel.management.v1.ManagementService.ListAllProjectGrants) method with the `x-zitadel-orgid` header set to the granted organization ID, you will not retrieve any results.
## Mitigation
To ensure your application continues to function correctly after the release of v2.66.6, implement the following changes:
1. **Conditional Header:** Only set the `x-zitadel-orgid` header to the project owner's organization ID if the user executing the [`ListAllProjectGrants`](/reference/api/management/zitadel.management.v1.ManagementService.ListAllProjectGrants) method belongs to a different organization than the project.
2. **Use `grantedOrgIdQuery`:** Utilize the `grantedOrgIdQuery` parameter to filter grants for the specific granted organization.
@@ -0,0 +1,21 @@
---
title: Technical Advisory 10015
---
## Date
Versions: >= v3.0.0
Date: 2025-03-31
## Description
CockroachDB was initially chosen due to its distributed nature and SQL compatibility. However, over time, it became apparent that the operational complexity and specific compatibility issues outweighed the benefits for our use case. We decided to focus on PostgreSQL to simplify our infrastructure and leverage its mature ecosystem.
## Impact
Zitadel v3 requires PostgreSQL as a database. Therefore, Zitadel v3 will not start if CockroachDB is configured as the database.
## Mitigation
To upgrade your self-hosted deployment to Zitadel v3 migrate to PostgreSQL. Please refer to [this guide](/self-hosting/manage/cli/mirror) to mirror the data to PostgreSQL before you deploy Zitadel v3.
@@ -0,0 +1,103 @@
---
title: Technical Advisory 10016
---
## Date
- v2.65.x: > v2.65.9
- v2.66.x > v2.66.17
- v2.67.x > v2.67.14
- v2.68.x > v2.68.10
- v2.69.x > v2.69.10
- v2.70.x > v2.70.11
- v2.71.x > v2.71.10
- v3.x > v3.2.1
Date: 2025-05-14
Last updated: 2025-05-19
## Description
### Background
Zitadel uses a eventstore table as main source of truth for state changes.
Projections are tables which provide alternative views of state, which are built using events.
In order to know which events are reduced into projections, we use a `position` column in the eventstore and a dedicated table which records the current state.
### Problem
Zitadel prior to the listed version had a precision bug. The `position` column uses a fixed-point numeric type. In Zitadel's Go code we used a `float64`. In certain cases we noticed a precision loss when Zitadel updated the `current_states` table.
## Impact
During a past attempt to fix this, we got reports of failing projections inside Zitadel. Because the precision became exact certain compare operations like *equal*, *less then*, etc would now return different results. This was because the values in `current_states` would already have lost precision from a broken version. This might happen to **some** deployments or projections: there is only a small probability.
We are releasing the fix again and your system might get affected.
- Original issue: [8671](https://github.com/zitadel/zitadel/issues/8671)
- Follow-up issue: [8863](https://github.com/zitadel/zitadel/issues/8863)
## Mitigation
When **after** deploying a fixed version and only when experiencing problems described by issue [8863](https://github.com/zitadel/zitadel/issues/8863), the following queries can be executed to fix `current_state` rows which have "broken" values. We recommend doing this in a transaction in order to double-check the affected rows, before committing the update.
```sql
begin;
with
broken as (
select
s.projection_name,
s.instance_id,
s.aggregate_id,
s.aggregate_type,
s.sequence,
s."position" as old_position,
e."position" as new_position
from
projections.current_states s
join eventstore.events2 e on s.instance_id = e.instance_id
and s.aggregate_id = e.aggregate_id
and s.aggregate_type = e.aggregate_type
and s.sequence = e.sequence
and s."position" != e."position"
where
s."position" != 0
and projection_name != 'projections.execution_handler'
),fixed as (
update projections.current_states s
set
"position" = b.new_position
from
broken b
where
s.instance_id = b.instance_id
and s.projection_name = b.projection_name
and s.aggregate_id = b.aggregate_id
and s.aggregate_type = b.aggregate_type
and s.sequence = b.sequence
returning *
)
select
b.projection_name,
b.instance_id,
b.aggregate_id,
b.aggregate_type,
b.sequence,
b.old_position,
b.new_position,
b.old_position - b.new_position difference
from
broken b;
```
If the output from the above looks reasonable, for example not a huge number in the `difference` column, commit the transaction:
```sql
commit;
```
When there are no rows returned, your system was not affected by precision loss.
When there's unexpected output, use `rollback;` instead.
@@ -0,0 +1,275 @@
---
title: ZITADEL Technical Advisories
sidebar_label: Technical Advisories
---
Technical advisories are notices that report major issues with ZITADEL Self-Hosted or the ZITADEL Cloud platform that could potentially impact security or stability in production environments.
These advisories may include details about the nature of the issue, its potential impact, and recommended mitigation actions.
Users are strongly encouraged to evaluate these advisories and consider the recommended mitigation actions independently from their version upgrade schedule.
We understand that these advisories may include breaking changes, and we aim to provide clear guidance on how to address these changes.
<table>
<tr>
<th>Advisory</th>
<th>Name</th>
<th>Type</th>
<th>Summary</th>
<th>Affected versions</th>
<th>Date</th>
</tr>
<tr>
<td>
<a href="./advisory/a10000">A-10000</a>
</td>
<td>Reusing user session</td>
<td>Breaking Behavior Change</td>
<td>
The default behavior for users logging in is to be directed to the Select
Account Page on the Login. With the upcoming changes, users will be
automatically authenticated when logging into a second application, as
long as they only have one active session. No action is required on your
part if this is the intended behavior.
</td>
<td>2.32.0</td>
<td>Calendar week 32</td>
</tr>
<tr>
<td>
<a href="./advisory/a10001">A-10001</a>
</td>
<td>Login Policy - Allow Register</td>
<td>Breaking Behavior Change</td>
<td>
When disabling the option, users are currently not able to register
locally and also not through an external IDP. With the upcoming change,
the setting will only prevent local registration. Restriction to Identity
Providers can be managed through the corresponding IDP Template. No action
is required on your side if this is the intended behavior or if you
already disabled registration on your IDP.
</td>
<td>2.35.0</td>
<td>Calendar week 34</td>
</tr>
<tr>
<td>
<a href="./advisory/a10002">A-10002</a>
</td>
<td>Console - Branding</td>
<td>Breaking Design Change</td>
<td>
Since Angular Material v15 many of the UI components have been refactored
to be based on the official Material Design Components for Web (MDC).
These refactored components do not support dynamic styling, so in order to
keep the library up-to-date, the management console UI will loose its dynamic theming
capability. If you need users to have your branding settings (background-,
button-, link and text coloring) you should implement your own user
facing UI yourself and not use ZITADELs management console UI.
ZITADEL hosted Login-UI is not affected by this change.
</td>
<td>2.40.0</td>
<td>Calendar week 44</td>
</tr>
<tr>
<td>
<a href="./advisory/a10003">A-10003</a>
</td>
<td>Login-UI - Default Context</td>
<td>Breaking Behavior Change</td>
<td>
When users are redirected to the ZITADEL Login-UI without any organizational context,
they're currently presented a login screen, based on the default settings,
e.g. available IDPs and possible login mechanisms. If the user will then register themselves,
by the registration form or through an IDP, the user will always be created on the default organization.
With the introduced change, the settings will no longer be loaded from the instance, but rather the default organization directly.
</td>
<td>2.38.0</td>
<td>Calendar week 41</td>
</tr>
<tr>
<td>
<a href="./advisory/a10004">A-10004</a>
</td>
<td>Sequence uniqueness</td>
<td>Breaking Behavior Change</td>
<td>
Due to storage optimizations ZITADEL changes the behavior of sequences.
This change improves command (create, update, delete) performance of ZITADEL.
Sequences are no longer unique inside an instance.
From now on sequences are upcounting per aggregate id.
For example sequences of newly created users begin at 1.
Existing sequences remain untouched.
</td>
<td>2.39.0</td>
<td>2023-10-14</td>
</tr>
<tr>
<td>
<a href="./advisory/a10005">A-10005</a>
</td>
<td>Expected downtime during upgrade</td>
<td>Expected downtime during upgrade</td>
<td>
Migrating to versions &gt;= 2.39 from &lt; 2.39 will cause down time during setup starts and the new version is started.
This is caused by storage optimizations which replace the `eventstore.events` database table with the new `eventstore.events2` table.
All existing events are migrated during the execution of the `zitadel setup` command.
New events will be inserted into the new `eventstore.events2` table. The old table `evetstore.events` is renamed to `eventstore.events_old` and will be dropped in a future release of ZITADEL.
</td>
<td>2.39.0</td>
<td>Calendar week 41/42 2023</td>
</tr>
<tr>
<td>
<a href="./advisory/a10006">A-10006</a>
</td>
<td>Additional grant to cockroach database user</td>
<td>Breaking Behavior Change</td>
<td>
Versions &gt;= 2.39.0 require the cockroach database user of ZITADEL to be granted to the `VIEWACTIVITY` grant. This can either be reached by grant the role manually or execute the `zitadel init` command.
</td>
<td>2.39.0</td>
<td>Calendar week 41/42 2023</td>
</tr>
<tr>
<td>
<a href="./advisory/a10007">A-10007</a>
</td>
<td>Additional grant to cockroach database user</td>
<td>Breaking Behavior Change</td>
<td>
Upcoming Versions require the SYSTEM_OWNER role to be available in the permission role mappings. Self-hosting ZITADEL users who define custom permission role mappings need to make sure their system users don't lose access to the system API.
</td>
<td>Upcoming</td>
<td>Upcoming</td>
</tr>
<tr>
<td>
<a href="./advisory/a10008">A-10008</a>
</td>
<td>New flag to prefill projections during setup instead of after start</td>
<td>Feature description</td>
<td>
New flag `--init-projections` introduced to `zitadel setup` commands (`setup`, `start-from-setup`, `start-from-init`)
</td>
<td>2.44.0, 2.43.6, 2.42.12</td>
<td>2024-01-25</td>
</tr>
<tr>
<td>
<a href="./advisory/a10009">A-10009</a>
</td>
<td>Ensure lock distribution for `FOR UPDATE`-statements on Cockroachdb</td>
<td>Feature description</td>
<td>
Fixes rare cases where updating projections was blocked by a `WRITE_TOO_OLD`-error when using cockroachdb.
</td>
<td>2.53.0</td>
<td>2024-05-28</td>
</tr>
<tr>
<td>
<a href="./advisory/a10010">A-10010</a>
</td>
<td>Event type of token added event changed</td>
<td>Breaking Behavior Change</td>
<td>
Version 2.53.0 improves the token issuance. Due to this there are changes to the event types created on token creation.
</td>
<td>2.53.0</td>
<td>2024-05-28</td>
</tr>
<tr>
<td>
<a href="./advisory/a10011">A-10011</a>
</td>
<td>Identity Provider options: allow "auto" only</td>
<td>Breaking Behavior Change</td>
<td>
Version 2.59.0 allows more combinations in the identity provider options. Due to this there might be unexpected behavior changes.
</td>
<td>2.59.0</td>
<td>2024-08-19</td>
</tr>
<tr>
<td>
<a href="./advisory/a10012">A-10012</a>
</td>
<td>Increased transaction duration for projections</td>
<td>Breaking Behavior Change</td>
<td>
In version 2.63.0 we've increased the transaction duration for projections to resolve outdated projections or dead-locks.
</td>
<td>2.63.0</td>
<td>2024-09-26</td>
</tr>
<tr>
<td>
<a href="./advisory/a10013">A-10013</a>
</td>
<td>Deprecation of "stable" version</td>
<td>Breaking Behavior Change</td>
<td>
The "stable" version will no longer be published or updated, and the corresponding Docker image tag will not be maintained anymore.
</td>
<td>-</td>
<td>2024-12-09</td>
</tr>
<tr>
<td>
<a href="./advisory/a10014">A-10014</a>
</td>
<td>Correction of project grant owner</td>
<td>Breaking Behavior Change</td>
<td>
Correct project grant owners, ensuring they are correctly associated with the projects organization.
</td>
<td>-</td>
<td>2025-01-10</td>
</tr>
<tr>
<td>
<a href="./advisory/a10015">A-10015</a>
</td>
<td>Drop CockroachDB support</td>
<td>Breaking Behavior Change</td>
<td>
CockroachDB is no longer supported by Zitadel.
</td>
<td>3.0.0</td>
<td>2025-03-31</td>
</tr>
<tr>
<td>
<a href="./advisory/a10016">A-10016</a>
</td>
<td>Position precision fix</td>
<td>Manual Intervention</td>
<td>
</td>
<td>2.65.10</td>
<td>2025-05-14</td>
</tr>
</table>
## Subscribe to our Mailing List
If you want to stay up to date on our technical advisories, we recommend subscribing to our [Technical Advisory mailing list](https://share-eu1.hsforms.com/1NAOUyAMmRzSnB7uP665xHA2dljbt) or visiting our [website](https://zitadel.com/technical-advisory).
As ZITADEL Cloud customer, you can also login to the <a href="https://zitadel.com/admin">ZITADEL Customer Portal</a> and enable the Technical Advisory <a href="https://zitadel.com/admin/settings">Notifications</a> in your settings.
## Categories
### Breaking Behavior Change
A breaking behavior change refers to a modification or update that changes the behavior of ZITADEL.
This change does not necessarily affect the APIs or any functions you are calling, so it may not require an update to your code.
However, if you rely on specific results or behaviors, they may no longer be guaranteed after the change is implemented.
Therefore, it is important to be aware of breaking behavior changes and their potential impact on your use of ZITADEL, and to take appropriate action if needed to ensure continued functionality.
### Expected downtime during upgrade
Expected downtime during upgrade means that ZITADEL might become unavailable during an upgrade.
ZITADEL is built for [zero downtime upgrades](/concepts/architecture/solution#zero-downtime-updates) at upgrades can be executed without downtime by just updating to a more recent version.
When deploying certain changes a zero downtime upgrade might not be possible, for example to guarantee data integrity.
In such cases we will issue a technical advisory to make you aware of this unexpected behavior.
@@ -0,0 +1,110 @@
---
title: Troubleshoot ZITADEL
sidebar_label: Troubleshoot
---
You will find some possible error messages here, what the problem is and what some possible solutions can be.
<Callout title="Didn't find an answer? ">
Join or [Chat](https://zitadel.com/chat) or open a [Discussion](https://github.com/zitadel/zitadel/discussions).
</Callout>
## User agent does not correspond
This error appeared for some users as soon as they were redirected to the login page of ZITADEL.
ZITADEL uses some cookies to identify the browser/user agent of the user, so it is able to store the active user sessions. By blocking the cookies the functions of ZITADEL will be affected.
We only found this issue with iPhone users, and it was dependent on the settings of the device.
Go to the settings of the app Safari and check in the "Experimental WebKit Features" if SameSite strict enforcement (ITP) is disabled
Also check if "block all cookies" is active. If so please disable this setting.
To make sure, that your new settings will trigger, please restart your mobile phone and try it again.
**Settings > Safari > Advanced > Experimental Features > disable: „SameSite strict enforcement (ITP)“**
![Same Site Strict Enforcement](../../public/img/manuals/errors/same-site-strict.png)
**Settings > Safari > disable: "Block All cookies"**
![](../../public/img/manuals/errors/block-cookies.png)
Do you still face this issue? Please contact us, and we will help you find out what the problem is.
## Invalid audience
`invalid audience (APP-Zxfako)`
This error message refers to the audience claim (`aud`) of your token.
This claim identifies the audience, i.e. the resource server, that this token is intended for.
If a resource server does not identify itself with a value in the "aud" claim when this claim is present, then the must be rejected (see [RFC7519](https://www.rfc-editor.org/rfc/rfc7519#section-4.1.3) for more details).
You might encounter this error message from ZITADEL, typically when you authenticated with a client in one project and trying to access an application in another project.
You need add a specific [reserved scope](/apis/openidoauth/scopes#reserved-scopes) to add the projectID to the audience of the access token.
The two scenarios should help you troubleshoot this issue:
### Frontend to Backend
You have one project for your frontend application and one project for your backend application.
End-users authenticate to an application in your frontend project.
The frontend then sends requests to the backend, validates the token with ZITADEL's introspection endpoint, and returns a payload to the frontend.
The backend returns the error `invalid audience (APP-Zxfako)`.
You must add the scope `urn:zitadel:iam:org:project:id:{projectId}:aud` to the auth request that is send from the front end.
Replace `projectId` with the projectId of your backend.
### Accessing ZITADEL's APIs
You have a project for a frontend application.
The application should also access the API of your ZITADEL, for example to pull a list of all users and display them on a user page.
End-users authenticate to the application in the frontend project, but when calling the management API you get the error `invalid audience (APP-Zxfako)`.
You must add the scope `urn:zitadel:iam:org:project:id:zitadel:aud` to the auth request that is send from the front end.
When accessing your ZITADEL instance's APIs they act as a resource server.
You can check the Management Console or via API and see that when you open your default organization there exists a project "ZITADEL" that contains different applications for each API and the Console.
Like in the scenario above the access token requires to have an `aud` claim that includes the "ZITADEL" project.
Instead of `urn:zitadel:iam:org:project:id:zitadel:aud` you could also use `urn:zitadel:iam:org:project:id:{projectId}:aud`, where `projectId` is the projectId of the Project "ZITADEL".
## WebFinger requirement for Tailscale
The WebFinger requirement and setup is a step a user has to take outside of their IdP set-up. WebFinger is a protocol which supports the ability for OIDC issuer discovery, and we use it to prove that the user has administrative control over the domain and to retrieve the issuer. This is a requirement we have in place for all users, regardless of their IdP, who use custom OIDC with Tailscale.
On their Custom Domain, e.g example.com, users need to host a WebFinger endpoint at https://example.com/.well-known/webfinger. When queried, this endpoint returns a JSON response detailing the issuer. Users would need to host the endpoint with the link to the ZITADEL issuer. Tailscale only looks up this endpoint once when a user signs up, and will only look up this endpoint again if the user needs to make a configuration change to their identity provider.
The requirements and a set up guide is detailed in the [Tailscale documentation](https://tailscale.com/kb/1240/sso-custom-oidc/).
## Login not possible. The organization of the user must be granted to the project
![](../../public/img/support/login_not_possible_error.png)
ZITADEL is not only capable of handling authentication but also authorization.
This error message tells you, that a project grant is missing from the owner organization to the organization of the authenticating user.
You do have two organizations, an owner (Organization A) and a customer (Organization B).
The Organization A owns a Project, and has to grant it to Organization B, so users are allowed to authenticate.
The error message is shown to users of Organization B that the permission is required, but the project is not granted to Organization B.
![](../../public/img/support/project_grant_missing.png)
You do have two possibilities.
1. Disable the permission check
2. Give the permission to the organization
### Disable the permission check
1. Go to the organization, who owns the project, where the user tries to authenticate.
2. Navigate to the general settings of the needed project
3. Disable "Check for Project on Authentication"
![](../../public/img/support/check_for_project_on_authentication.png)
### Give the needed permission to the organization
1. Go to the organization, who owns the project, where the user tries to authenticate.
2. Navigate to the grants page of the needed project
3. Click on the "New" button
4. Search for the organization to which you want to grant the project (e.g Organization B)
5. Select the roles you want to grant
6. Click save
![](../../public/img/support/project_grant_for_customer_b.png)