mirror of
https://github.com/andwati/zitadel.git
synced 2026-11-01 11:52:15 +00:00
Florian ForsterandGitHub
92a628d892
Users deploying ZITADEL against a managed PostgreSQL service (RDS, Cloud SQL, Azure Database, etc.) often do not have superuser access and cannot provide `Admin.*` credentials. The documented workaround — provisioning the user and database manually and then running `start-from-setup` — silently skips schema bootstrapping, causing `relation "eventstore.events" does not exist` errors with no clear recovery path. The root cause is that `zitadel init` conflates two steps that require different privileges without exposing them separately: - **Provisioning step** (`CREATE ROLE`, `CREATE DATABASE`, `GRANT`) — requires superuser. - **Schema bootstrapping step** (create `eventstore`/`projections`/`system` schemas and base tables) — requires only DB owner. Users who handle the provisioning step externally have no supported way to run schema bootstrapping alone. ## Changes - **`cmd/initialise/verify_schema.go`** (renamed from `verify_zitadel.go`): Rename `newZitadel()` → `newSchema()` (internal); rename the `init zitadel` sub-command to `zitadel init schema` (backwards-compatible alias kept) with a clear description that it bootstraps the ZITADEL database schema without admin/superuser privileges. Fix stale error log message to reference `init schema`. - **`cmd/initialise/verify_schema_test.go`** (renamed from `verify_zitadel_test.go`): Test file renamed to match source file. - **`cmd/initialise/init.go`**: Update call site to `newSchema()`. Add guidance in the `init` command's Long description about using `zitadel init schema` for users without admin credentials. - **`cmd/initialise/verify_database.go`**: Add a `pg_database` catalog pre-check before attempting `CREATE DATABASE`, so `zitadel init` with `ADMIN=service_user` no longer fails with `permission denied to create database` when the database was already provisioned externally. - **`cmd/initialise/verify_database_test.go`**: Add test cases covering the new catalog-check skip path, the existing error-skip path, and the error-propagation path when the `pg_database` query itself fails. - **`apps/docs/content/self-hosting/manage/database/index.mdx`**: Inline the `_postgres.mdx` partial (now only PostgreSQL is supported), add a top-level callout, and add a **Managed PostgreSQL / No Admin Access** section with explicit 3-step instructions and security guidance (strong passwords, SSL, TLS). Additional improvements: add inline `# Use 'require' or 'verify-full' for production` comments on `Mode: disable` lines in the YAML example; add clarifying comment to the redundant `GRANT` in the SQL snippet; replace admonition syntax with proper Fumadocs `<Callout>` components; add full database connection env vars to the `start-from-setup` example. - **`apps/docs/content/self-hosting/manage/updating_scaling.mdx`**: Rewrite the init phase description to clearly distinguish the provisioning and schema bootstrapping steps, and document `zitadel init schema` as the path for manual provisioning. Replace admonition syntax with proper Fumadocs `<Callout>` components. - **`apps/docs/content/self-hosting/manage/database/_postgres.mdx`**: Deleted (content merged into `index.mdx`). ## Problem This relates to https://github.com/zitadel/zitadel/discussions/9363 I think we can improve our UX in cases where a user wants to use an external DB and/or does not want to share too broad permissions with zitadel ## Related problems * https://github.com/zitadel/zitadel/issues/10432 * https://github.com/zitadel/zitadel/discussions/8583 * https://github.com/zitadel/zitadel/issues/7903 * https://github.com/zitadel/zitadel/issues/9718 * https://github.com/zitadel/zitadel/issues/8012 * https://github.com/zitadel/zitadel/issues/8558 ## Related PRs https://github.com/zitadel/zitadel/pull/11021
106 lines
5.2 KiB
Plaintext
106 lines
5.2 KiB
Plaintext
---
|
||
title: Update and Scale ZITADEL
|
||
description: "Guidelines for updating and scaling ZITADEL instances, utilizing separate initialization and runtime phases for high availability."
|
||
sidebar_label: Update and Scale
|
||
---
|
||
|
||
## TL;DR
|
||
|
||
For getting started easily, you can just combine the init, setup and runtime phases
|
||
by executing the `zitadel` binary with the argument `start-from-init`.
|
||
However, to improve day-two-operations,
|
||
we recommend you run the init phase and the setup phase
|
||
separately using `zitadel init` and `zitadel setup` respectively.
|
||
Both the init and the setup phases are idempotent,
|
||
meaning you can run them as many times as you want without unexpectedly changing any state.
|
||
Nevertheless, they just __need__ to be run at certain points in time, whereas further runs are obsolete.
|
||
The init phase has to run once over the full ZITADEL lifecycle, before the setup and runtime phases run first.
|
||
The setup phase has to run each time a new ZITADEL version is deployed.
|
||
After init and setup is done, you can start the runtime phase, that immediately accepts traffic, using `zitadel start`.
|
||
|
||
## Scaling ZITADEL
|
||
|
||
For production setups, we recommend that you [run ZITADEL in a highly available mode](/self-hosting/manage/production).
|
||
As all shared state is managed in the database,
|
||
the ZITADEL binary itself is stateless.
|
||
You can easily run as many ZITADEL binaries in parallel as you want.
|
||
If you use [Knative](https://knative.dev/docs/)
|
||
or [Google Cloud Run](https://cloud.google.com/run) (which uses Knative, too),
|
||
you can even scale to zero.
|
||
Especially if you use an autoscaler that scales to zero,
|
||
it is crucial that you minimize startup times.
|
||
|
||
## Updating ZITADEL
|
||
|
||
When want to deploy a new ZITADEL version,
|
||
the new versions setup phase takes care of database migrations.
|
||
You generally want to run the job in a controlled manner,
|
||
so multiple executions don’t interfere with each other.
|
||
Also, after the setup is done,
|
||
rolling out a new ZITADEL version is much faster
|
||
when the runtime processes are just executed with `zitadel start`.
|
||
|
||
## Separating Init and Setup from the Runtime
|
||
|
||
If you use the [official ZITADEL Helm chart](/self-hosting/deploy/kubernetes),
|
||
then you can stop reading now.
|
||
The init and setup phases are already separated and executed in dedicated Kubernetes Jobs.
|
||
If you use another orchestrator and want to separate the phases yourself,
|
||
you should know what happens during these phases.
|
||
|
||
### The Init Phase
|
||
|
||
The command `zitadel init` ensures the database connection is ready to use for the subsequent phases.
|
||
It just needs to be executed once over ZITADELs full life cycle,
|
||
when you install ZITADEL from scratch.
|
||
|
||
`zitadel init` performs two steps that require different privileges:
|
||
|
||
**Provisioning step** (requires a PostgreSQL superuser via `Database.postgres.Admin.Username`):
|
||
- If it doesn't exist already, it creates the unprivileged user configured in `Database.postgres.User.Username`.
|
||
Subsequent phases connect to the database with this user's credentials only.
|
||
- If it doesn't exist already, it creates a database with the configured database name.
|
||
- If not already done, it grants the necessary permissions to the unprivileged user.
|
||
|
||
**Schema bootstrapping step** (requires database owner privileges, uses the service user):
|
||
- If they don't exist already, it creates all schemas (`eventstore`, `projections`, `system`) and base tables.
|
||
|
||
<Callout type="note">
|
||
**Manual provisioning:** If you handle the provisioning step yourself (e.g. on a managed PostgreSQL service where you don't have superuser access),
|
||
you can skip it entirely by running only the schema bootstrapping step:
|
||
|
||
```bash
|
||
zitadel init schema
|
||
```
|
||
|
||
This sub-command connects using the service user credentials and only performs schema bootstrapping.
|
||
No `Admin.*` credentials are required.
|
||
See the [database documentation](/self-hosting/manage/database) for full details.
|
||
</Callout>
|
||
|
||
The init phase is idempotent if executed with the same binary version.
|
||
|
||
### The Setup Phase
|
||
|
||
During `zitadel setup`, ZITADEL creates projection tables and migrates existing data, if `--init-projections=true` is set.
|
||
Depending on the ZITADEL version and the runtime resources,
|
||
this step can take several minutes.
|
||
When deploying a new ZITADEL version,
|
||
make sure the setup phase runs before you roll out the new `zitadel start` processes.
|
||
The setup phase is executed in subsequent steps
|
||
whereas a new version's execution takes over where the last execution stopped.
|
||
|
||
Some configuration changes are only applied during the setup phase, like ExternalDomain, ExternalPort and ExternalSecure.
|
||
|
||
The setup phase is idempotent if executed with the same binary version.
|
||
|
||
### The Runtime Phase
|
||
|
||
The `zitadel start` command assumes the database is already initialized and set up.
|
||
It starts serving requests within fractions of a second.
|
||
Beware, in the background, out-of-date projections
|
||
[recompute their state by replaying all missed events](/concepts/eventstore/implementation#projections).
|
||
If a new ZITADEL version is deployed, this can take quite a long time,
|
||
depending on the amount of events to catch up.
|
||
You probably should consider providing `--init-projections=true`-flag to the [Setup Phase](#the-setup-phase) to shift the synchronization time to previous steps and delay the startup phase until events are caught up.
|