mirror of
https://github.com/andwati/zitadel.git
synced 2026-10-25 00:32:15 +00:00
75bc058bce
# Which Problems Are Solved
In Zitadel's append-only event-sourced architecture, the
`eventstore.events2` table grows indefinitely. PostgreSQL's default
autovacuum uses a percentage-based scale factor, so as the table grows,
the number of changed rows required to trigger a `VACUUM` or `ANALYZE`
drifts towards infinity. Without regular vacuums, the table's Visibility
Map becomes stale, disabling fast Index-Only Scans and forcing expensive
heap reads. Without regular analyzes, query planner statistics become
stale, leading to suboptimal execution plans.
This causes eventstore operations to progressively degrade as `events2`
grows, even without CPU, memory, or I/O saturation. A manual `VACUUM
ANALYZE` immediately restores performance, confirming the root cause.
- If `Eventstore.Autovacuum` is left at its default, `events2` keeps
using PostgreSQL's default, percentage-based autovacuum/autoanalyze
scale factors, which become impractically infrequent on large tables.
- There was previously no supported way to apply static, table-level
autovacuum tuning to `events2` through Zitadel's own configuration/setup
process.
# How the Problems Are Solved
- Added an `Eventstore.Autovacuum` runtime configuration block to
`cmd/defaults.yaml` (disabled by default):
```yaml
Eventstore:
Autovacuum:
Enabled: false # ZITADEL_EVENTSTORE_AUTOVACUUM_ENABLED
VacuumThreshold: 50000 # ZITADEL_EVENTSTORE_AUTOVACUUM_VACUUMTHRESHOLD
AnalyzeThreshold: 50000 # ZITADEL_EVENTSTORE_AUTOVACUUM_ANALYZETHRESHOLD
```
- Added a repeatable `zitadel setup` migration step
(`cmd/setup/eventstore_autovacuum.go`) that:
- When `Enabled: true`, disables the percentage-based
`autovacuum_vacuum_scale_factor`, `autovacuum_analyze_scale_factor`, and
`autovacuum_vacuum_insert_scale_factor` on `eventstore.events2`, and
applies static thresholds (`autovacuum_vacuum_insert_threshold`,
`autovacuum_vacuum_threshold`, `autovacuum_analyze_threshold`) from the
config instead.
- When `Enabled: false`, resets those storage parameters on
`eventstore.events2` back to the cluster defaults.
- Implements `Repeatable.Check()` so the step only re-runs when the
configuration actually changed since the last `zitadel setup` run.
- Added documentation in the new "Performance tuning" page.
# Additional Changes
- Move the projection documentation into performance tuning page
- Expand and update the projection documentation to the latest state in
zitadel. (contained some stale information)
# Additional Context
Several other issues describe read-performance symptoms consistent with
this same root cause (stale `events2` visibility map / planner
statistics at scale, without resource saturation). Since this PR
addresses the shared root cause:
- Closes #12448
- Closes #10754
- Closes #10260
- Closes #8585
- Closes #9239
---
_Generated by [Claude
Code](https://claude.ai/code/session_01HoKvEY7niCVgLCz7CajBwW)_
---------
Co-authored-by: Claude <[email protected]>
Co-authored-by: Copilot Autofix powered by AI <[email protected]>
Co-authored-by: Silvan <[email protected]>
222 lines
12 KiB
Plaintext
222 lines
12 KiB
Plaintext
---
|
|
title: ZITADEL Production Setup
|
|
description: "Configure self-hosted ZITADEL for production, including high availability, networking, monitoring, and logging best practices."
|
|
sidebar_label: Production Setup
|
|
---
|
|
|
|
As soon as you successfully deployed ZITADEL as a proof of concept using one of our [deployment guides](/self-hosting/deploy/overview),
|
|
you are ready to configure ZITADEL for production usage.
|
|
|
|
## TL;DR
|
|
We created a [Production Checklist](./productionchecklist) as a technical step by step guide.
|
|
|
|
## High Availability
|
|
|
|
We recommend running ZITADEL highly available using an orchestrator that schedules ZITADEL on multiple servers,
|
|
like [Kubernetes](/self-hosting/deploy/kubernetes).
|
|
For keeping startup times fast when scaling ZITADEL,
|
|
you should also consider using separate jobs with `zitadel init` and `zitadel setup`,
|
|
so your workload containers just have to execute `zitadel start`.
|
|
Read more about separating the init and setup phases on the [Updating and Scaling page](/self-hosting/manage/updating_scaling).
|
|
|
|
## Setup
|
|
|
|
Read [on the configure page](/self-hosting/manage/configure) about the available options you have to configure ZITADEL.
|
|
Prefer passing .yaml files to the ZITADEL binary instead of environment variables.
|
|
Restricting access to these files to avoid leaking sensitive information is easier than restricting access to environment variables.
|
|
Also, not all configuration options are available as environment variables.
|
|
|
|
## Networking
|
|
|
|
- To make ZITADEL available at the domain of your choice, [you need to configure the ExternalDomain property](/self-hosting/manage/custom-domain).
|
|
- To enable and restrict access to **HTTPS**, head over to [the description of your TLS options](/self-hosting/manage/tls_modes).
|
|
- If you want to front ZITADEL with a reverse proxy, web application firewall or content delivery network, make sure to support **[HTTP/2](/self-hosting/manage/http2)**.
|
|
- You can also refer to some **[example reverse proxy configurations](/self-hosting/manage/reverseproxy/reverse_proxy)**.
|
|
- The ZITADEL Management Console web GUI uses many gRPC-Web stubs. This results in a fairly big JavaScript bundle. You might want to compress it using [Gzip](/self-hosting/manage/custom-domain) or [Brotli](https://www.gnu.org/software/gzip/) or [Brotli](https://github.com/google/brotli).
|
|
- Serving and caching the assets using a content delivery network could improve network latencies and shield your ZITADEL runtime.
|
|
|
|
## Monitoring
|
|
|
|
By default, [**metrics**](/apis/observability/metrics) are exposed at /debug/metrics in OpenTelemetry (otel) format.
|
|
|
|
Also, you can enable **tracing** in the ZITADEL configuration.
|
|
|
|
```yaml
|
|
Tracing:
|
|
# Choose one in "otel", "google", "log" and "none"
|
|
Type: google
|
|
Fraction: 1
|
|
MetricPrefix: zitadel
|
|
```
|
|
|
|
## Logging
|
|
|
|
ZITADEL follows the principles that guide cloud-native and twelve factor applications.
|
|
Logs are a stream of time-ordered events collected from all running processes.
|
|
|
|
[ZITADEL is configurable](#default-zitadel-logging-config) to write the following events to the standard output:
|
|
|
|
- Runtime Logs: Define the log level and record format in the `Log` configuration section.
|
|
- Access Logs: Enable logging all HTTP and gRPC responses from the ZITADEL binary by setting `LogStore.Access.Stdout.Enabled` to true.
|
|
- Actions Execution Logs: Actions can emit custom logs at different levels. For example, a log record can be emitted each time a user is created or authenticated. If you don't want to have these logs in STDOUT, you can disable this by setting `LogStore.Execution.Stdout.Enabled` to true.
|
|
|
|
### Default ZITADEL Logging Config
|
|
|
|
```yaml
|
|
Log:
|
|
Level: info # ZITADEL_LOG_LEVEL
|
|
Formatter:
|
|
Format: text # ZITADEL_LOG_FORMATTER_FORMAT
|
|
|
|
LogStore:
|
|
Access:
|
|
Stdout:
|
|
# If enabled, all access logs are printed to the binary's standard output
|
|
Enabled: false # ZITADEL_LOGSTORE_ACCESS_STDOUT_ENABLED
|
|
Execution:
|
|
Stdout:
|
|
# If enabled, all execution logs are printed to the binary's standard output
|
|
Enabled: true # ZITADEL_LOGSTORE_EXECUTION_STDOUT_ENABLED
|
|
|
|
```
|
|
|
|
### Why ZITADEL does not write logs to files
|
|
|
|
Log file management should not be in each business apps responsibility.
|
|
Instead, your execution environment should provide tooling for managing logs in a generic way.
|
|
This includes tasks like rotating files, routing, collecting, archiving and cleaning-up.
|
|
For example, systemd has journald and kubernetes has fluentd and fluentbit.
|
|
|
|
## Telemetry
|
|
|
|
If you want to have some data about reached usage milestones pushed to external systems, enable telemetry in the ZITADEL configuration.
|
|
|
|
The following table describes the milestones that are sent to the endpoints:
|
|
|
|
| Trigger | Description |
|
|
|-----------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------|
|
|
| A virtual instance is created. | This data point is also sent when the first instance is automatically created during the ZITADEL binaries setup phase in a self-hosting scenario. |
|
|
| An authentication succeeded for the first time on an instance. | This is the first authentication with the instances automatically created admin user during the instance setup, which can be a user or a service account. |
|
|
| A project is created for the first time in a virtual instance. | The ZITADEL project that is automatically created during the instance setup is omitted. |
|
|
| An application is created for the first time in a virtual instance. | The applications in the ZITADEL project that are automatically created during the instance setup are omitted. |
|
|
| An authentication succeeded for the first time in a virtual instances application. | This is the first authentication using a ZITADEL application that is not created during the instance setup phase. |
|
|
| A virtual instance is deleted. | This data point is sent when a virtual instance is deleted via ZITADELs system API |
|
|
|
|
ZITADEL pushes the metrics by projecting certain events.
|
|
Therefore, you can configure delivery guarantees not in the Telemetry section of the ZITADEL configuration,
|
|
but in the Projections.Customizations.Telemetry section
|
|
|
|
## Database
|
|
|
|
### Prefer PostgreSQL
|
|
|
|
ZITADEL supports [PostgreSQL](https://www.postgresql.org/).
|
|
|
|
<Callout>
|
|
ZITADEL supports PostgreSQL. For migrating from CockroachDB to PostgreSQL, please refer to [the mirror guide](/self-hosting/manage/cli/mirror).
|
|
</Callout>
|
|
|
|
The indexes for the database are optimized using load tests from [ZITADEL Cloud](https://zitadel.com),
|
|
which runs with PostgreSQL.
|
|
If you identify problems with your database during load tests that indicate that the indexes are not optimized,
|
|
please create an issue in our [github repository](https://github.com/zitadel/zitadel).
|
|
|
|
### Configure ZITADEL
|
|
|
|
Depending on your environment, you maybe would want to tweak some settings about how ZITADEL interacts with the database in the database section of your ZITADEL configuration. Read more about your [database configuration options](/self-hosting/manage/database).
|
|
|
|
Set the database connection via environment variable:
|
|
|
|
```bash
|
|
export ZITADEL_DATABASE_POSTGRES_DSN=postgresql://zitadel:password@localhost:5432/zitadel?sslmode=disable
|
|
export ZITADEL_DATABASE_POSTGRES_MAXOPENCONNS=10
|
|
export ZITADEL_DATABASE_POSTGRES_MAXIDLECONNS=5
|
|
export ZITADEL_DATABASE_POSTGRES_MAXCONNLIFETIME=30m
|
|
export ZITADEL_DATABASE_POSTGRES_MAXCONNIDLETIME=5m
|
|
```
|
|
|
|
Or in a configuration file:
|
|
|
|
```yaml
|
|
Database:
|
|
postgres:
|
|
DSN: postgresql://zitadel:password@localhost:5432/zitadel?sslmode=disable
|
|
//highlight-start
|
|
MaxOpenConns: 10
|
|
MaxIdleConns: 5
|
|
MaxConnLifetime: 30m
|
|
MaxConnIdleTime: 5m
|
|
//highlight-end
|
|
```
|
|
|
|
### Manage your data
|
|
|
|
When designing your backup strategy,
|
|
it is worth knowing that
|
|
[ZITADEL is event sourced](/concepts/eventstore/overview).
|
|
That means, ZITADEL itself is able to recompute its
|
|
whole state from the records in the table `eventstore.events2`.
|
|
The timestamp of your last record in the events table
|
|
defines up to which point in time ZITADEL can restore its state.
|
|
|
|
The ZITADEL binary itself is stateless,
|
|
so there is no need for a special backup job.
|
|
|
|
Generally, for maintaining your database management system in production,
|
|
please refer to the corresponding docs [for PostgreSQL](https://www.postgresql.org/docs/current/admin.html).
|
|
|
|
## Data initialization
|
|
|
|
- You can configure instance defaults in the DefaultInstance section.
|
|
If you plan to eventually create [multiple virtual instances](/concepts/structure/instance#multiple-virtual-instances), these defaults take effect.
|
|
Also, these settings apply to the first instance, that ZITADEL automatically creates for you.
|
|
Especially the following properties are of special interest for your production setup.
|
|
|
|
```yaml
|
|
DefaultInstance:
|
|
OIDCSettings:
|
|
AccessTokenLifetime: 12h
|
|
IdTokenLifetime: 12h
|
|
RefreshTokenIdleExpiration: 720h #30d
|
|
RefreshTokenExpiration: 2160h #90d
|
|
# this sets the default email settings
|
|
SMTPConfiguration:
|
|
# settings of the host
|
|
SMTP:
|
|
#for example smtp.mailtrap.io:2525
|
|
Host:
|
|
User:
|
|
Password:
|
|
TLS:
|
|
# if the host of the sender is different from ExternalDomain set DefaultInstance.DomainPolicy.SMTPSenderAddressMatchesInstanceDomain to false
|
|
From:
|
|
FromName:
|
|
ReplyToAddress:
|
|
```
|
|
|
|
- If you don't want to use the DefaultInstance configuration for the first instance that ZITADEL automatically creates for you during the [setup phase](/self-hosting/manage/configure/configure#database-initialization-file), you can provide a FirstInstance YAML section using the --steps argument.
|
|
- Learn how to configure ZITADEL via the [Console user interface](/guides/manage/console/console-overview).
|
|
- Probably, you also want to [apply your custom branding](/guides/manage/customize/branding), [hook into certain events](/guides/manage/customize/behavior), [customize texts](/guides/manage/customize/texts) or [add metadata to your users](/guides/manage/customize/user-metadata).
|
|
- If you want to automatically create ZITADEL resources, you can use the [ZITADEL Terraform Provider](/guides/manage/terraform-provider).
|
|
|
|
## Limits and Quotas
|
|
|
|
If you host ZITADEL as a service,
|
|
you might want to [limit usage and/or execute tasks on certain usage units and levels](/self-hosting/manage/usage_control).
|
|
|
|
## Minimum system requirements
|
|
|
|
### General resource usage
|
|
|
|
ZITADEL itself requires approximately 512MB of RAM and can operate with less than one CPU core. The database component, under typical conditions, utilizes about one CPU core per 100 requests per second (req/s) and 4GB of RAM per core, which includes some caching.
|
|
|
|
<Callout title="Password hashing">
|
|
Be aware of CPU spikes when hashing passwords. We recommend to have 4 CPU cores available for this purpose.
|
|
</Callout>
|
|
|
|
### Production HA cluster
|
|
|
|
For a minimal high-availability setup, we recommend a cluster of 3 nodes, each with 4 CPU cores and 16GB of memory.
|
|
|
|
If you exclude non-essential services like log collection and metrics, you can reduce the resources to approximately 4 CPU cores and 8GB of memory per node.
|