andwati/zitadel
1
0
Fork 0
mirror of https://github.com/andwati/zitadel.git synced 2026-10-18 05:52:51 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(eventstore): autovacuum tuning for events2 table (#12449)

Browse Source 
# Which Problems Are Solved

In Zitadel's append-only event-sourced architecture, the
`eventstore.events2` table grows indefinitely. PostgreSQL's default
autovacuum uses a percentage-based scale factor, so as the table grows,
the number of changed rows required to trigger a `VACUUM` or `ANALYZE`
drifts towards infinity. Without regular vacuums, the table's Visibility
Map becomes stale, disabling fast Index-Only Scans and forcing expensive
heap reads. Without regular analyzes, query planner statistics become
stale, leading to suboptimal execution plans.

This causes eventstore operations to progressively degrade as `events2`
grows, even without CPU, memory, or I/O saturation. A manual `VACUUM
ANALYZE` immediately restores performance, confirming the root cause.

- If `Eventstore.Autovacuum` is left at its default, `events2` keeps
using PostgreSQL's default, percentage-based autovacuum/autoanalyze
scale factors, which become impractically infrequent on large tables.
- There was previously no supported way to apply static, table-level
autovacuum tuning to `events2` through Zitadel's own configuration/setup
process.

# How the Problems Are Solved

- Added an `Eventstore.Autovacuum` runtime configuration block to
`cmd/defaults.yaml` (disabled by default):
  ```yaml
  Eventstore:
    Autovacuum:
      Enabled: false # ZITADEL_EVENTSTORE_AUTOVACUUM_ENABLED
VacuumThreshold: 50000 # ZITADEL_EVENTSTORE_AUTOVACUUM_VACUUMTHRESHOLD
AnalyzeThreshold: 50000 # ZITADEL_EVENTSTORE_AUTOVACUUM_ANALYZETHRESHOLD
  ```
- Added a repeatable `zitadel setup` migration step
(`cmd/setup/eventstore_autovacuum.go`) that:
- When `Enabled: true`, disables the percentage-based
`autovacuum_vacuum_scale_factor`, `autovacuum_analyze_scale_factor`, and
`autovacuum_vacuum_insert_scale_factor` on `eventstore.events2`, and
applies static thresholds (`autovacuum_vacuum_insert_threshold`,
`autovacuum_vacuum_threshold`, `autovacuum_analyze_threshold`) from the
config instead.
- When `Enabled: false`, resets those storage parameters on
`eventstore.events2` back to the cluster defaults.
- Implements `Repeatable.Check()` so the step only re-runs when the
configuration actually changed since the last `zitadel setup` run.
  - Added documentation in the new "Performance tuning" page.
 
# Additional Changes

- Move the projection documentation into performance tuning page
- Expand and update the projection documentation to the latest state in
zitadel. (contained some stale information)

# Additional Context

Several other issues describe read-performance symptoms consistent with
this same root cause (stale `events2` visibility map / planner
statistics at scale, without resource saturation). Since this PR
addresses the shared root cause:

- Closes #12448
- Closes #10754
- Closes #10260
- Closes #8585
- Closes #9239


---
_Generated by [Claude
Code](https://claude.ai/code/session_01HoKvEY7niCVgLCz7CajBwW)_

---------

Co-authored-by: Claude <[email protected]>
Co-authored-by: Copilot Autofix powered by AI <[email protected]>
Co-authored-by: Silvan <[email protected]>
This commit is contained in:
Tim Möhlmann
2026-07-16 14:42:11 +00:00
committed by GitHub
co-authored by Claude Copilot Autofix powered by AI Silvan
parent 6bd46c5fb6
commit 75bc058bce
8 changed files with 615 additions and 39 deletions
Download Patch File Download Diff File Expand all files Collapse all files
apps/docs/content/self-hosting/manage/production.mdx
+1 -39
View File
@@ -149,51 +149,13 @@ Database:
//highlight-end
```
You also might want to configure how [projections](/concepts/eventstore/implementation#projections) are computed. These are the default values:
```yaml
# The Projections section defines the behavior for the scheduled and synchronous events projections.
Projections:
# Time interval between scheduled projections
RequeueEvery: 60s
# Time between retried database statements resulting from projected events
RetryFailedAfter: 1s
# Retried execution number of database statements resulting from projected events
MaxFailureCount: 5
# Number of concurrent projection routines. Values of 0 and below are overwritten to 1
ConcurrentInstances: 1
# Limit of returned events per query
BulkLimit: 200
# Only instance are projected, for which at least a projection relevant event exists withing the timeframe
# from HandleActiveInstances duration in the past until the projections current time
# Defaults to twice the RequeueEvery duration
HandleActiveInstances: 120s
# In the Customizations section, all settings from above can be overwritten for each specific projection
Customizations:
Projects:
BulkLimit: 2000
# The Notifications projection is used for sending emails and SMS to users
Notifications:
# As notification projections don't result in database statements, retries don't have an effect
MaxFailureCount: 0
# The NotificationsQuotas projection is used for calling quota webhooks
NotificationsQuotas:
# Delivery guarantee requirements are probably higher for quota webhooks
# Defaults to 45 days
HandleActiveInstances: 1080h
# As quota notification projections don't result in database statements, retries don't have an effect
MaxFailureCount: 0
# Quota notifications are not so time critical. Setting RequeueEvery every five minutes doesn't annoy the db too much.
RequeueEvery: 300s
```
### Manage your data
When designing your backup strategy,
it is worth knowing that
[ZITADEL is event sourced](/concepts/eventstore/overview).
That means, ZITADEL itself is able to recompute its
whole state from the records in the table eventstore.events.
whole state from the records in the table `eventstore.events2`.
The timestamp of your last record in the events table
defines up to which point in time ZITADEL can restore its state.
apps/docs/content/self-hosting/manage/tuning.mdx
+253
View File
@@ -0,0 +1,253 @@
---
title: ZITADEL Performance Tuning
description: "Tune ZITADEL's eventstore and projections for large-scale, high-throughput deployments."
sidebar_label: Performance Tuning
---
With [ZITADEL being event sourced](/concepts/eventstore/overview), a lot of the performance throughput depends on how the events table is accessed.
The events table is an append-only table which will grow over the lifetime of a deployment.
ZITADEL uses a [CQRS architecture](/concepts/architecture/software), which results in concurrent reads and writes against this table.
The eventstore is used to make consistent real-time decisions when executing "Commands". (For example: is a user not locked just before creating a token)
Projections store computed state of an object, which allows us to "Query" data more efficiently. (For example: finding a user by their email).
Because projections depend on events being reduced by a spooler, they are eventually consistent and may fall behind the state in the eventstore.
We counter eventual consistency by pub-sub and triggers.
The defaults for the configuration sections discussed below typically work well for small to medium sized deployments and do not need tuning.
We try to provide some general guidelines here.
However we recommend reaching out to support if your installation runs at a scale that requires tuning these values,
so our engineers can provide you with proper guidance based on your workload.
## Eventstore
<details>
<summary>Eventstore default configuration</summary>
```yaml
Eventstore:
# Sets the maximum duration of transactions pushing events
PushTimeout: 15s #ZITADEL_EVENTSTORE_PUSHTIMEOUT
# Maximum amount of push retries in case of primary key violation on the sequence
MaxRetries: 5 #ZITADEL_EVENTSTORE_MAXRETRIES
# Autovacuum tunes PostgreSQL's autovacuum and autoanalyze behavior of the
# eventstore.events2 table. As the table grows, PostgreSQL's default,
# percentage-based scale factors delay vacuum and analyze runs more and
# more, which can degrade read performance on large instances. Enabling
# this replaces the scale factors with static thresholds, so vacuum and
# analyze keep running at a predictable cadence regardless of table size.
# Settings are (re-)applied to the table as a repeatable step during `zitadel setup`.
Autovacuum:
# If enabled, the thresholds below are applied to the eventstore.events2 table
# and the default, percentage-based autovacuum scale factors are disabled.
# If disabled, the table is reset to the cluster's default autovacuum settings.
Enabled: false # ZITADEL_EVENTSTORE_AUTOVACUUM_ENABLED
# Number of modified rows that triggers an autovacuum run, regardless of table size (applied in both autovacuum_vacuum_threshold and autovacuum_vacuum_insert_threshold settings).
# Must be greater than 10000.
VacuumThreshold: 50000 # ZITADEL_EVENTSTORE_AUTOVACUUM_VACUUMTHRESHOLD
# Number of changed rows that triggers an autoanalyze run, regardless of table size.
# Must be greater than 10000.
AnalyzeThreshold: 50000 # ZITADEL_EVENTSTORE_AUTOVACUUM_ANALYZETHRESHOLD
```
</details>
### Pushing events
Pushing events happens when a resource is created or modified in a "Command".
The sequence in which events are pushed for an object is important.
A push is typically sub-millisecond if the database has sufficient resources and disk I/O available,
but may become slower when operating close to the resource limits during traffic spikes.
We employ row-level locking and unique constraint checks to ensure the sequence value of events is never duplicated on a single object.
Events for objects of different resources or distinct IDs happen concurrently. Some example cases:
- Two processes updating the metadata of the same organization at the same time: this cannot happen concurrently. One of two requests will take longer, waiting for the first to complete.
- Two independent users logging-in at the same time: happens concurrently. There is no lock or unique constraint stopping the two independent transactions.
The following options control the behavior of the pusher when the database is under strain:
- `Eventstore.PushTimeout`: How long the push transaction is allowed to run before it's canceled.
This protects against dead-locks, network timeouts or queries that simply run too long.
This acts as a circuit breaker in the case transactions freeze up.
Once the timeout expires the transaction is aborted,
and the underlying database connection discarded, instead of reused through the connection pool.
The push may be retried, or an error is returned to the caller / API client.
- A smaller value allows for fail-fast and faster retries.
However a value that's too small may result in unneeded retries,
exhausting the DBs resources even more.
- A larger value may result in unnecessarily long request lockups when the outcome is failure anyway.
- `Eventstore.MaxRetries`: If somehow, despite row-level locking, there is a duplicate sequence on an object's events,
the push transaction fails with a distinct unique constraint error.
ZITADEL will retry the push with an incremented sequence based on events previously inserted by the other transaction.
Retry is aggressive and does not use throttling!
Many concurrent retries will lead to resource starvation on the DB and ZITADEL.
When the pusher is out of retries, the last error is returned to the caller / API client.
- A smaller value protects the database against resource starvation.
However, running out of retries means an API request will fail, which may require the API client to retry instead.
- A larger value may result in resource starvation of the database.
API requests will take as long as the retries to execute.
This may result in a lot of back-pressure, making things only worse.
In other words, **do not touch** these values unless dealing with some exotic issues or instructed so by ZITADEL engineers.
### Reading events
Events are read by both the projection spoolers and during "Command" execution.
During command execution ZITADEL first reads events to determine the latest state of an object (for example "is a user not locked")
and a second time during a [push](#pushing-events), to establish the latest sequence of an object.
Because the eventstore is append-only, the table can grow to 100s of millions of rows on large production systems.
Keeping the table and index statistics up-to-date is paramount for PostgreSQL to choose the correct indexes.
When the table and index statistics are stale, PostgreSQL will use suboptimal plans and will become slow.
This has major performance impact on ZITADEL: pushes will take longer and spoolers will become slower,
resulting in larger eventual consistency gaps and higher resource consumption overall.
PostgreSQL can [`ANALYZE`](https://www.postgresql.org/docs/current/sql-analyze.html) the table to rebuild those statistics,
keeping the eventstore reads performant. PostgreSQL uses [routine vacuuming](https://www.postgresql.org/docs/current/routine-vacuuming.html)
jobs to automate analyzing of all tables.
However, PostgreSQL defaults to a percentage-based row count for it to trigger the vacuum and analyze routines,
with a default value of 10% for analyze routines.
Please consult your database admin or hosting platform, as they may use a different value for `autovacuum_analyze_scale_factor`.
Keeping the default is usually fine for smaller deployments, and tuning the routines may even reduce performance.
However, for medium to large systems and hyperscalers, the routine intervals will progressively become longer as the table grows,
to the point where they almost never run. For example:
1. With 100k existing events, 10k new events will trigger the analyze routine.
2. With 10M existing events, 1M new events will trigger the analyze routine.
3. With 100M existing events, 10M new events will trigger the analyze routine.
Cases 1 and 2 will probably not experience any noticeable performance impact.
However, case 3 will notice a significant slowdown of the eventstore.
We found the tipping point lying between 2M-3M new rows inserted on a 250M row eventstore table.
ZITADEL provides options to switch to static routine thresholds instead,
solving the problem of the routine intervals going to infinity.
- `Eventstore.Autovacuum.Enabled`: switches on the ZITADEL customization of threshold based routines.
This option defaults to `false`, as enabling this with the wrong threshold value will have adverse effects on performance.
Small systems don't need this as the ratio-based routines are actually better for small tables.
- `Eventstore.Autovacuum.VacuumThreshold`: Number of modified rows that triggers an autovacuum run.
- `Eventstore.Autovacuum.AnalyzeThreshold`: Number of changed rows that triggers an autoanalyze run.
Both thresholds have the same performance consequences:
- Pick a value that's too large and the routines never run, leading to stale planner statistics and bad performance.
- Pick a value that's too small and the routines will run too often, consuming valuable database resources,
even risking routines running back-to-back continuously.
The correct value of these settings highly depends on the amount of events pushed over a certain timeframe,
not the current amount of events present in the table.
The amount of events pushed then again depends on the amount and type of requests served. For example:
- Service account authentication pushes 2 events per request
- Human user authentication may push 5-10 events per successful flow,
depending on the amount of factors used and requested token types.
As a rough target, aim for the routines to trigger every few minutes to a low tens-of-minutes at your expected peak traffic.
This is mainly relevant for Enterprise and Mega Enterprise scale deployments, where the eventstore table is large enough that stale
statistics or visibility maps have a noticeable performance impact within minutes. Medium and Homelab deployments naturally land on
longer intervals with the same threshold, and that's fine: their eventstore tables are much smaller, so infrequent maintenance doesn't
let statistics drift far enough to matter.
The following table provides example values for various deployment sizes:
| Scale Profile | Events / Sec during peak traffic | Recommended Vacuum Threshold | Expected Trigger Frequency | Why This Setting? |
| :---- | :---- | :---- | :---- | :---- |
| **Homelab / Proof of Concept** | 0-10 | Keep switched off / use ratio based. | Every few hours | Keeps things snappy on weak CPUs without letting statistics get stale. |
| **Medium / Growth** | 10-100 | 50,000 | Every 8 to 80 minutes | Perfect balance of maintaining the visibility map for index-only scans without I/O pressure. |
| **Enterprise / High Scale** | 1,000 | 500,000 | Every 8 minutes | Prevents CPU thrashing while keeping index-only scans functional. |
| **Mega Enterprise** | 10,000 | 2,000,000 | Every 3.3 minutes | Keeps a massive 64-core PG instance highly optimized without queuing up concurrent autovacuum processes. |
## Projections
You also might want to configure how [projections](/concepts/eventstore/implementation#projections) are computed.
Real-time consistency is primarily driven by pub/sub: as soon as an event is pushed, subscribed projections are notified and reduce it immediately.
The settings below mostly tune a *secondary*, scheduled catch-up mechanism that acts as a safety net for cases where a pub/sub notification
is missed, for example due to a network hiccup or a restart. Most of these settings therefore affect how quickly a projection recovers
from a missed notification, not how fast a healthy system reacts to new events.
<details>
<summary>Projections default configuration</summary>
```yaml
# The Projections section defines the behavior for the scheduled and synchronous events projections.
Projections:
# The maximum duration a single projection transaction remains open while
# folding a batch of events into the projection table.
# 0 means no timeout is applied: the transaction runs until its batch (bounded by BulkLimit) finishes.
TransactionDuration: 1m # ZITADEL_PROJECTIONS_TRANSACTIONDURATION
# Time interval between scheduled catch-up sweeps
RequeueEvery: 60s # ZITADEL_PROJECTIONS_REQUEUEEVERY
# Time between retried database statements resulting from projected events
RetryFailedAfter: 1s # ZITADEL_PROJECTIONS_RETRYFAILEDAFTER
# Retried execution number of database statements resulting from projected events
MaxFailureCount: 5 # ZITADEL_PROJECTIONS_MAXFAILURECOUNT
# Limit of returned events per query
BulkLimit: 200 # ZITADEL_PROJECTIONS_BULKLIMIT
# Only instances are considered by the scheduled sweep, for which at least one API request was served within the
# timeframe from HandleActiveInstances duration in the past until now.
# If set to 0 (default), every instance ever seen is always considered active.
HandleActiveInstances: 0s # ZITADEL_PROJECTIONS_HANDLEACTIVEINSTANCES
# Maximum amount of instances cached as active. If set to 0 (default), every instance is always considered active.
MaxActiveInstances: 0 # ZITADEL_PROJECTIONS_MAXACTIVEINSTANCES
# Limits the amount of concurrently running projection triggers, shared across all projections and instances.
# If set to 0, 1/3 of Database.MaxOpenConns is used. Must be lower than Database.MaxOpenConns.
MaxParallelTriggers: 0 # ZITADEL_PROJECTIONS_MAXPARALLELTRIGGERS
# In the Customizations section, all settings from above (except HandleActiveInstances, MaxActiveInstances and
# MaxParallelTriggers, which are global) can be overwritten for each specific projection
Customizations:
# The Notifications projection prepares emails and SMS messages sent to users
Notifications:
MaxFailureCount: 10 # ZITADEL_PROJECTIONS_CUSTOMIZATIONS_NOTIFICATIONS_MAXFAILURECOUNT
# The NotificationsQuotas projection is used for calling quota webhooks
NotificationsQuotas:
MaxFailureCount: 10 # ZITADEL_PROJECTIONS_CUSTOMIZATIONS_NOTIFICATIONSQUOTAS_MAXFAILURECOUNT
# Quota notifications are not so time critical. Setting RequeueEvery every five minutes doesn't annoy the db too much.
RequeueEvery: 300s # ZITADEL_PROJECTIONS_CUSTOMIZATIONS_NOTIFICATIONSQUOTAS_REQUEUEEVERY
# Sending emails can take longer than 500ms
TransactionDuration: 5s # ZITADEL_PROJECTIONS_CUSTOMIZATIONS_NOTIFICATIONQUOTAS_TRANSACTIONDURATION
```
</details>
- `Projections.RequeueEvery`: Interval between scheduled catch-up sweeps.
- Too low: sweeps run needlessly often even though pub/sub has already kept every projection up to date, adding avoidable database load.
- Too high: if a pub/sub notification is ever missed, a projection can silently lag for up to this long before the next sweep corrects it.
Some projections override this to a much longer interval when their events are inherently low-urgency, for example
`Customizations.NotificationsQuotas.RequeueEvery: 300s` (5 minutes), since a delayed quota webhook doesn't justify sweeping every minute.
- `Projections.RetryFailedAfter` and `Projections.MaxFailureCount`: Control retries of an individual **statement** derived from
reducing one event, not of the whole projection or instance.
When a statement fails, ZITADEL waits `RetryFailedAfter` and retries; while retrying, it blocks further processing of the other
instances queued in the same catch-up batch, so a single stuck event can delay unrelated tenants.
Once a statement has failed `MaxFailureCount` times, it's recorded as permanently skipped and never retried again, letting the
projection move past it instead of getting stuck forever.
- Too small `RetryFailedAfter`: hammers the database with retries with no meaningful backoff.
- Too large `RetryFailedAfter`: a brief, transient error (deadlock, network blip) needlessly stalls the rest of that batch.
- Too small `MaxFailureCount`: a recoverable failure (e.g. a brief DB hiccup) may be abandoned before it would have succeeded.
- Too large `MaxFailureCount`: a truly broken event keeps retrying, and blocking its batch, for longer before being given up on.
As not every projection results in database statements, some override `MaxFailureCount`, for example `Customizations.Notifications`
and `Customizations.NotificationsQuotas` set it to `10`, since notification delivery only produces side effects (sending an
email, SMS, or webhook) rather than statements that could deadlock or violate a constraint.
- `Projections.BulkLimit`: Number of events fetched, and processed within one transaction, per catch-up iteration.
- Larger values: fewer round-trips when catching up a large backlog, but each transaction (and the locks it holds) runs longer.
- Smaller values: shorter transactions and more frequent progress checkpoints, at the cost of more round-trip overhead to consume the same backlog.
- `Projections.TransactionDuration`: Maximum time a single projection transaction may stay open while folding a batch of events.
- Too short: the transaction can be cut off mid-batch before it commits, wasting the work done so far and requiring that batch to be reprocessed.
- Too long / unbounded (`0`): a single slow or stuck batch can hold its transaction, and the row locks it acquired, open indefinitely.
- `Projections.HandleActiveInstances` and `Projections.MaxActiveInstances`: Limit which instances (ZITADEL tenants) the *scheduled*
sweep bothers with. An instance is considered "active" if it has served API traffic within the last `HandleActiveInstances`
duration — this is a cache of recently-seen tenants, not a scan of the events table, and it has no effect on the real-time
pub/sub trigger.
- The defaults (`0`/`0`) mean every instance ever seen stays "active" indefinitely and the cache never evicts. This is fine
for most deployments, but on an instance with heavy tenant churn, the periodic sweep keeps visiting every tenant it's
ever seen, including long-decommissioned ones.
- Setting `HandleActiveInstances` too short can drop a legitimately idle-but-still-in-use tenant out of the sweep list early;
this only matters if that tenant also happens to miss a pub/sub notification while considered inactive.
- `MaxActiveInstances` caps how many tenant IDs are cached (evicting the least-recently-used), bounding memory use on very
large multi-tenant deployments.
- `Projections.MaxParallelTriggers`: Size of the shared worker pool that processes projection triggers, across *all* projections
and *all* instances combined — not a per-projection setting.
- Must stay below `Database.MaxOpenConns`; a value of `0` auto-sizes it to a third of `Database.MaxOpenConns`.
- Too low: projection catch-up throughput becomes a bottleneck under load, widening eventual-consistency gaps.
- Too high: starves database connections needed for pushes, queries, and notifications.
- `Projections.Customizations`: Overrides `BulkLimit`, `MaxFailureCount`, `RequeueEvery`, `RetryFailedAfter`, and
`TransactionDuration` per named projection. `HandleActiveInstances`, `MaxActiveInstances`, and `MaxParallelTriggers` are
global settings and cannot be overridden per projection.
apps/docs/lib/sidebar-data.ts
+1
View File
@@ -813,6 +813,7 @@ export const guidesSidebar: readonly SidebarItem[] = [
"self-hosting/manage/upgrade-v3-to-v4",
"self-hosting/manage/database/database",
"self-hosting/manage/cache",
"self-hosting/manage/tuning",
],
},