# Which Problems Are Solved
In Zitadel's append-only event-sourced architecture, the
`eventstore.events2` table grows indefinitely. PostgreSQL's default
autovacuum uses a percentage-based scale factor, so as the table grows,
the number of changed rows required to trigger a `VACUUM` or `ANALYZE`
drifts towards infinity. Without regular vacuums, the table's Visibility
Map becomes stale, disabling fast Index-Only Scans and forcing expensive
heap reads. Without regular analyzes, query planner statistics become
stale, leading to suboptimal execution plans.
This causes eventstore operations to progressively degrade as `events2`
grows, even without CPU, memory, or I/O saturation. A manual `VACUUM
ANALYZE` immediately restores performance, confirming the root cause.
- If `Eventstore.Autovacuum` is left at its default, `events2` keeps
using PostgreSQL's default, percentage-based autovacuum/autoanalyze
scale factors, which become impractically infrequent on large tables.
- There was previously no supported way to apply static, table-level
autovacuum tuning to `events2` through Zitadel's own configuration/setup
process.
# How the Problems Are Solved
- Added an `Eventstore.Autovacuum` runtime configuration block to
`cmd/defaults.yaml` (disabled by default):
```yaml
Eventstore:
Autovacuum:
Enabled: false # ZITADEL_EVENTSTORE_AUTOVACUUM_ENABLED
VacuumThreshold: 50000 # ZITADEL_EVENTSTORE_AUTOVACUUM_VACUUMTHRESHOLD
AnalyzeThreshold: 50000 # ZITADEL_EVENTSTORE_AUTOVACUUM_ANALYZETHRESHOLD
```
- Added a repeatable `zitadel setup` migration step
(`cmd/setup/eventstore_autovacuum.go`) that:
- When `Enabled: true`, disables the percentage-based
`autovacuum_vacuum_scale_factor`, `autovacuum_analyze_scale_factor`, and
`autovacuum_vacuum_insert_scale_factor` on `eventstore.events2`, and
applies static thresholds (`autovacuum_vacuum_insert_threshold`,
`autovacuum_vacuum_threshold`, `autovacuum_analyze_threshold`) from the
config instead.
- When `Enabled: false`, resets those storage parameters on
`eventstore.events2` back to the cluster defaults.
- Implements `Repeatable.Check()` so the step only re-runs when the
configuration actually changed since the last `zitadel setup` run.
- Added documentation in the new "Performance tuning" page.
# Additional Changes
- Move the projection documentation into performance tuning page
- Expand and update the projection documentation to the latest state in
zitadel. (contained some stale information)
# Additional Context
Several other issues describe read-performance symptoms consistent with
this same root cause (stale `events2` visibility map / planner
statistics at scale, without resource saturation). Since this PR
addresses the shared root cause:
- Closes #12448
- Closes #10754
- Closes #10260
- Closes #8585
- Closes #9239
---
_Generated by [Claude
Code](https://claude.ai/code/session_01HoKvEY7niCVgLCz7CajBwW)_
---------
Co-authored-by: Claude <[email protected]>
Co-authored-by: Copilot Autofix powered by AI <[email protected]>
Co-authored-by: Silvan <[email protected]>
@zitadel/docs
Documentation for Zitadel, built with Fumadocs and Next.js.
Getting Started
Ensure you have followed the root quick start to set up dependencies.
Local Development
Ensure all dependencies are installed:
pnpm install
Start the development server:
pnpm nx run @zitadel/docs:dev
The site will be available at http://localhost:3000.
Scripts
Key scripts for documentation workflows:
| Script | Description |
|---|---|
dev |
Starts the development server. |
build |
Builds the production application. |
fetch:remote-content |
Fetches remote tags and referenced content. |
generate |
Runs all generation steps (fetch:remote-content, generate:proto-docs, generate:api-reference, generate:index-pages). |
check:links |
Validates content integrity (broken links, missing frontmatter, schema errors). |
check-types |
Validates typescript types. |
test |
Runs all validation steps (check-types, check:links). |
lint |
checks for code style and syntax errors (ESLint). |
clean |
Cleans the build output and generated files. |
Validation
- Code Quality: Run
pnpm nx run @zitadel/docs:lintto check for syntax and style issues in JS/TS/MDX files. - Content Integrity: Run
pnpm nx run @zitadel/docs:check-linksto validate content structure, including:- Broken internal links
- Missing required front-matter (e.g.,
title) - Image references
Troubleshooting
Issue: Nx configuration not found or "nx" command not found
If you see errors like:
NX Cannot find configuration for task @zitadel/zitadel:@zitadel/docs:generate
Pass --verbose to see the stacktrace.
or
pnpm nx run @zitadel/docs:dev
ERR_PNPM_RECURSIVE_EXEC_FIRST_FAIL Command "nx" not found
Did you mean "pnpm nx"?
Root cause: Your dependency cache or node_modules is out of sync.
Solution: Run these commands from the repository root in order:
- Clear node_modules:
rm -rf node_modules— removes the installed dependency folder - Clear pnpm cache:
pnpm cache delete— clears pnpm's internal package cache - Reinstall dependencies:
pnpm install— reinstalls all dependencies fresh - Test the setup:
pnpm nx run @zitadel/docs:generate— verifies everything works
This resolves 99% of setup-related issues.
Contributing
Build Process
The docs build process automatically handles the following steps via generate:
- Downloads required protoc plugins.
- Generates gRPC documentation from proto files.
- Generates API documentation from OpenAPI specs.
- Generates index files for directory structures.
Style Guide
- Variables: Use environment variables in code snippets where possible.
- Embedded Content: Use
_filename.mdxfor content embedded in other pages (not indexed individually). - Code Embedding: Use the
fileproperty in code blocks to embed code from the repo. - Voice: Use active voice and sentence case for titles.
Refer to the Google Developer Style Guide for general guidelines.
Adding Content
All documentation content is located in the content directory. Note that the system strictly accepts only .mdx files.
To add a new page:
- Create a
.mdxfile in the appropriate subdirectory ofcontent. - Register the new page in the sidebar settings at
lib/sidebar-data.tsto make it accessible in the navigation.
Pull Requests
Use docs(<scope>): <short summary> for PR titles.
Pass quality checks before submitting:
pnpm nx run @zitadel/docs:build