27 Commits

Include renames

Author	SHA1	Message	Date
3 people	3be21ccbf0	docs(perf): v4.17.1 results and harness fixes (#12651) ... Publishes the v4.17.1 benchmark sweep and the harness fixes the sweep uncovered. ## Results All 11 published targets ran for 30 minutes each against a fixed 7-container deployment (vCPU 6 / 6 Gi, psql v17.4, US1 client and server), at 600 VUs except where noted. Latencies are `http_req_duration` over the test phase. | target | VUs | iter/s | p50 | p95 | p99 | failed requests | |:---|---:|---:|---:|---:|---:|:---| | machine_pat_login | 600 | 3452 | 24.53ms | 872ms | 1.00s | 29 / 6,335,633 (0.00%) | | user_info | 600 | 3217 | 29.32ms | 943ms | 1.13s | 66 / 5,800,788 (0.00%) | | introspect | 600 | 2297 | 46.49ms | 1.16s | 1.36s | 1,400 / 4,159,863 (0.03%) | | machine_jwt_profile_grant | 600 | 1076 | 85.34ms | 2.57s | 2.93s | 97 / 1,986,156 (0.00%) | | machine_client_credentials_login | 600 | 834 | 56.78ms | 1.74s | 2.42s | 1,789 / 3,073,497 (0.05%) | | add_session | 600 | 767 | 95.25ms | 3.53s | 4.15s | 1 / 1,416,546 (0.00%) | | oidc_session | 600 | 326 | 77.47ms | 3.41s | 4.43s | 3 / 1,767,162 (0.00%) | | otp_session | 600 | 84 | 47.80ms | 13.83s | 20.22s | 182 / 466,756 (0.03%) | | password_session | 600 | 68 | 408.63ms | 20.00s | 24.05s | 1 / 253,280 (0.00%) | | human_password_login | 600 | 49 | 146.67ms | 4.08s | 1m0s | 9,227 / 683,256 (1.35%) | | manipulate_user | **200** | 19 | 304.58ms | 10.23s | 38.43s | 225 / 171,987 (0.13%) | Against the goals: - **#8352 is met.** `machine_jwt_profile_grant` reached **1076/s** against the 1000/s goal, with p50 at 85ms. - **#4424 is not met.** `human_password_login` runs at **49/s** against a 1200/s goal. The sweep ran with `improvedPerformance` enabled (`PROJECT_GRANT`, `PROJECT`, `USER_GRANT`, `ORG_DOMAIN_VERIFIED`), captured from the instance before the first target rather than assumed; the full flag set is on every page. ## What the observability rows show The containers were never the constraint. ZITADEL CPU sat at 9–50% (p50), memory at 6–7% (p50, peaking at 16%) of 6 GiB, and instance count was flat at 7 with no container startups in any window. Database CPU was at **86–98% median on nine of eleven targets**. Any "ZITADEL achieves N/s" figure that omits which resource ran out is true and misleading, so the pages say. The query breakdown then shows what the CPU numbers cannot: **advisory lock wait is the single largest category of database time across the sweep (91,857 s accumulated)**, ahead of event reads (64,805 s), event push (26,411 s) and projection reads (14,625 s). The three locks are the projection handler lock (`handler/v2/handler.go`), the eventstore read barrier (`repository/sql/query.go`) and the shared lock every push takes (`eventstore/v3/push.go`). That is what sets the `flowchart outcome` on each page, and it is not uniform: - **Scale** on the seven targets where a resource genuinely ran out — database CPU at 92–98% with 0% lock share. - **Potential issues** on `human_password_login`, `password_session`, `otp_session` and `manipulate_user`, where 45–74% of the top query time is lock wait. `password_session` runs at 51% database CPU and 50% container CPU and still only manages 68 iter/s; adding vCPU there buys nothing. Autovacuum is cleared as a contributor: 93.9% median DB CPU in buckets containing a vacuum against 93.1% without, within noise. The new v4.17 eventstore autovacuum is barely triggered — `events2` is not in the top twenty vacuumed tables. ## Harness fixes Defects found during the sweep, all of which corrupted published numbers or aborted runs: 1. **Colliding check names.** `updateHuman`, `lockUser` and `deleteUser` all registered checks as `update user is status ok`, two of them asserting `status === 201` against endpoints returning 200. k6 aggregates checks by name, so this produced a permanent, fictional 28% failure rate. Each check now has its own name and accepts 2xx. 2. **User-creation read-after-write race**, in three variants, each found after the previous fix. The read-back resolved `undefined` when the projection was behind, surfacing much later as `Cannot read property '0' of undefined`; then an unguarded `res.json()` turned every request timeout into a killed iteration; then a *partial* read — a valid user object with `loginNames` not yet projected — passed an `if (user)` guard and failed identically. `readUserAfterCreate()` now guards status and body before parsing, requires the field rather than the object, and retries with backoff, rejecting with the user id and username. 3. **Metric cardinality.** Id-bearing URLs produced 800k+ time series against k6's suggested 100k, driving k6 to 6.8 GB and an OOM mid-run. All 16 such call sites now set a grouped `name` tag. 4. **`password_session` wrote its CSV to `otp_session`'s filename** in the Makefile, silently overwriting the previous target's output. The read-back race is not only a harness defect: at 600 VUs it measures real load-induced projection lag, exceeding 25 s for login-name projection and 100 s for the user projection, against instant reads on an unloaded instance. ## Stable v2 user API Four calls in `benchmark/src/user.ts` still went to `/v2beta/users` — update, lock, delete and the machine read-back. They date from the original load-test bootstrap in April 2024 (#7635), when the user service was still v2beta, and were never moved when it was promoted; they surfaced in this diff only because the read-after-write fix rewrote the surrounding functions. All four are now on stable v2. `updateHuman` is the one that changed shape: `PUT /v2beta/users/{id}` is `UpdateHumanUser`, whose stable-v2 counterpart `PUT /v2/users/human/{id}` is deprecated in the proto, so it now calls `PATCH /v2/users/{id}` (`UpdateUser`) and `manipulate_user` nests the profile under `human`. Both RPCs land on `command.ChangeUserHuman`, so the measured work is unchanged. `lockUser`, `deleteUser` and the `createMachine` read-back move over unchanged in method and payload. No published result is affected: per-endpoint latencies come from custom Trends, and `run-bench.sh` drops the `name` system tag, so no artefact ever recorded a v2beta URL. The remaining `/management/v1` and `/admin/v1` calls are left alone — migrating those changes which API surface is benchmarked and would need a re-run. ## Timestamps Pages reported `Test start` as a bare `HH:MM`, so a run could not be correlated against monitoring the next day. Pages now carry the full UTC date and time and a `Test end` row, both derived from the first and last per-second row of `output.json`. `_template.mdx` is updated to match. ## Post-mortem `benchmark/postmortems/2026-08-27-503-burst.md` records two bursts of HTTP 503 `Service error -27` — an HTML page from the Google frontend, not a ZITADEL error — on `manipulate_user` and `human_password_login`, six hours apart. **The investigation is open**, and the note is explicit about what is established (no container startups; the two bursts have opposite server-side signatures), what is provisionally ruled out (restart, panic), and what is not yet known. It is included because both affected pages report the 503s as a separate population from the ZITADEL failures beside them, and that split needs its evidence somewhere. It also records two defects found by accident in the collected logs, unrelated to the 503s: `sql: transaction has already been committed or rolled back` (232 occurrences under load) and a missing i18n key, `"unable to get user by id" not found in language "en"`. ## Notes for review - **`manipulate_user` ran at 200 VUs, not 600, and the page says so.** Throughput saturates near 100 VUs; at 600 the same target produced 4,283 failed requests for identical throughput against 225 at 200 VUs. Its failure mode is degradation over sustained single-org churn — per-operation p50 climbs 1.7x–6x between the first and last five minutes of a run — not a concurrency threshold. - **Two `Observed errors` rows are hand-corrected** and deliberately differ from the generated text, on `manipulate_user` and `human_password_login`. Both cases are one generated number covering two unrelated failure populations; each page says it was corrected and why. - Org deletion failing with a 500 (`V3-C8l3V`) after heavy user churn is already tracked as #8510 and was hit repeatedly during cleanup. - **The run/publish tooling is no longer part of this PR.** It moved to a private repository, along with the cloud collection scripts, because deployment identity does not belong here. What remains is the published pages, the post-mortem findings and the harness fixes. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5 (1M context) <[email protected]>	2026-09-01 19:12:37 +00:00
5 people	27ce10276b	docs: create knowledge gap ID 65 (#12646) ... Automatically generated PR resolving Knowledge Gap ID 65. **Thread ID:** manual-1787771291264 **Action:** CREATE **New File:** `apps/docs/content/guides/integrate/login-ui/fork-and-deploy-login-app.mdx` ### AI Summary > My recommendation is to publish a a new Deploy & Operate guide for the Login App, then link to it from the existing Login App page. The existing pages address architecture/development, login-client authorization, and Login V2 adoption separately, but none currently explains long-term fork ownership and upgrades. (Contribution guide, login-client setup, adoption guide) The article should cover: Full-repository fork as the simplest current recommendation. The customer’s history-preserving slim-fork approach as an advanced alternative. Why copying only apps/login is unsafe while the current npm packages remain unavailable, linked to #12250. Release-tag pinning and versioning such as v4.17.0-custom.1. Development, generation, linting, tests, and Docker builds. An EKS/Kubernetes Deployment and Service example. PAT handling through ZITADEL_SERVICE_USER_TOKEN_FILE. Same-origin versus separate-domain proxy requirements. Gradual rollout, break-glass access, rollback, and troubleshooting. A repeatable upstream synchronization and release process. ### Human Reviewer Instructions > Sample content: > Maintain and deploy a customized ZITADEL Login UI > > The ZITADEL Login UI is an MIT-licensed Next.js application maintained in the ZITADEL monorepo. You can customize and deploy it independently, but your fork becomes an application that your team must build, test, secure, monitor, and update. > > This guide describes the current workflow for maintaining a customized Login UI. It is intended for teams that need source-code changes beyond the branding and text customization available through ZITADEL settings. > > If you only need to change colors, logos, fonts, or translated login text, first use ZITADEL's supported branding and customization settings. A source fork creates an ongoing maintenance responsibility. > > Understand the source dependencies > > The Login UI is not currently an isolated project in the repository. Its relevant dependency chain is: > > apps/login > └── packages/zitadel-client > └── packages/zitadel-proto > └── proto > > The root pnpm workspace and Nx configuration also participate in installing, generating, testing, and building these projects. Copying only apps/login can therefore fail or silently separate the UI from the API definitions against which it was developed. > > For the current monorepo workflow, keep the Login UI and these workspace dependencies on the same upstream ZITADEL revision. > > Choose a fork strategy > > Full repository fork — recommended > > Fork the complete zitadel/zitadel repository and limit your own changes to apps/login wherever possible. This is the simplest model to understand and the least likely to miss a new build-time dependency when upstream changes. > > Use two remotes: > > git clone [email protected]:YOUR_ORGANIZATION/zitadel.git > cd zitadel > git remote add upstream https://github.com/zitadel/zitadel.git > git fetch upstream --tags > > Start customization from a released tag rather than an arbitrary commit on main: > > git checkout -b custom-login/vX.Y.Z vX.Y.Z > > Replace vX.Y.Z with the ZITADEL release you have selected. A release tag gives you a reproducible compatibility baseline and a corresponding changelog. > > History-preserving slim fork — advanced > > If repository size or internal policy makes a full fork impractical, you can maintain a filtered fork containing only: > > apps/login > > packages/zitadel-client > > packages/zitadel-proto > > proto > > the root pnpm, lockfile, Node, and Nx configuration required by those projects > > Use git filter-repo identically both when bootstrapping the fork and during every later synchronization. The list of retained paths and all filter arguments must remain stable so that freshly filtered upstream history shares commit ancestry with your fork. > > Important constraints for this model: > > Bootstrap and update with one shared filtering implementation; do not duplicate the path list in separate scripts. > > Merge the initial import and subsequent update branches with a merge commit or fast-forward. Do not squash them, because squashing removes the filtered upstream ancestry needed for future three-way merges. > > Treat any change to the retained path list as a new bootstrap. Adding or removing paths rewrites the filtered commit IDs. > > Re-run pnpm install after filtering or merging so the lockfile is reconciled with the retained workspace. > > Review the retained path list whenever upstream changes the Login build graph. > > This is a valid Git maintenance technique, but it adds machinery that ZITADEL's normal build does not require. Own and test the filtering automation as part of your fork. > > Why a standalone apps/login copy is not recommended today > > At the time of writing, current @zitadel/client and @zitadel/proto releases are not available as version-matched npm dependencies for the Login UI. The application consumes their workspace sources from the monorepo. Until version-matched packages are published, retain those sources rather than replacing the workspace dependencies with old npm versions. Progress is tracked in zitadel/zitadel#12250. > > Configure a development environment > > The current contribution guide requires Node.js 22.x, Corepack/pnpm, and Nx. From the repository root: > > corepack enable > pnpm install > pnpm nx run-many --target generate > > To develop the Login UI against an existing ZITADEL instance, create apps/login/.env.dev.local: > > ZITADEL_API_URL=https://your-zitadel-instance.example.com > ZITADEL_SERVICE_USER_TOKEN=your-login-client-pat > > Do not commit this file or its token. ZITADEL_API_URL must not have a trailing slash. > > Start the development server: > > pnpm nx run @zitadel/login:dev > > The token must belong to a machine user with the Instance Login Client role (IAM_LOGIN_CLIENT). See Connect your self-hosted Login UI to ZITADEL for the supported setup. > > Run the required quality checks > > Run the checks for all three affected projects, not only the Next.js application: > > pnpm nx run-many \ > --projects @zitadel/login @zitadel/client @zitadel/proto \ > --targets lint build test > > If your update includes .proto changes, regenerate the TypeScript clients before testing: > > pnpm nx run @zitadel/proto:generate > > For meaningful Login UI changes, also perform end-to-end tests against a staging ZITADEL instance. At minimum, cover the authentication methods and policies you enable in production: > > Authorization Code Flow with PKCE and callback to the application > > password, passkey, external identity provider, and applicable MFA/OTP flows > > registration, account selection, password reset, verification, and logout > > organization-specific policies, branding, and translated text > > expected negative cases such as invalid or expired codes > > desktop and mobile browsers supported by your application > > Build an immutable image > > The repository's documented Docker build flow is: > > pnpm nx run @zitadel/login:build > docker build -t registry.example.com/identity/zitadel-login:vX.Y.Z-custom.1 apps/login > docker push registry.example.com/identity/zitadel-login:vX.Y.Z-custom.1 > > Use an immutable tag that records both the upstream ZITADEL baseline and your customization revision. For example: > > v4.17.0-custom.1 > > Also retain the source commit and image digest in your release record. Do not deploy a mutable latest tag to production. > > Deploy outside Vercel > > Vercel is not required. The Login UI can run as the Node.js application produced by the Next.js build or as a container. In Kubernetes or EKS, run it as a separate Deployment and Service behind your ingress or load balancer. > > The following fragment shows the important application settings. Adapt registry authentication, resources, topology, and ingress to your platform: > > apiVersion: apps/v1 > kind: Deployment > metadata: > name: zitadel-login > spec: > replicas: 2 > selector: > matchLabels: > app: zitadel-login > template: > metadata: > labels: > app: zitadel-login > spec: > containers: > - name: login > image: registry.example.com/identity/zitadel-login:vX.Y.Z-custom.1 > ports: > - name: http > containerPort: 3000 > env: > - name: ZITADEL_API_URL > value: https://zitadel.example.com > - name: NEXT_PUBLIC_BASE_PATH > value: /ui/v2/login > - name: ZITADEL_SERVICE_USER_TOKEN_FILE > value: /var/run/secrets/zitadel/login-client.pat > volumeMounts: > - name: login-client > mountPath: /var/run/secrets/zitadel > readOnly: true > readinessProbe: > tcpSocket: > port: http > livenessProbe: > tcpSocket: > port: http > volumes: > - name: login-client > secret: > secretName: zitadel-login-client > --- > apiVersion: v1 > kind: Service > metadata: > name: zitadel-login > spec: > selector: > app: zitadel-login > ports: > - name: http > port: 3000 > targetPort: http > > Store the PAT in a Kubernetes Secret or external secrets manager and expose it as a read-only file. Restrict access to the Secret and plan token rotation. Do not bake the PAT into the image. > > Add CPU and memory requests and limits based on load tests from your own authentication traffic. Use multiple replicas, disruption controls, and multi-zone scheduling according to your availability requirements. > > Configure routing and domains > > The Login UI contains a server-side component and Next.js middleware that proxies OIDC-related requests to ZITADEL. Correct proxy routing is part of the authentication flow, not only static frontend delivery. > > Same public origin > > When the Login UI and ZITADEL share a public origin, route: > > /ui/v2/login to the Login UI service > > all other ZITADEL paths to the ZITADEL API service > > Set NEXT_PUBLIC_BASE_PATH=/ui/v2/login at build and runtime as required by your packaging. Keep the value consistent with the Login V2 base URI configured in ZITADEL. > > Separate Login domain > > If the Login UI uses a separate public domain such as login.example.com: > > Serve it over HTTPS. > > Add the Login domain to the instance's Trusted Domains. > > Configure the proxy in front of the Login UI as documented by ZITADEL: > > x-zitadel-public-host: the public Login UI host, for example login.example.com > > x-zitadel-instance-host: the ZITADEL instance host, for example auth.example.com > > Test OIDC and SAML callbacks and every configured external identity provider through the public route. > > Do not assume that setting only ZITADEL_API_URL replaces the required host-routing behavior for every topology. The Login App architectural and proxy guidance is the source of truth for these headers. > > Enable Login V2 safely > > For an existing self-hosted ZITADEL installation, follow Adopt Login V2 on an existing installation. > > Prefer a gradual rollout: > > Deploy the custom Login UI without changing the instance-wide setting. > > Enable the new Login UI for a non-production test application and set its custom base URL. > > Run the complete authentication test matrix. > > Expand to selected applications. > > Enable it instance-wide only after validation, if that is your intended topology. > > Keep a break-glass PAT for a machine user with the Instance Owner role (IAM_OWNER) so that you can revert the Login V2 setting if interactive login becomes unavailable. Store and audit this credential according to your incident-access policy. > > Keep the fork synchronized > > Treat upstream synchronization as a normal application dependency update, not an automatic deployment. > > For a full repository fork: > > git fetch upstream --tags > git checkout -b update-login-vX.Y.Z > git merge vX.Y.Z > > Resolve conflicts while preserving the intent of your customizations. Then install, generate, test, and build again. For a slim fork, fetch a freshly filtered copy of the same release and merge its history using the identical filter configuration used during bootstrap. > > Recommended update workflow: > > Select a released ZITADEL version and read its release notes and migration guidance. > > Merge that release into a dedicated update branch. > > Review changes to apps/login, packages/zitadel-client, packages/zitadel-proto, proto, and the root build files. > > Reconcile dependencies and generated code. > > Run lint, build, unit, and staging end-to-end tests. > > Build an immutable custom image and scan it using your normal supply-chain controls. > > Deploy to staging, then canary or gradually roll out to production. > > Retain the previous image digest and configuration for rollback. > > Do not merge upstream continuously into production without qualification. Login changes can affect cookies, redirects, external IdP callbacks, API contracts, and authentication-policy behavior. > > Recommended release policy for your fork > > ZITADEL releases are the upstream input; your custom Login UI image is your own release artifact. A useful version format is: > > <upstream-zitadel-version>-custom.<revision> > > Maintain a short release record containing: > > upstream tag and commit > > custom fork commit > > container image digest > > applicable ZITADEL backend version or tested range > > configuration changes and migrations > > test evidence and known limitations > > Subscribe to ZITADEL releases and security notices. Decide deliberately which releases to adopt; prioritize security fixes and changes that affect Login V2 or the APIs it consumes. > > Troubleshooting checklist > > If the Login UI returns an internal error or an authentication flow redirects incorrectly, verify: > > the PAT is readable and belongs to a user with IAM_LOGIN_CLIENT > > ZITADEL_API_URL is reachable from the Login pod and has no trailing slash > > the public Login domain is trusted by the ZITADEL instance > > the configured Login V2 base URI exactly matches the public route and base path > > the ingress sends Login UI traffic to the correct service > > separate-domain deployments set the public-host and instance-host headers correctly > > the selected Login source, client package, proto package, and ZITADEL API version were tested together > > external IdP callback URLs reach the expected public endpoint > > the prior known-good image and configuration can still be restored > > Related documentation > > Login App architecture and setup > > Contribute to the Login UI > > Connect your self-hosted Login UI to ZITADEL > > Adopt Login V2 on an existing installation > > ZITADEL releases ### ⚠️ Manual Action Required > **Sidebar Update Needed:** Because this PR creates a brand new document, it will not automatically appear in the site's navigation. Please review the file location and manually update `apps/docs/lib/sidebar-data.ts` to include a reference to this new page before merging. --- 🤖 **Need adjustments?** Leave a comment below and tag **@zitadel-knowledge-bot** with your requested changes, and I will automatically update the files and push a new commit! --------- Co-authored-by: zitadel-knowledge-bot[bot] <296006658+zitadel-knowledge-bot[bot]@users.noreply.github.com> Co-authored-by: Matías Racedo <[email protected]> Co-authored-by: Copilot Autofix powered by AI <[email protected]>	2026-08-27 17:13:04 -03:00
3 people	318bfc36d7	feat: native app links for passkeys (#12580) ... # Which Problems Are Solved - Native iOS/Android passkeys need OS trust files on the ZITADEL domain; without them, Associated Domains / App Links verification fails. - Operators could not configure iOS Team ID + Bundle ID or Android package name + SHA-256 fingerprints on OIDC apps. - `/.well-known/apple-app-site-association` and `/.well-known/assetlinks.json` were not served from application config. # How the Problems Are Solved - Add iOS/Android app-link fields on OIDC app create/update (Application API v2 + Management), with validation. - Persist and project those fields; query active app-link configs instance-wide. - Serve AASA (`webcredentials`) and Digital Asset Links (`get_login_creds`) from well-known paths, with configurable `Cache-Control` and fingerprint normalization at serve time. - Console UI to edit the fields, with links to the well-known endpoints. - Operator docs for configuration, endpoints, caching, and verification. # Additional Changes - Document on API fields that well-known responses may be HTTP-cached and platform verifiers may delay propagation. - Runtime config: `WellKnown.AppLinksCacheControlMaxAge` (default `5m`; `0` → `no-store`). # Additional Context - Closes #12497 - Implemented and reviewed as stack: - #12531 API contract - #12532 storage wiring - #12536 well-known endpoints - #12537 console - #12547 docs --------- Co-authored-by: Cursor <[email protected]>	2026-08-12 08:04:27 +02:00
3 people	2b9b9856b4	docs: remove changelog link from sidebar (#12459) ... The changelog page on zitadel.com was removed for being unmaintained, leaving the docs sidebar link 404ing. Co-authored-by: Rajat Singh <[email protected]>	2026-07-17 14:26:01 +02:00
3 people	94ac4237a9	docs: add user self-deletion guide (#12458) ... ## Summary - Add a new guide documenting how to let users delete their own account via the User v2 API's `DeleteUser` endpoint, gated by the `user.self.delete` permission (granted via `ORG_USER_SELF_MANAGER`) - Link the new guide from the self-service concepts page and sidebar - Add the `ORG_USER_SELF_MANAGER` role to the administrators reference table ## Test plan - [x] Verified the new and updated pages render correctly in local dev (`pnpm dev`) replaces #12455 and #12457 (branch got renamed from `docs/human-pat-clarification`, which auto-closed #12457). Co-authored-by: Rajat Singh <[email protected]>	2026-07-16 16:35:44 +00:00
5 people	75bc058bce	feat(eventstore): autovacuum tuning for events2 table (#12449) ... # Which Problems Are Solved In Zitadel's append-only event-sourced architecture, the `eventstore.events2` table grows indefinitely. PostgreSQL's default autovacuum uses a percentage-based scale factor, so as the table grows, the number of changed rows required to trigger a `VACUUM` or `ANALYZE` drifts towards infinity. Without regular vacuums, the table's Visibility Map becomes stale, disabling fast Index-Only Scans and forcing expensive heap reads. Without regular analyzes, query planner statistics become stale, leading to suboptimal execution plans. This causes eventstore operations to progressively degrade as `events2` grows, even without CPU, memory, or I/O saturation. A manual `VACUUM ANALYZE` immediately restores performance, confirming the root cause. - If `Eventstore.Autovacuum` is left at its default, `events2` keeps using PostgreSQL's default, percentage-based autovacuum/autoanalyze scale factors, which become impractically infrequent on large tables. - There was previously no supported way to apply static, table-level autovacuum tuning to `events2` through Zitadel's own configuration/setup process. # How the Problems Are Solved - Added an `Eventstore.Autovacuum` runtime configuration block to `cmd/defaults.yaml` (disabled by default): ```yaml Eventstore: Autovacuum: Enabled: false # ZITADEL_EVENTSTORE_AUTOVACUUM_ENABLED VacuumThreshold: 50000 # ZITADEL_EVENTSTORE_AUTOVACUUM_VACUUMTHRESHOLD AnalyzeThreshold: 50000 # ZITADEL_EVENTSTORE_AUTOVACUUM_ANALYZETHRESHOLD ``` - Added a repeatable `zitadel setup` migration step (`cmd/setup/eventstore_autovacuum.go`) that: - When `Enabled: true`, disables the percentage-based `autovacuum_vacuum_scale_factor`, `autovacuum_analyze_scale_factor`, and `autovacuum_vacuum_insert_scale_factor` on `eventstore.events2`, and applies static thresholds (`autovacuum_vacuum_insert_threshold`, `autovacuum_vacuum_threshold`, `autovacuum_analyze_threshold`) from the config instead. - When `Enabled: false`, resets those storage parameters on `eventstore.events2` back to the cluster defaults. - Implements `Repeatable.Check()` so the step only re-runs when the configuration actually changed since the last `zitadel setup` run. - Added documentation in the new "Performance tuning" page. # Additional Changes - Move the projection documentation into performance tuning page - Expand and update the projection documentation to the latest state in zitadel. (contained some stale information) # Additional Context Several other issues describe read-performance symptoms consistent with this same root cause (stale `events2` visibility map / planner statistics at scale, without resource saturation). Since this PR addresses the shared root cause: - Closes #12448 - Closes #10754 - Closes #10260 - Closes #8585 - Closes #9239 --- _Generated by [Claude Code](https://claude.ai/code/session_01HoKvEY7niCVgLCz7CajBwW)_ --------- Co-authored-by: Claude <[email protected]> Co-authored-by: Copilot Autofix powered by AI <[email protected]> Co-authored-by: Silvan <[email protected]>	2026-07-16 14:42:11 +00:00
3 people	8395d4326e	docs: add v3 to v4 upgrade and Login V2 adoption guides (#12443) ... <!-- Please inform yourself about the contribution guidelines on submitting a PR here: https://github.com/zitadel/zitadel/blob/main/CONTRIBUTING.md#submit-a-pull-request-pr. Take note of how PR/commit titles should be written and replace the template texts in the sections below. Don't remove any of the sections. It is important that the commit history clearly shows what is changed and why. Important: By submitting a contribution you agree to the terms from our Licensing Policy as described here: https://github.com/zitadel/zitadel/blob/main/LICENSING.md#community-contributions. --> # Which Problems Are Solved - Operators upgrading from ZITADEL v3 to v4 lacked a clear, docs-backed upgrade path covering web keys (A-10017), Login V1 vs Login V2 expectations, and post-upgrade options. - There was no standalone guide for adopting Login V2 after already running on v4. - Related ops/docs pages did not consistently point readers to the upgrade and Login V2 adoption material. - Kubernetes/Helm operators upgrading to a chart that ships ZITADEL v4 had no ops-page section linking the advisory, upgrade guide, optional Login V2 deferral (`login.enabled: false`), and chart README upgrade notes. # How the Problems Are Solved - Adds technical advisory **A-10017** documenting OIDC web key staging requirements before upgrading to v4. - Adds an **upgrade-v3-to-v4** guide with the recommended upgrade path (Login V1 remains supported; Login V2 is optional at upgrade time). - Adds a **v4-only adopt-login-v2** guide for teams adopting Login V2 later, with steps and caveats separate from the version upgrade. - Adds an **Upgrading to ZITADEL v4** subsection under `self-hosting/deploy/kubernetes/operations.mdx` that links to A-10017, the upgrade guide, Adopt Login V2, notes `login.enabled: false` when not adopting Login V2 yet, and points to the [Helm chart README](https://github.com/zitadel/zitadel-charts/blob/main/charts/zitadel/README.md) for chart-specific breaking changes. # Additional Changes - Registers new docs in the sidebar and wires Related / cross-links from upgrade and ops pages so the advisory and guides are discoverable together. - Aligns language so upgrade vs. Login V2 adoption are clearly separated concerns. - Light cross-link updates on troubleshooting / updating-scaling / login-client pages where related. # Additional Context - Related: A-10017 (web keys advisory) - Docs paths introduced/updated: - `apps/docs/content/support/advisory/a10017.mdx` - `apps/docs/content/self-hosting/manage/upgrade-v3-to-v4.mdx` - `apps/docs/content/self-hosting/manage/adopt-login-v2.mdx` - `apps/docs/content/self-hosting/deploy/kubernetes/operations.mdx` (new "Upgrading to ZITADEL v4" section) - Sidebar and Related / cross-links - Companion Helm chart README PR: https://github.com/zitadel/zitadel-charts/pull/608 --------- Co-authored-by: Cursor <[email protected]>	2026-07-15 15:14:18 +00:00
4 people	d15b4e2da2	docs: create knowledge gap ID 34 (#12388) ... Automatically generated PR resolving Knowledge Gap ID 34. **Thread ID:** manual-1783339162398 **Action:** CREATE **New File:** `content/docs/drafts/gap-manual-1783339162398.mdx` ### AI Summary > Add a user migration guide for firebase to Zitadel ### Human Reviewer Instructions > Firebase to Zitadel Migration Summary > The Blocker: Incompatible Password Hashes > > Firebase uses a proprietary, modified scrypt algorithm requiring project-specific keys. > > Zitadel supports standard algorithms but lacks a verifier for Firebase's custom format. > > Result: Passwords cannot be directly imported. Firebase hashes must be discarded. > > Strategy 1: Bulk Import & Password Reset (Standard) > Requires users to set a new password on their first login. > > Export: Run firebase auth:export users.json --format=json. > > Map: Convert Firebase fields (e.g., localId) to Zitadel's schema. > > Import to Zitadel: Call the Zitadel import API. > > Action: Omit the hashedPassword object entirely. > > Action: Include "passwordChangeRequired": true in the JSON payload to trigger a reset flow via email or login prompt. > > Strategy 2: Just-In-Time (JIT) Migration (Seamless) > Migrates users transparently behind the scenes during an active grace period. > > Intercept Login: Your backend captures the plain-text password during login. > > Verify: Backend POSTs credentials to Firebase Auth REST API: > > https://identitytoolkit.googleapis.com/v1/accounts:signInWithPassword?key=[WEB_API_KEY] > > Migrate to Zitadel: > > If Valid (200 OK): Create the user in Zitadel immediately using the plain-text password (Zitadel will natively hash it). > > If Invalid (400): Reject login or check if the user is already in Zitadel. > > Sunset: After the grace period ends, migrate remaining inactive users using Strategy 1. > (Note: Do not log plain-text passwords and strictly enforce HTTPS during this phase). --- 🤖 **Need adjustments?** Leave a comment below and tag **@zitadel-knowledge-bot** with your requested changes, and I will automatically update the files and push a new commit! --------- Co-authored-by: zitadel-knowledge-bot[bot] <296006658+zitadel-knowledge-bot[bot]@users.noreply.github.com> Co-authored-by: Copilot Autofix powered by AI <[email protected]>	2026-07-07 12:43:28 +00:00
5 people	6b629c2290	docs: update roadmap and remove release cycle page (#12373) ... Removes the release cycle page and sidebar entry. Replaces the roadmap page with updated strategic roadmap content and renames its sidebar label to Roadmap. --------- Co-authored-by: Elina Sokolovska <[email protected]> Co-authored-by: Claude Sonnet 4.6 <[email protected]> Co-authored-by: Fabienne Bühler <[email protected]>	2026-07-03 21:06:22 +07:00
Federico CoppedeandGitHub	1b8b0ac410	docs: move customer portal docs to knowledge base (#12147) ... Moved the Customer Portal Documentation to the knowledge base (help.zitadel.com)	2026-05-10 21:42:08 -03:00
Florian ForsterandGitHub	24d0f34f39	perf: optimize docs static generation cost (#12108) ... ## Summary - Switch the docs catch-all route to full static generation and prebuild both latest and versioned docs paths. - Prebuild OG images for all docs pages, make sitemap and LLM export static, and remove nondeterministic sitemap timestamps. - Reduce build-time overhead by memoizing docs sidebar trees and skipping processed markdown generation for versioned docs. ## Testing - `pnpm nx run @zitadel/docs:build` - `pnpm nx run @zitadel/docs:lint` - `pnpm nx run @zitadel/docs:check-types` - Verified the prerender manifest contains 8,816 prerendered routes, 3,293 versioned docs routes, 4,406 OG routes, and zero revalidating docs routes.	2026-04-27 15:34:00 +00:00
Fabienne BühlerandGitHub	a592fd7293	docs: update naming on authorizations requests to role assignments (#11850) ... # Which Problems Are Solved API Requests in the docs, still had old naming # How the Problems Are Solved Updated Authorization Reuqest naming in docs with role assignments	2026-03-19 09:45:21 +01:00
Federico CoppedeandGitHub	80a20e9680	docs: added link to sidebar title for benchmarks (#11870) ... This PR adds a link to the benchmarks title on the sidebar	2026-03-18 15:29:53 +00:00
3 people	3fdac54878	docs: simplify helm chart guide (#11781) ... ## Summary Rewrites the Kubernetes quickstart (Stage 1) to match the simplicity of the Docker Compose quickstart. With the new bundled chart, users deploy the full stack — PostgreSQL, Traefik, ZITADEL API, ZITADEL Login — with two copy-paste blocks and no prerequisites beyond a Kubernetes cluster. ## Before / After **Before:** - Requires a pre-installed ingress controller - 4 manual steps: install PostgreSQL separately, create secrets, wire DB config, install ZITADEL - Users need to understand Helm dependency management **After:** ```bash mkdir zitadel-helm && cd zitadel-helm && curl -fsSLO https://raw.githubusercontent.com/zitadel/zitadel-charts/main/examples/0-quickstart/quickstart-values.yaml ``` ```bash helm repo add zitadel https://charts.zitadel.com && helm repo add bitnami https://charts.bitnami.com/bitnami && helm repo add traefik https://traefik.github.io/charts && helm repo update && helm upgrade --install zitadel zitadel/zitadel --values quickstart-values.yaml --wait ``` Same UX as the Docker Compose quickstart. Mirrors the GitLab chart pattern. ## Changes to `kubernetes/index.mdx` ### Stage 1 — Quickstart (full rewrite) - Prerequisites reduced to: a Kubernetes cluster (1.30+), kubectl, Helm - k3d moved to an optional callout tip (no longer required) - Install section: `mkdir` → `curl` → `helm upgrade --install --wait` (two copy-paste blocks, same pattern as compose) - `helm upgrade --install` instead of `helm install` — idempotent, safe to re-run - "Swap out components" table: shows how to replace PostgreSQL, Traefik, or add Redis - Callouts: masterkey warning, stack architecture diagram ### Stage 2+ (unchanged) Production cluster setup, TLS, cert-manager, etc. are untouched. ## Related - Chart changes PR: [zitadel/zitadel-charts#560](https://github.com/zitadel/zitadel-charts/pull/560) --------- Co-authored-by: Copilot <[email protected]>	2026-03-17 10:36:23 +00:00
Fabienne BühlerandGitHub	9a4530bf77	docs: Add note for updated terminology (#11802) ... # Which Problems Are Solved For existing customers it could be hard to understand the new aligned terms we have defined, if they are already used to the "old term" # How the Problems Are Solved Adding notes at the top of docs pages to state clearliy new and old terms.	2026-03-16 09:10:48 +00:00
3 people	574e23e834	fix: Separate Example Applications and SDKs/Integrations more clearly (#11569) ... # Which Problems Are Solved The distinction between Example Applications and SDKs has not been clear. In the SDKs section a lot of Example applications where listed. Also In the SDKs it was not really clear what the SDKs actually are. Also some examples and skds are outdated and not well maintained anymore. # How the Problems Are Solved - Moved all example app guides to the Example Applications Section - Restructured SDKs Nav --> SDKs & Integrations: Frontend & Mobile, Backend & API, Management API Clients - Removed unnecessary / outdated guides - Remove ids in framework.json for frameworks which shouldn't be rendered in console ## Todos: - Update SDKs Introduction page - Add links to common oidc libs for most used frameworks --------- Co-authored-by: Mridang Agarwalla <[email protected]>	2026-03-11 14:57:15 +01:00
Federico CoppedeandGitHub	7db9af486f	docs: add guest user auth use case (#11715) ... this PR adds a guest user auth doc to the use cases sidebar section	2026-02-27 15:57:09 +00:00
3 people	bac224c56d	chore: improve docker compose template, tests and docs (#11593) ... Replaces the single-file `docker-compose.yaml` quickstart with a production-aware, Traefik-based compose pack in `deploy/compose/`. The pack covers the full arc from a 2-minute localhost quickstart to a hardened homelab or semi-production deployment. ### What's in the pack **Stack**: Traefik (proxy) → ZITADEL API (Go `:8080`) + ZITADEL Login (Next.js `:3000`) → PostgreSQL All HTTP/gRPC routing is handled by Traefik via Docker labels — no manual proxy config needed. The Login V2 UI is enabled by default. Login URLs are derived automatically from `ZITADEL_DOMAIN`, `ZITADEL_EXTERNALPORT`, and `ZITADEL_PUBLIC_SCHEME` — no separate URL variables needed. **Compose files** | File | Purpose | |------|---------| | `docker-compose.yml` | Base stack — works standalone. Uses explicit `name: zitadel` network for reliable Traefik service discovery. | | `docker-compose.mode-letsencrypt.yml` | TLS overlay: ACME HTTP challenge | | `docker-compose.mode-external-tls.yml` | TLS overlay: upstream LB/CDN terminates TLS. Uses `forwardedHeaders.trustedIPs` (configurable via `TRAEFIK_TRUSTED_IPS`) instead of `insecure=true`. | | `docker-compose.mode-local-tls.yml` | TLS overlay: self-signed certs for LAN | | `docker-compose.prodlike.yml` | Splits init / setup / start for controlled upgrades | | `docker-compose.test.yml` | CI overlay: swaps images to locally-built `:local` tags | **Optional profiles**: `cache` (Redis), `observability` (OpenTelemetry Collector) ### Build infra - New `@zitadel/api:pack` and `@zitadel/login:pack` Nx targets build local Docker images (`zitadel/zitadel:local`, `zitadel/zitadel-login:local`) for use in CI and local testing - `apps/api/Dockerfile` now accepts a `BINARY` build arg so local and release builds share the same image ### Testing - New `@zitadel/compose` Nx project with targets: `test-config` (validates all overlay combinations using `--quiet`), `test-run` (starts full stack with local images), `test-e2e` (Playwright wiring + protocol matrix tests through Traefik), `test-full` (end-to-end: build → start → test → teardown), `stop` - **`@zitadel/compose` is explicitly excluded from `nx affected` in CI for now** — the full stack smoke test requires a Docker daemon and significant resources. The intent is to add a dedicated `compose_smoke_test` CI job in a follow-up. The targets can be run locally with `pnpm nx run @zitadel/compose:test-full`. ### Documentation - **`compose.mdx`**: Complete rewrite with a staged structure (Stage 1 Quickstart → Stage 2 Homelab → Stage 3 Beyond Compose). Documents TLS modes, profiles, secrets hardening, ExternalDomain/Port/Secure invariant, upgrades, and the path to Kubernetes - **New `requirements.mdx`**: Lists supported PostgreSQL versions (14–18), Redis (standalone), Docker Compose v2.x, and reverse proxy h2c requirements - **`reverse_proxy.mdx`**: Added intro covering h2c requirements, TLS modes table, and Login UI routing split - **`troubleshooting.mdx`**: New sections for container restarts on upgrade, FIRSTINSTANCE env vars not taking effect, and diagnosing unhealthy containers - **`caddy/index.mdx`**: Known issue and workaround for the `TE: trailers` header hang - Removed the old `apps/docs/content/self-hosting/deploy/docker-compose.yaml` embedded in the docs ### Breaking change The old `apps/docs/content/self-hosting/deploy/docker-compose.yaml` file is deleted. The getting-started docs page (`/self-hosting/deploy/compose`) now points to the new pack via a `curl | tar` download command. --- ### Checklist - [x] `deploy/compose/` smoke test passes end-to-end locally (`pnpm nx run @zitadel/compose:test-full`) - [x] Docs build passes (`pnpm nx run @zitadel/docs:build`) - [ ] Follow-up issue created to add `compose_smoke_test` CI job --------- Co-authored-by: Mridang Agarwalla <[email protected]>	2026-02-24 16:33:12 -08:00
4 people	6ce78152e9	docs: add cloud production checklist (#11671) ... # Which Problems Are Solved We do have a production checklist for self-hosters which lacks some important things for our cloud customers. # How the Problems Are Solved Adding a production checklist specifically for cloud users --------- Co-authored-by: Copilot <[email protected]> Co-authored-by: Florian Forster <[email protected]>	2026-02-23 23:25:24 +00:00
Mridang AgarwallaandGitHub	6150a61216	docs: added docs for the new dotnet example (#11541)	2026-02-23 07:55:18 +00:00
Florian ForsterandGitHub	2c260a1f10	docs: add description frontmatter to content pages for SEO (#11654)	2026-02-21 07:44:42 -08:00
3 people	943d9e0967	docs: fix broken docs links and insecure redirects (#11591) ... Summary - audited doc CSVs to replace insecure http links and ensure paths point to the migrated `apps/docs` structure - preserved downstream link intent while keeping references aligned with the `/docs` deployment prefix --------- Co-authored-by: Copilot <[email protected]>	2026-02-13 11:01:11 -08:00
3 people	cc7a6145f1	chore: rename manager/member to administrator (#11542) ... # Which Problems Are Solved As part of the naming consistency initiative, the terms `Manager`, `Member` and other similar terms that are used in the context of an administrator are replaced by `Administrator` in the UI and documentation. # How the Problems Are Solved By replacing the following terms with Administrator / Administrator Roles in docs, Console UI, and guides. - Manager - Add Manager - Add a Manager - Administrator (used as a person type) - Membership - Member grants # Additional Changes minor grammatical edits in the docs # Additional Context - Closes https://github.com/zitadel/zitadel/issues/11291 --------- Co-authored-by: Marco A. <[email protected]>	2026-02-11 17:43:06 +01:00
Mridang AgarwallaandGitHub	e545d7c5b1	docs: added docs for confguiring caching when provisioning zitadel on k8 (#11492)	2026-02-11 07:46:16 +01:00
3 people	1f0d54a978	chore(docs): User (Human) naming consistency (#11512) ... # Which Problems Are Solved Naming inconsistencies - User (Human) # How the Problems Are Solved Most of the occurrences have not been changed. `User (Human)` was mostly changed when talking about code objects and where I felt it was necessary to distinguish them from machine users. When both human and machine user occurrences were found, the machine user has been changed to service account (see https://github.com/zitadel/zitadel/issues/11285) # Additional Context - Closes #11284 --------- Co-authored-by: Wim Van Laer <[email protected]>	2026-02-06 09:46:24 +00:00
Florian ForsterandGitHub	413f8ba585	docs: add missing k8s example (#11474) ... This pull request significantly expands and restructures the Kubernetes deployment documentation for Zitadel. It introduces detailed, task-focused guides for installation, configuration, ingress, and database setup, replacing the previous minimal documentation. The new docs provide comprehensive instructions and best practices for deploying Zitadel on Kubernetes using Helm, including security, scaling, and high availability considerations. The most important changes are: **Documentation Restructuring and Overview** - Added a new `index.mdx` providing an overview of the Zitadel Helm chart, architecture, prerequisites, and quick links to relevant resources and next steps. - Removed the old minimal Kubernetes deployment page and replaced it with a structured, multi-page documentation set. **Installation and Setup** - Added an `installation.mdx` guide with step-by-step instructions for preparing prerequisites, creating Kubernetes secrets, configuring `values.yaml`, installing the Helm chart, and verifying the deployment. **Configuration and Best Practices** - Added a `configuration.mdx` guide covering Helm chart configuration options, including replica count, container images, security contexts, external domain setup, secrets management, scaling, autoscaling, pod disruption budgets, and anti-affinity rules. **Ingress and Database Configuration** - Added an `ingress.mdx` guide detailing how to configure Kubernetes ingress resources for Zitadel and Login containers, including TLS termination and certificate management. - Added a `database.mdx` guide explaining how to connect Zitadel to PostgreSQL with various TLS/security options and how to manage database credentials and certificates using Kubernetes secrets.	2026-01-30 08:27:01 +01:00
3 people	d8ea041c56	docs: introduce versioned docs and migrate to fuma (#11166) ... ## Todos for release - [x] Configure Env in docs project on vercel - [x] Configure Root Path in the docs project on vercel - [ ] Remove old CSP https://github.com/zitadel/website/pull/1592 ## What we did This pull request migrates the project documentation from the old `docs/` directory to the new `apps/docs/` directory, introduces a new documentation system built with Next.js and Fumadocs, and updates all relevant references, configuration files, and documentation to reflect this change. It also adds new configuration and ignore files for the new documentation app, updates CI and linting to exclude the new docs from certain checks, and revises the contributing guidelines accordingly. **Documentation System Migration and New Docs App** * Migrated all documentation from `docs/` to `apps/docs/`, and updated all references in `README.md`, `CONTRIBUTING.md`, and other files to point to the new location. [[1]](diffhunk://#diff-eca12c0a30e25b4b46522ebf89465a03ba72a03f540796c979137931d8f92055L585-L640) [[2]](diffhunk://#diff-eca12c0a30e25b4b46522ebf89465a03ba72a03f540796c979137931d8f92055L660-R607) [[3]](diffhunk://#diff-eca12c0a30e25b4b46522ebf89465a03ba72a03f540796c979137931d8f92055L740-R687) [[4]](diffhunk://#diff-b335630551682c19a781afebcf4d07bf978fb1f8ac04c6bf87428ed5106870f5L2-R3) [[5]](diffhunk://#diff-b335630551682c19a781afebcf4d07bf978fb1f8ac04c6bf87428ed5106870f5L30-R30) * Added a new Next.js/Fumadocs-based documentation app under `apps/docs/`, including core app files, layouts, routing, search API, and a comprehensive `README.md` with development and contribution instructions. [[1]](diffhunk://#diff-5a1b07344a2c1b4d3f37b23ff1388b62cd9f57dea3c1cd23d8a0412b7602b132R1-R74) [[2]](diffhunk://#diff-462b9ad1eabbb7d1c29bb9c36e4931eb180fd7190900e6d2babf8f4d66ad1c28R1-R39) [[3]](diffhunk://#diff-e16ae25660ded787b10ac35dea96d5ecaacf895dae0afc8a9bd4382dc79a8c87R1-R7) [apps/docs/app/[[...slug]]/layout.tsxR1-R81](diffhunk://#diff-59e08acde4e805b7aeccef1dcf98f1d71dfc550777e6b9402085cee0e9fa4e0aR1-R81), [apps/docs/app/[[...slug]]/page.tsxR1-R79](diffhunk://#diff-e5df3f80d0fa01e12d63d81f29c57a9d14e846c78fd3beabb2ef768e38fd9580R1-R79), [[4]](diffhunk://#diff-389b34918e040cacaa87cd7201ffa462cc2b0b716736f537e3d3c660ac69353bR1-R7) [[5]](diffhunk://#diff-d3b03416d1c457b19f1c27b26f2db741412df841b875c26ccadc36e4522247f4R1-R29) [[6]](diffhunk://#diff-c8fb8339570a5305809be7c618e14705fd86390dc278e6ce8ba224a7bc8b0c3cR1-R25) **Configuration and Tooling Updates** * Updated `.github/workflows/codeql.yml`, `.golangci.yaml`, and `.github/dependabot.yml` to properly handle the new docs app: excluded `apps/docs` from certain checks, added npm dependency updates for the docs app, and excluded generated content. [[1]](diffhunk://#diff-12783128521e452af0cfac94b99b8d250413c516ec71fe6d97dbea666ff7ba27L8-R14) [[2]](diffhunk://#diff-9917ddc9f1c3304218f7269265b746d997c5c0615478177b5fceecd33ef47cb5R5-R6) [[3]](diffhunk://#diff-9917ddc9f1c3304218f7269265b746d997c5c0615478177b5fceecd33ef47cb5R126-R129) [[4]](diffhunk://#diff-dd4fbda47e51f1e35defb9275a9cd9c212ecde0b870cba89ddaaae65c5f3cd28R89-R106) * Updated `.devcontainer/devcontainer.json` to use the latest Go 1.25.3 version for consistency. **Licensing and Miscellaneous** * Added `apps/docs/` to the list of licensed directories in `LICENSING.md`. These changes ensure the documentation is now maintained in a modern, scalable system and all project tooling is updated to support the new structure. --------- Co-authored-by: Federico Coppede <[email protected]>	2026-01-29 08:38:36 -08:00