mirror of
https://github.com/andwati/zitadel.git
synced 2026-10-18 05:52:51 +00:00
4bc484dac9
# Which Problems Are Solved - A projection that subscribes to several event types gets slower the further it falls behind. - The handler reads its events with one query: `aggregate_type = ANY(...) AND event_type = ANY(...)`, ordered by position and limited to the bulk limit. - The `es_projection` index returns each (aggregate type, event type) range in position order, but Postgres cannot merge those ranges. - So each batch reads every matching event after the projection's position, sorts them all, and keeps only the bulk limit. - A projection that is N events behind therefore reads about N rows per batch. Catching up costs about N² in total, and once events arrive faster than such batches can process them, the projection does not recover. - Every query that triggers the projection waits for that catch-up. Examples are `GetSession` and the notification handlers. # How the Problems Are Solved - **New opt-in, `SearchQueryBuilder.ScanEventTypesSeparately()`.** It applies to event queries for a single instance, ascending and limited, whose sub-queries filter only by aggregate and event types. - For such a query, the SQL layer reads each distinct (aggregate type, event type) combination in its own `ORDER BY … LIMIT` subquery, using `UNNEST … CROSS JOIN LATERAL`. - It then merges the results with the same sort key and limit. - Each subquery is one `es_projection` range read in position order, so it stops at the limit. A batch reads at most `limit × combinations` rows, however far behind the projection is. - Every other query renders exactly as before. - **The projection handler** now passes its real combinations, one sub-query per aggregate type instead of the cross product of all types, and sets the opt-in. - **Results are identical to the single query.** - The sort key (`position, in_tx_order, instance_id, aggregate_type, aggregate_id, sequence`) is a total order, and the global top n is contained in the union of each combination's top n. - The cursor from #12703 and the open-transaction cap apply inside every subquery. - A test against a real Postgres compares both shapes from every cursor and every position, for several limits. The test events include positions shared by several transactions, whose `in_tx_order` restarts. # Additional Changes This is a separate commit and can be split into its own PR: - `cmd/defaults.yaml` sets `BulkLimit: 1000` for the `sessions` projection. - `projection_state_latency` gets histogram buckets up to one week. Until now, every projection more than 30 minutes behind was reported in the same top bucket. # Additional Context These numbers come from a local, CPU-pinned sandbox, for comparing builds only. - **Database:** Postgres 17 in a VM with 8 vCPUs and 16 GiB. The `Eventstore.Autovacuum` and `Projections` settings mirror a production-like setup. - **Starting state:** `sessions8` 4.8M events behind, restored byte-identically for every run. Batch query for `sessions8`, `EXPLAIN ANALYZE`, median of 5 runs, parallel workers at the default: | | before | after | | --- | ---: | ---: | | at the backlog, limit 1000 | 832 ms | 4.0 ms | | at the backlog, limit 200 | 832 ms | 1.0 ms | | at the head (idle poll) | 0.14 ms | 0.19 ms | | generic plan (`plan_cache_mode=force_generic_plan`), at the backlog | 1,042 ms | 4.0 ms | Draining `sessions8` from that backlog for 600 s, 3 runs per build (median, with the range in brackets): | | before | after | | --- | ---: | ---: | | events/s | 975 (932-1,032) | 7,811 (7,523-7,955) | | mean batch time, including writes | 1.03 s | 0.13 s | | still behind after 600 s | 4.2M | 17k-277k | A new index `(instance_id, position, in_tx_order, aggregate_type, event_type)` with the old query was evaluated as an alternative and rejected. Under a generic plan, Postgres chose it for projections whose events are rare and walked the whole event stream: 430 ms instead of 0.15 ms. This is the behaviour that led to dropping `es_instance_position` in #10626. - Follow-up for #12703 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5 <[email protected]> Co-authored-by: Copilot Autofix powered by AI <[email protected]>
The Identity Infrastructure for Developers
ZITADEL is an open-source identity and access management platform built for teams that need more than basic auth. Whether you're securing a SaaS product, building a B2B platform, or self-hosting a production IAM stack — ZITADEL gives you everything out of the box: SSO, MFA, Passkeys, OIDC, SAML, SCIM, and a battle-tested multi-tenancy model.
No vendor lock-in. No compromise on control. Just a robust, API-first identity platform you can own.
🏡 Website | 💬 Chat | 📋 Docs | 🧑💻 Blog | 📞 Contact
Why ZITADEL
We built ZITADEL to handle the hardest IAM challenges at scale — starting with multi-tenancy.
| ZITADEL | FusionAuth | Keycloak | Auth0/Okta | |
|---|---|---|---|---|
| Open-source | ✅ | ❌ | ✅ | ❌ |
| Self-hostable | ✅ | ✅ | ✅ | ❌ |
| Infrastructure-level tenants | ✅ Instances (High scale) | ✅ Tenants | 🟡 Realms (Scaling limits) | ❌ (Multi-tenant = multi-account) |
| B2B Organizations | ✅ Native & Unlimited | 🟡 via Entity Management | ✅ (Recent addition) | 🟡 (Plan/Account dependent) |
| Full audit trail | ✅ Comprehensive Event Stream* | 🟡 Audit logs | 🟡 Audit logs | 🟡 Audit logs |
| Passkeys (FIDO2) | ✅ | ✅ | ✅ | ✅ |
| Actions / webhooks | ✅ | ✅ | 🟡 via SPI | ✅ |
| API-first (gRPC + REST) | ✅ | 🟡 REST only | 🟡 REST only | 🟡 REST only |
| SaaS + self-host parity | ✅ | ✅ | ➖ N/A | ➖ N/A |
ZITADEL Cloud and self-hosted ZITADEL run the same codebase.
Key differentiators for architects:
- Relational core, event-driven soul — every mutation is written as an immutable event for a complete, API-accessible audit trail. Unlike systems that log only select activities, ZITADEL provides a comprehensive event stream that can be audited or streamed to external systems via Webhooks.
- Strict multi-tenant hierarchy — Identity System → Organizations → Projects, with isolated data and policy scoping at multiple levels
- API-first design — every resource and action is available via connectRPC, gRPC, and HTTP/JSON APIs
- Zero-downtime updates and horizontal scalability without external session stores
Get Started in 3 Minutes
ZITADEL Self-Hosted
# Docker Compose — up and running in under 3 minutes
curl -LO https://raw.githubusercontent.com/zitadel/zitadel/main/deploy/compose/docker-compose.yml \
&& curl -LO https://raw.githubusercontent.com/zitadel/zitadel/main/deploy/compose/.env.example \
&& cp .env.example .env \
&& docker compose up -d --wait
Full deployment guides:
Need professional support for your self-hosted deployment? Contact us.
ZITADEL Cloud (SaaS)
Start for free at zitadel.com — no credit card required. Available in US · EU · AU · CH. Pay-as-you-go pricing.
Integrate with the V2 API
ZITADEL exposes every capability over a typed API. Here's how to create a user with the V2 REST API:
curl -X POST https://$ZITADEL_DOMAIN/v2/users/human \
-H "Authorization: Bearer $ACCESS_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"username": "[email protected]",
"profile": { "givenName": "Alice", "familyName": "Smith" },
"email": { "email": "[email protected]", "sendCode": {} }
}'
Explore the full API reference — including connectRPC and gRPC transports — or jump straight to quickstart examples.
Features
Authentication
- Single Sign On (SSO) · Username/Password · Passkeys (FIDO2 / WebAuthn)
- MFA: OTP, U2F, OTP Email, OTP SMS
- LDAP · Enterprise IdPs and social logins
- OpenID Connect certified · SAML 2.0 · Device authorization
- Machine-to-machine: JWT Profile, PAT, Client Credentials
- Token exchange and impersonation
- Custom sessions for flows beyond OIDC/SAML
- Hosted Login V2
Multi-Tenancy
- Identity brokering with pre-built IdP templates
- Customizable B2B onboarding with self-service for customers
- Delegated role management to third parties
- Domain discovery
Integration
- gRPC, connectRPC, and REST APIs for every resource
- Actions: webhooks, custom code, token enrichment
- RBAC · SCIM 2.0 Server
- Audit log and SOC/SIEM integration
- SDKs and example apps
Self-Service & Admin
- Self-registration with email/phone verification
- Administration Console for orgs and projects
- Custom branding per organization
Deployment
- PostgreSQL (≥ 14) · Zero-downtime updates · High scalability
Track upcoming features on our roadmap and follow our changelog for recent updates.
Showcase
Login V2
Our new, fully customizable login experience — documentation
Adopters & Ecosystem
Used in production by organizations worldwide. See the full Adopters list — and add yours by submitting a pull request.
- SDKs: All supported languages and frameworks
- Examples: Clone and use our examples
How To Contribute
ZITADEL is built in the open and welcoming to contributions of all kinds.
- 📖 Read the Contribution Guide to get started
- 💬 Join the conversation on Discord
- 🐛 Report bugs or request features via GitHub Issues
Contributors
Made with contrib.rocks.
Security
Security policy: SECURITY.md
Vulnerability Disclosure Policy — how to responsibly report security issues.
Technical Advisories are published for major issues that could impact security or stability in production.
License
AGPL-3.0 — see LICENSING.md for the full licensing policy, including Apache 2.0 and MIT exceptions for specific directories.