2 Commits

Include renames

Author	SHA1	Message	Date
4 people	4bc484dac9	perf(eventstore): read projection events per event type (#12753) ... # Which Problems Are Solved - A projection that subscribes to several event types gets slower the further it falls behind. - The handler reads its events with one query: `aggregate_type = ANY(...) AND event_type = ANY(...)`, ordered by position and limited to the bulk limit. - The `es_projection` index returns each (aggregate type, event type) range in position order, but Postgres cannot merge those ranges. - So each batch reads every matching event after the projection's position, sorts them all, and keeps only the bulk limit. - A projection that is N events behind therefore reads about N rows per batch. Catching up costs about N² in total, and once events arrive faster than such batches can process them, the projection does not recover. - Every query that triggers the projection waits for that catch-up. Examples are `GetSession` and the notification handlers. # How the Problems Are Solved - **New opt-in, `SearchQueryBuilder.ScanEventTypesSeparately()`.** It applies to event queries for a single instance, ascending and limited, whose sub-queries filter only by aggregate and event types. - For such a query, the SQL layer reads each distinct (aggregate type, event type) combination in its own `ORDER BY … LIMIT` subquery, using `UNNEST … CROSS JOIN LATERAL`. - It then merges the results with the same sort key and limit. - Each subquery is one `es_projection` range read in position order, so it stops at the limit. A batch reads at most `limit × combinations` rows, however far behind the projection is. - Every other query renders exactly as before. - **The projection handler** now passes its real combinations, one sub-query per aggregate type instead of the cross product of all types, and sets the opt-in. - **Results are identical to the single query.** - The sort key (`position, in_tx_order, instance_id, aggregate_type, aggregate_id, sequence`) is a total order, and the global top n is contained in the union of each combination's top n. - The cursor from #12703 and the open-transaction cap apply inside every subquery. - A test against a real Postgres compares both shapes from every cursor and every position, for several limits. The test events include positions shared by several transactions, whose `in_tx_order` restarts. # Additional Changes This is a separate commit and can be split into its own PR: - `cmd/defaults.yaml` sets `BulkLimit: 1000` for the `sessions` projection. - `projection_state_latency` gets histogram buckets up to one week. Until now, every projection more than 30 minutes behind was reported in the same top bucket. # Additional Context These numbers come from a local, CPU-pinned sandbox, for comparing builds only. - **Database:** Postgres 17 in a VM with 8 vCPUs and 16 GiB. The `Eventstore.Autovacuum` and `Projections` settings mirror a production-like setup. - **Starting state:** `sessions8` 4.8M events behind, restored byte-identically for every run. Batch query for `sessions8`, `EXPLAIN ANALYZE`, median of 5 runs, parallel workers at the default: | | before | after | | --- | ---: | ---: | | at the backlog, limit 1000 | 832 ms | 4.0 ms | | at the backlog, limit 200 | 832 ms | 1.0 ms | | at the head (idle poll) | 0.14 ms | 0.19 ms | | generic plan (`plan_cache_mode=force_generic_plan`), at the backlog | 1,042 ms | 4.0 ms | Draining `sessions8` from that backlog for 600 s, 3 runs per build (median, with the range in brackets): | | before | after | | --- | ---: | ---: | | events/s | 975 (932-1,032) | 7,811 (7,523-7,955) | | mean batch time, including writes | 1.03 s | 0.13 s | | still behind after 600 s | 4.2M | 17k-277k | A new index `(instance_id, position, in_tx_order, aggregate_type, event_type)` with the old query was evaluated as an alternative and rejected. Under a generic plan, Postgres chose it for projections whose events are rare and walked the whole event stream: 430 ms instead of 0.15 ms. This is the behaviour that led to dropping `es_instance_position` in #10626. - Follow-up for #12703 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5 <[email protected]> Co-authored-by: Copilot Autofix powered by AI <[email protected]>	2026-09-22 17:09:42 +02:00
4 people	fbe7b96138	fix(eventstore): prevent projections from skipping events (#12703) ... <!-- CURSOR_AGENT_PR_BODY_BEGIN --> # Which Problems Are Solved Handler/v2 projections (`users14` and every other event projection) can overlook committed events under load: - `eventstore.commands_to_events` stamped `position` with `NOW()` (transaction start). A slow create TX could commit after a faster TX that started later, so the projector stored a later position and never saw the earlier one. - Resume used `OFFSET` on a **filtered** row count. That could skip `in_tx_order = 1` at the next position (for example `user.human.added`). - Two transactions in the same microsecond both start at `in_tx_order = 1`. A cursor of `in_tx_order > last` alone would drop the other TX’s first event. This does not repair rows that are already missing from projections. # How the Problems Are Solved - Setup step 77 replaces `eventstore.commands_to_events` so `created_at = statement_timestamp()` and `position = EXTRACT(EPOCH FROM clock_timestamp())` at INSERT time. Initialise `08_events_table.sql` matches that for new databases. The function is `VOLATILE` to match `clock_timestamp()`. - The projector still drains in-flight writers with `pg_advisory_lock` / `pg_advisory_unlock` in one statement, then caps with `position <= EXTRACT(EPOCH FROM now())` (projector TX start). The exclusive lock is **not** held across the event `SELECT`. The cap must stay `now()`, not `clock_timestamp()`, or later inserts become a high-water mark. - Resume is a lexicographic `EventSortKey` on `(position, in_tx_order, instance_id, aggregate_type, aggregate_id, sequence)` via `AfterEventSortKey`. SQL `OFFSET` is no longer used for projection catch-up. - Handler and FieldHandler persist the last event’s `in_tx_order` in `current_states.filter_offset` (existing column, new meaning). Step 77 backfills matching rows from `events2`; unmatched rows keep their previous value. # Additional Changes - `eventstore.Event` exposes `InTxOrder() uint32` (BaseEvent, repository Event, v3 event, v1 models). - v3 push SELECT/RETURNING now includes `in_tx_order` so newly written events carry a real ordinal. - `WithMinPosition` stays an inclusive `PositionAtLeast` floor and does not smash the sort-key cursor. - `SearchQuery.PositionAfter(decimal)` is unchanged (used by access-token lookup). Do not confuse it with `SearchQueryBuilder.AfterEventSortKey`. - `commands_to_events` in setup 77 and initialise 08 is `VOLATILE PARALLEL SAFE`. Shipped 40/64/70 stay `STABLE` because they still stamp with `NOW()`. - Drop unused `last_owner TEXT` DECLARE from setup 77 and initialise `08_events_table.sql` (`i INTEGER` remains for the enforce-owner loop). Shipped 40/64/70 keep the leftover. - This PR’s migration is numbered 77 because `main` already has setup 76 (`76_users14_login_equality_indexes`). First-loop order is s70, then main’s 76, then this 77. # Additional Context **Reviewer map** | Start here | Why | | --- | --- | | `cmd/setup/77.sql` + `cmd/setup/77.go` | Position stamp + atomic function replace/backfill after shipped `s70` and main’s s76 | | `internal/eventstore/event_sort_key.go` + `search_query.go` | Canonical cursor and `AfterEventSortKey` | | `internal/eventstore/handler/v2/{state.go,handler.go}` | Persist/resume cursor; `eventQuery` | | `internal/eventstore/repository/sql/{query.go,postgres.go}` | Tuple `>` filter, 6-column `ORDER BY`, await cap | | `internal/eventstore/v3/push.go` / `push.sql` | Scan `in_tx_order` on write | Mechanical / low-risk: handler and query test fixtures, Event interface stubs (`event_base.go`, repository Event, mocks, v1 models). **Do not look for** a `70.sql` change — shipped migrations stay as on `main`. Setup 40/64 still use `NOW()`; step 77 overwrites the function afterward. Fresh `FirstInstance` (runs before `s70`) can still stamp `NOW()` until 77; runtime pushes after 77 use `clock_timestamp()`. Main’s `cmd/setup/76.go` is the users14 login-equality indexes, not this stamp. **Rollout** 1. Run setup so 77 replaces `commands_to_events` and backfills `filter_offset`. 2. Restart ZITADEL so projectors load the new resume path. 3. Existing missing `users14` rows are not repaired by this PR. **Tests** - `go test ./internal/eventstore/... ./cmd/setup/` - Sort-key SQL fixtures in `query_sort_key_test.go` (no `OFFSET`; aggregate-id queries keep 6-column `ORDER BY`) - `cmd/setup/77_test.go` asserts the embed has no unused `last_owner` and still uses `clock_timestamp()` / `VOLATILE PARALLEL SAFE` Related: #8509, #10480 (remove OFFSET), #10560 (advisory locks), #10710 (stable ORDER BY). Out of scope: repairing already-missing projection rows, holding the exclusive lock during SELECT, rewriting historical setup 40/64 SQL. <!-- CURSOR_AGENT_PR_BODY_END --> <div><a href="https://cursor.com/agents/bc-3da6eb51-11d4-4a8a-8ba0-e190f116df05?cursor_ref=pr_footer&cursor_cta=open_in_web"><picture><source media="(prefers-color-scheme: dark)" srcset="https://cursor.com/assets/images/open-in-web-dark.png"><source media="(prefers-color-scheme: light)" srcset="https://cursor.com/assets/images/open-in-web-light.png"><img alt="Open in Web" width="114" height="28" src="https://cursor.com/assets/images/open-in-web-dark.png"></picture></a>&nbsp;<a href="https://cursor.com/background-agent?bcId=bc-3da6eb51-11d4-4a8a-8ba0-e190f116df05&cursor_ref=pr_footer&cursor_cta=open_in_cursor"><picture><source media="(prefers-color-scheme: dark)" srcset="https://cursor.com/assets/images/open-in-cursor-dark.png"><source media="(prefers-color-scheme: light)" srcset="https://cursor.com/assets/images/open-in-cursor-light.png"><img alt="Open in Cursor" width="131" height="28" src="https://cursor.com/assets/images/open-in-cursor-dark.png"></picture></a>&nbsp;</div> --------- Co-authored-by: Cursor Agent <[email protected]> Co-authored-by: Silvan <[email protected]>	2026-09-09 11:08:50 +02:00