Commit Graph
16 Commits
Author SHA1 Message Date
fbe7b96138 fix(eventstore): prevent projections from skipping events (#12703)
<!-- CURSOR_AGENT_PR_BODY_BEGIN -->
# Which Problems Are Solved

Handler/v2 projections (`users14` and every other event projection) can
overlook committed events under load:

- `eventstore.commands_to_events` stamped `position` with `NOW()`
(transaction start). A slow create TX could commit after a faster TX
that started later, so the projector stored a later position and never
saw the earlier one.
- Resume used `OFFSET` on a **filtered** row count. That could skip
`in_tx_order = 1` at the next position (for example `user.human.added`).
- Two transactions in the same microsecond both start at `in_tx_order =
1`. A cursor of `in_tx_order > last` alone would drop the other TX’s
first event.

This does not repair rows that are already missing from projections.

# How the Problems Are Solved

- Setup step 77 replaces `eventstore.commands_to_events` so `created_at
= statement_timestamp()` and `position = EXTRACT(EPOCH FROM
clock_timestamp())` at INSERT time. Initialise `08_events_table.sql`
matches that for new databases. The function is `VOLATILE` to match
`clock_timestamp()`.
- The projector still drains in-flight writers with `pg_advisory_lock` /
`pg_advisory_unlock` in one statement, then caps with `position <=
EXTRACT(EPOCH FROM now())` (projector TX start). The exclusive lock is
**not** held across the event `SELECT`. The cap must stay `now()`, not
`clock_timestamp()`, or later inserts become a high-water mark.
- Resume is a lexicographic `EventSortKey` on `(position, in_tx_order,
instance_id, aggregate_type, aggregate_id, sequence)` via
`AfterEventSortKey`. SQL `OFFSET` is no longer used for projection
catch-up.
- Handler and FieldHandler persist the last event’s `in_tx_order` in
`current_states.filter_offset` (existing column, new meaning). Step 77
backfills matching rows from `events2`; unmatched rows keep their
previous value.

# Additional Changes

- `eventstore.Event` exposes `InTxOrder() uint32` (BaseEvent, repository
Event, v3 event, v1 models).
- v3 push SELECT/RETURNING now includes `in_tx_order` so newly written
events carry a real ordinal.
- `WithMinPosition` stays an inclusive `PositionAtLeast` floor and does
not smash the sort-key cursor.
- `SearchQuery.PositionAfter(decimal)` is unchanged (used by
access-token lookup). Do not confuse it with
`SearchQueryBuilder.AfterEventSortKey`.
- `commands_to_events` in setup 77 and initialise 08 is `VOLATILE
PARALLEL SAFE`. Shipped 40/64/70 stay `STABLE` because they still stamp
with `NOW()`.
- Drop unused `last_owner TEXT` DECLARE from setup 77 and initialise
`08_events_table.sql` (`i INTEGER` remains for the enforce-owner loop).
Shipped 40/64/70 keep the leftover.
- This PR’s migration is numbered 77 because `main` already has setup 76
(`76_users14_login_equality_indexes`). First-loop order is s70, then
main’s 76, then this 77.

# Additional Context

**Reviewer map**

| Start here | Why |
| --- | --- |
| `cmd/setup/77.sql` + `cmd/setup/77.go` | Position stamp + atomic
function replace/backfill after shipped `s70` and main’s s76 |
| `internal/eventstore/event_sort_key.go` + `search_query.go` |
Canonical cursor and `AfterEventSortKey` |
| `internal/eventstore/handler/v2/{state.go,handler.go}` |
Persist/resume cursor; `eventQuery` |
| `internal/eventstore/repository/sql/{query.go,postgres.go}` | Tuple
`>` filter, 6-column `ORDER BY`, await cap |
| `internal/eventstore/v3/push.go` / `push.sql` | Scan `in_tx_order` on
write |

Mechanical / low-risk: handler and query test fixtures, Event interface
stubs (`event_base.go`, repository Event, mocks, v1 models).

**Do not look for** a `70.sql` change — shipped migrations stay as on
`main`. Setup 40/64 still use `NOW()`; step 77 overwrites the function
afterward. Fresh `FirstInstance` (runs before `s70`) can still stamp
`NOW()` until 77; runtime pushes after 77 use `clock_timestamp()`.
Main’s `cmd/setup/76.go` is the users14 login-equality indexes, not this
stamp.

**Rollout**

1. Run setup so 77 replaces `commands_to_events` and backfills
`filter_offset`.
2. Restart ZITADEL so projectors load the new resume path.
3. Existing missing `users14` rows are not repaired by this PR.

**Tests**

- `go test ./internal/eventstore/... ./cmd/setup/`
- Sort-key SQL fixtures in `query_sort_key_test.go` (no `OFFSET`;
aggregate-id queries keep 6-column `ORDER BY`)
- `cmd/setup/77_test.go` asserts the embed has no unused `last_owner`
and still uses `clock_timestamp()` / `VOLATILE PARALLEL SAFE`

Related: #8509, #10480 (remove OFFSET), #10560 (advisory locks), #10710
(stable ORDER BY).
Out of scope: repairing already-missing projection rows, holding the
exclusive lock during SELECT, rewriting historical setup 40/64 SQL.
<!-- CURSOR_AGENT_PR_BODY_END -->

<div><a
href="https://cursor.com/agents/bc-3da6eb51-11d4-4a8a-8ba0-e190f116df05?cursor_ref=pr_footer&cursor_cta=open_in_web"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://cursor.com/assets/images/open-in-web-dark.png"><source
media="(prefers-color-scheme: light)"
srcset="https://cursor.com/assets/images/open-in-web-light.png"><img
alt="Open in Web" width="114" height="28"
src="https://cursor.com/assets/images/open-in-web-dark.png"></picture></a>&nbsp;<a
href="https://cursor.com/background-agent?bcId=bc-3da6eb51-11d4-4a8a-8ba0-e190f116df05&cursor_ref=pr_footer&cursor_cta=open_in_cursor"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://cursor.com/assets/images/open-in-cursor-dark.png"><source
media="(prefers-color-scheme: light)"
srcset="https://cursor.com/assets/images/open-in-cursor-light.png"><img
alt="Open in Cursor" width="131" height="28"
src="https://cursor.com/assets/images/open-in-cursor-dark.png"></picture></a>&nbsp;</div>

---------

Co-authored-by: Cursor Agent <[email protected]>
Co-authored-by: Silvan <[email protected]>
2026-09-09 11:08:50 +02:00
131f70db34 fix(eventstore): use decimal, correct mirror (#9914)
# Eventstore fixes

- `event.Position` used float64 before which can lead to [precision
loss](https://github.com/golang/go/issues/47300). The type got replaced
by [a type without precision
loss](https://github.com/jackc/pgx-shopspring-decimal)
- the handler reported the wrong error if the current state was updated
and therefore took longer to retry failed events.

# Mirror fixes

- max age of auth requests can be configured to speed up copying data
from `auth.auth_requests` table. Auth requests last updated before the
set age will be ignored. Default is 1 month
- notification projections are skipped because notifications should be
sent by the source system. The projections are set to the latest
position
- ensure that mirror can be executed multiple times

---------

Co-authored-by: Livio Spring <[email protected]>
2025-05-28 21:54:18 +00:00
Tim MöhlmannandGitHub aeb379e7de fix(eventstore): revert precise decimal (#8527) (#8679) 2024-09-24 18:43:29 +02:00
SilvanandGitHub b522588d98 fix(eventstore): precise decimal (#8527)
# Which Problems Are Solved

Float64 which was used for the event.Position field is [not precise in
go and gets rounded](https://github.com/golang/go/issues/47300). This
can lead to unprecies position tracking of events and therefore
projections especially on cockcoachdb as the position used there is a
big number.

example of a unprecies position:
exact: 1725257931223002628
float64: 1725257931223002624.000000

# How the Problems Are Solved

The float64 was replaced by
[github.com/jackc/pgx-shopspring-decimal](https://github.com/jackc/pgx-shopspring-decimal).

# Additional Changes

Correct behaviour of makefile for load tests.
Rename `latestSequence`-queries to `latestPosition`
2024-09-06 12:19:19 +03:00
f680dd934d refactor: rename package errors to zerrors (#7039)
* chore: rename package errors to zerrors

* rename package errors to gerrors

* fix error related linting issues

* fix zitadel error assertion

* fix gosimple linting issues

* fix deprecated linting issues

* resolve gci linting issues

* fix import structure

---------

Co-authored-by: Elio Bischof <[email protected]>
2023-12-08 15:30:55 +01:00
SilvanandGitHub b5564572bc feat(eventstore): increase parallel write capabilities (#5940)
This implementation increases parallel write capabilities of the eventstore.
Please have a look at the technical advisories: [05](https://zitadel.com/docs/support/advisory/a10005) and  [06](https://zitadel.com/docs/support/advisory/a10006).
The implementation of eventstore.push is rewritten and stored events are migrated to a new table `eventstore.events2`.
If you are using cockroach: make sure that the database user of ZITADEL has `VIEWACTIVITY` grant. This is used to query events.
2023-10-19 12:19:10 +02:00
SilvanandGitHub dd84b93ac3 fix(token): filter users by instance id (#5596)
* fix(token): filter users by instance id
2023-04-03 12:56:37 +00:00
aed7010508 fix: scheduling (#3978)
* fix: improve scheduling

* build pre-release

* fix: locker

* fix: user handler and print stack in case of panic in reducer

* chore: remove sentry

* fix: improve handler projection and implement tests

* more tests

* fix: race condition in tests

* Update internal/eventstore/repository/sql/query.go

Co-authored-by: Silvan <[email protected]>

* fix: implemented suggested changes

* fix: lock statement

Co-authored-by: Silvan <[email protected]>
2022-07-22 10:08:39 +00:00
fa9f581d56 chore(v2): move to new org (#3499)
* chore: move to new org

* logging

* fix: org rename caos -> zitadel

Co-authored-by: adlerhurst <[email protected]>
2022-04-26 23:01:45 +00:00
Livio AmstutzandGitHub 1305c14e49 feat: handle instanceID in projections (#3442)
* feat: handle instanceID in projections

* rename functions

* fix key lock

* fix import
2022-04-19 08:26:12 +02:00
958362e6c9 feat: handle instance from context (#3382)
* commander

* commander

* selber!

* move to packages

* fix(errors): implement Is interface

* test: command

* test: commands

* add init steps

* setup tenant

* add default step yaml

* possibility to set password

* merge v2 into v2-commander

* fix: rename iam command side to instance

* fix: rename iam command side to instance

* fix: rename iam command side to instance

* fix: rename iam command side to instance

* fix: search query builder can filter events in memory

* fix: filters for add member

* fix(setup): add `ExternalSecure` to config

* chore: name iam to instance

* fix: matching

* remove unsued func

* base url

* base url

* test(command): filter funcs

* test: commands

* fix: rename orgiampolicy to domain policy

* start from init

* commands

* config

* fix indexes and add constraints

* fixes

* fix: merge conflicts

* fix: protos

* fix: md files

* setup

* add deprecated org iam policy again

* typo

* fix search query

* fix filter

* Apply suggestions from code review

* remove custom org from org setup

* add todos for verification

* change apps creation

* simplify package structure

* fix error

* move preparation helper for tests

* fix unique constraints

* fix config mapping in setup

* fix error handling in encryption_keys.go

* fix projection config

* fix query from old views to projection

* fix setup of mgmt api

* set iam project and fix instance projection

* fix tokens view

* fix steps.yaml and defaults.yaml

* fix projections

* change instance context to interface

* instance interceptors and additional events in setup

* cleanup

* tests for interceptors

* fix label policy

* add todo

* single api endpoint in environment.json

Co-authored-by: adlerhurst <[email protected]>
Co-authored-by: fabi <[email protected]>
2022-03-29 11:53:19 +02:00
Livio AmstutzandGitHub 56b916a2b0 feat: projections auto create their tables (#3324)
* begin init checks for projections

* first projection checks

* debug notification providers with query fixes

* more projections and first index

* more projections

* more projections

* finish projections

* fix tests (remove db name)

* create tables in setup

* fix logging / error handling

* add tenant to views

* rename tenant to instance_id

* add instance_id to all projections

* add instance_id to all queries

* correct instance_id on projections

* add instance_id to failed_events

* use separate context for instance

* implement features projection

* implement features projection

* remove unique constraint from setup when migration failed

* add error to failed setup event

* add instance_id to primary keys

* fix IAM projection

* remove old migrations folder

* fix keysFromYAML test
2022-03-23 09:02:39 +01:00
5132ebe07c feat: add tenant column to eventstore (#3314)
* feat: add tenant column to eventstore

* feat: read tenant from context on push and filter

* Update 07_events_table.sql

* pass tenant to queryFactory

* fix some query tests

* init in tests

* add missing sql files

Co-authored-by: Livio Amstutz <[email protected]>
2022-03-15 07:19:02 +01:00
48456b84b9 fix: changes (#1482)
* fix: changes

* change listquery to changequery

* fix: changes in auth

Co-authored-by: Max Peintner <[email protected]>
2021-03-29 10:32:23 +02:00
Livio AmstutzandGitHub a4763b1e4c feat: features (#1427)
* features

* features

* features

* fix json tags

* add features handler to auth

* mocks for tests

* add setup step

* fixes

* add featurelist to auth api

* grandfather state and typos

* typo

* merge new-eventstore

* fix login policy tests

* label policy in features

* audit log retention
2021-03-25 17:26:21 +01:00
FabiandGitHub d8e42744b4 fix: move v2 pkgs (#1331)
* fix: move eventstore pkgs

* fix: move eventstore pkgs

* fix: remove v2 view

* fix: remove v2 view
2021-02-23 15:13:04 +01:00