mirror of
https://github.com/andwati/zitadel.git
synced 2026-11-22 04:48:46 +00:00
fix(eventstore): prevent projections from skipping events (#12703)
<!-- CURSOR_AGENT_PR_BODY_BEGIN -->
# Which Problems Are Solved
Handler/v2 projections (`users14` and every other event projection) can
overlook committed events under load:
- `eventstore.commands_to_events` stamped `position` with `NOW()`
(transaction start). A slow create TX could commit after a faster TX
that started later, so the projector stored a later position and never
saw the earlier one.
- Resume used `OFFSET` on a **filtered** row count. That could skip
`in_tx_order = 1` at the next position (for example `user.human.added`).
- Two transactions in the same microsecond both start at `in_tx_order =
1`. A cursor of `in_tx_order > last` alone would drop the other TX’s
first event.
This does not repair rows that are already missing from projections.
# How the Problems Are Solved
- Setup step 77 replaces `eventstore.commands_to_events` so `created_at
= statement_timestamp()` and `position = EXTRACT(EPOCH FROM
clock_timestamp())` at INSERT time. Initialise `08_events_table.sql`
matches that for new databases. The function is `VOLATILE` to match
`clock_timestamp()`.
- The projector still drains in-flight writers with `pg_advisory_lock` /
`pg_advisory_unlock` in one statement, then caps with `position <=
EXTRACT(EPOCH FROM now())` (projector TX start). The exclusive lock is
**not** held across the event `SELECT`. The cap must stay `now()`, not
`clock_timestamp()`, or later inserts become a high-water mark.
- Resume is a lexicographic `EventSortKey` on `(position, in_tx_order,
instance_id, aggregate_type, aggregate_id, sequence)` via
`AfterEventSortKey`. SQL `OFFSET` is no longer used for projection
catch-up.
- Handler and FieldHandler persist the last event’s `in_tx_order` in
`current_states.filter_offset` (existing column, new meaning). Step 77
backfills matching rows from `events2`; unmatched rows keep their
previous value.
# Additional Changes
- `eventstore.Event` exposes `InTxOrder() uint32` (BaseEvent, repository
Event, v3 event, v1 models).
- v3 push SELECT/RETURNING now includes `in_tx_order` so newly written
events carry a real ordinal.
- `WithMinPosition` stays an inclusive `PositionAtLeast` floor and does
not smash the sort-key cursor.
- `SearchQuery.PositionAfter(decimal)` is unchanged (used by
access-token lookup). Do not confuse it with
`SearchQueryBuilder.AfterEventSortKey`.
- `commands_to_events` in setup 77 and initialise 08 is `VOLATILE
PARALLEL SAFE`. Shipped 40/64/70 stay `STABLE` because they still stamp
with `NOW()`.
- Drop unused `last_owner TEXT` DECLARE from setup 77 and initialise
`08_events_table.sql` (`i INTEGER` remains for the enforce-owner loop).
Shipped 40/64/70 keep the leftover.
- This PR’s migration is numbered 77 because `main` already has setup 76
(`76_users14_login_equality_indexes`). First-loop order is s70, then
main’s 76, then this 77.
# Additional Context
**Reviewer map**
| Start here | Why |
| --- | --- |
| `cmd/setup/77.sql` + `cmd/setup/77.go` | Position stamp + atomic
function replace/backfill after shipped `s70` and main’s s76 |
| `internal/eventstore/event_sort_key.go` + `search_query.go` |
Canonical cursor and `AfterEventSortKey` |
| `internal/eventstore/handler/v2/{state.go,handler.go}` |
Persist/resume cursor; `eventQuery` |
| `internal/eventstore/repository/sql/{query.go,postgres.go}` | Tuple
`>` filter, 6-column `ORDER BY`, await cap |
| `internal/eventstore/v3/push.go` / `push.sql` | Scan `in_tx_order` on
write |
Mechanical / low-risk: handler and query test fixtures, Event interface
stubs (`event_base.go`, repository Event, mocks, v1 models).
**Do not look for** a `70.sql` change — shipped migrations stay as on
`main`. Setup 40/64 still use `NOW()`; step 77 overwrites the function
afterward. Fresh `FirstInstance` (runs before `s70`) can still stamp
`NOW()` until 77; runtime pushes after 77 use `clock_timestamp()`.
Main’s `cmd/setup/76.go` is the users14 login-equality indexes, not this
stamp.
**Rollout**
1. Run setup so 77 replaces `commands_to_events` and backfills
`filter_offset`.
2. Restart ZITADEL so projectors load the new resume path.
3. Existing missing `users14` rows are not repaired by this PR.
**Tests**
- `go test ./internal/eventstore/... ./cmd/setup/`
- Sort-key SQL fixtures in `query_sort_key_test.go` (no `OFFSET`;
aggregate-id queries keep 6-column `ORDER BY`)
- `cmd/setup/77_test.go` asserts the embed has no unused `last_owner`
and still uses `clock_timestamp()` / `VOLATILE PARALLEL SAFE`
Related: #8509, #10480 (remove OFFSET), #10560 (advisory locks), #10710
(stable ORDER BY).
Out of scope: repairing already-missing projection rows, holding the
exclusive lock during SELECT, rewriting historical setup 40/64 SQL.
<!-- CURSOR_AGENT_PR_BODY_END -->
<div><a
href="https://cursor.com/agents/bc-3da6eb51-11d4-4a8a-8ba0-e190f116df05?cursor_ref=pr_footer&cursor_cta=open_in_web"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://cursor.com/assets/images/open-in-web-dark.png"><source
media="(prefers-color-scheme: light)"
srcset="https://cursor.com/assets/images/open-in-web-light.png"><img
alt="Open in Web" width="114" height="28"
src="https://cursor.com/assets/images/open-in-web-dark.png"></picture></a> <a
href="https://cursor.com/background-agent?bcId=bc-3da6eb51-11d4-4a8a-8ba0-e190f116df05&cursor_ref=pr_footer&cursor_cta=open_in_cursor"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://cursor.com/assets/images/open-in-cursor-dark.png"><source
media="(prefers-color-scheme: light)"
srcset="https://cursor.com/assets/images/open-in-cursor-light.png"><img
alt="Open in Cursor" width="131" height="28"
src="https://cursor.com/assets/images/open-in-cursor-dark.png"></picture></a> </div>
---------
Co-authored-by: Cursor Agent <[email protected]>
Co-authored-by: Silvan <[email protected]>
This commit is contained in:
co-authored by
Cursor Agent
Silvan
parent
684de2e691
commit
fbe7b96138
@@ -15,7 +15,9 @@ import (
|
||||
"github.com/zitadel/zitadel/internal/telemetry/tracing"
|
||||
)
|
||||
|
||||
// awaitOpenTransactions ensures event ordering, so we don't events younger that open transactions
|
||||
// awaitOpenTransactions drains in-flight writers, then caps at projector TX start (now()),
|
||||
// not clock_timestamp(). A wall-clock cap at SELECT time would include rows inserted after
|
||||
// BEGIN and recreate position overtake.
|
||||
var (
|
||||
awaitOpenTransactionsV1 = ` AND created_at <= now()`
|
||||
awaitOpenTransactionsV2 = ` AND "position" <= EXTRACT(EPOCH FROM now())`
|
||||
@@ -38,6 +40,8 @@ func NewPostgres(client *database.DB) *Postgres {
|
||||
|
||||
func (db *Postgres) Health(ctx context.Context) error { return db.Ping() }
|
||||
|
||||
const eventSortKeySQL = `"position", in_tx_order, instance_id, aggregate_type, aggregate_id, "sequence"`
|
||||
|
||||
// FilterToReducer finds all events matching the given search query and passes them to the reduce function.
|
||||
func (psql *Postgres) FilterToReducer(ctx context.Context, searchQuery *eventstore.SearchQueryBuilder, reduce eventstore.Reducer) (err error) {
|
||||
ctx, span := tracing.NewSpan(ctx)
|
||||
@@ -91,9 +95,9 @@ func (db *Postgres) orderByEventSequence(desc, shouldOrderBySequence, useV1 bool
|
||||
}
|
||||
|
||||
if desc {
|
||||
return ` ORDER BY "position" DESC, in_tx_order DESC, instance_id, aggregate_type, aggregate_id`
|
||||
return ` ORDER BY (` + eventSortKeySQL + `) DESC`
|
||||
}
|
||||
return ` ORDER BY "position", in_tx_order, instance_id, aggregate_type, aggregate_id`
|
||||
return ` ORDER BY ` + eventSortKeySQL
|
||||
}
|
||||
|
||||
func (db *Postgres) eventQuery(useV1 bool) string {
|
||||
@@ -123,6 +127,7 @@ func (db *Postgres) eventQuery(useV1 bool) string {
|
||||
", aggregate_type" +
|
||||
", aggregate_id" +
|
||||
", revision" +
|
||||
", in_tx_order" +
|
||||
" FROM eventstore.events2"
|
||||
}
|
||||
|
||||
|
||||
@@ -103,7 +103,7 @@ func query(ctx context.Context, criteria querier, searchQuery *eventstore.Search
|
||||
|
||||
// if there is only one subquery we can optimize the query ordering by sequence
|
||||
var shouldOrderBySequence bool
|
||||
if len(q.SubQueries) == 1 {
|
||||
if q.EventSortKeyAfter == nil && len(q.SubQueries) == 1 {
|
||||
for _, filter := range q.SubQueries[0] {
|
||||
if filter.Field == repository.FieldAggregateID {
|
||||
shouldOrderBySequence = filter.Operation == repository.OperationEquals
|
||||
@@ -226,6 +226,7 @@ func eventsScanner(useV1 bool) func(scanner scan, dest interface{}) (err error)
|
||||
&event.AggregateType,
|
||||
&event.AggregateID,
|
||||
&revision,
|
||||
&event.InTx,
|
||||
)
|
||||
event.Version = eventstore.Version("v" + strconv.Itoa(int(revision)))
|
||||
}
|
||||
@@ -279,6 +280,21 @@ func prepareConditions(criteria querier, query *repository.SearchQuery, useV1 bo
|
||||
args = append(args, additionalArgs...)
|
||||
}
|
||||
|
||||
if query.EventSortKeyAfter != nil && !useV1 {
|
||||
if clauses != "" {
|
||||
clauses += " AND "
|
||||
}
|
||||
clauses += `(` + eventSortKeySQL + `) > (?, ?, ?, ?, ?, ?)`
|
||||
args = append(args,
|
||||
query.EventSortKeyAfter.Position,
|
||||
query.EventSortKeyAfter.InTxOrder,
|
||||
query.EventSortKeyAfter.InstanceID,
|
||||
query.EventSortKeyAfter.AggregateType,
|
||||
query.EventSortKeyAfter.AggregateID,
|
||||
query.EventSortKeyAfter.Sequence,
|
||||
)
|
||||
}
|
||||
|
||||
excludeAggregateIDs := query.ExcludeAggregateIDs
|
||||
if len(excludeAggregateIDs) > 0 {
|
||||
excludeAggregateIDs = append(excludeAggregateIDs, query.InstanceID, query.InstanceIDs, query.Position, query.CreatedAfter, query.CreatedBefore)
|
||||
|
||||
@@ -0,0 +1,144 @@
|
||||
package sql
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql/driver"
|
||||
"regexp"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/shopspring/decimal"
|
||||
|
||||
"github.com/zitadel/zitadel/internal/database"
|
||||
"github.com/zitadel/zitadel/internal/eventstore"
|
||||
"github.com/zitadel/zitadel/internal/eventstore/repository"
|
||||
)
|
||||
|
||||
func Test_query_event_sort_key(t *testing.T) {
|
||||
cursor := eventstore.EventSortKey{
|
||||
Position: decimal.NewFromFloat(1.5),
|
||||
InTxOrder: 5,
|
||||
InstanceID: "instanceID",
|
||||
AggregateType: "user",
|
||||
AggregateID: "agg-a",
|
||||
Sequence: 5,
|
||||
}
|
||||
tuple := `(` + eventSortKeySQL + `) > (`
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
query *eventstore.SearchQueryBuilder
|
||||
sql string
|
||||
args []driver.Value
|
||||
awaitLock bool
|
||||
}{
|
||||
{
|
||||
name: "after sort key with await cap",
|
||||
query: eventstore.NewSearchQueryBuilder(eventstore.ColumnsEvent).
|
||||
InstanceID("instanceID").
|
||||
OrderAsc().
|
||||
AwaitOpenTransactions().
|
||||
Limit(200).
|
||||
AfterEventSortKey(cursor).
|
||||
AddQuery().
|
||||
AggregateTypes("user").
|
||||
EventTypes("user.human.added").
|
||||
Builder(),
|
||||
sql: `SELECT created_at, event_type, "sequence", "position", payload, creator, "owner", instance_id, aggregate_type, aggregate_id, revision, in_tx_order FROM eventstore.events2 WHERE instance_id = $1 AND aggregate_type = $2 AND event_type = $3 AND (` + eventSortKeySQL + `) > ($4, $5, $6, $7, $8, $9) AND "position" <= EXTRACT(EPOCH FROM now()) ORDER BY ` + eventSortKeySQL + ` LIMIT $10`,
|
||||
args: []driver.Value{
|
||||
"instanceID",
|
||||
eventstore.AggregateType("user"),
|
||||
eventstore.EventType("user.human.added"),
|
||||
cursor.Position,
|
||||
cursor.InTxOrder,
|
||||
cursor.InstanceID,
|
||||
cursor.AggregateType,
|
||||
cursor.AggregateID,
|
||||
cursor.Sequence,
|
||||
uint64(200),
|
||||
},
|
||||
awaitLock: true,
|
||||
},
|
||||
{
|
||||
name: "after sort key without await",
|
||||
query: eventstore.NewSearchQueryBuilder(eventstore.ColumnsEvent).
|
||||
InstanceID("instanceID").
|
||||
OrderAsc().
|
||||
Limit(200).
|
||||
AfterEventSortKey(cursor).
|
||||
AddQuery().
|
||||
AggregateTypes("user").
|
||||
Builder(),
|
||||
sql: `SELECT created_at, event_type, "sequence", "position", payload, creator, "owner", instance_id, aggregate_type, aggregate_id, revision, in_tx_order FROM eventstore.events2 WHERE instance_id = $1 AND aggregate_type = $2 AND (` + eventSortKeySQL + `) > ($3, $4, $5, $6, $7, $8) ORDER BY ` + eventSortKeySQL + ` LIMIT $9`,
|
||||
args: []driver.Value{
|
||||
"instanceID",
|
||||
eventstore.AggregateType("user"),
|
||||
cursor.Position,
|
||||
cursor.InTxOrder,
|
||||
cursor.InstanceID,
|
||||
cursor.AggregateType,
|
||||
cursor.AggregateID,
|
||||
cursor.Sequence,
|
||||
uint64(200),
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "after sort key with aggregate id still uses sort-key order",
|
||||
query: eventstore.NewSearchQueryBuilder(eventstore.ColumnsEvent).
|
||||
InstanceID("instanceID").
|
||||
OrderAsc().
|
||||
Limit(200).
|
||||
AfterEventSortKey(cursor).
|
||||
AddQuery().
|
||||
AggregateTypes("user").
|
||||
AggregateIDs("agg-a").
|
||||
Builder(),
|
||||
sql: `SELECT created_at, event_type, "sequence", "position", payload, creator, "owner", instance_id, aggregate_type, aggregate_id, revision, in_tx_order FROM eventstore.events2 WHERE instance_id = $1 AND aggregate_type = $2 AND aggregate_id = $3 AND (` + eventSortKeySQL + `) > ($4, $5, $6, $7, $8, $9) ORDER BY ` + eventSortKeySQL + ` LIMIT $10`,
|
||||
args: []driver.Value{
|
||||
"instanceID",
|
||||
eventstore.AggregateType("user"),
|
||||
"agg-a",
|
||||
cursor.Position,
|
||||
cursor.InTxOrder,
|
||||
cursor.InstanceID,
|
||||
cursor.AggregateType,
|
||||
cursor.AggregateID,
|
||||
cursor.Sequence,
|
||||
uint64(200),
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
client := NewPostgres(&database.DB{Database: new(testDB)})
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
if strings.Contains(tt.sql, "OFFSET") {
|
||||
t.Fatal("sort-key resume must not use OFFSET")
|
||||
}
|
||||
if strings.Contains(tt.sql, `ORDER BY "sequence"`) {
|
||||
t.Fatal("sort-key resume must not order by sequence")
|
||||
}
|
||||
if !strings.Contains(tt.sql, tuple) {
|
||||
t.Fatalf("expected sort-key tuple %s in SQL", tuple)
|
||||
}
|
||||
|
||||
mock := newMockClient(t)
|
||||
if tt.awaitLock {
|
||||
mock = mock.expectExec(regexp.QuoteMeta(
|
||||
`select pg_advisory_lock('eventstore.events2'::REGCLASS::OID::INTEGER, hashtext($1)), pg_advisory_unlock('eventstore.events2'::REGCLASS::OID::INTEGER, hashtext($1))`),
|
||||
[]driver.Value{"instanceID"},
|
||||
)
|
||||
}
|
||||
mock = mock.expectQuery(regexp.QuoteMeta(tt.sql), tt.args)
|
||||
client.DB.DB = mock.client
|
||||
|
||||
err := query(context.Background(), client, tt.query, &[]*repository.Event{}, false)
|
||||
if err != nil {
|
||||
t.Fatalf("query() error = %v", err)
|
||||
}
|
||||
if err := mock.mock.ExpectationsWereMet(); err != nil {
|
||||
t.Errorf("not all expectations met: %v", err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -181,13 +181,13 @@ func Test_prepareColumns(t *testing.T) {
|
||||
}),
|
||||
},
|
||||
res: res{
|
||||
query: `SELECT created_at, event_type, "sequence", "position", payload, creator, "owner", instance_id, aggregate_type, aggregate_id, revision FROM eventstore.events2`,
|
||||
query: `SELECT created_at, event_type, "sequence", "position", payload, creator, "owner", instance_id, aggregate_type, aggregate_id, revision, in_tx_order FROM eventstore.events2`,
|
||||
expected: []eventstore.Event{
|
||||
&repository.Event{AggregateID: "hodor", AggregateType: "user", Seq: 5, Pos: decimal.NewFromInt(42), Data: nil, Version: "v1"},
|
||||
&repository.Event{AggregateID: "hodor", AggregateType: "user", Seq: 5, Pos: decimal.NewFromInt(42), Data: nil, Version: "v1", InTx: 3},
|
||||
},
|
||||
},
|
||||
fields: fields{
|
||||
dbRow: []interface{}{time.Time{}, eventstore.EventType(""), uint64(5), decimal.NewNullDecimal(decimal.NewFromInt(42)), sql.RawBytes(nil), "", sql.NullString{}, "", eventstore.AggregateType("user"), "hodor", uint8(1)},
|
||||
dbRow: []interface{}{time.Time{}, eventstore.EventType(""), uint64(5), decimal.NewNullDecimal(decimal.NewFromInt(42)), sql.RawBytes(nil), "", sql.NullString{}, "", eventstore.AggregateType("user"), "hodor", uint8(1), uint32(3)},
|
||||
},
|
||||
},
|
||||
{
|
||||
@@ -200,13 +200,13 @@ func Test_prepareColumns(t *testing.T) {
|
||||
}),
|
||||
},
|
||||
res: res{
|
||||
query: `SELECT created_at, event_type, "sequence", "position", payload, creator, "owner", instance_id, aggregate_type, aggregate_id, revision FROM eventstore.events2`,
|
||||
query: `SELECT created_at, event_type, "sequence", "position", payload, creator, "owner", instance_id, aggregate_type, aggregate_id, revision, in_tx_order FROM eventstore.events2`,
|
||||
expected: []eventstore.Event{
|
||||
&repository.Event{AggregateID: "hodor", AggregateType: "user", Seq: 5, Pos: decimal.Decimal{}, Data: nil, Version: "v1"},
|
||||
},
|
||||
},
|
||||
fields: fields{
|
||||
dbRow: []interface{}{time.Time{}, eventstore.EventType(""), uint64(5), decimal.NullDecimal{}, sql.RawBytes(nil), "", sql.NullString{}, "", eventstore.AggregateType("user"), "hodor", uint8(1)},
|
||||
dbRow: []interface{}{time.Time{}, eventstore.EventType(""), uint64(5), decimal.NullDecimal{}, sql.RawBytes(nil), "", sql.NullString{}, "", eventstore.AggregateType("user"), "hodor", uint8(1), uint32(0)},
|
||||
},
|
||||
},
|
||||
{
|
||||
@@ -904,7 +904,7 @@ func Test_query_events_mocked(t *testing.T) {
|
||||
},
|
||||
fields: fields{
|
||||
mock: newMockClient(t).expectQuery(
|
||||
regexp.QuoteMeta(`SELECT created_at, event_type, "sequence", "position", payload, creator, "owner", instance_id, aggregate_type, aggregate_id, revision FROM eventstore.events2 WHERE instance_id = $1 AND aggregate_type = $2 AND event_type = $3 AND "position" >= $4 AND aggregate_id NOT IN (SELECT aggregate_id FROM eventstore.events2 WHERE aggregate_type = $5 AND event_type = ANY($6) AND instance_id = $7 AND "position" >= $8) ORDER BY "position" DESC, in_tx_order DESC, instance_id, aggregate_type, aggregate_id LIMIT $9`),
|
||||
regexp.QuoteMeta(`SELECT created_at, event_type, "sequence", "position", payload, creator, "owner", instance_id, aggregate_type, aggregate_id, revision, in_tx_order FROM eventstore.events2 WHERE instance_id = $1 AND aggregate_type = $2 AND event_type = $3 AND "position" >= $4 AND aggregate_id NOT IN (SELECT aggregate_id FROM eventstore.events2 WHERE aggregate_type = $5 AND event_type = ANY($6) AND instance_id = $7 AND "position" >= $8) ORDER BY (`+eventSortKeySQL+`) DESC LIMIT $9`),
|
||||
[]driver.Value{"instanceID", eventstore.AggregateType("notify"), eventstore.EventType("notify.foo.bar"), decimal.NewFromFloat(123.456), eventstore.AggregateType("notify"), []eventstore.EventType{"notification.failed", "notification.success"}, "instanceID", decimal.NewFromFloat(123.456), uint64(5)},
|
||||
),
|
||||
},
|
||||
@@ -933,7 +933,7 @@ func Test_query_events_mocked(t *testing.T) {
|
||||
},
|
||||
fields: fields{
|
||||
mock: newMockClient(t).expectQuery(
|
||||
regexp.QuoteMeta(`SELECT created_at, event_type, "sequence", "position", payload, creator, "owner", instance_id, aggregate_type, aggregate_id, revision FROM eventstore.events2 WHERE instance_id = $1 AND aggregate_type = $2 AND event_type = $3 AND created_at > $4 AND aggregate_id NOT IN (SELECT aggregate_id FROM eventstore.events2 WHERE aggregate_type = $5 AND event_type = ANY($6) AND instance_id = $7 AND created_at > $8) ORDER BY "position" DESC, in_tx_order DESC, instance_id, aggregate_type, aggregate_id LIMIT $9`),
|
||||
regexp.QuoteMeta(`SELECT created_at, event_type, "sequence", "position", payload, creator, "owner", instance_id, aggregate_type, aggregate_id, revision, in_tx_order FROM eventstore.events2 WHERE instance_id = $1 AND aggregate_type = $2 AND event_type = $3 AND created_at > $4 AND aggregate_id NOT IN (SELECT aggregate_id FROM eventstore.events2 WHERE aggregate_type = $5 AND event_type = ANY($6) AND instance_id = $7 AND created_at > $8) ORDER BY (`+eventSortKeySQL+`) DESC LIMIT $9`),
|
||||
[]driver.Value{"instanceID", eventstore.AggregateType("notify"), eventstore.EventType("notify.foo.bar"), time.Unix(123, 456), eventstore.AggregateType("notify"), []eventstore.EventType{"notification.failed", "notification.success"}, "instanceID", time.Unix(123, 456), uint64(5)},
|
||||
),
|
||||
},
|
||||
|
||||
Reference in New Issue
Block a user