Files
zitadel/internal/eventstore/repository/search_query_test.go
T
4bc484dac9 perf(eventstore): read projection events per event type (#12753)
# Which Problems Are Solved

- A projection that subscribes to several event types gets slower the
further it falls behind.
- The handler reads its events with one query: `aggregate_type =
ANY(...) AND event_type = ANY(...)`, ordered by position and limited to
the bulk limit.
- The `es_projection` index returns each (aggregate type, event type)
range in position order, but Postgres cannot merge those ranges.
- So each batch reads every matching event after the projection's
position, sorts them all, and keeps only the bulk limit.
- A projection that is N events behind therefore reads about N rows per
batch. Catching up costs about N² in total, and once events arrive
faster than such batches can process them, the projection does not
recover.
- Every query that triggers the projection waits for that catch-up.
Examples are `GetSession` and the notification handlers.

# How the Problems Are Solved

- **New opt-in, `SearchQueryBuilder.ScanEventTypesSeparately()`.** It
applies to event queries for a single instance, ascending and limited,
whose sub-queries filter only by aggregate and event types.
- For such a query, the SQL layer reads each distinct (aggregate type,
event type) combination in its own `ORDER BY … LIMIT` subquery, using
`UNNEST … CROSS JOIN LATERAL`.
  - It then merges the results with the same sort key and limit.
- Each subquery is one `es_projection` range read in position order, so
it stops at the limit. A batch reads at most `limit × combinations`
rows, however far behind the projection is.
  - Every other query renders exactly as before.
- **The projection handler** now passes its real combinations, one
sub-query per aggregate type instead of the cross product of all types,
and sets the opt-in.
- **Results are identical to the single query.**
- The sort key (`position, in_tx_order, instance_id, aggregate_type,
aggregate_id, sequence`) is a total order, and the global top n is
contained in the union of each combination's top n.
- The cursor from #12703 and the open-transaction cap apply inside every
subquery.
- A test against a real Postgres compares both shapes from every cursor
and every position, for several limits. The test events include
positions shared by several transactions, whose `in_tx_order` restarts.

# Additional Changes

This is a separate commit and can be split into its own PR:
- `cmd/defaults.yaml` sets `BulkLimit: 1000` for the `sessions`
projection.
- `projection_state_latency` gets histogram buckets up to one week.
Until now, every projection more than 30 minutes behind was reported in
the same top bucket.

# Additional Context

These numbers come from a local, CPU-pinned sandbox, for comparing
builds only.
- **Database:** Postgres 17 in a VM with 8 vCPUs and 16 GiB. The
`Eventstore.Autovacuum` and `Projections` settings mirror a
production-like setup.
- **Starting state:** `sessions8` 4.8M events behind, restored
byte-identically for every run.

Batch query for `sessions8`, `EXPLAIN ANALYZE`, median of 5 runs,
parallel workers at the default:

| | before | after |
| --- | ---: | ---: |
| at the backlog, limit 1000 | 832 ms | 4.0 ms |
| at the backlog, limit 200 | 832 ms | 1.0 ms |
| at the head (idle poll) | 0.14 ms | 0.19 ms |
| generic plan (`plan_cache_mode=force_generic_plan`), at the backlog |
1,042 ms | 4.0 ms |

Draining `sessions8` from that backlog for 600 s, 3 runs per build
(median, with the range in brackets):

| | before | after |
| --- | ---: | ---: |
| events/s | 975 (932-1,032) | 7,811 (7,523-7,955) |
| mean batch time, including writes | 1.03 s | 0.13 s |
| still behind after 600 s | 4.2M | 17k-277k |

A new index `(instance_id, position, in_tx_order, aggregate_type,
event_type)` with the old query was evaluated as an alternative and
rejected. Under a generic plan, Postgres chose it for projections whose
events are rare and walked the whole event stream: 430 ms instead of
0.15 ms. This is the behaviour that led to dropping
`es_instance_position` in #10626.

- Follow-up for #12703

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5 <[email protected]>
Co-authored-by: Copilot Autofix powered by AI <[email protected]>
2026-09-22 17:09:42 +02:00

214 lines
4.9 KiB
Go

package repository
import (
"reflect"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/zitadel/zitadel/internal/eventstore"
"github.com/zitadel/zitadel/internal/zerrors"
)
func TestNewFilter(t *testing.T) {
type args struct {
field Field
value interface{}
operation Operation
}
tests := []struct {
name string
args args
want *Filter
}{
{
name: "aggregateID equals",
args: args{
field: FieldAggregateID,
value: "hodor",
operation: OperationEquals,
},
want: &Filter{Field: FieldAggregateID, Operation: OperationEquals, Value: "hodor"},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
if got := NewFilter(tt.args.field, tt.args.value, tt.args.operation); !reflect.DeepEqual(got, tt.want) {
t.Errorf("NewFilter() = %v, want %v", got, tt.want)
}
})
}
}
func TestFilter_Validate(t *testing.T) {
type fields struct {
field Field
value interface{}
operation Operation
isNil bool
}
tests := []struct {
name string
fields fields
wantErr bool
}{
{
name: "correct filter",
fields: fields{
field: FieldSequence,
operation: OperationGreater,
value: uint64(235),
},
wantErr: false,
},
{
name: "filter is nil",
fields: fields{isNil: true},
wantErr: true,
},
{
name: "no field error",
fields: fields{
operation: OperationGreater,
value: uint64(235),
},
wantErr: true,
},
{
name: "no value error",
fields: fields{
field: FieldSequence,
operation: OperationGreater,
},
wantErr: true,
},
{
name: "no operation error",
fields: fields{
field: FieldSequence,
value: uint64(235),
},
wantErr: true,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
var f *Filter
if !tt.fields.isNil {
f = &Filter{
Field: tt.fields.field,
Value: tt.fields.value,
Operation: tt.fields.operation,
}
}
if err := f.Validate(); (err != nil) != tt.wantErr {
t.Errorf("Filter.Validate() error = %v, wantErr %v", err, tt.wantErr)
}
})
}
}
func TestColumns_Validate(t *testing.T) {
type fields struct {
columns eventstore.Columns
}
tests := []struct {
name string
fields fields
wantErr bool
}{
{
name: "correct filter",
fields: fields{
columns: eventstore.ColumnsEvent,
},
wantErr: false,
},
{
name: "columns too low",
fields: fields{
columns: 0,
},
wantErr: true,
},
{
name: "columns too high",
fields: fields{
columns: 100,
},
wantErr: true,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
if err := tt.fields.columns.Validate(); (err != nil) != tt.wantErr {
t.Errorf("Columns.Validate() error = %v, wantErr %v", err, tt.wantErr)
}
})
}
}
func TestQueryFromBuilder_eventTypeScans(t *testing.T) {
base := func() *eventstore.SearchQueryBuilder {
return eventstore.NewSearchQueryBuilder(eventstore.ColumnsEvent).ScanEventTypesSeparately()
}
errScan := zerrors.ThrowInvalidArgument(nil, "REPO-Ahx6e", "")
tests := []struct {
name string
builder *eventstore.SearchQueryBuilder
want []EventTypeScan
wantErr error
}{
{
name: "not requested",
builder: eventstore.NewSearchQueryBuilder(eventstore.ColumnsEvent).
AddQuery().AggregateTypes("user").EventTypes("user.locked").Builder(),
want: nil,
},
{
name: "event types of each sub query, sorted and distinct",
builder: base().
AddQuery().AggregateTypes("user").EventTypes("user.locked", "user.removed").Builder().
AddQuery().AggregateTypes("org").EventTypes("org.removed").Builder().
AddQuery().AggregateTypes("user").EventTypes("user.locked").Builder(),
want: []EventTypeScan{
{AggregateType: "org", EventType: "org.removed"},
{AggregateType: "user", EventType: "user.locked"},
{AggregateType: "user", EventType: "user.removed"},
},
},
{
name: "sub query with several aggregate types",
builder: base().AddQuery().AggregateTypes("user", "org").EventTypes("user.locked", "org.removed").Builder(),
wantErr: errScan,
},
{
name: "sub query without event types",
builder: base().AddQuery().AggregateTypes("user").Builder(),
wantErr: errScan,
},
{
name: "sub query with aggregate ids",
builder: base().AddQuery().AggregateTypes("user").AggregateIDs("id").EventTypes("user.locked").Builder(),
wantErr: errScan,
},
{
name: "sub query with event data",
builder: base().AddQuery().AggregateTypes("user").EventTypes("user.locked").EventData(map[string]any{"key": "value"}).Builder(),
wantErr: errScan,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
query, err := QueryFromBuilder(tt.builder)
if tt.wantErr != nil {
require.ErrorIs(t, err, tt.wantErr)
return
}
require.NoError(t, err)
assert.Equal(t, tt.want, query.EventTypeScans)
})
}
}