Files
3ff4fbc7e4 fix(eventstore): add owners to unique constraints (#12740)
<!-- CURSOR_AGENT_PR_BODY_BEGIN -->
# Which Problems Are Solved

Bulk unique-constraint deletes (org, IDP with many links, user, project)
reconstruct every uniqueness key in Go and emit N `DELETE`s. That can
OOM on large tenants.

A one-shot owner backfill is also not enough: during a rolling deploy,
older binaries still insert `owners = '{}'`, so a later owner-only
delete would miss those rows.

Owner-only org/user/grant delete only matches rows whose `owners` bag
contains that tag. Several grant and member types stamped the aggregate
resource owner, which is not always the user’s home org or the granted
org. Username-scope rewrites also dropped `user:` tags, so owner-only
user delete could miss rewritten username rows.

Command must not switch to owner-only delete on the first 79 of a
version, because leftover `'{}'` rows from rolling-update overlap are
only backfilled on a later 79. A failed 79 must not turn owner-delete
on, and backfill joins must not stamp the wrong org (unverified domains,
org-scoped usernames colliding with instance-scoped ones).

# How the Problems Are Solved

Uniqueness stays the existing primary key `(instance_id, unique_type,
unique_field)`. `owners` is only a lifecycle bag of tags (`org:<id>`,
`user:<id>`, `idp:<id>`, `project:<id>`, `grant:<id>`).

- Add `owners TEXT[] NOT NULL DEFAULT '{}'` on
`eventstore.unique_constraints` (init SQL for greenfield, setup 78 for
existing clusters).
- Create `unique_constraints_owners_gin` with `CREATE INDEX
CONCURRENTLY` **outside a transaction** (same pattern as step 76).
Built-in `GIN (owners)` only; no `btree_gin`.
- Stamp `WithOwners` on **add** events. Field-level removes ignore
owners. `RemoveByOwner` stores the tag in `Owners` and both v1/v3 and v2
pushers run `DELETE … WHERE instance_id = $1 AND owners @>
ARRAY[$tag]::text[]`.
- Setup 79 backfills `owners` from projections via numbered
`cmd/setup/79/*.sql` files. Table names are interpolated from
`internal/query/projection` constants. Matches use `unique_field IN (x,
lower(x))` so the primary key can be used. Missing projection tables
(`42P01`) are skipped.
- Org domains are tagged only when `is_verified`. Instance-scoped
usernames join only users that are not org-scoped (organization settings
or domain policy `user_login_must_be_domain`). Every username row still
gets `org:` + `user:` of that user.
- 79 is a **repeatable** migration. `Check` re-runs when
`lastRun["version"]` changed or when setup-only `ForceFinalize` is set
and `lastRun["finalized"]` is still false. `Finalized` is written only
after Execute succeeds.
- Failed setup events keep the last successful `lastRun`. Command reads
`finalized` only from `system.migration.repeatable.done`.
- Org / user / project remove skip N-key event replay **when that
`finalized` bit is true**. Otherwise they keep today’s N-key lists
**and** still emit `RemoveByOwner`. The “ready” result is cached in
memory once true.

Setup writes `finalized` on the 79 repeatable.done payload:

- First 79 of a version: backfill only, `finalized=false` (unless
`ForceFinalize` is already set).
- Later 79: version change or
`BackfillUniqueConstraintOwners.ForceFinalize` /
`ZITADEL_BACKFILLUNIQUECONSTRAINTOWNERS_FORCEFINALIZE` after the rolling
update is done. That run backfills leftovers and writes
`finalized=true`.

Member and grant owner bags now include every org that should delete the
row:

- Instance, org, and project members also stamp
`org:<user_resource_owner>`.
- User grants stamp grant, user, project, and granted orgs.
- Project grant members stamp the project org, user org, and granted
org.
- Project grants stamp both orgs plus `grant:<id>`. Grant remove emits
`RemoveByOwner(grant)`.
- Username-scope rewrites keep `user:<id>` as well as `org:`.
- User/grant orgs are carried as unexported add-event fields (not
persisted JSON). `WithOwners` skips empty and duplicate tags.

# Additional Changes

- Table-driven owner-tag tests, including project grant (two `org:` tags
plus `grant:`), cross-org members, user grants, and username-scope
rewrite `user:` tags.
- Table-driven 79 `Check` + `finalizeAfterSuccess` tests for version
change, sequential version bumps, ForceFinalize restart, and
already-finalized skip.
- Command tests: a failed 79 event with `finalized=true` does not enable
owner-delete.
- Integration fixtures (verified vs unverified org domain;
instance-scoped `bob` vs org-scoped `bob`).
- Instance-only and global unique types stay untagged (`'{}'`). Instance
wipe still uses `UniqueConstraintInstanceRemove`.
- Pusher owner cases live in `eventstore_pusher_owners_test.go` so
`eventstore_pusher_test.go` stays under 1k lines.

# Additional Context

A later cleanup can delete the N-key replay helpers once a release that
ran 79 a second time (or an operator who set `ForceFinalize`) is in
production. That is not required for correctness.

<!-- CURSOR_AGENT_PR_BODY_END -->

<div><a
href="https://cursor.com/agents/bc-44f9bc9b-9a93-4cd9-8dd1-663a7b9aad2a?cursor_ref=pr_footer&cursor_cta=open_in_web"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://cursor.com/assets/images/open-in-web-dark.png"><source
media="(prefers-color-scheme: light)"
srcset="https://cursor.com/assets/images/open-in-web-light.png"><img
alt="Open in Web" width="114" height="28"
src="https://cursor.com/assets/images/open-in-web-dark.png"></picture></a>&nbsp;<a
href="https://cursor.com/background-agent?bcId=bc-44f9bc9b-9a93-4cd9-8dd1-663a7b9aad2a&cursor_ref=pr_footer&cursor_cta=open_in_cursor"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://cursor.com/assets/images/open-in-cursor-dark.png"><source
media="(prefers-color-scheme: light)"
srcset="https://cursor.com/assets/images/open-in-cursor-light.png"><img
alt="Open in Cursor" width="131" height="28"
src="https://cursor.com/assets/images/open-in-cursor-dark.png"></picture></a>&nbsp;</div>

---------

Co-authored-by: Cursor Agent <[email protected]>
Co-authored-by: Silvan <[email protected]>
2026-09-18 15:08:42 +02:00

214 lines
9.0 KiB
Go

package setup
import (
"encoding/json"
"strings"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/zitadel/zitadel/internal/eventstore"
"github.com/zitadel/zitadel/internal/migration"
"github.com/zitadel/zitadel/internal/query/projection"
)
var _ migration.RepeatableMigration = (*BackfillUniqueConstraintOwners)(nil)
func TestBackfillUniqueConstraintOwnersStmts(t *testing.T) {
statements, err := backfillUniqueConstraintOwnersStatements()
require.NoError(t, err)
want := []struct {
file string
uniqueType string
field string
table string
extra []string
}{
{"01_usernames_org_scoped.sql", "usernames", "u.username || u.resource_owner", projection.UserTable, []string{"'org:' || u.resource_owner"}},
{"02_usernames.sql", "usernames", "u.username, lower(u.username)", projection.UserTable, []string{"'org:' || u.resource_owner", "organization_scoped_usernames", "user_login_must_be_domain"}},
{"03_external_idps.sql", "external_idps", "l.idp_id || l.external_user_id", projection.IDPUserLinkTable, []string{"'idp:' || l.idp_id"}},
{"04_org_name.sql", "org_name", "o.name", projection.OrgProjectionTable, nil},
{"05_org_domain.sql", "org_domain", "d.domain", projection.OrgDomainTable, []string{"d.is_verified"}},
{"06_project_names.sql", "project_names", "p.name || p.resource_owner", projection.ProjectProjectionTable, nil},
{"07_appname.sql", "appname", "a.name || ':' || a.project_id", projection.AppProjectionTable, nil},
{"08_project_role.sql", "project_role", "r.role_key || ':' || r.project_id", projection.ProjectRoleProjectionTable, nil},
{"09_entity_ids.sql", "entity_ids", "s.entity_id", projection.AppProjectionTable, []string{projection.AppSAMLTable}},
{"10_project_grant.sql", "project_grant", "g.granted_org_id || ':' || g.project_id", projection.ProjectGrantProjectionTable, []string{"'org:' || g.granted_org_id", "'grant:' || g.grant_id"}},
{"11_project_grant_member.sql", "project_grant_member", "m.project_id || ':' || m.user_id || ':' || m.grant_id", projection.ProjectGrantMemberProjectionTable, []string{"'grant:' || m.grant_id", "m.user_resource_owner", "m.granted_org"}},
{"12_user_grant.sql", "user_grant", "g.resource_owner || ':' || g.user_id || ':' || g.project_id || ':' || g.grant_id", projection.UserGrantProjectionTable, []string{"'grant:' || g.grant_id", "g.resource_owner_user", "g.granted_org"}},
{"13_member_org.sql", "member", "m.org_id || ':' || m.user_id", projection.OrgMemberProjectionTable, []string{"m.user_resource_owner"}},
{"14_member_project.sql", "member", "m.project_id || ':' || m.user_id", projection.ProjectMemberProjectionTable, []string{"m.user_resource_owner"}},
{"15_member_instance.sql", "member", "m.id || ':' || m.user_id", projection.InstanceMemberProjectionTable, []string{"m.user_resource_owner"}},
{"16_group_name.sql", "group_name", "g.name || ':' || g.resource_owner", projection.GroupProjectionTable, nil},
{"17_action_names.sql", "action_names", "a.name || ':' || a.resource_owner", projection.ActionTable, nil},
{"18_idp_config_names.sql", "idp_config_names", "i.name || i.resource_owner", projection.IDPTable, []string{"'idp:' || i.id"}},
{"19_idp_config_names_templates.sql", "idp_config_names", "t.name || t.resource_owner", projection.IDPTemplateTable, []string{"'idp:' || t.id"}},
{"20_mail_text.sql", "mail_text", "t.aggregate_id || ':' || t.type || ':' || t.language", projection.MessageTextTable, []string{"t.aggregate_id <> t.instance_id"}},
}
require.Len(t, statements, len(want))
for i, stmt := range statements {
assert.Equal(t, want[i].file, stmt.file)
assert.Contains(t, stmt.query, "unique_type = '"+want[i].uniqueType+"'")
assert.Contains(t, stmt.query, want[i].field)
assert.Contains(t, stmt.query, want[i].table)
assert.Contains(t, stmt.query, "uc.unique_field IN (")
assert.NotContains(t, stmt.query, "lower(uc.unique_field)")
assert.NotContains(t, stmt.query, "{{.")
for _, extra := range want[i].extra {
assert.Contains(t, stmt.query, extra)
}
}
}
func TestBackfillUniqueConstraintOwners_CheckAndFinalize(t *testing.T) {
tests := []struct {
name string
forceFinalize bool
lastRun map[string]interface{}
wantRun bool
wantFinalized bool
}{
{
name: "missing lastRun runs without finalizing",
lastRun: nil,
wantRun: true,
wantFinalized: false,
},
{
name: "empty lastRun runs without finalizing",
lastRun: map[string]interface{}{},
wantRun: true,
wantFinalized: false,
},
{
name: "missing lastRun with ForceFinalize finalizes",
forceFinalize: true,
lastRun: nil,
wantRun: true,
wantFinalized: true,
},
{
name: "same version not finalized skips",
lastRun: map[string]interface{}{"version": "v2.0.0", "finalized": false},
wantRun: false,
wantFinalized: false,
},
{
name: "same version missing finalized skips",
lastRun: map[string]interface{}{"version": "v2.0.0"},
wantRun: false,
wantFinalized: false,
},
{
name: "same version not finalized with ForceFinalize runs and finalizes",
forceFinalize: true,
lastRun: map[string]interface{}{"version": "v2.0.0", "finalized": false},
wantRun: true,
wantFinalized: true,
},
{
name: "same version already finalized skips",
lastRun: map[string]interface{}{"version": "v2.0.0", "finalized": true},
wantRun: false,
wantFinalized: true,
},
{
name: "same version already finalized with ForceFinalize skips",
forceFinalize: true,
lastRun: map[string]interface{}{"version": "v2.0.0", "finalized": true},
wantRun: false,
wantFinalized: true,
},
{
name: "version changed runs and finalizes",
lastRun: map[string]interface{}{"version": "v1.0.0", "finalized": false},
wantRun: true,
wantFinalized: true,
},
{
name: "version changed keeps finalized true",
lastRun: map[string]interface{}{"version": "v1.0.0", "finalized": true},
wantRun: true,
wantFinalized: true,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
mig := &BackfillUniqueConstraintOwners{Version: "v2.0.0", ForceFinalize: tt.forceFinalize}
assert.Equal(t, tt.wantRun, mig.Check(tt.lastRun))
assert.False(t, mig.Finalized, "Check must not set finalized before Execute")
mig.finalizeAfterSuccess()
assert.Equal(t, tt.wantFinalized, mig.Finalized)
})
}
}
func TestBackfillUniqueConstraintOwners_SequentialVersionChanges(t *testing.T) {
mig := &BackfillUniqueConstraintOwners{Version: "v1.0.0"}
require.True(t, mig.Check(nil))
mig.finalizeAfterSuccess()
assert.False(t, mig.Finalized)
lastRun := map[string]interface{}{"version": mig.Version, "finalized": mig.Finalized}
mig = &BackfillUniqueConstraintOwners{Version: "v2.0.0"}
require.True(t, mig.Check(lastRun))
mig.finalizeAfterSuccess()
assert.True(t, mig.Finalized)
lastRun = map[string]interface{}{"version": mig.Version, "finalized": mig.Finalized}
mig = &BackfillUniqueConstraintOwners{Version: "v3.0.0"}
require.True(t, mig.Check(lastRun))
mig.finalizeAfterSuccess()
assert.True(t, mig.Finalized)
}
func TestBackfillUniqueConstraintOwners_ForceFinalizeRestart(t *testing.T) {
mig := &BackfillUniqueConstraintOwners{Version: "v1.0.0"}
require.True(t, mig.Check(nil))
mig.finalizeAfterSuccess()
require.False(t, mig.Finalized)
lastRun := map[string]interface{}{"version": "v1.0.0", "finalized": false}
mig = &BackfillUniqueConstraintOwners{Version: "v1.0.0", ForceFinalize: true}
require.True(t, mig.Check(lastRun), "restart with ForceFinalize should re-run")
assert.False(t, mig.Finalized, "Check must not set finalized before Execute")
mig.finalizeAfterSuccess()
assert.True(t, mig.Finalized)
lastRun = map[string]interface{}{"version": "v1.0.0", "finalized": true}
mig = &BackfillUniqueConstraintOwners{Version: "v1.0.0", ForceFinalize: true}
assert.False(t, mig.Check(lastRun), "later restart with ForceFinalize should skip")
}
func TestUniqueTypesWithOwnersOmitsMailText(t *testing.T) {
assert.NotContains(t, eventstore.UniqueTypesWithOwners, "mail_text")
assert.Contains(t, eventstore.UniqueTypesWithOwners, "usernames")
assert.Contains(t, eventstore.UniqueTypesWithOwners, "idp_config_names")
}
func TestBackfillUniqueConstraintOwnersJSONOmitsForceFinalize(t *testing.T) {
mig := &BackfillUniqueConstraintOwners{
Version: "v2.0.0",
Finalized: true,
ForceFinalize: true,
lastVersion: "v1.0.0",
lastFinalized: true,
}
data, err := json.Marshal(mig)
require.NoError(t, err)
var payload map[string]any
require.NoError(t, json.Unmarshal(data, &payload))
assert.Equal(t, "v2.0.0", payload["version"])
assert.Equal(t, true, payload["finalized"])
assert.NotContains(t, payload, "ForceFinalize")
assert.NotContains(t, payload, "forceFinalize")
assert.NotContains(t, payload, "lastVersion")
assert.NotContains(t, payload, "lastFinalized")
assert.False(t, strings.Contains(string(data), "v1.0.0"))
}