mirror of
https://github.com/andwati/zitadel.git
synced 2026-10-18 05:52:51 +00:00
<!-- CURSOR_AGENT_PR_BODY_BEGIN -->
# Which Problems Are Solved
Bulk unique-constraint deletes (org, IDP with many links, user, project)
reconstruct every uniqueness key in Go and emit N `DELETE`s. That can
OOM on large tenants.
A one-shot owner backfill is also not enough: during a rolling deploy,
older binaries still insert `owners = '{}'`, so a later owner-only
delete would miss those rows.
Owner-only org/user/grant delete only matches rows whose `owners` bag
contains that tag. Several grant and member types stamped the aggregate
resource owner, which is not always the user’s home org or the granted
org. Username-scope rewrites also dropped `user:` tags, so owner-only
user delete could miss rewritten username rows.
Command must not switch to owner-only delete on the first 79 of a
version, because leftover `'{}'` rows from rolling-update overlap are
only backfilled on a later 79. A failed 79 must not turn owner-delete
on, and backfill joins must not stamp the wrong org (unverified domains,
org-scoped usernames colliding with instance-scoped ones).
# How the Problems Are Solved
Uniqueness stays the existing primary key `(instance_id, unique_type,
unique_field)`. `owners` is only a lifecycle bag of tags (`org:<id>`,
`user:<id>`, `idp:<id>`, `project:<id>`, `grant:<id>`).
- Add `owners TEXT[] NOT NULL DEFAULT '{}'` on
`eventstore.unique_constraints` (init SQL for greenfield, setup 78 for
existing clusters).
- Create `unique_constraints_owners_gin` with `CREATE INDEX
CONCURRENTLY` **outside a transaction** (same pattern as step 76).
Built-in `GIN (owners)` only; no `btree_gin`.
- Stamp `WithOwners` on **add** events. Field-level removes ignore
owners. `RemoveByOwner` stores the tag in `Owners` and both v1/v3 and v2
pushers run `DELETE … WHERE instance_id = $1 AND owners @>
ARRAY[$tag]::text[]`.
- Setup 79 backfills `owners` from projections via numbered
`cmd/setup/79/*.sql` files. Table names are interpolated from
`internal/query/projection` constants. Matches use `unique_field IN (x,
lower(x))` so the primary key can be used. Missing projection tables
(`42P01`) are skipped.
- Org domains are tagged only when `is_verified`. Instance-scoped
usernames join only users that are not org-scoped (organization settings
or domain policy `user_login_must_be_domain`). Every username row still
gets `org:` + `user:` of that user.
- 79 is a **repeatable** migration. `Check` re-runs when
`lastRun["version"]` changed or when setup-only `ForceFinalize` is set
and `lastRun["finalized"]` is still false. `Finalized` is written only
after Execute succeeds.
- Failed setup events keep the last successful `lastRun`. Command reads
`finalized` only from `system.migration.repeatable.done`.
- Org / user / project remove skip N-key event replay **when that
`finalized` bit is true**. Otherwise they keep today’s N-key lists
**and** still emit `RemoveByOwner`. The “ready” result is cached in
memory once true.
Setup writes `finalized` on the 79 repeatable.done payload:
- First 79 of a version: backfill only, `finalized=false` (unless
`ForceFinalize` is already set).
- Later 79: version change or
`BackfillUniqueConstraintOwners.ForceFinalize` /
`ZITADEL_BACKFILLUNIQUECONSTRAINTOWNERS_FORCEFINALIZE` after the rolling
update is done. That run backfills leftovers and writes
`finalized=true`.
Member and grant owner bags now include every org that should delete the
row:
- Instance, org, and project members also stamp
`org:<user_resource_owner>`.
- User grants stamp grant, user, project, and granted orgs.
- Project grant members stamp the project org, user org, and granted
org.
- Project grants stamp both orgs plus `grant:<id>`. Grant remove emits
`RemoveByOwner(grant)`.
- Username-scope rewrites keep `user:<id>` as well as `org:`.
- User/grant orgs are carried as unexported add-event fields (not
persisted JSON). `WithOwners` skips empty and duplicate tags.
# Additional Changes
- Table-driven owner-tag tests, including project grant (two `org:` tags
plus `grant:`), cross-org members, user grants, and username-scope
rewrite `user:` tags.
- Table-driven 79 `Check` + `finalizeAfterSuccess` tests for version
change, sequential version bumps, ForceFinalize restart, and
already-finalized skip.
- Command tests: a failed 79 event with `finalized=true` does not enable
owner-delete.
- Integration fixtures (verified vs unverified org domain;
instance-scoped `bob` vs org-scoped `bob`).
- Instance-only and global unique types stay untagged (`'{}'`). Instance
wipe still uses `UniqueConstraintInstanceRemove`.
- Pusher owner cases live in `eventstore_pusher_owners_test.go` so
`eventstore_pusher_test.go` stays under 1k lines.
# Additional Context
A later cleanup can delete the N-key replay helpers once a release that
ran 79 a second time (or an operator who set `ForceFinalize`) is in
production. That is not required for correctness.
<!-- CURSOR_AGENT_PR_BODY_END -->
<div><a
href="https://cursor.com/agents/bc-44f9bc9b-9a93-4cd9-8dd1-663a7b9aad2a?cursor_ref=pr_footer&cursor_cta=open_in_web"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://cursor.com/assets/images/open-in-web-dark.png"><source
media="(prefers-color-scheme: light)"
srcset="https://cursor.com/assets/images/open-in-web-light.png"><img
alt="Open in Web" width="114" height="28"
src="https://cursor.com/assets/images/open-in-web-dark.png"></picture></a> <a
href="https://cursor.com/background-agent?bcId=bc-44f9bc9b-9a93-4cd9-8dd1-663a7b9aad2a&cursor_ref=pr_footer&cursor_cta=open_in_cursor"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://cursor.com/assets/images/open-in-cursor-dark.png"><source
media="(prefers-color-scheme: light)"
srcset="https://cursor.com/assets/images/open-in-cursor-light.png"><img
alt="Open in Cursor" width="131" height="28"
src="https://cursor.com/assets/images/open-in-cursor-dark.png"></picture></a> </div>
---------
Co-authored-by: Cursor Agent <[email protected]>
Co-authored-by: Silvan <[email protected]>
214 lines
9.0 KiB
Go
214 lines
9.0 KiB
Go
package setup
|
|
|
|
import (
|
|
"encoding/json"
|
|
"strings"
|
|
"testing"
|
|
|
|
"github.com/stretchr/testify/assert"
|
|
"github.com/stretchr/testify/require"
|
|
|
|
"github.com/zitadel/zitadel/internal/eventstore"
|
|
"github.com/zitadel/zitadel/internal/migration"
|
|
"github.com/zitadel/zitadel/internal/query/projection"
|
|
)
|
|
|
|
var _ migration.RepeatableMigration = (*BackfillUniqueConstraintOwners)(nil)
|
|
|
|
func TestBackfillUniqueConstraintOwnersStmts(t *testing.T) {
|
|
statements, err := backfillUniqueConstraintOwnersStatements()
|
|
require.NoError(t, err)
|
|
|
|
want := []struct {
|
|
file string
|
|
uniqueType string
|
|
field string
|
|
table string
|
|
extra []string
|
|
}{
|
|
{"01_usernames_org_scoped.sql", "usernames", "u.username || u.resource_owner", projection.UserTable, []string{"'org:' || u.resource_owner"}},
|
|
{"02_usernames.sql", "usernames", "u.username, lower(u.username)", projection.UserTable, []string{"'org:' || u.resource_owner", "organization_scoped_usernames", "user_login_must_be_domain"}},
|
|
{"03_external_idps.sql", "external_idps", "l.idp_id || l.external_user_id", projection.IDPUserLinkTable, []string{"'idp:' || l.idp_id"}},
|
|
{"04_org_name.sql", "org_name", "o.name", projection.OrgProjectionTable, nil},
|
|
{"05_org_domain.sql", "org_domain", "d.domain", projection.OrgDomainTable, []string{"d.is_verified"}},
|
|
{"06_project_names.sql", "project_names", "p.name || p.resource_owner", projection.ProjectProjectionTable, nil},
|
|
{"07_appname.sql", "appname", "a.name || ':' || a.project_id", projection.AppProjectionTable, nil},
|
|
{"08_project_role.sql", "project_role", "r.role_key || ':' || r.project_id", projection.ProjectRoleProjectionTable, nil},
|
|
{"09_entity_ids.sql", "entity_ids", "s.entity_id", projection.AppProjectionTable, []string{projection.AppSAMLTable}},
|
|
{"10_project_grant.sql", "project_grant", "g.granted_org_id || ':' || g.project_id", projection.ProjectGrantProjectionTable, []string{"'org:' || g.granted_org_id", "'grant:' || g.grant_id"}},
|
|
{"11_project_grant_member.sql", "project_grant_member", "m.project_id || ':' || m.user_id || ':' || m.grant_id", projection.ProjectGrantMemberProjectionTable, []string{"'grant:' || m.grant_id", "m.user_resource_owner", "m.granted_org"}},
|
|
{"12_user_grant.sql", "user_grant", "g.resource_owner || ':' || g.user_id || ':' || g.project_id || ':' || g.grant_id", projection.UserGrantProjectionTable, []string{"'grant:' || g.grant_id", "g.resource_owner_user", "g.granted_org"}},
|
|
{"13_member_org.sql", "member", "m.org_id || ':' || m.user_id", projection.OrgMemberProjectionTable, []string{"m.user_resource_owner"}},
|
|
{"14_member_project.sql", "member", "m.project_id || ':' || m.user_id", projection.ProjectMemberProjectionTable, []string{"m.user_resource_owner"}},
|
|
{"15_member_instance.sql", "member", "m.id || ':' || m.user_id", projection.InstanceMemberProjectionTable, []string{"m.user_resource_owner"}},
|
|
{"16_group_name.sql", "group_name", "g.name || ':' || g.resource_owner", projection.GroupProjectionTable, nil},
|
|
{"17_action_names.sql", "action_names", "a.name || ':' || a.resource_owner", projection.ActionTable, nil},
|
|
{"18_idp_config_names.sql", "idp_config_names", "i.name || i.resource_owner", projection.IDPTable, []string{"'idp:' || i.id"}},
|
|
{"19_idp_config_names_templates.sql", "idp_config_names", "t.name || t.resource_owner", projection.IDPTemplateTable, []string{"'idp:' || t.id"}},
|
|
{"20_mail_text.sql", "mail_text", "t.aggregate_id || ':' || t.type || ':' || t.language", projection.MessageTextTable, []string{"t.aggregate_id <> t.instance_id"}},
|
|
}
|
|
|
|
require.Len(t, statements, len(want))
|
|
for i, stmt := range statements {
|
|
assert.Equal(t, want[i].file, stmt.file)
|
|
assert.Contains(t, stmt.query, "unique_type = '"+want[i].uniqueType+"'")
|
|
assert.Contains(t, stmt.query, want[i].field)
|
|
assert.Contains(t, stmt.query, want[i].table)
|
|
assert.Contains(t, stmt.query, "uc.unique_field IN (")
|
|
assert.NotContains(t, stmt.query, "lower(uc.unique_field)")
|
|
assert.NotContains(t, stmt.query, "{{.")
|
|
for _, extra := range want[i].extra {
|
|
assert.Contains(t, stmt.query, extra)
|
|
}
|
|
}
|
|
}
|
|
|
|
func TestBackfillUniqueConstraintOwners_CheckAndFinalize(t *testing.T) {
|
|
tests := []struct {
|
|
name string
|
|
forceFinalize bool
|
|
lastRun map[string]interface{}
|
|
wantRun bool
|
|
wantFinalized bool
|
|
}{
|
|
{
|
|
name: "missing lastRun runs without finalizing",
|
|
lastRun: nil,
|
|
wantRun: true,
|
|
wantFinalized: false,
|
|
},
|
|
{
|
|
name: "empty lastRun runs without finalizing",
|
|
lastRun: map[string]interface{}{},
|
|
wantRun: true,
|
|
wantFinalized: false,
|
|
},
|
|
{
|
|
name: "missing lastRun with ForceFinalize finalizes",
|
|
forceFinalize: true,
|
|
lastRun: nil,
|
|
wantRun: true,
|
|
wantFinalized: true,
|
|
},
|
|
{
|
|
name: "same version not finalized skips",
|
|
lastRun: map[string]interface{}{"version": "v2.0.0", "finalized": false},
|
|
wantRun: false,
|
|
wantFinalized: false,
|
|
},
|
|
{
|
|
name: "same version missing finalized skips",
|
|
lastRun: map[string]interface{}{"version": "v2.0.0"},
|
|
wantRun: false,
|
|
wantFinalized: false,
|
|
},
|
|
{
|
|
name: "same version not finalized with ForceFinalize runs and finalizes",
|
|
forceFinalize: true,
|
|
lastRun: map[string]interface{}{"version": "v2.0.0", "finalized": false},
|
|
wantRun: true,
|
|
wantFinalized: true,
|
|
},
|
|
{
|
|
name: "same version already finalized skips",
|
|
lastRun: map[string]interface{}{"version": "v2.0.0", "finalized": true},
|
|
wantRun: false,
|
|
wantFinalized: true,
|
|
},
|
|
{
|
|
name: "same version already finalized with ForceFinalize skips",
|
|
forceFinalize: true,
|
|
lastRun: map[string]interface{}{"version": "v2.0.0", "finalized": true},
|
|
wantRun: false,
|
|
wantFinalized: true,
|
|
},
|
|
{
|
|
name: "version changed runs and finalizes",
|
|
lastRun: map[string]interface{}{"version": "v1.0.0", "finalized": false},
|
|
wantRun: true,
|
|
wantFinalized: true,
|
|
},
|
|
{
|
|
name: "version changed keeps finalized true",
|
|
lastRun: map[string]interface{}{"version": "v1.0.0", "finalized": true},
|
|
wantRun: true,
|
|
wantFinalized: true,
|
|
},
|
|
}
|
|
|
|
for _, tt := range tests {
|
|
t.Run(tt.name, func(t *testing.T) {
|
|
mig := &BackfillUniqueConstraintOwners{Version: "v2.0.0", ForceFinalize: tt.forceFinalize}
|
|
assert.Equal(t, tt.wantRun, mig.Check(tt.lastRun))
|
|
assert.False(t, mig.Finalized, "Check must not set finalized before Execute")
|
|
mig.finalizeAfterSuccess()
|
|
assert.Equal(t, tt.wantFinalized, mig.Finalized)
|
|
})
|
|
}
|
|
}
|
|
|
|
func TestBackfillUniqueConstraintOwners_SequentialVersionChanges(t *testing.T) {
|
|
mig := &BackfillUniqueConstraintOwners{Version: "v1.0.0"}
|
|
require.True(t, mig.Check(nil))
|
|
mig.finalizeAfterSuccess()
|
|
assert.False(t, mig.Finalized)
|
|
|
|
lastRun := map[string]interface{}{"version": mig.Version, "finalized": mig.Finalized}
|
|
mig = &BackfillUniqueConstraintOwners{Version: "v2.0.0"}
|
|
require.True(t, mig.Check(lastRun))
|
|
mig.finalizeAfterSuccess()
|
|
assert.True(t, mig.Finalized)
|
|
|
|
lastRun = map[string]interface{}{"version": mig.Version, "finalized": mig.Finalized}
|
|
mig = &BackfillUniqueConstraintOwners{Version: "v3.0.0"}
|
|
require.True(t, mig.Check(lastRun))
|
|
mig.finalizeAfterSuccess()
|
|
assert.True(t, mig.Finalized)
|
|
}
|
|
|
|
func TestBackfillUniqueConstraintOwners_ForceFinalizeRestart(t *testing.T) {
|
|
mig := &BackfillUniqueConstraintOwners{Version: "v1.0.0"}
|
|
require.True(t, mig.Check(nil))
|
|
mig.finalizeAfterSuccess()
|
|
require.False(t, mig.Finalized)
|
|
|
|
lastRun := map[string]interface{}{"version": "v1.0.0", "finalized": false}
|
|
mig = &BackfillUniqueConstraintOwners{Version: "v1.0.0", ForceFinalize: true}
|
|
require.True(t, mig.Check(lastRun), "restart with ForceFinalize should re-run")
|
|
assert.False(t, mig.Finalized, "Check must not set finalized before Execute")
|
|
mig.finalizeAfterSuccess()
|
|
assert.True(t, mig.Finalized)
|
|
|
|
lastRun = map[string]interface{}{"version": "v1.0.0", "finalized": true}
|
|
mig = &BackfillUniqueConstraintOwners{Version: "v1.0.0", ForceFinalize: true}
|
|
assert.False(t, mig.Check(lastRun), "later restart with ForceFinalize should skip")
|
|
}
|
|
|
|
func TestUniqueTypesWithOwnersOmitsMailText(t *testing.T) {
|
|
assert.NotContains(t, eventstore.UniqueTypesWithOwners, "mail_text")
|
|
assert.Contains(t, eventstore.UniqueTypesWithOwners, "usernames")
|
|
assert.Contains(t, eventstore.UniqueTypesWithOwners, "idp_config_names")
|
|
}
|
|
|
|
func TestBackfillUniqueConstraintOwnersJSONOmitsForceFinalize(t *testing.T) {
|
|
mig := &BackfillUniqueConstraintOwners{
|
|
Version: "v2.0.0",
|
|
Finalized: true,
|
|
ForceFinalize: true,
|
|
lastVersion: "v1.0.0",
|
|
lastFinalized: true,
|
|
}
|
|
data, err := json.Marshal(mig)
|
|
require.NoError(t, err)
|
|
|
|
var payload map[string]any
|
|
require.NoError(t, json.Unmarshal(data, &payload))
|
|
assert.Equal(t, "v2.0.0", payload["version"])
|
|
assert.Equal(t, true, payload["finalized"])
|
|
assert.NotContains(t, payload, "ForceFinalize")
|
|
assert.NotContains(t, payload, "forceFinalize")
|
|
assert.NotContains(t, payload, "lastVersion")
|
|
assert.NotContains(t, payload, "lastFinalized")
|
|
assert.False(t, strings.Contains(string(data), "v1.0.0"))
|
|
}
|