andwati/zitadel
1
0
Fork 0
mirror of https://github.com/andwati/zitadel.git synced 2026-10-18 05:52:51 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat: allow transactional table setup step to recreate the whole schema (#11833)

Browse Source 
## Problem description

- Setup currently creates/updates relational tables, but there is no
built-in way to fully reset the relational schema during iterative
development.
- Re-running setup after schema/projection changes can leave stale
relational objects and projection state behind, which makes local/dev
validation harder.
- There is no explicit, configurable switch in setup steps for
destructive schema recreation behavior.

## How the Problems Are Solved

- Adds a new setup step configuration section for relational tables with
a `ShouldRecreateSchema` flag (default `false`).
- Wires the new config flag into the transactional tables setup step.
- Extends the transactional tables execution logic to optionally:
  - Drop the `zitadel` schema with `CASCADE`
- Clean related projection state entries for relational tables in
`projections.current_states`
  - Recreate tables through the existing setup flow afterward
- Wraps the destructive operations in a transaction and keeps
error/rollback handling plus logging for visibility.

## Additional Changes

- Adds an explicit warning in step configuration that schema recreation
is intended for development and not production use.
- Keeps behavior fully backward-compatible by default
(`ShouldRecreateSchema: false`), so existing setups are unchanged unless
the flag is enabled.

## Additional Context

- Follow-up for relational setup/dev workflow improvements.
- PR: #11833

## How to use it

### Env `export ZITADEL_RELATIONALTABLES_SHOULDRECREATESCHEMA=true`

### Config

Add the following to your custom setup steps config:

```yaml
RelationalTables:
  ShouldRecreateSchema: true
```
This commit is contained in:
Silvan
2026-03-18 03:53:57 +00:00
committed by GitHub
parent a864434b4b
commit 6d90a120a6
5 changed files with 72 additions and 27 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CONTRIBUTING.md
+31 -25
View File
@@ -12,8 +12,8 @@ Zitadel is an open-source identity and access management platform built with a m
- [Cypress runtime dependencies](https://docs.cypress.io/guides/continuous-integration/introduction#Dependencies) - Required for Browser UI tests
<details>
<summary>WSL2 on Windows 10 users (click to expand)</summary>
For Cypress tests on WSL2, you may need to configure X11 forwarding. Following suggestions [here](https://stackoverflow.com/questions/62641553/setup-cypress-on-wsl-ubuntu-for-windows-10) and [here](https://github.com/microsoft/WSL/issues/4106). Use at your own risk.
For Cypress tests on WSL2, you may need to configure X11 forwarding. Following suggestions [on Stackoverflow](https://stackoverflow.com/questions/62641553/setup-cypress-on-wsl-ubuntu-for-windows-10) and [Github](https://github.com/microsoft/WSL/issues/4106). Use at your own risk.
1. Install `VcXsrv Windows X Server`
2. Set shortcut target to `"C:\Program Files\VcXsrv\xlaunch.exe" -ac`
@@ -34,7 +34,7 @@ Jump to the dedicated sections for developing a specific project:
- [Contributing to the API](#contribute-to-api)
- [Contributing to the Login](#contribute-to-login)
- [Contributing to the Management Console](#contribute-to-management-console)
- [Contributing to the Management Console](#contribute-to-the-management-console)
- [Contributing to the Docs](#contribute-to-docs)
- [Contributing translations](#contribute-translations)
@@ -49,11 +49,11 @@ You can build and start any project with Nx commands.
| **Develop** | `pnpm nx run PROJECT:dev` | Development server | |
| **Generate** | `pnpm nx run PROJECT:generate` | Generate .gitignored files | |
| **Generate Go Files** | `pnpm nx run @zitadel/api:generate-go` | Regenerate checked-in files | This is needed to generate files using [Stringer](https://pkg.go.dev/golang.org/x/tools/cmd/stringer), [Enumer](https://github.com/dmarkham/enumer) or [gomock](https://github.com/uber-go/mock) |
| **Install Proto Plugins** | `pnpm nx run @zitadel/api:generate-install` | Install proto toolchain | Installs Go-based plugins (protoc-gen-go, connect-go, …) to `.artifacts/bin/`. Run automatically by `generate` targets; Nx caches the outputs. |
| **Install Proto Plugins** | `pnpm nx run @zitadel/api:generate-install` | Install proto toolchain | Installs Go-based plugins (protoc-gen-go, connect-go, …) to `.artifacts/bin/`. Run automatically by `generate` targets; Nx caches the outputs. |
| **Test - Unit** | `pnpm nx run PROJECT:test-unit` | Run unit tests | |
| **Test - Integration** | `pnpm nx run PROJECT:test-integration` | Run integration tests | Learn more about how to [debug API integration tests](#run-api-integration-tests) |
| **Test - Integration** | `pnpm nx run PROJECT:test-integration` | Run integration tests | Learn more about how to [debug API integration tests](#run-api-integration-tests) |
| **Test - Integration Stop** | `pnpm nx run PROJECT:test-integration-stop` | Stop integration containers | |
| **Test - Functional UI** | `pnpm nx run @zitadel/functional-ui:test` | Run functional UI tests | Learn more about how to [develop the Management Console and opening the interactive Test Suite](#pass-management-console-quality-checks) |
| **Test - Functional UI** | `pnpm nx run @zitadel/functional-ui:test` | Run functional UI tests | Learn more about how to [develop the Management Console and opening the interactive Test Suite](#pass-the-management-console-quality-checks) |
| **Test - Functional UI Stop** | `pnpm nx run @zitadel/functional-ui:stop` | Run functional UI containers | |
| **Test** | `pnpm nx run PROJECT:test` | Run all tests | |
| **Lint** | `pnpm nx run PROJECT:lint` | Check code style | |
@@ -106,7 +106,7 @@ Follow [@zitadel](https://twitter.com/zitadel) on twitter
[Contribute](#how-to-contribute)
- [Contribute API code](#contribute-to-api)
- [Contribute frontend code](#contribute-to-frontend)
- [Contribute frontend code](#contribute-frontend-code)
- If you found a mistake on our [Docs page](https://zitadel.com/docs) or something is missing please read [the Docs section](#contribute-to-docs)
- [Translate](#contribute-translations) and improve texts
@@ -128,9 +128,9 @@ The code consists of the following parts:
| ------------------ | -------------------------------------------------- | --------------------------------------------------------------------------------------------------------- | --------------------------------------------------- | --------------------------------------------------- |
| API implementation | Service that serves the grpc(-web) and RESTful API | [go](https://go.dev) | [API implementation](./internal/api/grpc) | [Contribute to API](#contribute-to-api) |
| API definitions | Specifications of the API | [Protobuf](https://developers.google.com/protocol-buffers) | [./proto/zitadel](./proto/zitadel) | [Contribute to API](#contribute-to-api) |
| Management Console | Frontend the user interacts with after log in | [Angular](https://angular.io), [Typescript](https://www.typescriptlang.org) | [./console](./console) | [Contribute to Frontend](#contribute-to-frontend) |
| Login | Modern authentication UI built with Next.js | [Next.js](https://nextjs.org), [React](https://reactjs.org), [TypeScript](https://www.typescriptlang.org) | [./apps/login](./apps/login) | [Contribute to Frontend](#contribute-to-frontend) |
| Docs | Project documentation made with Fumadocs | [Fumadocs](https://fumadocs.dev/) | [./apps/docs](./apps/docs) | [Contribute to Frontend](#contribute-to-frontend) |
| Management Console | Frontend the user interacts with after log in | [Angular](https://angular.io), [Typescript](https://www.typescriptlang.org) | [./console](./console) | [Contribute to Frontend](#contribute-frontend-code) |
| Login | Modern authentication UI built with Next.js | [Next.js](https://nextjs.org), [React](https://reactjs.org), [TypeScript](https://www.typescriptlang.org) | [./apps/login](./apps/login) | [Contribute to Frontend](#contribute-frontend-code) |
| Docs | Project documentation made with Fumadocs | [Fumadocs](https://fumadocs.dev/) | [./apps/docs](./apps/docs) | [Contribute to Frontend](#contribute-frontend-code) |
| translations | Internationalization files for default languages | YAML | [./console](./console) and [./internal](./internal) | [Contribute Translations](#contribute-translations) |
Please follow the guides to validate and test the code before you contribute.
@@ -226,6 +226,14 @@ Please check out the dedicated [API guidelines](./API_DESIGN.md) page when contr
To start developing, make sure you followed the [quick start](#quick-start) steps.
> [!IMPORTANT]
> The relational database migrations are work in progress on main. Therefore they are objectives to change. To simplify the development process you can add the following configuration to your setup step config (`--steps /path/to/config.yaml`). This re-initiates the relational tables on each restart.
```yaml
RelationalTables:
ShouldRecreateSchema: true
```
### Develop the API
Optionally build the Management Console
@@ -271,7 +279,7 @@ For example, in VSCode, you can use a `launch.json` configuration like this.
}
```
If you have built the Management Console and started the Login, visit http://localhost:8080/ui/console?login_hint=zitadel-admin@zitadel.localhost and enter `Password1!` to log in.
If you have built the Management Console and started the Login, visit [http://localhost:8080/ui/console?login_hint=zitadel-admin@zitadel.localhost](http://localhost:8080/ui/console?login_hint=zitadel-admin@zitadel.localhost) and enter `Password1!` to log in.
Call the API using the generated [](./admin.pat) with [grpcurl](https://github.com/fullstorydev/grpcurl) or [grpcui](https://github.com/fullstorydev/grpcui), for example:
@@ -294,7 +302,6 @@ pnpm nx run @zitadel/api:test-unit
API tests are run as gRPC clients against a running Zitadel server binary.
The server binary is [built with coverage enabled](https://go.dev/doc/build-cover).
```bash
pnpm nx run @zitadel/api:test-integration
```
@@ -366,9 +373,9 @@ This repository uses **pnpm** as package manager and **Nx** for build orchestrat
Choose your contribution area:
- **[Login App](#contribute-to-login)** (Next.js/React) - Modern authentication flows
- **[Console](#contribute-to-console)** (Angular) - Admin dashboard and user management
- **[Console](#contribute-to-the-management-console)** (Angular) - Admin dashboard and user management
- **[Docs](#contribute-to-docs)** (Fumadocs) - Project documentation
- **[Client Packages](#client-packages)** - Shared libraries for API communication
- **[Client Packages](#contribute-to-client-packages)** - Shared libraries for API communication
### Project Dependencies
@@ -409,7 +416,7 @@ In another terminal, start the Login development server
pnpm nx run @zitadel/login:dev
```
Visit http://localhost:8080/ui/console?login_hint=zitadel-admin@zitadel.localhost and enter `Password1!` to log in.
Visit [http://localhost:8080/ui/console?login_hint=zitadel-admin@zitadel.localhost](http://localhost:8080/ui/console?login_hint=zitadel-admin@zitadel.localhost) and enter `Password1!` to log in.
Make some changes to the source code and see how the browser is automatically updated.
@@ -431,7 +438,7 @@ ZITADEL_SERVICE_USER_TOKEN=[personal access token for an instance Login Client]
pnpm nx run @zitadel/login:dev
```
Visit http://localhost:8080/ui/console?login_hint=zitadel-admin@zitadel.localhost and enter `Password1!` to log in.
Visit [http://localhost:8080/ui/console?login_hint=zitadel-admin@zitadel.localhost](http://localhost:8080/ui/console?login_hint=zitadel-admin@zitadel.localhost) and enter `Password1!` to log in.
#### Login Architecture
@@ -486,7 +493,7 @@ In another terminal, start the Login
pnpm nx run @zitadel/login:prod
```
Allow the API [to redirect to your dev server](#configure-console-dev-server-redirects).
Allow the API [to redirect to your dev server](#configure-the-management-console-dev-server-redirects).
In another terminal, start the Management Console development server
@@ -494,7 +501,7 @@ In another terminal, start the Management Console development server
pnpm nx run @zitadel/console:dev
```
Visit http://localhost:4200/?login_hint=zitadel-admin@zitadel.localhost and enter `Password1!` to log in.
Visit [http://localhost:4200/?login_hint=zitadel-admin@zitadel.localhost](http://localhost:4200/?login_hint=zitadel-admin@zitadel.localhost) and enter `Password1!` to log in.
Make some changes to the source code and see how the browser is automatically updated.
@@ -514,15 +521,15 @@ Start the development server.
pnpm nx run @zitadel/console:dev
```
Allow the API [to redirect to your dev server](#configure-console-dev-server-redirects).
Allow the API [to redirect to your dev server](#configure-the-management-console-dev-server-redirects).
Visit http://localhost:4200/?login_hint=zitadel-admin@zitadel.localhost and enter `Password1!` to log in.
Visit [http://localhost:4200/?login_hint=zitadel-admin@zitadel.localhost](http://localhost:4200/?login_hint=zitadel-admin@zitadel.localhost) and enter `Password1!` to log in.
#### Configure the Management Console Dev Server Redirects
To allow the Management Console access via http://localhost:4200, you have to configure the Zitadel API.
To allow the Management Console access via [http://localhost:4200](http://localhost:4200), you have to configure the Zitadel API.
1. Navigate to http://localhost:8080/ui/console/projects.
1. Navigate to [http://localhost:8080/ui/console/projects](http://localhost:8080/ui/console/projects).
2. Select the _ZITADEL_ project.
3. Select the _Console_ application.
4. Select _Redirect Settings_
@@ -541,7 +548,7 @@ pnpm nx run-many --projects @zitadel/console @zitadel/client @zitadel/proto @zit
Run functional UI tests against a locally built API and a dev server Management Console.
Allow the API [to redirect to your dev server](#configure-console-dev-server-redirects).
Allow the API [to redirect to your dev server](#configure-the-management-console-dev-server-redirects).
Alternatively, create the file `tests/functional-ui/.env.open.local` with the following content:
```conf
@@ -584,7 +591,7 @@ pnpm nx run @zitadel/client:build # Build after changes
All binary proto plugins are installed to `.artifacts/bin/<GOOS>/<GOARCH>/` and declared as Nx target outputs, making them eligible for Nx remote cache.
| Scope | Target | Installs |
|---|---|---|
| --- | --- | --- |
| `@zitadel/api` | `generate-install` | Go-based plugins: `buf`, `protoc-gen-go`, `protoc-gen-connect-go`, `protoc-gen-openapiv2`, `protoc-gen-validate`, `protoc-gen-authoption`, … |
| `@zitadel/console` | `install-proto-plugins` | `protoc-gen-grpc-web`, `protoc-gen-js`, `protoc-gen-openapiv2` (pre-built binaries, no Go required) |
| `@zitadel/docs` | `install-proto-plugins` | `protoc-gen-connect-openapi` (pre-built binary, no Go required) |
@@ -596,7 +603,6 @@ All binary proto plugins are installed to `.artifacts/bin/<GOOS>/<GOARCH>/` and
Project documentation is located under [./apps/docs](./apps/docs).
Please refer to the [Docs README](./apps/docs/README.md) for detailed instructions on how to contribute to the documentation.
## Contribute Translations
Zitadel loads translations from four files:
cmd/setup/config.go
+1
View File
@@ -185,6 +185,7 @@ type Steps struct {
s67SyncMemberRoleFields *SyncMemberRoleFields
s68TargetAddPayloadTypeColumn *TargetAddPayloadTypeColumn
s69CacheTablesLogged *CacheTablesLogged
RelationalTables *TransactionalTables
}
func NewSteps(ctx context.Context, v *viper.Viper) (*Steps, error) {
cmd/setup/setup.go
+2 -1
View File
@@ -324,7 +324,8 @@ func Setup(ctx context.Context, config *Config, steps *Steps, masterKey string)
defaults: config.SystemDefaults,
},
&TransactionalTables{
dbClient: dbClient,
dbClient: dbClient,
ShouldRecreateSchema: steps.RelationalTables.ShouldRecreateSchema,
},
&projectionTables{
es: eventstoreClient,
cmd/setup/steps.yaml
+5 -1
View File
@@ -180,4 +180,8 @@ AddEventCreatedAt:
BulkAmount: 100 # ZITADEL_ADDEVENTCREATEDAT_BULKAMOUNT
FillFields:
BatchSize: 1000 # ZITADEL_EVENTSTORE_FILLFIELDS_BULKLIMIT
BatchSize: 1000 # ZITADEL_EVENTSTORE_FILLFIELDS_BULKLIMIT
RelationalTables:
# This is used for development DO NOT USE IN PRODUCTION
ShouldRecreateSchema: false # ZITADEL_RELATIONALTABLES_SHOULDRECREATESCHEMA
cmd/setup/transactional_tables.go
+33
View File
@@ -4,6 +4,7 @@ import (
"context"
_ "embed"
"github.com/zitadel/zitadel/backend/v3/instrumentation/logging"
"github.com/zitadel/zitadel/backend/v3/storage/database/dialect/postgres"
"github.com/zitadel/zitadel/internal/database"
"github.com/zitadel/zitadel/internal/eventstore"
@@ -11,9 +12,19 @@ import (
type TransactionalTables struct {
dbClient *database.DB
ShouldRecreateSchema bool
}
func (mig *TransactionalTables) Execute(ctx context.Context, _ eventstore.Event) error {
// TODO(adlerhurst): revert changes made in https://github.com/zitadel/zitadel/pull/11833 before v5 release.
if mig.ShouldRecreateSchema {
logging.Info(ctx, "dropping schema of relational tables")
if err := mig.dropSchema(ctx); err != nil {
return err
}
}
config := &postgres.Config{Pool: mig.dbClient.Pool}
pool, err := config.Connect(ctx)
if err != nil {
@@ -29,3 +40,25 @@ func (mig *TransactionalTables) String() string {
func (mig *TransactionalTables) Check(lastRun map[string]interface{}) bool {
return true
}
func (mig *TransactionalTables) dropSchema(ctx context.Context) (err error) {
tx, err := mig.dbClient.BeginTx(ctx, nil)
if err != nil {
return err
}
defer func() {
if err != nil {
rollbackErr := tx.Rollback()
logging.OnError(ctx, rollbackErr).Debug("rollback failed")
return
}
err = tx.Commit()
}()
_, err = tx.ExecContext(ctx, "DROP SCHEMA IF EXISTS zitadel CASCADE")
if err != nil {
return err
}
_, err = tx.ExecContext(ctx, "DELETE FROM projections.current_states WHERE projection_name LIKE $1 || '%' OR projection_name = $2", "zitadel.", "relational_tables")
return err
}