# Which Problems Are Solved
Zitadel did not provide easy correlation between errors, logs, traces
and metrics. The configuration for those instrumentations was also not
consistent, with some supporting different exporters then others.
Implementation and parsing of config was also spaghettified over
multiple packages, with awkward parsing and inconsistent naming of
options.
# How the Problems Are Solved
All telemetry is now merged under the name "instrumentation". Why?
1. We thought it was a good idea in the past to call the milestone
exporter `Telemtry` in the runtime config. Calling this `TelemetryV2`
looks weird.
2. Not everything is a meter and not everything is sent (tele...).
3. It's also
[defined](https://opentelemetry.io/docs/concepts/instrumentation/) as
such by the OTEL documentation.
## New features
- Adds structured, context based logging with trace-ID awareness
- Static log fields are added to the context, such as service and
request path
- Static log fields are injected in each logline emitted by the
application
- Structured logs can also be send to an otel exporter
- Structured logs can be printed to StdErr in text and JSON format
- Error sinks make sure every error is logged at the correct level:
- Warnings for client side errors (HTTP 400 range, Invalid request etc)
- Error for server side errors (Internal server errors)
- Metrics can now also be send to a OTEL collector. (previously they
could only be scraped from `/debug/metrics` with prometheus)
## Exporters
This change adds all the exporters supported by OTEL upstream and some
google specific exporters for our cloud deployment.
- StdOut / StdErr: all instrumentations
- OTEL gRPC / HTTP: all instrumentations
- Google: all instrumentations except logging
- Prometheus (pull-based): only metrics
The exception is profiling, which only supports the google exporting due
to lack of support by OTEL upstream.
## Configuration and structure
- All instrumentation is moved into the new `backend/v3/instrumentation`
package. It reuses configuration types, so both code and runtime
configuration are easier to understand.
- The `internal/telemetry` packages are removed.
- Instrumentation is started with a single function and a proper
shutdown function is now provided.
- Legacy configuration is still parsed from the runtime config, as long
as the new configuration is disabled. This allows backporting this
feature to v4 without breaking existing configurations.
# Additional Changes
- Devcontainer: set `$PATH` variable so installed go binaries can be run
individually, without NX.
- NX: install GCI tool to fix imports
# Additional Context
- Closes https://github.com/zitadel/zitadel/issues/8408
- Closes https://github.com/zitadel/zitadel/issues/6664
- Backport to v4
# Which Problems Are Solved
Replaces Turbo by Nx and lays the foundation for the next CI
improvements. It enables using Nx Cloud to speed the up the pipelines
that affect any node package.
It streamlines the dev experience for frontend and backend developers by
providing the following commands:
| Task | Command | Notes |
|------|---------|--------|
| **Production** | `nx run PROJECT:prod` | Production server |
| **Develop** | `nx run PROJECT:dev` | Hot reloading development server
|
| **Test** | `nx run PROJECT:test` | Run all tests |
| **Lint** | `nx run PROJECT:lint` | Check code style |
| **Lint Fix** | `nx run PROJECT:lint-fix` | Auto-fix style issues |
The following values can be used for PROJECT:
- @zitadel/zitadel (root commands)
- @zitadel/api,
- @zitadel/login,
- @zitadel/console,
- @zitadel/docs,
- @zitadel/client
- @zitadel/proto
The project names and folders are streamlined:
| Old Folder | New Folder |
| --- | --- |
| ./e2e | ./tests/functional-ui |
| ./load-test | ./benchmark |
| ./build/zitadel | ./apps/api |
| ./console | ./apps/console (postponed so the PR is reviewable) |
Also, all references to the TypeScript repo are removed so we can
archive it.
# How the Problems Are Solved
- Ran `npx nx@latest init`
- Replaced all turbo.json by project.json and fixed the target configs
- Removed Turbo dependency
- All JavaScript related code affected by a PRs changes is
quality-checked using the `nx affected` command
- We move PR checks that are runnable using Nx into the `check`
workflow. For workflows where we don't use Nx, yet, we restore
previously built dependency artifacts from Nx.
- We only use a single and easy to understand dev container
- The CONTRIBUTING.md is streamlined
- The setup with a generated client pat is orchestrated with Nx
- Everything related to the TypeScript repo is updated or removed. A
**Deploy with Vercel** button is added to the docs and the
CONTRIBUTING.md.
# Additional Changes
- NPM package names have a consistent pattern.
- Docker bake is removed. The login container is built and released like
the core container.
- The integration tests build the login container before running, so
they don't rely on the login container action anymore. This fixes
consistently failing checks on PRs from forks.
- The docs build in GitHub actions is removed, as we already build on
Vercel.
# Additional Context
- Internal discussion:
https://zitadel.slack.com/archives/C087ADF8LRX/p1756277884928169
- Workflow dispatch test:
https://github.com/zitadel/zitadel/actions/runs/17760122959
---------
Co-authored-by: Florian Forster <[email protected]>
Co-authored-by: Tim Möhlmann <[email protected]>
Co-authored-by: Copilot <[email protected]>
# Which Problems Are Solved
Login integration tests are not executed in the pipeline
# How the Problems Are Solved
The login integration tests are fixed and added as a pipeline workflow.
It tests against the built login docker image.
On pipeline failures, developers are guided on how to fix them using a
dev container configured for this purpose.
# Additional Changes
- email domains are replaced by example.com. In case the tests were
accidentally run against a cloud instance, it wouldn't cause bounces.
- pnpm is upgraded, because the --filter argument doesn't work for the
install command on the old version.
- The login Dockerfile is optimized for docker image builds
# Additional Changes From Review for
https://github.com/zitadel/zitadel/pull/10305
These changes were requested from @peintnermax
- The base dev container starts without any services besides the
database and the dev container itself
- CONTRIBUTING.md is restructured
- To reproduce pipeline checks, only the devcontainer CLI and Docker are
needed. This is described in the CONTRIBUTING.md
- The convenience npm script "generate" is added
# Additional Context
- Follow-up for PR https://github.com/zitadel/zitadel/pull/10305
- Base for https://github.com/zitadel/zitadel/issues/10277
# Which Problems Are Solved
Building the login container in the pipeline without build layer caches
takes about 20 minutes.
# How the Problems Are Solved
We use cache-from and cache-to arguments to use the GitHub cache API.
Compare 1st and 2nd run [of this PRs
pipeline](https://github.com/zitadel/zitadel/actions/runs/16590808510/job/46927893304)
# Additional Context
- Follows up on #10343
# Which Problems Are Solved
- The previous monorepo in monorepo structure for the login app and its
related packages was fragmented, complicated and buggy.
- The process for building and testing the login container was
inconsistent between local development and CI.
- Lack of clear documentation as well as easy and reliable ways for
non-frontend developers to reproduce and fix failing PR checks locally.
# How the Problems Are Solved
- Consolidated the login app and its related npm packages by moving the
main package to `apps/login/apps/login` and merging
`apps/login/packages/integration` and `apps/login/packages/acceptance`
into the main `apps/login` package.
- Migrated from Docker Compose-based test setups to dev container-based
setups, adding support for multiple dev container configurations:
- `.devcontainer/base`
- `.devcontainer/turbo-lint-unit`
- `.devcontainer/turbo-lint-unit-debug`
- `.devcontainer/login-integration`
- `.devcontainer/login-integration-debug`
- Added npm scripts to run the new dev container setups, enabling exact
reproduction of GitHub PR checks locally, and updated the pipeline to
use these containers.
- Cleaned up Dockerfiles and docker-bake.hcl files to only build the
production image for the login app.
- Cleaned up compose files to focus on dev environments in dev
containers.
- Updated `CONTRIBUTING.md` with guidance on running and debugging PR
checks locally using the new dev container approach.
- Introduced separate Dockerfiles for the login app to distinguish
between using published client packages and building clients from local
protos.
- Ensured the login container is always built in the pipeline for use in
integration and acceptance tests.
- Updated Makefile and GitHub Actions workflows to use
`--frozen-lockfile` for installing pnpm packages, ensuring reproducible
installs.
- Disabled GitHub release creation by the changeset action.
- Refactored the `/build` directory structure for clarity and
maintainability.
- Added a `clean` command to `docks/package.json`.
- Experimentally added `knip` to the `zitadel-client` package for
improved linting of dependencies and exports.
# Additional Changes
- Fixed Makefile commands for consistency and reliability.
- Improved the structure and clarity of the `/build` directory to
support seamless integration of the login build.
- Enhanced documentation and developer experience for running and
debugging CI checks locally.
# Additional Context
- See updated `CONTRIBUTING.md` for new local development and debugging
instructions.
- These changes are a prerequisite for further improvements to the CI
pipeline and local development workflow.
- Closes#10276
# Which Problems Are Solved
1. Postgres spams FATAL: role "root" does not exist as mentioned in
https://github.com/zitadel/zitadel/discussions/7832 (even with -U)
2. The compose commands for a ZITADEL deployment with initial service
account key don't work out-of-the box with a non-root user, because
docker creates non-existing directories to bind-mount with root
ownership.
```
time="2024-10-29T09:37:13Z" level=error msg="migration failed" caller="/home/runner/work/zitadel/zitadel/internal/migration/migration.go:68" error="open /machinekey/zitadel-admin-sa.json: permission denied" name=03_default_instance
time="2024-10-29T09:37:13Z" level=fatal msg="migration failed" caller="/home/runner/work/zitadel/zitadel/cmd/setup/setup.go:248" error="open /machinekey/zitadel-admin-sa.json: permission denied" name=03_default_instance
```
# How the Problems Are Solved
1. The branch bases on https://github.com/zitadel/zitadel/pull/8826. The
env vars are cleaned up and prettified across compose files.
2. A command is added to the docs that creates the directory with the
current users permission. The ZITADEL container runs with the current
users ID.
# Additional Context
- Replaces https://github.com/zitadel/zitadel/pull/8826
- Discussion https://github.com/zitadel/zitadel/discussions/7832
- Closes https://github.com/zitadel/zitadel/issues/7725
---------
Co-authored-by: m4tu4g <[email protected]>
# Which Problems Are Solved
Node devcontainer still used node version 18.
# How the Problems Are Solved
Use default (lts) version to mitigate problems.
# Additional Changes
None
# Additional Context
- reference:
https://github.com/devcontainers/features/tree/main/src/node
Co-authored-by: Tim Möhlmann <[email protected]>
# Which Problems Are Solved
Describes how to develop ZITADEL using dev containers.
# Additional Changes
Sets default env variables for using postgres as database in the dev
container.
Tim MöhlmannandGitHub
SilvanandGitHub