andwati/zitadel
1
0
Fork 0
mirror of https://github.com/andwati/zitadel.git synced 2026-10-18 13:52:51 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
23c31ab2cbfbe554cfd36bcb5a46f01203ae511e
zitadel/cmd/setup/steps.yaml
T
Tim MöhlmannandGitHub eb22b58756 feat(telemetry): improved instrumentation for observability (#11159) 
# Which Problems Are Solved

Zitadel did not provide easy correlation between errors, logs, traces
and metrics. The configuration for those instrumentations was also not
consistent, with some supporting different exporters then others.
Implementation and parsing of config was also spaghettified over
multiple packages, with awkward parsing and inconsistent naming of
options.

# How the Problems Are Solved

All telemetry is now merged under the name "instrumentation". Why?
1. We thought it was a good idea in the past to call the milestone
exporter `Telemtry` in the runtime config. Calling this `TelemetryV2`
looks weird.
2. Not everything is a meter and not everything is sent (tele...). 
3. It's also
[defined](https://opentelemetry.io/docs/concepts/instrumentation/) as
such by the OTEL documentation.

## New features

- Adds structured, context based logging with trace-ID awareness
- Static log fields are added to the context, such as service and
request path
- Static log fields are injected in each logline emitted by the
application
- Structured logs can also be send to an otel exporter
- Structured logs can be printed to StdErr in text and JSON format
- Error sinks make sure every error is logged at the correct level:
- Warnings for client side errors (HTTP 400 range, Invalid request etc)
  - Error for server side errors (Internal server errors)
- Metrics can now also be send to a OTEL collector. (previously they
could only be scraped from `/debug/metrics` with prometheus)

## Exporters

This change adds all the exporters supported by OTEL upstream and some
google specific exporters for our cloud deployment.

- StdOut / StdErr: all instrumentations
- OTEL gRPC / HTTP: all instrumentations
- Google: all instrumentations except logging
- Prometheus (pull-based): only metrics

The exception is profiling, which only supports the google exporting due
to lack of support by OTEL upstream.

## Configuration and structure 

- All instrumentation is moved into the new `backend/v3/instrumentation`
package. It reuses configuration types, so both code and runtime
configuration are easier to understand.
- The `internal/telemetry` packages are removed.
- Instrumentation is started with a single function and a proper
shutdown function is now provided.
- Legacy configuration is still parsed from the runtime config, as long
as the new configuration is disabled. This allows backporting this
feature to v4 without breaking existing configurations.

# Additional Changes

- Devcontainer: set `$PATH` variable so installed go binaries can be run
individually, without NX.
- NX: install GCI tool to fix imports

# Additional Context

- Closes https://github.com/zitadel/zitadel/issues/8408
- Closes https://github.com/zitadel/zitadel/issues/6664
- Backport to v4
2026-01-12 05:51:39 +00:00

144 lines
7.7 KiB
YAML
Raw Blame History

Instrumentation:
ServiceName: "zitadel-setup" # ZITADEL_INSTRUMENTATION_SERVICENAME
Trace:
Fraction: 1.0 # ZITADEL_INSTRUMENTATION_TRACE_FRACTION
Exporter:
# The following exporter types are supported:
# "none": Disables trace exporting
# "stdOut": Exports traces to standard output
# "stdErr": Exports traces to standard error
# "grpc": Exports traces using the OTEL gRPC exporter (recommended)
# "http": Exports traces using the OTEL HTTP exporter
# "google": Exports traces to Google Cloud Trace
Type: "none" # ZITADEL_INSTRUMENTATION_TRACE_EXPORTER_TYPE
# Endpoint of the OTEL collector for grpc and http exporters
Endpoint: "" # ZITADEL_INSTRUMENTATION_TRACE_EXPORTER_ENDPOINT
# Disable https for grpc and http exporters
Insecure: false # ZITADEL_INSTRUMENTATION_TRACE_EXPORTER_INSECURE
# Interval for batching traces before export
BatchDuration: 1s # ZITADEL_INSTRUMENTATION_TRACE_EXPORTER_BATCHDURATION
# Project ID for Google Cloud Trace exporter
GoogleProjectID: "" # ZITADEL_INSTRUMENTATION_TRACE_EXPORTER_GOOGLEPROJECTID
Metric:
Exporter:
# The following exporter types are supported:
# "none": Disables metric exporting
# "stdOut": Exports metrics to standard output
# "stdErr": Exports metrics to standard error
# "grpc": Exports metrics using the OTEL gRPC exporter (recommended)
# "http": Exports metrics using the OTEL HTTP exporter
# "google": Exports metrics to Google Cloud Trace
# "prometheus": Exposes metrics via an HTTP endpoint for Prometheus to scrape
Type: "none" # ZITADEL_INSTRUMENTATION_METRIC_EXPORTER_TYPE
# Endpoint of the OTEL collector for grpc and http exporters
Endpoint: "" # ZITADEL_INSTRUMENTATION_METRIC_EXPORTER_ENDPOINT
# Disable https for grpc and http exporters
Insecure: false # ZITADEL_INSTRUMENTATION_METRIC_EXPORTER_INSECURE
# Interval at which metrics are exported
BatchDuration: 1s # ZITADEL_INSTRUMENTATION_METRIC_EXPORTER_BATCHDURATION
# Project ID for Google Cloud Trace exporter
GoogleProjectID: "" # ZITADEL_INSTRUMENTATION_METRIC_EXPORTER_GOOGLEPROJECTID
# Structured logging can print logs on standard error and/or export them using OTEL.
Log:
# Log lines lower than this level are not emitted.
Level: "INFO" # ZITADEL_INSTRUMENTATION_LOG_LEVEL
# Enable printing structured logs to standard error in the specified format.
# The following formats are supported:
# - "disabled": Disables logging to standard error
# - "text": Logs are printed as human-readable text
# - "json": Logs are printed as JSON objects
# - "gcp": JSON formatted logs compatible with Google Cloud Platform logging
StdErr: "text" # ZITADEL_INSTRUMENTATION_LOG_STDERR
# Adds the source file and line number where the log line was emitted.
# Not to be confused with the source of an error.
AddSource: true # ZITADEL_INSTRUMENTATION_LOG_ADDSOURCE
Exporter:
# The following exporter types are supported:
# - "none": Disables log exporting
# - "stdOut": Exports logs to standard output
# - "stdErr": Exports logs to standard error
# - "grpc": Exports logs using the OTEL gRPC exporter (recommended)
# - "http": Exports logs using the OTEL HTTP exporter
Type: "none" # ZITADEL_INSTRUMENTATION_LOG_EXPORTER_TYPE
# Endpoint of the OTEL collector for grpc and http exporters
Endpoint: "" # ZITADEL_INSTRUMENTATION_LOG_EXPORTER_ENDPOINT
# Disable https for grpc and http exporters
Insecure: false # ZITADEL_INSTRUMENTATION_LOG_EXPORTER_INSECURE
# Interval at which metrics are exported
BatchDuration: 1s # ZITADEL_INSTRUMENTATION_LOG_EXPORTER_BATCHDURATION
# Project ID for Google Cloud Trace exporter
GoogleProjectID: "" # ZITADEL_INSTRUMENTATION_LOG_EXPORTER_GOOGLEPROJECTID
Profile:
# The following profiler types are supported:
# "none": Disables profiling
# "google": Exports profiling data to Google Cloud Profiler
Type: "none" # ZITADEL_INSTRUMENTATION_PROFILE_TYPE
# Project ID for Google Cloud Profiler
GoogleProjectID: "" # ZITADEL_INSTRUMENTATION_PROFILE_GOOGLEPROJECTID
# By using the FirstInstance section, you can overwrite the DefaultInstance configuration for the first instance created by zitadel setup.
FirstInstance:
# If set to true zitadel is setup without initial data
Skip: false
# The machine key from the section FirstInstance.Org.Machine.MachineKey is written to the MachineKeyPath.
MachineKeyPath: # ZITADEL_FIRSTINSTANCE_MACHINEKEYPATH
# The personal access token from the section FirstInstance.Org.Machine.Pat is written to the PatPath.
PatPath: # ZITADEL_FIRSTINSTANCE_PATPATH
LoginClientPatPath: # ZITADEL_FIRSTINSTANCE_LOGINCLIENTPATPATH
InstanceName: ZITADEL # ZITADEL_FIRSTINSTANCE_INSTANCENAME
DefaultLanguage: en # ZITADEL_FIRSTINSTANCE_DEFAULTLANGUAGE
Org:
Name: ZITADEL # ZITADEL_FIRSTINSTANCE_ORG_NAME
# In the FirstInstance.Org.Human section, the initial organization's admin user with the role IAM_OWNER is defined.
# If FirstInstance.Org.Machine.Machine is defined, a service user is created with the IAM_OWNER role.
Human:
# In case UserLoginMustBeDomain is false (default) and you don't overwrite the username with an email,
# it will be suffixed by the org domain (org-name + domain from config).
# for example zitadel-admin in org ZITADEL on domain.tld -> [email protected]
UserName: zitadel-admin # ZITADEL_FIRSTINSTANCE_ORG_HUMAN_USERNAME
FirstName: ZITADEL # ZITADEL_FIRSTINSTANCE_ORG_HUMAN_FIRSTNAME
LastName: Admin # ZITADEL_FIRSTINSTANCE_ORG_HUMAN_LASTNAME
NickName: # ZITADEL_FIRSTINSTANCE_ORG_HUMAN_NICKNAME
DisplayName: # ZITADEL_FIRSTINSTANCE_ORG_HUMAN_DISPLAYNAME
Email:
# uses the username if empty
Address: # ZITADEL_FIRSTINSTANCE_ORG_HUMAN_EMAIL_ADDRESS
Verified: true # ZITADEL_FIRSTINSTANCE_ORG_HUMAN_EMAIL_VERIFIED
PreferredLanguage: en # ZITADEL_FIRSTINSTANCE_ORG_HUMAN_PREFERREDLANGUAGE
Gender: # ZITADEL_FIRSTINSTANCE_ORG_HUMAN_GENDER
Phone:
Number: # ZITADEL_FIRSTINSTANCE_ORG_HUMAN_PHONE_NUMBER
Verified: # ZITADEL_FIRSTINSTANCE_ORG_HUMAN_PHONE_VERIFIED
Password: Password1! # ZITADEL_FIRSTINSTANCE_ORG_HUMAN_PASSWORD
PasswordChangeRequired: true # ZITADEL_FIRSTINSTANCE_ORG_HUMAN_PASSWORDCHANGEREQUIRED
# In the FirstInstance.Org.Machine section, the initial organization's admin user with the role IAM_OWNER is defined.
# If FirstInstance.Org.Machine.Machine is defined, a service user is created with the IAM_OWNER role.
Machine:
Machine:
Username: # ZITADEL_FIRSTINSTANCE_ORG_MACHINE_MACHINE_USERNAME
Name: # ZITADEL_FIRSTINSTANCE_ORG_MACHINE_MACHINE_NAME
MachineKey:
# date format: 2023-01-01T00:00:00Z
ExpirationDate: # ZITADEL_FIRSTINSTANCE_ORG_MACHINE_MACHINEKEY_EXPIRATIONDATE
# Currently, the only supported value is 1 for JSON
Type: # ZITADEL_FIRSTINSTANCE_ORG_MACHINE_MACHINEKEY_TYPE
Pat:
# date format: 2023-01-01T00:00:00Z
ExpirationDate: # ZITADEL_FIRSTINSTANCE_ORG_MACHINE_PAT_EXPIRATIONDATE
LoginClient:
Machine:
Username: # ZITADEL_FIRSTINSTANCE_ORG_LOGINCLIENT_MACHINE_USERNAME
Name: # ZITADEL_FIRSTINSTANCE_ORG_LOGINCLIENT_MACHINE_NAME
Pat:
# date format: 2023-01-01T00:00:00Z
ExpirationDate: # ZITADEL_FIRSTINSTANCE_ORG_LOGINCLIENT_PAT_EXPIRATIONDATE
CorrectCreationDate:
FailAfter: 5m # ZITADEL_CORRECTCREATIONDATE_FAILAFTER
AddEventCreatedAt:
BulkAmount: 100 # ZITADEL_ADDEVENTCREATEDAT_BULKAMOUNT
FillFields:
BatchSize: 1000 # ZITADEL_EVENTSTORE_FILLFIELDS_BULKLIMIT
Reference in New Issue View Git Blame Copy Permalink