andwati/zitadel
1
0
Fork 0
mirror of https://github.com/andwati/zitadel.git synced 2026-10-29 02:22:16 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(telemetry): improved instrumentation for observability (#11159)

Browse Source 
# Which Problems Are Solved

Zitadel did not provide easy correlation between errors, logs, traces
and metrics. The configuration for those instrumentations was also not
consistent, with some supporting different exporters then others.
Implementation and parsing of config was also spaghettified over
multiple packages, with awkward parsing and inconsistent naming of
options.

# How the Problems Are Solved

All telemetry is now merged under the name "instrumentation". Why?
1. We thought it was a good idea in the past to call the milestone
exporter `Telemtry` in the runtime config. Calling this `TelemetryV2`
looks weird.
2. Not everything is a meter and not everything is sent (tele...). 
3. It's also
[defined](https://opentelemetry.io/docs/concepts/instrumentation/) as
such by the OTEL documentation.

## New features

- Adds structured, context based logging with trace-ID awareness
- Static log fields are added to the context, such as service and
request path
- Static log fields are injected in each logline emitted by the
application
- Structured logs can also be send to an otel exporter
- Structured logs can be printed to StdErr in text and JSON format
- Error sinks make sure every error is logged at the correct level:
- Warnings for client side errors (HTTP 400 range, Invalid request etc)
  - Error for server side errors (Internal server errors)
- Metrics can now also be send to a OTEL collector. (previously they
could only be scraped from `/debug/metrics` with prometheus)

## Exporters

This change adds all the exporters supported by OTEL upstream and some
google specific exporters for our cloud deployment.

- StdOut / StdErr: all instrumentations
- OTEL gRPC / HTTP: all instrumentations
- Google: all instrumentations except logging
- Prometheus (pull-based): only metrics

The exception is profiling, which only supports the google exporting due
to lack of support by OTEL upstream.

## Configuration and structure 

- All instrumentation is moved into the new `backend/v3/instrumentation`
package. It reuses configuration types, so both code and runtime
configuration are easier to understand.
- The `internal/telemetry` packages are removed.
- Instrumentation is started with a single function and a proper
shutdown function is now provided.
- Legacy configuration is still parsed from the runtime config, as long
as the new configuration is disabled. This allows backporting this
feature to v4 without breaking existing configurations.

# Additional Changes

- Devcontainer: set `$PATH` variable so installed go binaries can be run
individually, without NX.
- NX: install GCI tool to fix imports

# Additional Context

- Closes https://github.com/zitadel/zitadel/issues/8408
- Closes https://github.com/zitadel/zitadel/issues/6664
- Backport to v4
This commit is contained in:
Tim Möhlmann
2026-01-12 05:51:39 +00:00
committed by GitHub
parent d7a9e9d48b
commit eb22b58756
160 changed files with 4320 additions and 2135 deletions
Download Patch File Download Diff File Expand all files Collapse all files
cmd/setup/cleanup.go
+10 -2
View File
@@ -2,6 +2,7 @@ package setup
import (
"context"
"errors"
"github.com/spf13/cobra"
"github.com/spf13/viper"
@@ -19,9 +20,16 @@ func NewCleanup() *cobra.Command {
Use: "cleanup",
Short: "cleans up migration if they got stuck",
Long: `cleans up migration if they got stuck`,
Run: func(cmd *cobra.Command, args []string) {
config := MustNewConfig(viper.GetViper())
RunE: func(cmd *cobra.Command, args []string) (err error) {
config, shutdown, err := NewConfig(cmd.Context(), viper.GetViper())
if err != nil {
return err
}
defer func() {
err = errors.Join(err, shutdown(cmd.Context()))
}()
Cleanup(cmd.Context(), config)
return nil
},
}
}
cmd/setup/config.go
+21 -9
View File
@@ -2,6 +2,9 @@ package setup
import (
"bytes"
"context"
"errors"
"fmt"
"strings"
"time"
@@ -9,6 +12,7 @@ import (
"github.com/spf13/viper"
"github.com/zitadel/logging"
"github.com/zitadel/zitadel/backend/v3/instrumentation"
"github.com/zitadel/zitadel/cmd/encryption"
"github.com/zitadel/zitadel/cmd/hooks"
"github.com/zitadel/zitadel/internal/actions"
@@ -27,7 +31,6 @@ import (
"github.com/zitadel/zitadel/internal/notification/handlers"
"github.com/zitadel/zitadel/internal/query/projection"
static_config "github.com/zitadel/zitadel/internal/static/config"
metrics "github.com/zitadel/zitadel/internal/telemetry/metrics/config"
)
type Config struct {
@@ -40,8 +43,9 @@ type Config struct {
ExternalDomain string
ExternalPort uint16
ExternalSecure bool
Instrumentation instrumentation.Config
Log *logging.Config
Metrics metrics.Config
Metrics *instrumentation.LegacyMetricConfig
EncryptionKeys *encryption.EncryptionKeyConfig
DefaultInstance command.InstanceSetup
Machine *id.Config
@@ -66,7 +70,7 @@ type InitProjections struct {
BulkLimit uint64
}
func MustNewConfig(v *viper.Viper) *Config {
func NewConfig(ctx context.Context, v *viper.Viper) (*Config, instrumentation.ShutdownFunc, error) {
config := new(Config)
err := v.Unmarshal(config,
viper.DecodeHook(mapstructure.ComposeDecodeHookFunc(
@@ -85,20 +89,28 @@ func MustNewConfig(v *viper.Viper) *Config {
mapstructure.TextUnmarshallerHookFunc(),
)),
)
logging.OnError(err).Fatal("unable to read default config")
if err != nil {
return nil, nil, fmt.Errorf("unable to read default config: %w", err)
}
config.Instrumentation.Metric.SetLegacyConfig(config.Metrics)
shutdown, err := instrumentation.Start(ctx, config.Instrumentation)
if err != nil {
return nil, nil, fmt.Errorf("unable to start instrumentation: %w", err)
}
err = config.Log.SetLogger()
logging.OnError(err).Fatal("unable to set logger")
err = config.Metrics.NewMeter()
logging.OnError(err).Fatal("unable to set meter")
if err != nil {
err = errors.Join(err, shutdown(ctx))
return nil, nil, fmt.Errorf("unable to set logger: %w", err)
}
id.Configure(config.Machine)
// Copy the global role permissions mappings to the instance until we allow instance-level configuration over the API.
config.DefaultInstance.RolePermissionMappings = config.InternalAuthZ.RolePermissionMappings
return config
return config, shutdown, nil
}
type Steps struct {
cmd/setup/config_test.go
+2 -1
View File
@@ -241,7 +241,8 @@ Actions:
v := viper.New()
v.SetConfigType("yaml")
require.NoError(t, v.ReadConfig(strings.NewReader(tt.args.yaml)))
got := MustNewConfig(v)
got, _, err := NewConfig(t.Context(), v)
require.NoError(t, err)
tt.want(t, got)
})
}
cmd/setup/setup.go
+30 -7
View File
@@ -6,6 +6,7 @@ import (
_ "embed"
"errors"
"fmt"
"log/slog"
"net/http"
"os"
"os/signal"
@@ -61,23 +62,45 @@ func New() *cobra.Command {
Long: `sets up data to start ZITADEL.
Requirements:
- postgreSQL`,
Run: func(cmd *cobra.Command, args []string) {
err := tls.ModeFromFlag(cmd)
logging.OnError(err).Fatal("invalid tlsMode")
RunE: func(cmd *cobra.Command, args []string) (err error) {
defer func() {
if err != nil {
slog.Error("zitadel setup command failed", "err", err)
}
}()
err = tls.ModeFromFlag(cmd)
if err != nil {
return fmt.Errorf("invalid tlsMode: %w", err)
}
err = BindInitProjections(cmd)
logging.OnError(err).Fatal("unable to bind \"init-projections\" flag")
if err != nil {
return fmt.Errorf("unable to bind \"init-projections\" flag: %w", err)
}
err = bindForMirror(cmd)
logging.OnError(err).Fatal("unable to bind \"for-mirror\" flag")
if err != nil {
return fmt.Errorf("unable to bind \"for-mirror\" flag: %w", err)
}
config, shutdown, err := NewConfig(cmd.Context(), viper.GetViper())
if err != nil {
return err
}
defer func() {
err = errors.Join(err, shutdown(cmd.Context()))
}()
config := MustNewConfig(viper.GetViper())
steps := MustNewSteps(viper.New())
masterKey, err := key.MasterKey(cmd)
logging.OnError(err).Panic("No master key provided")
if err != nil {
return fmt.Errorf("no master key provided: %w", err)
}
Setup(cmd.Context(), config, steps, masterKey)
return nil
},
}
cmd/setup/steps.yaml
+79
View File
@@ -1,3 +1,82 @@
Instrumentation:
ServiceName: "zitadel-setup" # ZITADEL_INSTRUMENTATION_SERVICENAME
Trace:
Fraction: 1.0 # ZITADEL_INSTRUMENTATION_TRACE_FRACTION
Exporter:
# The following exporter types are supported:
# "none": Disables trace exporting
# "stdOut": Exports traces to standard output
# "stdErr": Exports traces to standard error
# "grpc": Exports traces using the OTEL gRPC exporter (recommended)
# "http": Exports traces using the OTEL HTTP exporter
# "google": Exports traces to Google Cloud Trace
Type: "none" # ZITADEL_INSTRUMENTATION_TRACE_EXPORTER_TYPE
# Endpoint of the OTEL collector for grpc and http exporters
Endpoint: "" # ZITADEL_INSTRUMENTATION_TRACE_EXPORTER_ENDPOINT
# Disable https for grpc and http exporters
Insecure: false # ZITADEL_INSTRUMENTATION_TRACE_EXPORTER_INSECURE
# Interval for batching traces before export
BatchDuration: 1s # ZITADEL_INSTRUMENTATION_TRACE_EXPORTER_BATCHDURATION
# Project ID for Google Cloud Trace exporter
GoogleProjectID: "" # ZITADEL_INSTRUMENTATION_TRACE_EXPORTER_GOOGLEPROJECTID
Metric:
Exporter:
# The following exporter types are supported:
# "none": Disables metric exporting
# "stdOut": Exports metrics to standard output
# "stdErr": Exports metrics to standard error
# "grpc": Exports metrics using the OTEL gRPC exporter (recommended)
# "http": Exports metrics using the OTEL HTTP exporter
# "google": Exports metrics to Google Cloud Trace
# "prometheus": Exposes metrics via an HTTP endpoint for Prometheus to scrape
Type: "none" # ZITADEL_INSTRUMENTATION_METRIC_EXPORTER_TYPE
# Endpoint of the OTEL collector for grpc and http exporters
Endpoint: "" # ZITADEL_INSTRUMENTATION_METRIC_EXPORTER_ENDPOINT
# Disable https for grpc and http exporters
Insecure: false # ZITADEL_INSTRUMENTATION_METRIC_EXPORTER_INSECURE
# Interval at which metrics are exported
BatchDuration: 1s # ZITADEL_INSTRUMENTATION_METRIC_EXPORTER_BATCHDURATION
# Project ID for Google Cloud Trace exporter
GoogleProjectID: "" # ZITADEL_INSTRUMENTATION_METRIC_EXPORTER_GOOGLEPROJECTID
# Structured logging can print logs on standard error and/or export them using OTEL.
Log:
# Log lines lower than this level are not emitted.
Level: "INFO" # ZITADEL_INSTRUMENTATION_LOG_LEVEL
# Enable printing structured logs to standard error in the specified format.
# The following formats are supported:
# - "disabled": Disables logging to standard error
# - "text": Logs are printed as human-readable text
# - "json": Logs are printed as JSON objects
# - "gcp": JSON formatted logs compatible with Google Cloud Platform logging
StdErr: "text" # ZITADEL_INSTRUMENTATION_LOG_STDERR
# Adds the source file and line number where the log line was emitted.
# Not to be confused with the source of an error.
AddSource: true # ZITADEL_INSTRUMENTATION_LOG_ADDSOURCE
Exporter:
# The following exporter types are supported:
# - "none": Disables log exporting
# - "stdOut": Exports logs to standard output
# - "stdErr": Exports logs to standard error
# - "grpc": Exports logs using the OTEL gRPC exporter (recommended)
# - "http": Exports logs using the OTEL HTTP exporter
Type: "none" # ZITADEL_INSTRUMENTATION_LOG_EXPORTER_TYPE
# Endpoint of the OTEL collector for grpc and http exporters
Endpoint: "" # ZITADEL_INSTRUMENTATION_LOG_EXPORTER_ENDPOINT
# Disable https for grpc and http exporters
Insecure: false # ZITADEL_INSTRUMENTATION_LOG_EXPORTER_INSECURE
# Interval at which metrics are exported
BatchDuration: 1s # ZITADEL_INSTRUMENTATION_LOG_EXPORTER_BATCHDURATION
# Project ID for Google Cloud Trace exporter
GoogleProjectID: "" # ZITADEL_INSTRUMENTATION_LOG_EXPORTER_GOOGLEPROJECTID
Profile:
# The following profiler types are supported:
# "none": Disables profiling
# "google": Exports profiling data to Google Cloud Profiler
Type: "none" # ZITADEL_INSTRUMENTATION_PROFILE_TYPE
# Project ID for Google Cloud Profiler
GoogleProjectID: "" # ZITADEL_INSTRUMENTATION_PROFILE_GOOGLEPROJECTID
# By using the FirstInstance section, you can overwrite the DefaultInstance configuration for the first instance created by zitadel setup.
FirstInstance:
# If set to true zitadel is setup without initial data