Files
zitadel/backend/v3/instrumentation/instrumentation.go
T
997aa607c4 feat(telemetry): unify request details (#11509)
# Which Problems Are Solved

In the "new" structured logging, request details were added in different
middlewares, such as request, instance and user IDs. This meant the the
upstream request logging middleware did not have access to metadata that
got added later, resulting in incomplete logs. Furthermore it was not
possible to correlate an API error response to log output.

# How the Problems Are Solved

A mutable request details object is added to the context early on. When
the api authz function run, the instance and user IDs are added to this
object as they become available. Every logline emitted after this
(time-wise) will then all contain these details under the `request` log
group.

<details>

<summary>example output in JSON</summary>

```json
{
  "time": "2026-03-13T19:21:20.890428806Z",
  "level": "INFO",
  "source": {
    "function": "github.com/zitadel/zitadel/internal/api/grpc/server/connect_middleware.LogHandler.func1.1",
    "file": "/workspaces/zitadel/internal/api/grpc/server/connect_middleware/log_interceptor.go",
    "line": 34
  },
  "msg": "request served",
  "request": {
    "id": "d6q67c04vtjmi77cbbpg",
    "instance_host": "localhost:8080",
    "instance_id": "362349751439458307",
    "user_id": "362349751440048131"
  },
  "TraceID": "a9e0fee3522224f3583bbdcda737f4b4",
  "SpanID": "a563056eee36920a",
  "stream": "request",
  "version": "2026-03-13T19:20:59Z",
  "protocol": "connect",
  "service": "zitadel.user.v2.UserService",
  "http_method": "POST",
  "path": "/zitadel.user.v2.UserService/ListUsers",
  "code": "code_0",
  "duration": 12254350
}
```

</details>

Request IDs are now also returned with a response header or metadata.
Depending on the protocol:

- HTTP calls always return the request ID as header, regardless of
status
- gRPC calls always return the request ID as header, even if there was
an error
- connect RPC calls returns the request ID as header on success, trailer
in case of error. This is because header must be set on the response
object, which is nil in case of error. When there is an error, metadata
can be added which are then sent as trailers.

# Additional Changes

- Use the existing call duration middleware for both request ID and
logging for a consistent request start timestamp in all layers.
- Upgrade sloggcp for some fixes (notably TraceID)
- Modify the NoCache middleware so it uses `SetHeaders` instead of
`SendHeaders`. The latter prevented any other handler from setting
headers, including the new request ID middleware.

# Additional Context

Follow up on demo of:
  - https://github.com/zitadel/zitadel/pull/11159
  - https://github.com/zitadel/zitadel/pull/11435
  - backport to v4

---------

Co-authored-by: Copilot <[email protected]>
Co-authored-by: muhlemmer <[email protected]>
Co-authored-by: Copilot Autofix powered by AI <[email protected]>
Co-authored-by: Marco A. <[email protected]>
2026-03-18 14:07:33 +01:00

83 lines
2.3 KiB
Go

// Package instrumentation initializes and configures telemetry and logging instrumentation.
// After [Start] is called, the instrumentation providers can be accessed using global getters.
package instrumentation
import (
"context"
"net/http"
"slices"
"strings"
"sync"
"go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
"go.opentelemetry.io/otel"
"go.opentelemetry.io/otel/metric"
"go.opentelemetry.io/otel/sdk/log"
"go.opentelemetry.io/otel/trace"
)
const Name = "zitadel/backend/v3/instrumentation/tracing"
type Config struct {
ServiceName string
Trace TraceConfig
Metric MetricConfig
Log LogConfig
Profile ProfileConfig
}
type ShutdownFunc func(context.Context) error
var (
startOnce sync.Once
shutdown ShutdownFunc
errStart error
)
// Start initializes global instrumentation once, based on the provided configuration.
// It is safe to call multiple times; subsequent calls will return the same shutdown function and error as the first call.
func Start(ctx context.Context, cfg Config) (ShutdownFunc, error) {
startOnce.Do(func() {
errStart = startProfiler(cfg.Profile, cfg.ServiceName)
if errStart != nil {
return
}
var logProvider *log.LoggerProvider
logProvider, shutdown, errStart = setupOTelSDK(ctx, cfg)
if errStart != nil {
return
}
setLogger(logProvider, cfg.Log)
})
return shutdown, errStart
}
// NewTracer creates a new tracer from the global provider.
// See [trace.TracerProvider.Tracer] for details.
func NewTracer(name string, options ...trace.TracerOption) *Tracer {
provider := otel.GetTracerProvider()
return &Tracer{provider.Tracer(name, options...)}
}
// NewMeter creates a new meter from the global provider.
// See [metric.MeterProvider.Meter] for details.
func NewMeter(name string, options ...metric.MeterOption) *Meter {
provider := otel.GetMeterProvider()
return &Meter{
Meter: provider.Meter(name, options...),
}
}
// RequestFilter returns an [otelhttp.Filter] that excludes requests
// with paths starting with any of the specified prefixes.
func RequestFilter(ignoredPrefix ...string) otelhttp.Filter {
return func(r *http.Request) bool {
return !slices.ContainsFunc(
ignoredPrefix,
func(endpoint string) bool {
return strings.HasPrefix(r.URL.RequestURI(), endpoint)
},
)
}
}