# Which Problems Are Solved
- The `uri` label of `http.server.request_count` and
`http.server.return_code_counter` holds the concrete request path, so
every object ID served over a REST path becomes its own metric series.
On our v4.16.2 instance `/debug/metrics` exposes 8,000 series, 4,467 of
which (56%) belong to `http_server_return_code_counter_total` alone,
spread over 4,390 distinct `uri` values such as
`/v2/sessions/385063742120926058`. The set only grows with the number of
objects the installation has served.
- Paths matching no route are recorded verbatim, so unauthenticated
scanner traffic (`/wp-json/`, `/login.php`, …) keeps adding series.
- The grpc-gateway answers `405 Method Not Allowed` without going
through the configured `errorHandler`, so those responses never reported
a route pattern, not even before the regression.
This is a regression, first shipped in v4.11.0; v4.10.1 is unaffected.
#9286 introduced the mechanism, #9523 extended it to unknown paths, and
#11435 removed it while reorganising the middleware packages:
> Removed setting of URI to context in metric middleware. There were
only setters and no getters. (Unused value)
The getter was `*recorder.RequestURI` in `RegisterRequestCounter` /
`RegisterRequestCodeCounter`, which the same PR replaced with
`baseURI(r)`. Both ends went at once, so nothing failed.
# How the Problems Are Solved
- `metrics.WithRequestURIPattern` / `metrics.SetRequestURIPattern` are
back, and `metrics.RequestURI` is now the single place deciding what to
label with: the pattern a router reported, or the requested path if none
did. `UnknownPath` moved to the metrics package so every surface shares
one constant.
- The HTTP metrics middleware prepares the context before passing the
request on, which is the half that got dropped.
- `setRequestURIPattern` in the grpc-gateway reports the pattern to
metrics again, not only to tracing, and is now also called on the `405`
branch.
- Requests routed by chi (the OIDC endpoints) fall back to the pattern
chi matched. That covers the RFC 7592 client configuration routes added
in #12315 (`/oauth/v2/register/{client_id}`), which are templated on the
client ID, and collapses unknown paths below the OIDC prefixes to
`UNKNOWN_PATH` as well.
Against the metrics dump from our instance,
`http_server_return_code_counter_total` drops from 4,467 series to
roughly 35, and the endpoint as a whole from 8,000 to about 3,570.
# Additional Changes
- Regression tests, which were missing and are the reason the revert
went unnoticed:
- `metrics`: resolution of the `uri` label, including that a pattern set
on a derived context reaches the middleware.
- `gateway`: every way the gateway can answer — success, error,
unroutable path, wrong method.
- `middleware`: both routing styles, i.e. a router reporting its own
pattern and a chi routed request.
- A doc comment on `SetRequestURIPattern` explaining that it is written
by routers and read back by the middleware through the context, so it
does not read as an unused setter again.
# Additional Context
- Closes#12556
- Restores #9286 and #9523
- Regressed in #11435
Co-authored-by: Cursor <[email protected]>
# Which Problems Are Solved
Streams allow differentiating logs produced by different components of
Zitadel.
# How the Problems Are Solved
The `backend/v3/instrumentation/logging` package now exposes convenience
function for setting and getting a logger from the context. As well as
high-level functions to emit log records at various levels. When
constructing a new logger a "stream" needs to be specified:
- **runtime**: General runtime logs, such as startup and shutdown
messages. Default for logs that do not belong to the other categories.
- **request**: Logs for incoming API and HTTP requests.
- **event_handler**: Logs for event handling in projections.
- **queue**: Logs for the job queue processing.
- **event_pusher**: Logs for event pushing to the database. Disabled by
default, contains sensitive information.
Each line from the returned logger contains a `stream` field as well as
a `version` field with the current Zitadel version.
## Runtime config
Streams can be enabled by passing an array of stream names in the
runtime config. Because some log streams may contain sensitive data
(especially events), it is now also possible to mask values by their
key.
# Additional Changes
- Wrap `slogctx` in the `logging` package. (Except API error converter
packages, because of import cycle)
- Add some docs to `logging` package so other devs understand how to add
logging to Zitadel
- Add `logging.OnError` and `logging.WithError` helper functions with
`Panic()` and `Fatal()` methods, to preserve current calls in the `cmd`
packages.
- Add instance context extractor.
- Only output request details in the request info log. Request ID
remains propagated through context.
- Moved middleware functionality into protocol specific packages.
- Removed setting of URI to context in metric middleware. There were ony
setters and no getters. (Unused value)
- Reuse a single statusWriter in the middleware package for middlewares
that need to know the response status.
# Additional Context
- Closes#11333
- Closes#11331
- Partly #11330
---------
Co-authored-by: Silvan <[email protected]>
# Which Problems Are Solved
Zitadel did not provide easy correlation between errors, logs, traces
and metrics. The configuration for those instrumentations was also not
consistent, with some supporting different exporters then others.
Implementation and parsing of config was also spaghettified over
multiple packages, with awkward parsing and inconsistent naming of
options.
# How the Problems Are Solved
All telemetry is now merged under the name "instrumentation". Why?
1. We thought it was a good idea in the past to call the milestone
exporter `Telemtry` in the runtime config. Calling this `TelemetryV2`
looks weird.
2. Not everything is a meter and not everything is sent (tele...).
3. It's also
[defined](https://opentelemetry.io/docs/concepts/instrumentation/) as
such by the OTEL documentation.
## New features
- Adds structured, context based logging with trace-ID awareness
- Static log fields are added to the context, such as service and
request path
- Static log fields are injected in each logline emitted by the
application
- Structured logs can also be send to an otel exporter
- Structured logs can be printed to StdErr in text and JSON format
- Error sinks make sure every error is logged at the correct level:
- Warnings for client side errors (HTTP 400 range, Invalid request etc)
- Error for server side errors (Internal server errors)
- Metrics can now also be send to a OTEL collector. (previously they
could only be scraped from `/debug/metrics` with prometheus)
## Exporters
This change adds all the exporters supported by OTEL upstream and some
google specific exporters for our cloud deployment.
- StdOut / StdErr: all instrumentations
- OTEL gRPC / HTTP: all instrumentations
- Google: all instrumentations except logging
- Prometheus (pull-based): only metrics
The exception is profiling, which only supports the google exporting due
to lack of support by OTEL upstream.
## Configuration and structure
- All instrumentation is moved into the new `backend/v3/instrumentation`
package. It reuses configuration types, so both code and runtime
configuration are easier to understand.
- The `internal/telemetry` packages are removed.
- Instrumentation is started with a single function and a proper
shutdown function is now provided.
- Legacy configuration is still parsed from the runtime config, as long
as the new configuration is disabled. This allows backporting this
feature to v4 without breaking existing configurations.
# Additional Changes
- Devcontainer: set `$PATH` variable so installed go binaries can be run
individually, without NX.
- NX: install GCI tool to fix imports
# Additional Context
- Closes https://github.com/zitadel/zitadel/issues/8408
- Closes https://github.com/zitadel/zitadel/issues/6664
- Backport to v4
# Which Problems Are Solved
The metric `http_server_return_code_counter` doesn't record calls to the
gRPC gateway.
# How the Problems Are Solved
The DefaultMetricsHandler that is used for the gPRC gateway doesn't
record `http_server_return_code_counter`.
Instead of the DefaultMetricsHandler, a custom metrics handler which
includes `http_server_return_code_counter` is created for the gRPC
gateway
# Additional Changes
The DefaultMetricsHandler function is removed, as it is no longer used.
# Additional Context
Reported by a customer
---------
Co-authored-by: Silvan <[email protected]>
chore(fmt): run gci on complete project
Fix global import formatting in go code by running the `gci` command. This allows us to just use the command directly, instead of fixing the import order manually for the linter, on each PR.
Co-authored-by: Elio Bischof <[email protected]>
