mirror of
https://github.com/andwati/zitadel.git
synced 2026-11-17 19:05:13 +00:00
3be21ccbf07337ea9a7faa27a02f360ccb34638d
3
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
83854e80e0 |
fix(telemetry): record route patterns instead of request paths on HTTP metrics (#12557)
# Which Problems Are Solved - The `uri` label of `http.server.request_count` and `http.server.return_code_counter` holds the concrete request path, so every object ID served over a REST path becomes its own metric series. On our v4.16.2 instance `/debug/metrics` exposes 8,000 series, 4,467 of which (56%) belong to `http_server_return_code_counter_total` alone, spread over 4,390 distinct `uri` values such as `/v2/sessions/385063742120926058`. The set only grows with the number of objects the installation has served. - Paths matching no route are recorded verbatim, so unauthenticated scanner traffic (`/wp-json/`, `/login.php`, …) keeps adding series. - The grpc-gateway answers `405 Method Not Allowed` without going through the configured `errorHandler`, so those responses never reported a route pattern, not even before the regression. This is a regression, first shipped in v4.11.0; v4.10.1 is unaffected. #9286 introduced the mechanism, #9523 extended it to unknown paths, and #11435 removed it while reorganising the middleware packages: > Removed setting of URI to context in metric middleware. There were only setters and no getters. (Unused value) The getter was `*recorder.RequestURI` in `RegisterRequestCounter` / `RegisterRequestCodeCounter`, which the same PR replaced with `baseURI(r)`. Both ends went at once, so nothing failed. # How the Problems Are Solved - `metrics.WithRequestURIPattern` / `metrics.SetRequestURIPattern` are back, and `metrics.RequestURI` is now the single place deciding what to label with: the pattern a router reported, or the requested path if none did. `UnknownPath` moved to the metrics package so every surface shares one constant. - The HTTP metrics middleware prepares the context before passing the request on, which is the half that got dropped. - `setRequestURIPattern` in the grpc-gateway reports the pattern to metrics again, not only to tracing, and is now also called on the `405` branch. - Requests routed by chi (the OIDC endpoints) fall back to the pattern chi matched. That covers the RFC 7592 client configuration routes added in #12315 (`/oauth/v2/register/{client_id}`), which are templated on the client ID, and collapses unknown paths below the OIDC prefixes to `UNKNOWN_PATH` as well. Against the metrics dump from our instance, `http_server_return_code_counter_total` drops from 4,467 series to roughly 35, and the endpoint as a whole from 8,000 to about 3,570. # Additional Changes - Regression tests, which were missing and are the reason the revert went unnoticed: - `metrics`: resolution of the `uri` label, including that a pattern set on a derived context reaches the middleware. - `gateway`: every way the gateway can answer — success, error, unroutable path, wrong method. - `middleware`: both routing styles, i.e. a router reporting its own pattern and a chi routed request. - A doc comment on `SetRequestURIPattern` explaining that it is written by routers and read back by the middleware through the context, so it does not read as an unused setter again. # Additional Context - Closes #12556 - Restores #9286 and #9523 - Regressed in #11435 Co-authored-by: Cursor <[email protected]> |
||
|
|
11dbb1b277 |
feat(logging): add streams (#11435)
# Which Problems Are Solved Streams allow differentiating logs produced by different components of Zitadel. # How the Problems Are Solved The `backend/v3/instrumentation/logging` package now exposes convenience function for setting and getting a logger from the context. As well as high-level functions to emit log records at various levels. When constructing a new logger a "stream" needs to be specified: - **runtime**: General runtime logs, such as startup and shutdown messages. Default for logs that do not belong to the other categories. - **request**: Logs for incoming API and HTTP requests. - **event_handler**: Logs for event handling in projections. - **queue**: Logs for the job queue processing. - **event_pusher**: Logs for event pushing to the database. Disabled by default, contains sensitive information. Each line from the returned logger contains a `stream` field as well as a `version` field with the current Zitadel version. ## Runtime config Streams can be enabled by passing an array of stream names in the runtime config. Because some log streams may contain sensitive data (especially events), it is now also possible to mask values by their key. # Additional Changes - Wrap `slogctx` in the `logging` package. (Except API error converter packages, because of import cycle) - Add some docs to `logging` package so other devs understand how to add logging to Zitadel - Add `logging.OnError` and `logging.WithError` helper functions with `Panic()` and `Fatal()` methods, to preserve current calls in the `cmd` packages. - Add instance context extractor. - Only output request details in the request info log. Request ID remains propagated through context. - Moved middleware functionality into protocol specific packages. - Removed setting of URI to context in metric middleware. There were ony setters and no getters. (Unused value) - Reuse a single statusWriter in the middleware package for middlewares that need to know the response status. # Additional Context - Closes #11333 - Closes #11331 - Partly #11330 --------- Co-authored-by: Silvan <[email protected]> |
||
|
|
eb22b58756 |
feat(telemetry): improved instrumentation for observability (#11159)
# Which Problems Are Solved Zitadel did not provide easy correlation between errors, logs, traces and metrics. The configuration for those instrumentations was also not consistent, with some supporting different exporters then others. Implementation and parsing of config was also spaghettified over multiple packages, with awkward parsing and inconsistent naming of options. # How the Problems Are Solved All telemetry is now merged under the name "instrumentation". Why? 1. We thought it was a good idea in the past to call the milestone exporter `Telemtry` in the runtime config. Calling this `TelemetryV2` looks weird. 2. Not everything is a meter and not everything is sent (tele...). 3. It's also [defined](https://opentelemetry.io/docs/concepts/instrumentation/) as such by the OTEL documentation. ## New features - Adds structured, context based logging with trace-ID awareness - Static log fields are added to the context, such as service and request path - Static log fields are injected in each logline emitted by the application - Structured logs can also be send to an otel exporter - Structured logs can be printed to StdErr in text and JSON format - Error sinks make sure every error is logged at the correct level: - Warnings for client side errors (HTTP 400 range, Invalid request etc) - Error for server side errors (Internal server errors) - Metrics can now also be send to a OTEL collector. (previously they could only be scraped from `/debug/metrics` with prometheus) ## Exporters This change adds all the exporters supported by OTEL upstream and some google specific exporters for our cloud deployment. - StdOut / StdErr: all instrumentations - OTEL gRPC / HTTP: all instrumentations - Google: all instrumentations except logging - Prometheus (pull-based): only metrics The exception is profiling, which only supports the google exporting due to lack of support by OTEL upstream. ## Configuration and structure - All instrumentation is moved into the new `backend/v3/instrumentation` package. It reuses configuration types, so both code and runtime configuration are easier to understand. - The `internal/telemetry` packages are removed. - Instrumentation is started with a single function and a proper shutdown function is now provided. - Legacy configuration is still parsed from the runtime config, as long as the new configuration is disabled. This allows backporting this feature to v4 without breaking existing configurations. # Additional Changes - Devcontainer: set `$PATH` variable so installed go binaries can be run individually, without NX. - NX: install GCI tool to fix imports # Additional Context - Closes https://github.com/zitadel/zitadel/issues/8408 - Closes https://github.com/zitadel/zitadel/issues/6664 - Backport to v4 |