# Which Problems Are Solved
In the "new" structured logging, request details were added in different
middlewares, such as request, instance and user IDs. This meant the the
upstream request logging middleware did not have access to metadata that
got added later, resulting in incomplete logs. Furthermore it was not
possible to correlate an API error response to log output.
# How the Problems Are Solved
A mutable request details object is added to the context early on. When
the api authz function run, the instance and user IDs are added to this
object as they become available. Every logline emitted after this
(time-wise) will then all contain these details under the `request` log
group.
<details>
<summary>example output in JSON</summary>
```json
{
"time": "2026-03-13T19:21:20.890428806Z",
"level": "INFO",
"source": {
"function": "github.com/zitadel/zitadel/internal/api/grpc/server/connect_middleware.LogHandler.func1.1",
"file": "/workspaces/zitadel/internal/api/grpc/server/connect_middleware/log_interceptor.go",
"line": 34
},
"msg": "request served",
"request": {
"id": "d6q67c04vtjmi77cbbpg",
"instance_host": "localhost:8080",
"instance_id": "362349751439458307",
"user_id": "362349751440048131"
},
"TraceID": "a9e0fee3522224f3583bbdcda737f4b4",
"SpanID": "a563056eee36920a",
"stream": "request",
"version": "2026-03-13T19:20:59Z",
"protocol": "connect",
"service": "zitadel.user.v2.UserService",
"http_method": "POST",
"path": "/zitadel.user.v2.UserService/ListUsers",
"code": "code_0",
"duration": 12254350
}
```
</details>
Request IDs are now also returned with a response header or metadata.
Depending on the protocol:
- HTTP calls always return the request ID as header, regardless of
status
- gRPC calls always return the request ID as header, even if there was
an error
- connect RPC calls returns the request ID as header on success, trailer
in case of error. This is because header must be set on the response
object, which is nil in case of error. When there is an error, metadata
can be added which are then sent as trailers.
# Additional Changes
- Use the existing call duration middleware for both request ID and
logging for a consistent request start timestamp in all layers.
- Upgrade sloggcp for some fixes (notably TraceID)
- Modify the NoCache middleware so it uses `SetHeaders` instead of
`SendHeaders`. The latter prevented any other handler from setting
headers, including the new request ID middleware.
# Additional Context
Follow up on demo of:
- https://github.com/zitadel/zitadel/pull/11159
- https://github.com/zitadel/zitadel/pull/11435
- backport to v4
---------
Co-authored-by: Copilot <[email protected]>
Co-authored-by: muhlemmer <[email protected]>
Co-authored-by: Copilot Autofix powered by AI <[email protected]>
Co-authored-by: Marco A. <[email protected]>
# Which Problems Are Solved
Panics may cause a service discruption by unexpectedly closing an
request's connection. Or in the case of gRPC completely killing the
service.
Allthough panics are still individual bugs that need to be solved, this
PR makes sure a panic is gracefully handled and an understandable error
is returned to the client.
# How the Problems Are Solved
- Recover in the middleware interceptors for the 3 API protocols (HTTP,
gRPC, connect).
- HTTP middleware uses formatted responses for:
- OIDC errors (JSON formatted response)
- UI (error page rendering)
- SCIM
- Upon recovery an alert level log is printed (ERROR+4 for stdlib log
handlers)
# Additional Context
- internal observation
---------
Co-authored-by: Livio Spring <[email protected]>
Co-authored-by: Livio Spring <[email protected]>
# Which Problems Are Solved
Streams allow differentiating logs produced by different components of
Zitadel.
# How the Problems Are Solved
The `backend/v3/instrumentation/logging` package now exposes convenience
function for setting and getting a logger from the context. As well as
high-level functions to emit log records at various levels. When
constructing a new logger a "stream" needs to be specified:
- **runtime**: General runtime logs, such as startup and shutdown
messages. Default for logs that do not belong to the other categories.
- **request**: Logs for incoming API and HTTP requests.
- **event_handler**: Logs for event handling in projections.
- **queue**: Logs for the job queue processing.
- **event_pusher**: Logs for event pushing to the database. Disabled by
default, contains sensitive information.
Each line from the returned logger contains a `stream` field as well as
a `version` field with the current Zitadel version.
## Runtime config
Streams can be enabled by passing an array of stream names in the
runtime config. Because some log streams may contain sensitive data
(especially events), it is now also possible to mask values by their
key.
# Additional Changes
- Wrap `slogctx` in the `logging` package. (Except API error converter
packages, because of import cycle)
- Add some docs to `logging` package so other devs understand how to add
logging to Zitadel
- Add `logging.OnError` and `logging.WithError` helper functions with
`Panic()` and `Fatal()` methods, to preserve current calls in the `cmd`
packages.
- Add instance context extractor.
- Only output request details in the request info log. Request ID
remains propagated through context.
- Moved middleware functionality into protocol specific packages.
- Removed setting of URI to context in metric middleware. There were ony
setters and no getters. (Unused value)
- Reuse a single statusWriter in the middleware package for middlewares
that need to know the response status.
# Additional Context
- Closes#11333
- Closes#11331
- Partly #11330
---------
Co-authored-by: Silvan <[email protected]>
# Which Problems Are Solved
Zitadel did not provide easy correlation between errors, logs, traces
and metrics. The configuration for those instrumentations was also not
consistent, with some supporting different exporters then others.
Implementation and parsing of config was also spaghettified over
multiple packages, with awkward parsing and inconsistent naming of
options.
# How the Problems Are Solved
All telemetry is now merged under the name "instrumentation". Why?
1. We thought it was a good idea in the past to call the milestone
exporter `Telemtry` in the runtime config. Calling this `TelemetryV2`
looks weird.
2. Not everything is a meter and not everything is sent (tele...).
3. It's also
[defined](https://opentelemetry.io/docs/concepts/instrumentation/) as
such by the OTEL documentation.
## New features
- Adds structured, context based logging with trace-ID awareness
- Static log fields are added to the context, such as service and
request path
- Static log fields are injected in each logline emitted by the
application
- Structured logs can also be send to an otel exporter
- Structured logs can be printed to StdErr in text and JSON format
- Error sinks make sure every error is logged at the correct level:
- Warnings for client side errors (HTTP 400 range, Invalid request etc)
- Error for server side errors (Internal server errors)
- Metrics can now also be send to a OTEL collector. (previously they
could only be scraped from `/debug/metrics` with prometheus)
## Exporters
This change adds all the exporters supported by OTEL upstream and some
google specific exporters for our cloud deployment.
- StdOut / StdErr: all instrumentations
- OTEL gRPC / HTTP: all instrumentations
- Google: all instrumentations except logging
- Prometheus (pull-based): only metrics
The exception is profiling, which only supports the google exporting due
to lack of support by OTEL upstream.
## Configuration and structure
- All instrumentation is moved into the new `backend/v3/instrumentation`
package. It reuses configuration types, so both code and runtime
configuration are easier to understand.
- The `internal/telemetry` packages are removed.
- Instrumentation is started with a single function and a proper
shutdown function is now provided.
- Legacy configuration is still parsed from the runtime config, as long
as the new configuration is disabled. This allows backporting this
feature to v4 without breaking existing configurations.
# Additional Changes
- Devcontainer: set `$PATH` variable so installed go binaries can be run
individually, without NX.
- NX: install GCI tool to fix imports
# Additional Context
- Closes https://github.com/zitadel/zitadel/issues/8408
- Closes https://github.com/zitadel/zitadel/issues/6664
- Backport to v4
# Which Problems Are Solved
The payload in actions V2 is currently sent as JSON to the target
endpoint. It might get exposed to intermediary infrastructure or logging
systems.
For these scenarios there needs to be an application-layer encryption,
where the provider of the endpoint can define an key to be used for the
encryption.
# How the Problems Are Solved
- Added an additional option to the target to specify the payload type:
`JSON` (current and default), `JWT`, `JWE` (api and console)
- added endpoints to upload and manage public keys (to be used for
encryption) for a target
- updated all action v2 executions (interceptors, oidc, saml, ...) to
provide the `GetActiveSigningWebKey` from queries
- implemented jwt and jwe in the exections incl. refactoring of code and
tests
- changes to the authn_keys table:
- added a `fingerprint` column
- dropped not null constraint on expiration
- moved the `GetSignerOnce` into its own package to prevent circular
dependencies
# Additional Changes
None
# Additional Context
closes#11061
---------
Co-authored-by: Marco A. <[email protected]>
Co-authored-by: conblem <[email protected]>
Co-authored-by: Silvan <[email protected]>
# Which Problems Are Solved
As part of our efforts to simplify the structure and versions of our
APIs, were moving all existing v2beta endpoints to v2 and deprecate
them. They will be removed in Zitadel V5.
# How the Problems Are Solved
- This PR moves instance v2beta service and its endpoints to a
corresponding v2 version. The v2beta service and endpoints are
deprecated.
- The docs are moved to the new GA service and its endpoints. The v2beta
is not displayed anymore.
- The comments and have been improved and, where not already done, moved
from swagger annotations to proto.
- All required fields have been marked with (google.api.field_behavior)
= REQUIRED and validation rules have been added where missing
- `Domain` has been renamed to `CustomDomain` to align with naming
conventions
- `..Query` has been renamed to `..Filter` to align with other services
- The `instance_id` parameter can now passed on all endpoints and is
properly used, but requires `system` permissions. It can be omitted to
use the own instance (identified by context as any other service).
- The following endpoints are affected:
- GetInstance
- UpdateInstance
- ListCustomDomains
- AddTrustedDomain
- RemoveTrustedDomain
- ListTrustedDomains
- InstanceService has been added the InstanceInterceptor's
`explicitInstanceIdServices` to allow passing the id
- If the instance is not found by id, the error is not directly returned
to prevent enumeration.
- Permissions are checked in the API instead of the interceptor for
these calls.
- Setting the same instance name in the update no longer returns an
error, but the previous change date.
# Additional Changes
none
# Additional Context
- part of https://github.com/zitadel/zitadel/issues/10772
- requires backport to v4.x
# Which Problems Are Solved
Requests without errors were mapped being mapped to GRPC status code
`Unknown`, which were then being mapped as HTTP `500` status code.
# How the Problems Are Solved
By deriving the grpc status codes from the error only when there's an
error.
When the error is `nil`, the grpc status code is set to 0 (`OK`).
# Additional Changes
N/A
# Additional Context
- Closes#10884
# Which Problems Are Solved
Comparing the v3 and v4 deployments we noticed an increase in memory
usage. A first analysis revealed that it might be related to the
(multiple) initialization of the `i18n.Translator`, partially related
# How the Problems Are Solved
Initialize the tranlator once (apart from the translator interceptor,
which uses context / request specific information) and pass it to all
necessary middleware.
# Additional Changes
Removed unnecessary error return parameter from the translator
initialization.
# Additional Context
- noticed internally
- backport to v4.x
# Which Problems Are Solved
This PR adds functionality to propagate request headers in actions v2.
# How the Problems Are Solved
The new functionality is added to the`ExecutionHandler` interceptors,
where the incoming request headers (from a list of allowed headers to be
forwarded) are set in the payload of the request before calling the
target.
# Additional Changes
This PR also contains minor fixes to the Actions V2 example docs.
# Additional Context
- Closes#9941
---------
Co-authored-by: Marco A. <[email protected]>
# Which Problems Are Solved
The event execution system currently uses a projection handler that
subscribes to and processes all events for all instances. This creates a
high static cost because the system over-fetches event data, handling
many events that are not needed by most instances. This inefficiency is
also reflected in high "rows returned" metrics in the database.
# How the Problems Are Solved
Eliminate the use of a project handler. Instead, events for which
"execution targets" are defined, are directly pushed to the queue by the
eventstore. A Router is populated in the Instance object in the authz
middleware.
- By joining the execution targets to the instance, no additional
queries are needed anymore.
- As part of the instance object, execution targets are now cached as
well.
- Events are queued within the same transaction, giving transactional
guarantees on delivery.
- Uses the "insert many fast` variant of River. Multiple jobs are queued
in a single round-trip to the database.
- Fix compatibility with PostgreSQL 15
# Additional Changes
- The signing key was stored as plain-text in the river job payload in
the DB. This violated our [Secrets
Storage](https://zitadel.com/docs/concepts/architecture/secrets#secrets-storage)
principle. This change removed the field and only uses the encrypted
version of the signing key.
- Fixed the target ordering from descending to ascending.
- Some minor linter warnings on the use of `io.WriteString()`.
# Additional Context
- Introduced in https://github.com/zitadel/zitadel/pull/9249
- Closes https://github.com/zitadel/zitadel/issues/10553
- Closes https://github.com/zitadel/zitadel/issues/9832
- Closes https://github.com/zitadel/zitadel/issues/10372
- Closes https://github.com/zitadel/zitadel/issues/10492
---------
Co-authored-by: Stefan Benz <[email protected]>
# Which Problems Are Solved
The current maintained gRPC server in combination with a REST (grpc)
gateway is getting harder and harder to maintain. Additionally, there
have been and still are issues with supporting / displaying `oneOf`s
correctly.
We therefore decided to exchange the server implementation to
connectRPC, which apart from supporting connect as protocol, also also
"standard" gRCP clients as well as HTTP/1.1 / rest like clients, e.g.
curl directly call the server without any additional gateway.
# How the Problems Are Solved
- All v2 services are moved to connectRPC implementation. (v1 services
are still served as pure grpc servers)
- All gRPC server interceptors were migrated / copied to a corresponding
connectRPC interceptor.
- API.ListGrpcServices and API. ListGrpcMethods were changed to include
the connect services and endpoints.
- gRPC server reflection was changed to a `StaticReflector` using the
`ListGrpcServices` list.
- The `grpc.Server` interfaces was split into different combinations to
be able to handle the different cases (grpc server and prefixed gateway,
connect server with grpc gateway, connect server only, ...)
- Docs of services serving connectRPC only with no additional gateway
(instance, webkey, project, app, org v2 beta) are changed to expose that
- since the plugin is not yet available on buf, we download it using
`postinstall` hook of the docs
# Additional Changes
- WebKey service is added as v2 service (in addition to the current
v2beta)
# Additional Context
closes#9483
---------
Co-authored-by: Elio Bischof <[email protected]>
Mridang AgarwallaandGitHub
Gayathri VijayanandGitHub
