Observability

Skipper provides three observability pillars: OpenTelemetry tracing, Prometheus metrics, and structured logging. All are configurable and can be enabled independently.

Enabling telemetry

Flag	Default	Purpose
`--telemetry`	false	Master switch for all telemetry
`--telemetry-trace`	true (when enabled)	Enable distributed tracing
`--telemetry-metric`	true (when enabled)	Enable Prometheus metrics
`--telemetry-metric-otlp`	false	Export metrics via OTLP in addition to Prometheus
`--telemetry-prometheus-host`	0.0.0.0	Prometheus scrape endpoint host
`--telemetry-prometheus-port`	9090	Prometheus scrape endpoint port

Tracing

Skipper uses the OpenTelemetry SDK with an OTLP exporter. The service name is skipper.controller or skipper.router depending on the component. Resource metadata includes container, environment, process, and runtime information.

Key features:

Errors logged at error level are automatically recorded on the active span.
Trace ID and span ID are added to all error-level log records for correlation.
Context attributes propagate to child spans automatically.

The router's HTTP transport and the controller's gRPC server are both instrumented for automatic distributed tracing. /healthz is excluded from tracing to reduce noise.

Prometheus metrics

Controller metrics (prefix: `skipper_controller_`)

Metric	Type	Labels	Purpose
`assignments_total`	Counter	function_deployment	Total pod assignments performed by the controller.
`heartbeats_total`	Counter	function_deployment	Heartbeats received from routers.
`informer_event_lag_seconds`	Histogram	resource, event	Lag between Kubernetes object change and informer processing (add and delete events only). (buckets: 62.5ms--8m32s)
`informer_events_total`	Counter	resource, event	Kubernetes informer events processed by the controller.
`informer_last_event_time_seconds`	Gauge	--	Unix timestamp of the last informer event processed.
`scale_downs_total`	Counter	function_deployment	Total scale-down operations performed by the controller.
`scale_ups_total`	Counter	function_deployment	Total scale-up operations performed by the controller.
`waiting_for_unassigned_pods`	Gauge	function_deployment	Functions blocked waiting for pod assignment.

Router metrics (prefix: `skipper_router_`)

Metric	Type	Labels	Purpose
`heartbeats_total`	Counter	function_deployment	Heartbeats sent to the controller.
`requests_in_flight`	Gauge	function_deployment	Requests currently being handled by the router.
`requests_total`	Counter	function_deployment	Total requests processed by the router.

Key insights

waiting_for_unassigned_pods > 0 indicates pod supply issues. The underlying deployment may need more replicas.
informer_event_lag_seconds spikes suggest API server or network pressure.
requests_in_flight correlates directly with scaling decisions when in-flight request scaling is configured.
Compare scale_ups_total vs scale_downs_total over time to understand scaling churn.

Structured logging

Default format is JSON (--log-format, options: json or text). Log levels: trace, debug, info, warn, error (--log-level, default: info). A custom TRACE level below DEBUG is available for very verbose output.

Structured fields propagate through the call chain, so a log entry inherits context from its parent operation. Attribute names are consistent between logs and traces.

Log-trace correlation

Error-level logs automatically include trace_id and span_id fields when a trace is active, enabling you to jump from a log entry directly to its distributed trace. Errors are also recorded on the active span for bidirectional correlation.