Observability

Overview#

RunR installs its observability stack as part of every Knative Serving installation on a region (KnativeInstallService::installServing()); no user action is needed. A region can instead reuse an existing in-cluster Prometheus or Loki (prometheus_managed / loki_managed = false); RunR then skips that install and reads the configured Service.

All components run in the observability namespace of the region's cluster.

On top of metrics and logs, RunR provides:

Components#

1. cAdvisor (metrics collector)#

Kind: DaemonSet — one pod per node Image: gcr.io/cadvisor/cadvisor:v0.49.2 Namespace: observability

Details:

• --containerd=<socket> — the containerd socket is set explicitly and detected per distro from the nodes' containerRuntimeVersion (/run/containerd/containerd.sock for kubeadm/managed Kubernetes, /run/k3s/containerd/containerd.sock for k3s/rke2)
• tolerations: [{operator: Exists}] — tolerates every taint, including runr.dev/dedicated
• hostPID: true, SYS_PTRACE capability (all others dropped) — needed to read cgroups
• The kube-prometheus-stack kubelet cAdvisor scrape is disabled; this DaemonSet replaces it

Metrics produced:

container_cpu_usage_seconds_total
container_memory_working_set_bytes

2. Prometheus (metrics store)#

Helm chart: prometheus-community/kube-prometheus-stack Release: knative Namespace: observability

Scrape targets:

• cAdvisor → PodMonitor CRD (cadvisor-runr) — 1s interval
• Knative Serving pods (container port named metrics) → additionalScrapeConfigs job knative-serving — 15s interval

Knative Serving is configured to export Prometheus metrics (config-observability ConfigMap, metrics-protocol: prometheus).

Access: in-cluster ClusterIP Service only — prometheus_url = http://prometheus-operated.observability:9090. Prometheus is not exposed outside the cluster (a prometheus-nodeport Service left by older installs is deleted).

3. Prometheus recording rules#

CRD: PrometheusRule/runr-metrics Labels: release: knative, app.kubernetes.io/part-of: runr

runr.instant (2s interval)#

• sum by(...) → collapses duplicate label sets (cAdvisor can report the same container under several label sets)
• namespace!="" → filters out non-Kubernetes system containers
• irate([30s]) → uses the last two samples
• Quick API queries do not use these rules; they query the raw metrics (see Data flow)

runr.averages (1m interval)#

4. PodMonitor (scrape configuration)#

CRD: PodMonitor/cadvisor-runr

interval:      1s
scrapeTimeout: 1s
honorLabels:   true

relabelings: __meta_kubernetes_pod_node_name → node

metricRelabelings:

• container_label_io_kubernetes_pod_namespace → namespace
• container_label_io_kubernetes_pod_name → pod
• container_label_io_kubernetes_container_name → container
• empty container name with a pod name → container="kata" (cAdvisor cannot see inside Kata VM-isolated containers)
• container_label_io_kubernetes_pod_name =~ .+ → keep (drops non-pod system cgroups)

5. Loki Stack (log aggregation)#

Helm chart: grafana/loki-stack Release: loki-stack Namespace: observability

Important: with tolerations: [{operator: Exists}] Promtail also runs on worker nodes tainted runr.dev/dedicated; without it, project pod logs on those nodes would not be collected.

Retention: Loki's own retention period is effectively unlimited (876000h); the compactor runs with deletes enabled so that LogRetentionService can remove log streams older than the region's retention_days through the Loki delete API (loki/api/v1/delete), keeping the last N deployments per project.

Loki status is tracked in the DB:

• loki_status: installing → install started
• loki_status: ready → Loki pod is Running, logs are queryable

6. Observability HA#

PriorityClass: observability-critical

value:            1_000_000
preemptionPolicy: Never

Observability pods are not evicted under node memory pressure; with Never preemption they do not preempt user pods either.

PodDisruptionBudgets:

During node drains and rolling upgrades at least one pod of each stays up. Both are single-replica setups; the PDBs protect them from voluntary disruption.

7. Dedicated workers#

Prometheus, Loki and Tempo keep their data on a PVC. A node-bound volume (local-path, k3s's default class, any volume with node affinity) keeps its pod on the node it was provisioned on, and that node can be selected as a dedicated worker of the region later (taint runr.dev/dedicated=projects:NoSchedule, architecture.md). The taint leaves the running pod alone; the next one (a restart, a Helm upgrade) has to start on the same node again. All three therefore tolerate the taint and prefer the nodes that are not dedicated workers (a preferred node affinity, so they still run where every node with room is a dedicated worker). Promtail and cAdvisor tolerate every taint and run on every node.

Data flow — metrics#

Container
    │
    ▼
cAdvisor DaemonSet (housekeeping: 1s)
    │  container_cpu_usage_seconds_total
    │  container_memory_working_set_bytes
    ▼
Prometheus (scrape: 1s, via PodMonitor)
    │
    ├─ [Quick]     last_over_time(rate(...[30s])[30s:5s]) — raw query, no rule dependency
    │
    └─ [Avg / 1h]  Recording rules (eval: 2s)
                   │  runr:cpu_cores:rate2m  (irate[30s])
                   │  runr:memory_mb:current
                   ▼
                   runr:cpu_cores:avg1h  (1m eval)
                   runr:memory_mb:avg1h
    ▼
RunR API — SWR cache (quick: refreshed when older than 1s / full: 60s)
    │  1. Direct HTTP to the in-cluster Service (only when the region is RunR's own cluster)
    │  2. Kubernetes API service proxy of the region's cluster
    ▼
Vue UI (quick metrics poll: 2s, 1h averages poll: 60s)

GET /api/projects/{p}/deployments/{d}/metrics?quick=1 returns the quick set (cpu, memory, rps); without quick it adds the 1h averages. Schedule-triggered projects use a 6m window instead of 30s, since their CronJob pods are short-lived.

Data flow — logs#

Container stdout/stderr
    │
    ▼
Promtail DaemonSet (every node)
    │  tails /var/log/pods/*
    ▼
Loki (push)
    │
    ▼
RunR API (KnativeLogService)
    ├─ 1. Direct HTTP → loki_url (only when the region is RunR's own cluster)
    ├─ 2. Kubernetes API service proxy → loki-stack:3100
    └─ 3. kubectl logs fallback (Loki not ready, unreachable, or no lines on first load)
    │
    ▼
Vue UI (poll: 1s, incremental ?since=<ms>)

Incremental log fetching: the first load returns a 24-hour window (up to 2000 lines). Later polls send since=<last_ts_ms> and receive only newer lines.

LogQL query:

{namespace="<project-ns>", pod=~"<slug>-.*"}

The container filter is applied in the API (handler, or app for caas projects); queue-proxy logs are not shown. For an operator-supplied multi-tenant Loki, loki_org_id is sent as X-Scope-OrgID.

API access#

RunR reads a region's Prometheus and Loki as in-cluster Services on the region's own cluster (MonitoringEndpoint). Operator-supplied URLs must have the form http(s)://<service>.<namespace>.svc[.cluster.local]:<port>[/path]; the central API never fetches an arbitrary URL.

Install flow#

Inside KnativeInstallService::installServing(), in order:

1. configureKnativeMetrics()   → config-observability: metrics-protocol=prometheus

2. installPrometheus()         (prometheus_managed)
   ├─ helm install kube-prometheus-stack
   ├─ deployCAdvisor() → ServiceAccount, ClusterRole, ClusterRoleBinding, DaemonSet, PodMonitor, PrometheusRule
   └─ resolvePrometheusUrl() → http://prometheus-operated.observability:9090
   — or, for a reused Prometheus with Prometheus Operator CRDs:
   applyScrapeResourcesForExternalPrometheus() → cAdvisor, PodMonitors, runr:* rules
     (label app.kubernetes.io/part-of=runr; the operator's Prometheus must select them)

3. installLoki()               (loki_managed)
   └─ helm install loki-stack (Loki + Promtail, tolerations: Exists)

4. applyObservabilityHA()      (when Prometheus or Loki is managed)
   ├─ PriorityClass: observability-critical
   ├─ PDB: prometheus-pdb
   └─ PDB: loki-pdb

5. deployAgent()               → runr-agent (see monitoring.md)

6. DB update
   ├─ prometheus_url = http://prometheus-operated.observability:9090
   ├─ loki_url       = http://loki-stack.observability.svc.cluster.local:3100
   └─ loki_status    = 'installing' (managed) / 'ready' (reused)

7. TracingInstallService::installWithServing()   (observability.tracing.installWithServing)
   └─ OpenTelemetry Collector + Tempo for the region's tracing mode (see Tracing backend);
      a failed tracing install is logged on the Serving component, Serving stays installed

KnativeStatusService sets loki_status = 'ready' once the Loki pod is Running.

php artisan cluster:patch-observability {regionId?} re-applies the Prometheus and Loki Helm values on existing regions.

Tracing backend#

Every region has a tracing mode (Settings → Tracing, PUT /api/observability/regions/{region}/tracing). Regions without their own setting use observability.tracing.defaultMode (Helm).

Function pods always send to RunR's collector, whatever the mode, so a mode change needs no redeploy of projects. Export headers (API keys of a hosted backend) are stored encrypted, put into the Secret otel-collector-export and referenced from the collector as environment variables; the API returns their names only (export_header_names).

Status: installing → ready once the collector (and the managed Tempo) pods are Running and ready (observability:run tracing-status, every minute), or error with status_message. Projects can switch tracing on only in a region whose backend is ready.

Components#

The collector and Tempo request about as much as the other region add-ons; the region capacity offered to projects is the nodes' allocatable minus these requests, like every other platform pod (architecture.md). region:reconcile (step tracing) brings an installed backend to the chart versions, images, requests and init container above. A Tempo pod that is not ready with an earlier template (crash-looping) would keep its StatefulSet from rolling; the install deletes it and the StatefulSet starts it with the new template.

Function pods export to http://otel-collector.observability.svc.cluster.local:4318; RunR queries Tempo at http://tempo.observability.svc.cluster.local:3200 (managed) through the same direct / Kubernetes API service proxy chain as Prometheus and Loki (API access).

Knative spans#

The activator and queue-proxy spans go to the collector as well, so a trace shows the time a request waited for a cold start. RunR merges into KnativeServing.spec.config:

• observability (Knative ≥ 1.18): tracing-protocol: http/protobuf, tracing-endpoint: http://otel-collector.observability.svc.cluster.local:4318/v1/traces, tracing-sampling-rate
• tracing (older Knative): backend: zipkin, zipkin-endpoint: http://otel-collector.observability.svc.cluster.local:9411/api/v2/spans, sample-rate

The rate is the region's knative_sample_rate (default observability.tracing.knativeSampleRate, 0.1). Function spans sample at the project's own ratio: the universal wrapper follows a sampled parent and otherwise decides by trace id, so a Knative hop that did not sample does not switch a project's tracing off. Mode off sets tracing-protocol: none / backend: none.

Commands#

php artisan cluster:install-tracing {regionId?} {--mode=managed|reuse|external|off}

installs (or reconfigures) the backend of one region or of every Knative region; POST /api/observability/regions/{region}/tracing/install queues the same install (InstallTracingJob), DELETE /api/observability/regions/{region}/tracing removes it.

php artisan region:reconcile (the Helm bootstrap job runs it after every install and upgrade) runs the same install where tracing is enabled — the region's mode, or for a region without its own setting defaultMode while installWithServing is on — and the backend is missing or runs another chart version or image than the pinned ones (step tracing, TracingBackendStep; upgrade.md). Where tracing is not enabled it installs nothing.

Auto-instrumentation#

Tracing is switched on per project (project → Traces → "Tracing", or PUT /api/projects/{p}/observability/tracing {enabled, sample_ratio}) and applies with the next deploy (requires_redeploy in the response). With tracing off the pod spec carries no tracing variables and the images start as before.

With tracing on, ObservabilityPodSpec adds to the function container:

Variables the project sets itself win over these.

Server span of the universal wrapper#

Every code runtime runs behind the universal wrapper when tracing or replay is on: custom runtimes already do, built-in http runtimes are fronted by it in observed http mode (the wrapper listens on $PORT, the runtime's server on $PORT + 1). The wrapper creates one SERVER span per invocation, continues an incoming traceparent, passes its own context to the runtime (traceparent request header; TRACEPARENT for schedule runs) and returns traceresponse. Span attributes:

A 5xx or a timeout marks the span as error and adds an exception event with the handler's error output. Spans are batched (up to 256 per request, every second, queue of 2048) and sent as OTLP/HTTP JSON; the wrapper uses the Go standard library only.

Built-in runtimes#

The instrumentation packages are part of the images (build argument OTEL=0 builds without them) and are loaded only when tracing is on.

Traces and logs#

With tracing on the wrapper writes one access line per invocation to the function's log:

[runr-wrapper] trace_id=4bf92f3577b34da6a3ce929d0e0e4736 span_id=00f067aa0ba902b7 sampled=true GET /orders 500 41.3ms cold_start=false

Python's logging instrumentation adds otelTraceID / otelSpanID to log records. The trace view's "Logs" panel queries Loki for the project's lines that contain the trace id ({namespace=…, pod=~"<slug>-.*"} |= "<trace id>") from one minute before the trace to five minutes after it.

Traces tab and API#

Project → Traces (permission traces.view): filters for status (all / error / ok), minimum duration, path substring and time range (15 min – 7 days); the list shows start time, service and root span, path, status, duration and cold start; a trace opens as a waterfall (spans indented by parent, bars on the trace's time axis, errors marked) with the attributes, events, error and resource of each span, and the trace's log lines.

Queries run server-side against the region's Tempo; a trace id of another project answers 404. 409 not_queryable — the region's mode has no Tempo for RunR (external / off) or it is not ready; 503 — Tempo did not answer. The three routes answer Cache-Control: no-store, private without an ETag: spans keep arriving, so neither the API's response cache nor a proxy keeps a trace (or its 404) (API caching).

Request replay#

Opt-in per project (project → Replay → "Capture failed requests", or PUT /api/projects/{p}/observability/replay), code runtimes with http or queue triggers, applies with the next deploy. The universal wrapper posts every failed request — HTTP status ≥ 500 or a timeout — to RunR (POST /api/internal/observability/replay/{key}, a per-project HMAC token that changes whenever capture is switched on again).

Redaction happens in the wrapper, before a capture leaves the pod, and again in the API:

• never captured: Authorization, Proxy-Authorization, Cookie, Set-Cookie, X-Api-Key, X-Auth-Token, X-CSRF-Token, X-XSRF-Token, X-Amz-Security-Token, X-Goog-IAP-JWT-Assertion, hop-by-hop headers, every header whose name ends like a secret (the rule of secret environment variables: …token, …secret, …password, …api_key, …access_key, …credentials, …) and the project's own list; the capture records their names (redacted_headers)
• query values of access_token, api_key, apikey, password, secret, token and the project's own list are replaced by REDACTED
• headers, bodies and replay results are stored encrypted

Replay (POST /api/projects/{p}/replays/{id}/replay, permission replay.manage) re-sends the request to the project's endpoint (https://<slug>.<knative domain>), which serves the current revision. Redacted headers are not sent; headers needed for the replay (for example a test Authorization) can be given for that replay only (headers, never stored), as can a different query. The request carries X-RunR-Replay: 1 and is never captured again. The response is compared with the original: status, added / removed / changed headers (volatile ones such as Date ignored) and a line diff of the bodies (JSON pretty-printed first).

Privacy. Captures hold request bodies and the headers left after redaction, which can include personal data. Capture is off by default; switch it on only where storing failed requests is acceptable, add the project's own sensitive headers and query parameters to the redaction lists, keep the retention short and delete captures (one, or all of a project) once they are no longer needed. Captures are removed with their project and after their retention (observability:run prune, daily).

Uptime checks#

Project → Uptime (uptime.manage to change, metrics.view to read). A check probes either a path on the project's endpoint or an absolute url.

State per location: down after failure_threshold consecutive failures, up after recovery_threshold consecutive successes (the first success of a new check is up at once). The check is down while any location is down and up when every location is up. Down opens an incident and fires the uptime_down alert rules; up resolves the incident and the alert. An incident's title and message can be edited (PATCH /api/projects/{p}/uptime-incidents/{id}) and appear on the status page.

The control plane probes due checks every 30 seconds (observability:run uptime, parallel requests). Region agents fetch their checks (GET /api/internal/observability/agent/{region}/uptime-checks) and post results (POST …/uptime-results) with the agent token; the agent finds RunR's API through RUNR_API_URL or its WebSocket URL, and RUNR_UPTIME_DISABLED=true switches its prober off. The UI shows the uptime % of 24 h / 7 d / 30 d, the average response time, daily bars of the last 30 days, the latest results and the incidents. Results are kept resultsRetentionDays (90).

Status page#

One public page per organization (Settings → Status page, status_page.manage): slug, title, description, logo, optional custom domain, incident history on / off and its length (≤ 90 days). Components are the uptime checks with "show on status page" (own display name and order). The page shows the overall state, each component's state and daily uptime bars, and the incident history. It uses the organization's brand (product name, colours, logo) when white-label branding is set.

• https://<console host>/status/<slug> — no sign-in; only pages switched public
• custom domain — point it (CNAME) at the console host and add it to the console's ingress and certificate; the page then answers at https://<custom domain>/status
• API: GET /api/public/status-pages/{slug}, GET /api/public/status-pages?domain=<host> (no authentication, 60 requests / minute); the payload contains components and incidents only. It answers Cache-Control: public, max-age=30 (with an ETag), so a proxy or CDN in front of the console may cache it for 30 seconds

Alerting#

Settings → Alerts (alerts.manage).

Channels#

Webhook URLs, bot tokens and signing secrets are stored encrypted and never returned: the API shows the host of a URL and whether a secret is set. URLs on private addresses are refused unless alerts.allowPrivateTargets. "Send test" delivers a test message at once and records the result on the channel.

Webhook requests carry X-RunR-Event (alert.firing, alert.resolved, test), X-RunR-Timestamp, X-RunR-Alert-Id and, with a secret, X-RunR-Signature: sha256=<hex HMAC-SHA256(secret, "<timestamp>.<raw body>")>. The body:

{"event": "alert.firing", "id": "01J…", "type": "error_rate", "title": "…", "text": "…",
 "severity": "critical", "status": "firing", "fields": {"project": "orders", "value": "12.5 %"},
 "url": "https://console.example.com/projects/…", "fired_at": "…", "sent_at": "…"}

Rules#

The last three read the span metrics of the region's Prometheus (traced projects) every minute (observability:run alerts) and resolve when the value is back under the threshold. Every rule has a severity (info, warning, critical), a cooldown (default 15 minutes) and the channels it routes to. Alerts are deduplicated by rule and subject (project, region + resource, uptime check): a still-firing alert is sent again only after the cooldown; a resolve is always sent.

Delivery#

Every alert creates one delivery per channel, sent by a queued job with up to 4 tries (alerts.deliveryTries) and backoff 30 s / 2 min / 10 min. The delivery log (GET /api/observability/alert-deliveries) shows status, attempts, the last error and the response code; a failed delivery can be retried (POST …/alert-deliveries/{id}/retry). Alert events (GET /api/observability/alert-events) list firing and resolved alerts. Deliveries are kept 30 days.

MCP tools#

The RunR MCP server (AI agents) gives agents traces, request replay and uptime checks: observability_get / observability_update, traces_search, trace_get (span tree and correlated logs), replays_list, replay_get, replay_run, uptime_checks_list, uptime_check_create, uptime_check_update, uptime_check_delete, uptime_check_run, uptime_check_results, uptime_incidents_list, alert channels (alert_channels_list, alert_channel_create, alert_channel_update, alert_channel_delete, alert_channel_test), alert rules (alert_rules_list, alert_rule_create, alert_rule_update, alert_rule_delete), alert_events_list, and the status page (status_page_get, status_page_update) — through the routes above, with the same permissions.

Permissions#

Switching tracing or replay capture on for a project needs projects.update; reading the region tracing settings needs regions.view. Built-in roles: Developer has traces.view, replay.manage, uptime.manage; Viewer has traces.view; Owner and Admin have all. See permissions.md.

Audit log#

Recorded actions (never with secret values; channel changes record the changed field names): region.tracing.update, region.tracing.install, region.tracing.uninstall, project.tracing.update, project.replay.update, replay.replay, replay.delete, replay.delete_all, uptime_check.create, uptime_check.update, uptime_check.delete, uptime_check.run, uptime_incident.update, status_page.update, alert_channel.create, alert_channel.update, alert_channel.delete, alert_channel.test, alert_rule.create, alert_rule.update, alert_rule.delete, alert_delivery.retry. The internal ingest routes (replay captures from function pods, uptime results from region agents) are machine traffic and are not audited.

Configuration#

Helm values (observability.*, rendered into the API's OBS_* variables):

Further settings are environment variables of api/config/observability.php, set via extraEnv (for example OBS_TRACE_SEARCH_MAX_HOURS, OBS_REPLAY_PER_MINUTE, OBS_ALERT_DELIVERY_BACKOFF, OBS_NODE_INSTRUMENTATIONS, OBS_SPANMETRICS_*).

Scheduled tasks (ObservabilityServiceProvider): observability:run uptime every 30 s, observability:run alerts and observability:run tracing-status every minute, observability:run prune daily.

Related files#

See also: monitoring.md (runr-agent, metric snapshots, cost), kata-hypervisors.md.