Observability & Alerting
The agent ships metrics and eBPF traces to the hub, where they are stored in TimescaleDB and power charts, a service graph, alert rules and anomaly detection. You do not run Prometheus, Alertmanager, Jaeger or Hubble to get any of it.
Metrics, Traces & Health
Metrics without Prometheus
Spoke agents scrape node, pod and apiserver samples and ship them over the uplink. The hub stores them in TimescaleDB. There is no REST ingest, so one tenant cannot write into another tenant's charts.
Dashboards & Explorer
Observe → Metrics dashboards, a metric explorer over kube-state and scraped OpenMetrics series, and RED signals (rate, errors, latency) for the apiserver and individual pods.
Bounded Range Queries
Label-equality range reads with steps from 15s to 1h, capped at 50 series and 5,000 points per chart. Resource pages fall back to this history when a live query comes back empty.
eBPF Service Topology
The node agent's eBPF HTTP tracer builds a live service graph from real traffic. No Cilium or Hubble required. Opt-in per spoke, kernel 5.8 or newer.
Traces without Jaeger
The same eBPF spans feed Observe → Traces: a per-trace list and a span waterfall, stored in TimescaleDB on the hub.
Cluster Health Audits
Ten read-only audits in one findings table, worst first: Deployments, StatefulSets, DaemonSets, Jobs, CronJobs, HPAs, storage provisioners, service endpoints, aggregated APIs and cluster DNS.
Built-in Retention
Compression and retention are set by the hub schema. Nothing to size or operate separately.
| Data | Kept for |
|---|---|
| Raw node / pod samples | 7 days |
| Samples, 1-minute aggregate | 30 days |
| OpenMetrics & kube-state series | 14 days |
| Request metrics (RED) | 30 days |
| Trace spans | 7 days |
Native Alerting
Alert Rules
- Threshold rules over one cluster's metric series, evaluated by the hub every 30 seconds
- Counter transforms (rate, delta, increase), reducers (avg, max, min, sum, last) and six comparators
- Normal → Pending → Firing → Resolved, with a "for" duration before a rule fires
- Only the leader replica evaluates, so each Fire and Resolve is sent once
- Fire/resolve history is kept for every rule, even one with no channel attached
- Tenants see rules, states and history only for the clusters they own
Notification Channels
- SMTP, Slack, Microsoft Teams and generic webhook channels, each with a Test button
- Live alerts arrive as Slack Block Kit and Teams Adaptive Cards
- Webhook URLs pointing at private, loopback, link-local, .local or .internal hosts are refused
- Redirects are never followed, so a 302 cannot bounce a send to a blocked address
- Channel configs, including webhook URLs and SMTP passwords, are encrypted at rest
Anomalies, Digests & Postmortems
Anomaly Detection
- Learns each series' own normal from a 60-minute baseline (median ± MAD). No static thresholds, no ML
- Opens only after at least 3 consecutive anomalous minutes, with hysteresis on resolve
- Severity needs both an unusual and a material move, so a tiny wiggle on an idle series stays a Warning
- At most 8 firing series per metric, so one noisy metric cannot flood the feed
- Optional batched email of High and Critical anomalies, at most one every 15 minutes by default
Fleet-Health Digest
- Daily or weekly email: nodes, pods, firing alerts, firing anomalies and CPU / memory per cluster
- Registered clusters that are offline are still listed
- One-click unsubscribe (RFC 8058)
- Each send is claimed first, so two replicas never mail the same report
Incident Postmortems
- Draft a blameless RCA from a firing alert, built from alerts, significant events, error logs, anomalies and remediations in the window
- Timeline, detection time and blast radius are filled in by code, never by the model
- Every citation is checked against the evidence. Fabricated references are dropped, and root causes below 0.8 confidence are marked as hypotheses
- Person names are scrubbed. Works without an LLM, and stays a draft until an admin approves it