Hub Operations, Backup & DR
The hub runs as several replicas on a three-instance Postgres, drains without dropping the console, and backs its database up continuously with a weekly restore check. It installs air-gapped from your own registry, encrypts stored secrets and updates itself from the console.
High Availability
Readiness Without the Database
A Postgres primary switchover no longer fails every replica at once. Readiness reflects what each pod can serve and never queries the shared database on the probe path.
Proactive Draining
Handover starts as soon as a pod is marked for deletion. Spokes are told to reconnect elsewhere right away, and in-flight requests finish during a drain window.
mTLS Peer Forwarding
Each spoke uplink lives on one replica. The other replicas forward commands to it over mutual TLS on a dedicated peer port, not through the console Gateway.
Spread & Protected
A hard hostname topology spread keeps two replicas off the same node, and a PodDisruptionBudget covers voluntary disruptions.
Clean Cold Starts
Schema migrations are serialised across replicas with a Postgres advisory lock. Starting every replica at once causes no restarts and no crash loops.
Leader-Only Background Work
Alert evaluation, anomaly detection, remediation scans and scheduled digests run on the elected leader, so nothing is sent twice.
Verified live: draining the node that hosts the Postgres primary, a manager replica and the CloudNativePG operator at once, with the console polled every 250 ms, gave 0 failures across 186 samples.
Database Backup & Disaster Recovery
The audit log, IAM, the fleet registry and sealed credentials all live in the hub's Postgres. It is protected in layers.
| Layer | What it is | What it covers |
|---|---|---|
| In-cluster HA | 3 CloudNativePG instances, synchronous quorum | Pod and node loss with zero data loss, no backup needed |
| Continuous WAL archiving | WAL shipped to S3-compatible storage, 1-minute archive timeout | Point-in-time recovery to any second |
| Daily base backup | Full physical backup at 02:00, plus one at install | Bounds how much WAL a restore has to replay |
| Weekly restore check | Restores the latest backup into a throwaway cluster and checks it | Proves backups can actually be restored |
- Backups are on by default: the chart will not render without a backup destination unless you explicitly opt out
- 30-day recovery window
- The backup endpoint CA is pinned, so a DNS rebind becomes an archiving error, not a data leak
- Restore, including point-in-time recovery, is built into the chart
- Measured in a destroy-and-restore drill: 48.8 seconds of data lost with a writer running at the moment of loss, and 3 of 3 instances healthy about 4 minutes after starting the restore
- A manager started with the wrong at-rest key refuses to serve instead of writing unreadable data
Snapshots & Workload Backups
Management-Plane Snapshots
- JSON export of the hub's own state: fleet registry, IAM users and roles, GitOps repos, policies, placements, alert rules and more
- Credential-free: no password hashes, tokens, MFA secrets or sessions. A final check refuses any leftover secret-shaped field
- Restore is a dry run by default. A real write needs an explicit confirmation
- Snapshots are HMAC-sealed, and a tampered snapshot is refused
- People erased under GDPR after a snapshot was taken are never restored
etcd Snapshots
- A Job on a control-plane node snapshots stacked etcd
- Uploads to S3 destinations a platform admin configures once. Credentials never enter the API or cross the uplink
- The snapshot image comes from the live etcd Pod and is never caller-chosen
- Each snapshot records whether Secrets were encrypted at rest
- Restore uses your own tooling (etcdutl, kubeadm). Managed control planes should use the provider's backup
Velero & CSI Snapshots
- Live Velero Backup inventory per cluster: phase, schedule, TTL, errors and warnings
- Trigger a Velero backup from the API
- List and create CSI VolumeSnapshots
- Restore a snapshot into a new PVC
- Reports clearly when Velero or the snapshot CRDs are not installed
Hardening, Air Gap & Updates
Secrets at Rest
- Stored credentials are encrypted with AES-256-GCM under a dedicated key, separate from the session-signing key
- Optional envelope encryption with HashiCorp Vault Transit or AWS KMS; boot fails closed if the KMS cannot unwrap the key
- Previous-key window for rotation, with a background re-seal every 6 hours and progress in the console
- Postgres connections are TLS-only. Plaintext clients are rejected
- The plaintext console port can be locked to the Gateway namespace
Supply Chain & Air Gap
- Multi-arch images (amd64 and arm64) with SBOM attestations and SLSA provenance
- Images are signed with a cosign key you hold. Helm charts are keyless-signed and reach SLSA Build L2
- Every image can be pinned by digest, and signatures can be enforced at admission with a sigstore policy
- Air-gapped installs mirror into your registry, and the self-updater reads that mirror. Nothing calls out
- FIPS 140-3 manager build using AWS-LC-RS (CMVP #4759), built from source by you. The agent has no FIPS build
Console Self-Update
- The console checks your registry for new versions and upgrades itself from the Updates page
- Its RBAC can patch only the manager Deployment
- Preflight checks before the patch; the run is marked succeeded or failed within 15 minutes