Principal-level interviews leave the single cluster behind. The interviewer probes whether you can run many clusters as one system, operate a platform as a product, and reason about money — because at fleet scale, utilization and blast radius are financial statements. The mental model: the fleet is one distributed system whose nodes are clusters, and every platform change deserves the same rollout discipline you demand from application teams. Chapters 1–10 taught what happens inside a cluster; this chapter is about what happens to twenty of them at once.
Fleet architecture. A fleet treats clusters as cattle: any cluster is rebuildable from a Git repository that holds its full desired configuration (config-as-data). The default topology is replicated-independent clusters — no runtime coordination between them; a CD system applies the same configuration N times (the trade-offs against hub-spoke and federation are tabulated in Appendix D). Changes move through rings: ordered groups of clusters, starting with a canary, each gated by a bake period before the next ring proceeds (Flow 29). Drift is any live-cluster state that differs from the repo; without continuous enforcement, ring N no longer matches what ring 0 tested.
Capacity economics. A cluster is "full" when requests exhaust allocatable — not when usage does (Chapter 3). Fleet-wide, the gap between the two is the single biggest cost lever: systematic over-requesting shows up as clusters that are 90% requested and 40% used. The fixes are incentives and automation, not scheduler flags: showback or chargeback per namespace, quotas priced in real money, right-sizing recommendations, and Karpenter-class node provisioning that shapes the node pool to the actual pending pods. In accelerator fleets the stakes multiply: an idle GPU burns more money than an entire idle general-purpose node (Chapter 9).
Platform engineering. A platform is a product whose customers are product teams. The central decision is the abstraction: raw Kubernetes with guardrails, a golden-path layer (templates and defaults over full API access), or an internal PaaS that hides Kubernetes entirely. Whichever you expose, pair the paved road with policy: admission (Chapter 2) enforces what the templates encourage. Two disciplines keep a platform honest: escape hatches — teams can drop below the abstraction, and their usage rate tells you whether the abstraction is right — and platform SLOs with error budgets, so platform changes are governed by the same reliability math as any service.
Your platform team merges a new admission policy intended for all forty clusters.
- Platform engineer merges the change to the fleet repo — the single source of desired state for every cluster.
- CI validates offline: schema checks, policy dry-runs against recorded production objects, and an ephemeral test cluster when the change warrants one.
- CD (the GitOps engine — an agent per cluster pulling desired state from the repo) reconciles ring 0 only: the canary cluster.
- A canary proves nothing beyond syntax unless it runs representative workloads — same webhook consumers, same operators, same node shapes as the fleet.
- Bake gate (an automated check over SLOs and error budgets) watches ring 0 for its bake period: admission latency, error rates, controller health, rollout success of ordinary deploys.
- CD proceeds ring by ring on green. Rings are ordered by blast radius — bigger and more production-critical clusters later, with longer bakes.
- Bake gate halts the wave on regression anywhere and CD reverts the affected ring: rollback is a Git revert replayed by the same machinery, never a hand edit.
- Drift controller (the CD agent's enforcement mode) continuously reverts out-of-band edits, so every ring still matches the repo when its turn comes.
- Platform engineer lands fixes through the same rings on an expedited schedule. There is no kubectl fast path — the expedited ring exists precisely so bypassing is never the rational choice.
- CD reports fleet convergence: every cluster at the same revision, drift-free. That report — not the merge — is when the change is "done".
flowchart TD
COMMIT[merged and validated] --> RING[apply next ring]
RING --> GATE{bake gate}
GATE -->|healthy, rings left| RING
GATE -->|healthy, last ring| DONE[fleet converged]
GATE -->|regression| HALT[halt and revert]
HALT --> FIX[fix rides same rings]
FIX --> RING
classDef resource fill:#fb7185,stroke:#be123c,color:#fff
classDef process fill:#38bdf8,stroke:#0369a1,color:#fff
classDef lease fill:#f59e0b,stroke:#b45309,color:#fff
classDef standby fill:#94a3b8,stroke:#475569,color:#fff
classDef leader fill:#10b981,stroke:#047857,color:#fff
class COMMIT resource
class RING,FIX process
class GATE lease
class HALT standby
class DONE leader
Figure 11.1 — a fleet change is a gated loop over rings; the halt-and-revert path is as designed as the happy path.
Where this can fail
- Symptom: canary green, ring 2 bricked. Cause: the canary is not representative — no GPU nodes, no webhook consumers, no real operators. Where to look: workload census per ring vs the canary's.
- Symptom: waves halt on every rollout. Cause: bake gates keyed to noisy SLOs; the gate cries wolf. Where to look: gate signal precision; alert-to-regression ratio.
- Symptom: ring N behaves differently from ring 0 on the same commit. Cause: drift accumulated where enforcement was off. Where to look: drift reports, audit log for out-of-band writes.
- Symptom: a "quick emergency fix" via kubectl broke a cluster and the repo does not show it. Cause: bypass culture — the expedited path was slower than disobedience. Where to look: change records vs Git history; make the paved path the fastest path.
- Symptom: the config change locked the platform out of ring 0 — nothing can be applied, including the fix. Cause: an admission rule with
failurePolicy: Failmatching the CD system's own writes (Flow 4 at fleet scale). Where to look: webhook exclusions for the CD and platform namespaces.
Your platform team must move its widely-used Widget CRD from v1 to an incompatible v2, across forty clusters.
- Platform team ships v2 alongside v1 in the CRD, with a conversion webhook (Chapter 6); the API server serves both versions, converting through the hub version.
- CD rolls that CRD change through the rings (Flow 29). Conversion is now on the read and write path for every Widget in every cluster — an unavailable conversion webhook has the same failure shape as a down admission webhook (Flow 4), scoped to this API group.
- API server (in each cluster) serves v1 and v2 clients; stored objects remain v1 on disk until written.
- Platform team flips
storage: trueto v2: new writes persist as v2, old objects stay v1 until touched. - Storage-version migrator (a controller or batch job) rewrites every stored Widget in every cluster so etcd holds only v2.
- Platform team runs the consumer migration through a deprecation window: every controller, dashboard, and script reading v1 must move. Finding them is the hard part — audit logs grouped by API version are the census.
- Platform team stops serving v1, ring by ring. This is the point of no return: a consumer missed in step 6 breaks here, at whatever ring its cluster sits in.
- Platform team trims
status.storedVersionsto v2 (Chapter 6); a later release can drop v1 from the CRD entirely. - CD re-verifies the whole fleet afterward: a cluster that was offline during the wave or restored from backup can resurface v1 stored objects — reconcile the fleet, not just the wave.
flowchart TD
V2[add v2 and conversion] --> BOTH[serve both versions]
BOTH --> FLIP[storage flips to v2]
FLIP --> MIG[migrate stored objects]
MIG --> CONS[migrate consumers]
CONS --> DROP[stop serving v1]
CONS -->|pinned consumer found| WAIT[hold the wave]
DROP --> TRIM[trim storedVersions]
classDef process fill:#38bdf8,stroke:#0369a1,color:#fff
classDef resource fill:#fb7185,stroke:#be123c,color:#fff
classDef standby fill:#94a3b8,stroke:#475569,color:#fff
class V2,BOTH,FLIP,MIG,CONS,TRIM process
class DROP resource
class WAIT standby
Figure 11.2 — the version march is one-way after "stop serving v1"; everything before it is reversible, so spend the caution there.
Where this can fail
- Symptom: all Widget operations fail in one cluster. Cause: the conversion webhook is down there — conversion sits on every read and write of the group. Where to look: webhook backend health; treat it with Flow 4's mitigations.
- Symptom: a team's controller breaks days after the rollout finished. Cause: a v1 consumer missed by the census, discovered only when v1 stopped being served. Where to look: audit logs by API version before step 7, not after.
- Symptom:
storedVersionssays v2-only, yet v1 objects exist. Cause: a cluster restored from a pre-migration backup. Where to look: re-run the migrator after any restore; fleet-wide verification, not wave-completion, is the exit criterion. - Symptom: objects subtly corrupted after conversion. Cause: a lossy conversion webhook — round-trips must be lossless (Chapter 6's red flag). Where to look: conversion round-trip tests against production-shaped objects.
- Symptom: the fleet sits half-migrated for a year. Cause: no deadline and no owner — deprecation windows without teeth never close. Where to look: the deprecation policy; someone must own "v1 is gone".
A platform team presents this architecture for your review: six production clusters, one per region. All forty product teams share them, one namespace each. A hub cluster runs controllers managing all six spokes. An org-wide validating webhook with failurePolicy: Fail enforces naming and labels on every object in every cluster. Fleet configuration applies to all six clusters in one step. Two infra teams hold cluster-admin.
Critique. Start with blast radius; there are three fleet-wide single points. The webhook: one bad rule or backend outage stops every deploy in every region simultaneously (Flow 4), and Fail on cosmetic policy buys enforcement with availability. The hub: its outage freezes all fleet change, and its compromise is fleet compromise — the hub-spoke row of Appendix D's table priced this. The ring-less rollout: applying to all six regions at once makes regional isolation an illusion — the regions share fate through the config pipeline. What is right: forty trusted teams on shared clusters is the cheap, defensible tenancy default, and a hub is a legitimate pattern once its risk is priced. What to ask before redesigning: do any teams face compliance regimes needing hard isolation; has the webhook ever fired an incident; who reviews its rules; what is the recovery time if the hub dies. The migration order: rings first — the largest risk cut for the least work; then scope the webhook and demote cosmetic rules to Ignore or CEL warnings; then shrink the hub's blast radius (per-region hubs, or pull-based agents per cluster); re-open tenancy only if compliance forces it.
Q 11.1 — Why is replicated-independent the default fleet topology, rather than hub-spoke or a federated API?
Answer. Because coordination is the enemy of isolation. Independent clusters that each pull the same config share no runtime fate: one cluster's outage, upgrade, or compromise stays inside it, and the fleet's availability is the product of independent terms. A hub adds a component whose failure freezes change everywhere and whose compromise owns everything; a federated API adds a new single point and a version-skew surface on the request path. Both can be worth it — hubs for fleet-scale config logic, federation for genuine cross-cluster scheduling — but they must buy something, because they always sell isolation. Default to independence; add coordination only where a real requirement demands it.
Strong answers also mention: failover belongs above the fleet (DNS, global load balancing) precisely so that no in-fleet component must coordinate it.
Q 11.2 — Why is chronic low utilization an incentive problem more than a scheduler problem?
Answer. The scheduler packs what it is told (Chapter 3): requests, not usage. Teams set requests, and every incentive points up — padded requests mean fewer OOMKills, fewer noisy neighbors, easier on-call. Nobody sees the bill. So clusters run request-full and usage-empty, and buying a better bin-packer changes nothing because the inputs are inflated. The fixes act on the inputs: showback that puts a price on requests, quotas that make padding compete with the team's own roadmap, right-sizing bots that shrink requests with evidence, and node provisioning that shapes capacity to real demand. Only after the inputs are honest does scheduler tuning matter.
Strong answers also mention: the same logic inverted for accelerators — GPU idleness is so expensive that queueing (making workloads wait) beats reservation.
Q 11.3 — Why should a platform team publish SLOs and run on error budgets?
Answer. Because a platform without SLOs is accountable for everything and measurably responsible for nothing. An SLO ("namespace provisioning under a minute, 99.5% of deploys unimpeded by platform change") does three jobs: it converts "the platform feels slow" into a falsifiable claim; it gives the ring system its gate signal — a fleet rollout halts when it eats the error budget (Flow 29); and it makes the platform's reliability trade visible to leadership in the same currency as any product's. Without a budget, platform changes are governed by confidence; with one, they are governed by evidence.
Strong answers also mention: escape-hatch usage as a complementary metric — SLOs measure whether the platform works; hatch usage measures whether it is the right platform.
Q 11.4 — Design the ring system for a 40-cluster fleet. What makes it trustworthy?
Answer. Four rings by blast radius: ring 0, one canary cluster; ring 1, two or three low-stakes production clusters; ring 2, the general population; ring 3, the crown jewels (revenue-critical, compliance-bound). Trust comes from three properties. Representativeness: the canary runs a workload census matched to the fleet — same operators, webhook consumers, node shapes, a GPU pool if the fleet has one; review the census quarterly, because fleets drift. Gate quality: bakes watch golden signals plus platform-specific ones (admission latency, deploy success rate), with bake times growing per ring — an hour at ring 0 is worthless for a change whose failure mode is slow memory growth. An expedited path: emergencies ride the same rings with compressed bakes, and its existence is what makes bypass indefensible. The failure story: any gate red halts the wave, the ring reverts via Git, and the postmortem asks why the earlier rings did not catch it.
Strong answers also mention: changes that cannot ring safely — CRD storage flips (Flow 30), webhook additions — get their own playbooks; the ring system must know which changes it does not fit.
Q 11.5 — Your GPU fleet reports 30% utilization while teams complain about queue times. Walk through the diagnosis.
Answer. Both symptoms together mean capacity is trapped, not absent — find where. Measure in layers. Allocation vs usage: are GPUs allocated but idle (notebooks held overnight, jobs that reserve eight and use two) — that is quota and idle-reclamation policy. Fragmentation: multi-GPU jobs need co-located devices (topology, Chapter 9); four singles free on four nodes serve no 4-GPU job — that is bin-packing policy and DRA-aware scheduling. Gang waits: distributed jobs holding partial allocations while waiting for the rest waste everything they hold — that is gang scheduling and queueing (Kueue-class admission) rather than raw capacity. Padding: requests set to whole GPUs for fractional workloads — that is sharing/partitioning (MIG-class, DRA structured parameters). Rank the fixes by recovered dollars: usually idle reclamation first, then queueing, then packing policy, then sharing. Buying more GPUs comes last — it scales the waste along with the capacity.
Strong answers also mention: the political half — idle reclamation takes something away from teams, so it ships with grace periods, preemption classes, and an SLO for queue time, or it dies in review.
Q 11.6 — Forty teams, two under hard compliance regimes. One shared fleet, clusters per team, or something between? Commit.
Answer. Assumption: the other 38 teams mutually trust to the "noisy neighbor, not hostile actor" level; the two regulated teams cannot. Commit: shared clusters for the 38 — namespaces, quotas, and policy (Appendix D's tenancy table prices this as the cost floor) — and dedicated clusters for the two regimes, because an auditor accepts a cluster boundary in a way they will never accept a namespace. The counter-position — clusters for everyone — buys uniform isolation and simpler audits, and I would take it seriously at higher trust stakes; it loses here because forty control planes cost real money and real SRE attention, and the 38 teams' risk does not justify it. Kill-criteria, stated up front: a cross-namespace security incident that quotas and policy could not have contained; a third compliance regime appearing (the "special case" is now a pattern); or shared-control-plane outages exceeding the platform error budget twice in a year. Any of those, and the tenancy decision reopens with the balance shifted.
Strong answers also mention: virtual clusters as the middle option worth piloting for API isolation — while noting node sharing still fails the hard-compliance test.
Q 11.7 — Leadership asks: "Should we build an internal PaaS on top of Kubernetes?" Argue both sides, then commit.
Answer. For: most product teams want "deploy my service, give me a URL and logs"; a PaaS encodes the golden path, shrinks the Kubernetes learning surface to near zero, and turns a hundred bespoke YAML stacks into one supported product. Against: a PaaS is a product with a permanent team, a roadmap, and a support queue; the org now maintains Kubernetes and a platform on it, and every workload that does not fit the abstraction (the GPU pipeline, the stateful legacy system) either forks the PaaS or routes around it. Commit: build the thin version — golden-path templates, defaults, and policy over the raw API — only if three things hold: a platform team you will fund indefinitely, workload homogeneity above roughly 80%, and a real escape hatch to plain Kubernetes. Kill-criterion: if escape-hatch usage grows past a fifth of workloads, the abstraction is wrong — stop investing in the wrapper and reinvest in paved roads for the raw API.
Strong answers also mention: the decision is reversible in one direction only — retiring a PaaS teams depend on is a multi-year migration, so under-build first.
- "The canary was green, so the change is safe." A canary proves what it runs. An unrepresentative canary is a syntax checker with extra steps.
- "The hub is just another cluster." It is the fleet's control plane: its outage freezes change everywhere, its compromise owns everything. Price it like one.
- "Utilization is an ops metric." At fleet scale it is a financial statement — the requests-vs-usage gap is the cloud bill's error term.
- "One cluster per team ends the tenancy debate." It converts an isolation problem into a cost and fleet-management problem. The debate moves; it does not end.
- "GitOps means there is no drift." GitOps without continuous enforcement is documentation. Drift is what the enforcement loop deletes, not what the methodology prevents.
- "In an emergency we kubectl straight to prod." The expedited ring exists so that the paved path is always the fastest path. A bypass that beats it is a process bug — fix the process.