rfc: regional security principles and key management deployment proposal - #21
rfc: regional security principles and key management deployment proposal#21hannahhoward wants to merge 2 commits into
Conversation
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The regional security RFC moves the region wrap from A256KW to AES-256-GCM inside the region's secrets manager. Swap the algorithm in the keys table and point to the new RFC; tenant and Hilt wraps unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
bajtos
left a comment
bajtos
left a comment
There was a problem hiding this comment.
Great proposal, I agree with it!
I have a few comments to consider, see below.
|
|
||
| **2. Discovery is the caveat on both.** The revocation lever helps once we know to pull it. A malicious operator or a stolen machine that we discover is contained from the next boot. Discovered too late, there is very little left to do: the system has been serving that adversary plaintext all along. This is a limit of the problem itself, and no design spend moves it. Remote kill of a *running* appliance (a central-triggered seal) would narrow the window between discovery and containment; it is worth looking at and is lower priority than getting the at-rest and startup properties right. | ||
|
|
||
| **3. The running system has no defense against its own operator.** The appliance holds plaintext CEKs and object bytes in memory whenever it serves a read, and the operator controls the hypervisor, the kernel, and the network path. PR #19's threat model states this plainly: "The region operator has hypervisor access and can read the SSH host key and any authorized-keys material." Neither of the obvious conclusions follows. Reaching for exotic protection against this adversary (trusted execution environments as a requirement) spends complexity on a property we cannot have. Concluding that at-rest custody is pointless because memory is readable anyway gives away the two properties we can have. |
There was a problem hiding this comment.
Reaching for exotic protection against this adversary (trusted execution environments as a requirement) spends complexity on a property we cannot have. Concluding that at-rest custody is pointless because memory is readable anyway gives away the two properties we can have.
In my limited understanding of confidential computing, the host cannot read the memory of a VM running inside a TEE.
I believe that running our appliance inside a TEE would protect us from this class of attacks.
At the same time, I think we are better off trusting our region operators not to exfiltrate data, at least for the first few regions.
|
|
||
| ## Principles | ||
|
|
||
| **1. Two properties are winnable at the region, and we build for both.** A powered-off disk, imaged after theft, RMA, or decommissioning, yields no object plaintext, no key material, and no credential that still works. And a region we have decided to cut off does not come back up: revoking it centrally prevents the appliance from starting. Every design decision below serves these two properties. |
There was a problem hiding this comment.
I want to point out another property that should be easy to achieve and make our appliance meaningfully more secure:
- Do not keep any plaintext/key material/credentials anywhere on the filesystem.
Even if a malicious node operator sshs into the running VM and looks around, they cannot access any sensitive content by reading files. The only way to access sensitive stuff is by reading the VM's memory, and extracting secrets from raw memory bytes requires much more sophisticated effort.
|
|
||
| ### One home for secrets: a local OpenBao, rooted centrally | ||
|
|
||
| Each appliance runs OpenBao, listening on a unix socket, with raft storage on the appliance disk. It is the single home for regional secrets: the Region KEK (a non-exportable transit key), the provider wallet key, TLS leaf keys, Postgres credentials, service identity PEMs, and the S3 credentials the installer currently writes into config files. Nothing secret sits in a plain file. |
There was a problem hiding this comment.
TLS leaf keys
I would like to provision TLS certs using Caddy's built-in features.
It seems that Caddy can use Vault as the storage backend via 3rd-party plugin, so I guess we are good here.
|
|
||
| ### Host hardening replaces process-level memory protection | ||
|
|
||
| Following OpenBao's own post-mlock guidance: swap disabled (`memory.swap.max=0` in the unit's cgroup) or encrypted, core dumps off, seeded entropy at first boot, no snapshot/clone reuse. One host checklist protects OpenBao's process and Ingot's alike. The irreducible key material in Ingot is one object's CEK for the duration of its own request, which is the blast radius we accept for being the process that serves plaintext. The dead-disk property carries one wiring dependency alongside this checklist: the PUT pipeline must encrypt bytes before they rest in the spool (the [FIL-481](https://linear.app/filecoin-foundation/issue/FIL-481)/[FIL-482](https://linear.app/filecoin-foundation/issue/FIL-482) ordering), or the spool must live on a bound volume. |
There was a problem hiding this comment.
Following OpenBao's own post-mlock guidance: swap disabled (
memory.swap.max=0in the unit's cgroup) or encrypted, core dumps off, seeded entropy at first boot, no snapshot/clone reuse. One host checklist protects OpenBao's process and Ingot's alike.
Seems like another thing we need to incorporate in our deployment setup.
| - Audit device on the appliance: per-wrap audit costs about 30% of sustained throughput, which we do not currently need. On by default, or off? | ||
| - Spool and encryption ordering: the dead-disk property requires the PUT pipeline to encrypt before bytes rest in the spool, or the spool to live on a bound volume. This belongs to the [FIL-481](https://linear.app/filecoin-foundation/issue/FIL-481)/[FIL-482](https://linear.app/filecoin-foundation/issue/FIL-482) wiring. | ||
| - A separated Piri on its own machine would change what an intruder can reach: Piri holds only FEE ciphertext, so access to a running Piri host could yield the provider wallet but not decrypted customer data. The Principle 3 concession attaches to the machine that decrypts, so a split deployment could keep bulk storage on less-trusted hardware. Relevant to future deployments. | ||
| - Who operates the central OpenBao, and its availability target; ties into PR #19's open question of who operates which layer. |
There was a problem hiding this comment.
Note #19 is not covering the Forge Central services (Hilt, Sprue, piri-signing-service, OpenBao).
I think that does not matter in the context of this RFC. We already know that we need to run a central OpenBao for Hilt, adding appliance-local OpenBao as another client should be a trivial improvement.
bajtos
left a comment
bajtos
left a comment
There was a problem hiding this comment.
After I learned more about what it means to operate an OpenBao instance (see Upgrade OpenBao version in my FilOne Appliance Deployment RFC), I prefer not to add OpenBao to every Ingot instance.
Instead, I propose adopting https://developers.google.com/tink as an embedded alternative to Vault/OpenBao - similarly to how SQLite is an embedded alternative to Postgres.
We can still use the central OpenBao to "unseal" the secrets (unwrap the local KEK used to decrypt secrets).
The key issue is embedded KMS means single process. Tink is fundamentally not designed to read from multiple processes at once. I could see this working for Piri, in the future, as I see SQLite and Tink being a reasonable path for a future where scaling Piri is done through one Piri per machine connected to a storage array. Fundamentally Ingot isn't designed this way -- the target is a stateless process that talks to an out-of-process database, and therefore also, an out of process secrets manager. the scaling model is horizontal scaling a la traditional web applications. Zero downtime blue-green ingot deploys is a first order priority (maybe not v1 but the first zero downtime improvement we ship) Ultimately, I just don't see deploying ingot an embeeded secrets manager working, and since we're stuck with Postgres and OpenBao, I think the first version just has Piri talking to those. |
|
|
||
| ### One home for secrets: a local OpenBao, rooted centrally | ||
|
|
||
| Each appliance runs OpenBao, listening on a unix socket, with raft storage on the appliance disk. It is the single home for regional secrets: the Region KEK (a non-exportable transit key), the provider wallet key, TLS leaf keys, Postgres credentials, service identity PEMs, and the S3 credentials the installer currently writes into config files. Nothing secret sits in a plain file. |
There was a problem hiding this comment.
One gap worth specifying: how the non-transit secrets (service PEMs, DSN, S3 creds) travel from Bao into Piri/Ingot processes.
Proposal: systemd LoadCredential= — pid1 materializes each secret into the unit's private ramfs (can't swap, per-unit mount namespace, 0400, gone on stop), apps keep reading file paths, quadlet bind-mounts %d/ into the container. Source is pluggable (plain file for dev, TPM blob for standalone, socket→Bao for enrolled) so the unit files are identical across modes. This is a documented host-OS feature, per Principle 4 and the "stock machinery" criterion — no secrets client in either app.
There was a problem hiding this comment.
I'm finding this hard to read and understand. I assume this means that there's essentially no change to Piri/Ingot for these credentials? We keep reading from file paths (or in the case of postgres DSN we add load from file support).
|
|
||
| Each appliance runs OpenBao, listening on a unix socket, with raft storage on the appliance disk. It is the single home for regional secrets: the Region KEK (a non-exportable transit key), the provider wallet key, TLS leaf keys, Postgres credentials, service identity PEMs, and the S3 credentials the installer currently writes into config files. Nothing secret sits in a plain file. | ||
|
|
||
| The local OpenBao's storage is sealed by a transit key held at a central OpenBao (`seal "transit"`). At boot the appliance authenticates to central, unwraps its barrier key, and unseals. With central unreachable, or the seal credential revoked, OpenBao 2.6.1 refuses to start at all (verified behavior). This is the startup-kill lever from Principle 1: revoking one credential at central makes everything on the disk permanently unreadable. The boot credential is CIDR-bound to the region's egress and single-use where the deployment permits, so an imaged disk replayed elsewhere fails, and any use of a stolen credential is visible at central. Hardware binding of the boot identity (TPM sealing at enrollment) is the eventual strengthening; it needs host-image work and is out of scope here. The boot-time dependency on central has the same failure envelope the appliance already accepts for auth, where Hilt-derived credentials expire daily; steady-state reads never call central, preserving the encryption RFC's Independence criterion. |
There was a problem hiding this comment.
Above we say:
Each appliance runs OpenBao [...] It is the single home for regional secrets: the Region KEK (a non-exportable transit key), the provider wallet key, TLS leaf keys, Postgres credentials, service identity PEMs, and the S3 credentials the installer currently writes into config files. Nothing secret sits in a plain file.
So the appliance operator runs OpenBao and puts all their sensitve data in it, yes?
Then we go onto say:
The local OpenBao's storage is sealed by a transit key held at a central OpenBao (
seal "transit"). At boot the appliance authenticates to central, unwraps its barrier key, and unseals. With central unreachable, or the seal credential revoked, OpenBao 2.6.1 refuses to start at all (verified behavior). This is the startup-kill lever from Principle 1: revoking one credential at central makes everything on the disk permanently unreadable.
Am I reading/understanding this right(?): If the central OpenBao is down, or revoked, a provider cannot read their provider wallet key, Postgres credentials, service identity PEMs, and S3 credentials?
Replace smelt's single-VM Docker Compose deployment for the five central services with Terraform-managed infrastructure: ECS/Fargate behind an ALB, one RDS Postgres instance with a database per service, and OpenBao as the secrets manager. Supports multiple stages through a `stage` variable. The substantial change from smelt is where secrets come from. smelt mints keys on an operator's laptop and stores them in 1Password; here a Go Lambda runs inside the VPC, writes private material straight to SSM Parameter Store, and returns only DIDs, wallet addresses and database names. No private key reaches Terraform state or a local disk. - Port smelt's keygen: Ed25519 identities, secp256k1 wallets with the three serializations each consumer expects, and the UCAN startup proofs, now issued in-process through ucantool's new ucandelegate API rather than by shelling out to its CLI. - Never regenerate an existing secret. Writes use Overwrite=false, so re-applying cannot abandon a funded wallet or change a registered DID. - Namespace parameters per service, so a task execution role reads only /forge/<stage>/<service>/* and a compromised task cannot reach another service's credentials. - Store OpenBao's data in Postgres and seal it with KMS. Fargate has no durable local disk, and KMS removes the unseal key, the 1Password item holding it, and the sidecar that applied it. - Give hilt an AppRole scoped to forge/hilt/data/tenant/* instead of the Vault root token, which smelt tracks as debt. - Expose OpenBao publicly, since fil-one/RFC#21 makes it the transit seal regional appliances authenticate against to unseal at boot. - Add a fund phase that signs the three FilecoinPay transactions inside AWS, so the payer key never leaves the account. Terraform never invokes it; `make fund-payer` shows a plan and requires confirmation. - Pin images by manifest digest rather than git SHA, which names the last commit instead of the code just built and collides against a dirty tree. Use ucantool's identity helpers rather than libforge's: libforge formats the signer into its error text with %s, and an ed25519 signer is a []byte with no String method, so a marshalling failure renders the raw private key into a logged error. Reported separately; sprue, hilt and swarf still depend on it. Signed-off-by: Miroslav Bajtoš <oss@bajtos.net> Assisted-by: Claude:claude-opus-5
|
@bajtos can I get a removal on the requested change, unless you still feel strongly about embedded secrets? |
|
|
||
| ## TL;DR | ||
|
|
||
| A region can defend exactly two security properties, and we should commit to both: **nothing on the appliance's disks is readable at rest, and a revoked appliance never comes back up.** Both are gated on discovery. Against an operator with sustained, undiscovered access to the running system, no design protects the data: the appliance decrypts objects to serve them, and the operator owns the hardware it runs on. |
There was a problem hiding this comment.
Both are gated on discovery
What does this mean?
undiscovered access to the running system
Huh?
There was a problem hiding this comment.
Looking back on this I think it's the "Against an operator" bit that is confusing. The text should perhaps be "an attacker"...with undiscovered & sustained access. Then it makes sense - an operator always has access because, well, they are the operator.
|
|
||
| ### One home for secrets: a local OpenBao, rooted centrally | ||
|
|
||
| Each appliance runs OpenBao, listening on a unix socket, with raft storage on the appliance disk. It is the single home for regional secrets: the Region KEK (a non-exportable transit key), the provider wallet key, TLS leaf keys, Postgres credentials, service identity PEMs, and the S3 credentials the installer currently writes into config files. Nothing secret sits in a plain file. |
There was a problem hiding this comment.
I'm finding this hard to read and understand. I assume this means that there's essentially no change to Piri/Ingot for these credentials? We keep reading from file paths (or in the case of postgres DSN we add load from file support).
This PR adds an OpenBao `Provider` implementation for wrapping and unwrapping CEKs. The code related to holding the KEK in memory has been removed per the RFC amendment fil-one/RFC#21 The regional KEK is expected to be present in OpenBao under the key name passed to the constructor and it MUST be created with `derived=true, exportable=false`. Additionally: * Renames `SoftwareProvider` to `InProcessProvider` and removes the `KEKSource` interface as it would only have been implemented by `SoftwareProvider`. * Renames `Scope` to `BindingContext` since it is the context bound for key derivation. --------- Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
| - Audit device on the appliance: per-wrap audit costs about 30% of sustained throughput, which we do not currently need. On by default, or off? | ||
| - Spool and encryption ordering: the dead-disk property requires the PUT pipeline to encrypt before bytes rest in the spool, or the spool to live on a bound volume. This belongs to the [FIL-481](https://linear.app/filecoin-foundation/issue/FIL-481)/[FIL-482](https://linear.app/filecoin-foundation/issue/FIL-482) wiring. | ||
| - A separated Piri on its own machine would change what an intruder can reach: Piri holds only FEE ciphertext, so access to a running Piri host could yield the provider wallet but not decrypted customer data. The Principle 3 concession attaches to the machine that decrypts, so a split deployment could keep bulk storage on less-trusted hardware. Relevant to future deployments. | ||
| - Who operates the central OpenBao, and its availability target; ties into PR #19's open question of who operates which layer. |
4 participants
📖 Preview
Summary
Regional security principles — the two properties a region can actually defend (nothing readable from a powered-off disk, and a revoked appliance never starts again) — and the key-management deployment that delivers them: a local OpenBao on each appliance, auto-unsealed from a central OpenBao, with the region wrap moving from A256KW to context-bound transit AES-256-GCM. Includes measured throughput down to a two-core floor, and amends the encryption RFC's keys table for the algorithm swap (tenant and Hilt wraps unchanged). Picks up the Region-KEK open question from #19.
Read the docs rendered:
🤖 Generated with Claude Code