diff --git a/.context/TASKS.md b/.context/TASKS.md index fb2c6ddf..55c69f27 100644 --- a/.context/TASKS.md +++ b/.context/TASKS.md @@ -25,13 +25,44 @@ TASK STATUS LABELS: `#in-progress`: currently being worked on (add inline, don't move task) --> -### Phase 1: [Name] `#priority:high` -- [ ] Task 1 -- [ ] Task 2 + + +### Phase 1: Correctness & Broken Things `#priority:high` +- [ ] Fix broken CI integration test #source:jira.xml #added:2026-07-14 +- [ ] Fix broken recovery/restore flow #source:jira.xml #added:2026-07-14 +- [ ] Fix `spike cipher` stream mode (broken; owner: Murat); JSON mode fix unblocks encryption-as-a-service demo/docs #source:jira.xml #added:2026-07-14 +- [ ] Retry sqlite operations with exponential backoff on transient locks (all of `app/nexus/internal/state/persist`) → ideas/research-db-resilience.md #source:jira.xml #added:2026-07-14 +- [ ] Bound the Bootstrap keeper-wait loop with a configurable timeout/max-attempts instead of looping forever → ideas/research-db-resilience.md #source:jira.xml #added:2026-07-14 +- [ ] Make `env` accessors return sentinel errors instead of calling `log.FatalLn` (removes env→log circular dep, makes them testable) → ideas/research-env-error-handling.md #source:jira.xml #added:2026-07-14 + +### Phase 2: SDK Extraction `#priority:medium` +- [ ] Graduate generic internal helpers to spike-sdk-go (nonce/crypto, Shamir verify, permission (de)serialize, validation, trust/spiffeid, URL builders, `GCMNonceSize`, `Id()`, canonical permission set) → ideas/research-sdk-extraction.md #source:jira.xml #added:2026-07-14 + +### Phase 3: Testing `#priority:medium` +- [ ] Make `make test` concurrent again (currently serialized by env setup) → ideas/research-cli-testing.md #source:jira.xml #added:2026-07-14 +- [ ] Add integration tests: root key cached/recovered/not-re-initialized; secret & policy CRUD; Pilot denies when Nexus uninitialized / warns when unreachable → ideas/research-cli-testing.md #source:jira.xml #added:2026-07-14 +- [ ] Raise CLI command coverage to 60%+ via unit + HTTP-mock tests; fix `t.Skip()`ed tests; DI-refactor `sendShardsToKeepers` → ideas/research-cli-testing.md #source:jira.xml #added:2026-07-14 +- [ ] `start.sh` should exercise recovery/restore and encryption/decryption #source:jira.xml #added:2026-07-14 + +### Phase 4: Policy & Secrets `#priority:medium` +- [ ] Add a `list` permission type; scope `spike secret list` to the caller's allowed path patterns → ideas/research-list-permission.md #source:jira.xml #added:2026-07-14 +- [ ] Add `LoadPolicyByName` for indexed O(1) lookup (UpsertPolicy currently scans all policies) #source:jira.xml #added:2026-07-14 +- [ ] Reject empty id/path for both policies and secrets #source:jira.xml #added:2026-07-14 +- [ ] Substring/partial-match filtering for `policy list` and `secret list` (today requires the exact regex) #source:jira.xml #added:2026-07-14 +- [ ] Refactor `app/spike/internal/cmd/secret/get.go` to extract the repeated marshal-and-print helpers #source:jira.xml #added:2026-07-14 -### Phase 2: [Name] `#priority:medium` -- [ ] Task 1 -- [ ] Task 2 +### Phase 5: Ops & Config `#priority:low` +- [ ] Wire the unused `DatabaseOperationTimeout()` into the DB path → ideas/research-db-resilience.md #source:jira.xml #added:2026-07-14 +- [ ] Move `const maxShardID = 1000` to env-var configuration #source:jira.xml #added:2026-07-14 +- [ ] Add a configurable SVID-source acquisition timeout (go-spiffe context) so lookups don't wait forever #source:jira.xml #added:2026-07-14 +- [ ] Add a Nexus `/status` endpoint; Pilot warns when Nexus not ready; Bootstrap uses it instead of the k8s Jobs API #source:jira.xml #added:2026-07-14 ## Blocked diff --git a/jira.xml b/jira.xml deleted file mode 100644 index 92900988..00000000 --- a/jira.xml +++ /dev/null @@ -1,2550 +0,0 @@ - - - - - - Zero Turtle Pi Grid - Secure Secrets CLI UX (similar to Claude auth) - - - spike cipher (stream mode) is broken -> Murat. - - - PRs to watch: - https://github.com/spiffe/spike-sdk-go/pull/140 - https://github.com/spiffe/spike/pull/263 - https://github.com/spiffe/spike/pull/268 - https://github.com/spiffe/spike/pull/270 - https://github.com/spiffe/helm-charts-hardened/pull/665 - - - - - - fix broken things first: - * CI integration test is broken - * recovery/restore is broken - - - a demo just focus on the architecture and nothing else. (no live demo, - no k8s deployment, etc) - - - Create a new walk-trough video recording. - * Brief architecture overview - * Dev startup for bare-metal - * Dev startup for Minikube - * CLI overview - * Mini codewalk - - - // TDO: check all database operations (secrets, policies, metadata) - and - // ensure that they are retried with exponential backoff. - - add retries to everything under: - app/nexus/internal/state/persist - ^ they all talk to db; and sqlite can temporarily lock for - a variety of reasons. - - - Try SPIKE on a Mac (and create a video) - - - spike on windows video. - - - func initializeSqliteBackend(rootKey *[32]byte) backend.Backend { - panic if rootkey is nil or empty. - - - create a video about how to develop SPIKE on WSL. - - - demo: encryption as a service - - - - - const GCMNonceSize = 12 - to sdk. - - - once https://github.com/spiffe/spike-sdk-go/pull/140/changes - is merged, the following needs to be updated accordingly - as 1. it is faking a context with timeout without actually doing - anything at the http layer - 2. it makes things extra complicate without achieving proper - network-layer cancellation: - - - - `spike secret list` right now lists all secrets available and - requires system level access. - - we might want to restrict the "list" action to a regular expression - matcher with a policy, so for certain SPIFFE IDs, a `list` call - will only list the secrets that they are allowed to see. - - so that a workload (if allowed) can list secrets under - secret/tenants/acme/* - but not secret/tenants/bodega - - For that, we likely need to extend the policy structure too - - here is how the policies are specified currenty: - spike policy create --name=workload-can-write \ - --path-pattern="^tenants/demo/db/.*$" \ - --spiffeid-pattern="^spiffe://spike\.ist/workload/.*$" \ - --permissions="write" - - we may need to add a "list" permission type, and that would - limit the policy to list secrets that match to the path pattern only. - - if a workload does not have a list access given by a policy, it - won't be able to list secrets. - - - duplicate code fragment: - for _, policy := range *policies { - result.WriteString(fmt.Sprintf("ID: %s\n", policy.ID)) - result.WriteString(fmt.Sprintf("Name: %s\n", policy.Name)) - result.WriteString(fmt.Sprintf("SPIFFE ID Pattern: %s\n", - policy.SPIFFEIDPattern)) - result.WriteString(fmt.Sprintf("Path Pattern: %s\n", - policy.PathPattern)) - - perms := make([]string, 0, len(policy.Permissions)) - for _, p := range policy.Permissions { - perms = append(perms, string(p)) - } - - - duplicate code fragment - - func encryptStream(cmd *cobra.Command, api *sdk.API, inFile, outFile string) { - // Validate the input file exists before attempting encryption. - if inFile != "" { - if _, err := os.Stat(inFile); err != nil { - if os.IsNotExist(err) { - cmd.PrintErrf("Error: Input file does not exist: %s\n", inFile) - return - } - cmd.PrintErrf("Error: Cannot access input file: %s\n", inFile) - return - } - - - maybe have a dedicated RootKey type of *[32]byte - - - move internal stuff to the SDK. - - - duplicate code fragment - // Get the specific version - v, exists := secret.Versions[version] - if !exists { - failErr := *sdkErrors.ErrEntityNotFound.Clone() - failErr.Msg = fmt.Sprintf( - "secret with path %s not found for version %v", - path, version, - ) - return nil, &failErr - } - - - duplicated code fragment: - func (s *DataStore) DeletePolicy( - ctx context.Context, id string, - ) *sdkErrors.SDKError { - const fName = "DeletePolicy" - - validation.CheckContext(ctx, fName) - - s.mu.Lock() - defer s.mu.Unlock() - - tx, beginErr := s.db.BeginTx( - ctx, &sql.TxOptions{Isolation: sql.LevelSerializable}, - ) - if beginErr != nil { - failErr := sdkErrors.ErrTransactionBeginFailed.Wrap(beginErr) - return failErr - } - - - duplicated code fragment: - - func RouteDecrypt( - w http.ResponseWriter, r *http.Request, audit *journal.AuditEntry, - ) *sdkErrors.SDKError { - const fName = "routeDecrypt" - journal.AuditRequest(fName, r, audit, journal.AuditCreate) - - // Check if streaming mode based on Content-Type - contentType := r.Header.Get(headerKeyContentType) - streamModeActive := contentType == headerValueOctetStream - - if streamModeActive { - // Cipher getter for streaming mode - getCipher := func() (cipher.AEAD, *sdkErrors.SDKError) { - return getCipherOrFailStreaming(w) - } - return handleStreamingDecrypt(w, r, getCipher) - } - - - refactor with injections to make the code more testable: - - func TestSendShardsToKeepers_NetworkDependentFunction(t *testing.T) { - // The sendShardsToKeepers function has multiple external dependencies that make it - // difficult to test without a significant infrastructure: - // 1. Requires SPIFFE X509Source - // 2. Makes network calls via mTLS clients - // 3. Depends on state management for the root key - // 4. Calls computeShares() and sanityCheck() which have their own dependencies - t.Skip("Skipping sendShardsToKeepers test - requires SPIFFE infrastructure, network connectivity, and state management") - - // Note: To properly test this function, you would need to: - // 1. Mock the workloadapi.X509Source - // 2. Mock network.CreateMTLSClientWithPredicate - // 3. Mock net.Post - // 4. Mock state.RootKeyZero() - // 5. Mock computeShares() and sanityCheck() - // 6. Set up test HTTP servers - // 7. Or refactor the code for better testability with dependency injection - } - - - g := group.P256 - t := uint(env.ShamirThresholdVal() - 1) // Need t+1 shares to reconstruct - n := uint(env.ShamirSharesVal()) // Total number of shares - duplicated code fragment. - - - SPIKE Bootstrap tries to reach keepers forever, but it should have a max - timeout (like, say, 20mins), after which it gives up and crashes. - -- configurable. - - - - ensure db operations are retried a couple times with backoff - before giving up. - - SQLLite can error out if there is a blocked transaction or - a integrity issue, which a retry can fix it. - - - BREAKING: API Methods should accept a context, so that they can - be cancelled from the consumer without having to for a wrapper - goroutine - - - Add a SPIKE Lite section for the docs. - But before that crypto stream and json mode needs to be fixed. - - SPIKE Lite: - # get pilot's pem and key to test things out. - # this can be part of documentation too. to test the API directly, 1. extract the pem and key, and then do regular - curl. - curl -s -X POST --header "Content-Type:application/octet-stream" --data-binary "This is a test encryption" - https://spire-spike-nexus/v1/encrypt -k --cert /tmp/pem/svid.0.pem --key /tmp/pem/svid.0.key -o encrypted - curl -s -X POST --header "Content-Type:application/octet-stream" --data-binary @encrypted - https://spire-spike-nexus/v1/decrypt -k --cert /tmp/pem/svid.0.pem --key /tmp/pem/svid.0.key -o decrypted - cat decrypted; echo - - - start.sh should test recovery and restore - start.sh should test encryption and decryption - - - func DatabaseOperationTimeout() time.Duration { - ^ this is not used anywhere. find where it should be used and add it. - - - - code duplication - - // Initialize parameters - g := group.P256 - t := uint(env.ShamirThresholdVal() - 1) // Need t+1 shares to reconstruct - n := uint(env.ShamirSharesVal()) // Total number of shares - - // Create a secret from our 32-byte key: - rootSecret := g.NewScalar() - if err := rootSecret.UnmarshalBinary(rootKeySeed[:]); err != nil { - failErr := sdkErrors.ErrDataUnmarshalFailure.Wrap(err) - log.FatalErr(fName, *failErr) - } - - - * helm-charts-hardened is still waiting for spike-next merge. - * helm-charts-hardene/spike-next has incorrect assumptions about - components that might need update: - 1. ## @param trustRoot.keepers Override which trustRoot Keepers are in - for spike nexus keepers can have multiple trust roots (which is okay) - but for spike keeper, nexus is assumed to have a single trust root. - in a HA setup nexus can have multiple trust roots, so the value - in values.yaml should be an array. - 2. SPIKE Nexus statefulset uses both SPIKE_TRUST_ROOT_NEXUS and also - SPIKE_TRUST_ROOT (same for keeper, bootstrap, and pilot) - that's not a bug, but it's confusing and should be documented - in the charts which one is for what. SPIKE_TRUST_ROOT is for self - assignment and SPIKE_TRUST_ROOT_NEXUS etc is for validation functions. - - - `make test` is still not parallelized due to env setup. - - - when entries are not registered to the SPIRE Server and the - operator tries to use SPIKE, the error messages can be more - explanatory. - - - also, empty id or path should raise an error for policies - and also for secrets - - - spike-next is still an open branch on helm-charts main. - check if there are still things that need to be merged, and what - needs to be done to push the PR forward. - - - Go through GitHub issues and close the completed ones. - - - // It's unlikely to have 1000 SPIKE Keepers across the board. - // The indexes start from 1 and increase one-by-one by design. - const maxShardID = 1000 - to env var configuration. - - - additional memory clearing for the sdk: - https://github.com/spiffe/spike/issues/243 - - - VSecM has a missing video: - vsecm missing video(s) - https://vsecm.com/documentation/getting-started/overview/ - we can directly remove it too, since vscem is less of a focus nowadays. - - - move this to the SDK: app/spike/internal/trust/spiffeid.go - then any code that does self-authentication such as the following, - will use trust.AuthenticateForXyz() instead: - This... - // I should be SPIKE Nexus. - if !spiffeid.IsNexus(selfSPIFFEID){...crash if not nexus...} - will become - trust.AuthenticateForNexus(selfSPIFFEID) - - - SDK. duplicate code fragment: - if source == nil { - return nil, sdkErrors.ErrSPIFFENilX509Source.Clone() - } - - r := reqres.SecretMetadataRequest{Path: path, Version: version} - - mr, marshalErr := json.Marshal(r) - if marshalErr != nil { - failErr := sdkErrors.ErrDataMarshalFailure.Wrap(marshalErr) - failErr.Msg = "problem generating the payload" - return nil, failErr - } - - - - - check all test helpers. some of them can go to the SDK instead. - - - Integration test: Ensure SPIKE Nexus caches the root key in memory. - - - Integration test: Ensure SPIKE Nexus recovers root key from keepers. - - - Integration test: Ensure SPIKE Nexus does not (inadvertently) initialize twice. - Once it's initialized successfully it should not recompute root key - material without manual `spike operator` intervention (because rotating - the root key without re-encrypting secrets will turn the backing store - unreadable) - when the key is lost, it should wait it to be re-seeded by keepers, or - manually recovered via `spike operator`. - - - verify if the keeper has shard before resending it: - send hash of the shard first - if keeper says “I have it”, don’t send the actual shard. - this will make things extra secure. - - - Use Case: Each SPIKE Keeper in its own trust domain. - (we can also create a demo once this feature is established) - - Details: - - The current design assumes that all keepers are in the same trust boundary - (which defaults to spike.ist) - that can make sense for a single-cluster deployment - (each keeper can be tainted to deploy itself on a distinct node for - redundancy) - however; a keeper's SPIFFE ID does not have to have a single trust root. - each SPIKE keeper can (in theory) live in a separate cluster, in its own - trust boundary, depicted by a different trust root. - - If that's the case the below function will not be valid - - func IsKeeper(id string) bool { - return id == spiffeid.SpikeKeeper() - } - - Instead of this the validation should be done against SPIKE_NEXUS_KEEPER_PEERS - and the env var should also contain the trust root of each keeper. - - Similarly, the function cannot check the trust root. - it may however verify the part of spiffeID "after" the trust root. - - // I should be a SPIKE Keeper. - if !cfg.IsKeeper(selfSpiffeid) { - log.FatalF("Authenticate: SPIFFE ID %s is not valid.\n", selfSpiffeid) - } - - So for the `main` function of SPIKE Keeper we'll need a more relaxed - version of IsKeeper. - and the IsKeeper in SPIKE Nexus will validate looking at the env - config. - - which also means, SPIKE Keeper can require SPIKE_TRUST_ROOT along - with SPIKE_KEEPER_TLS_PORT to start. at start-keeper-1.sh - - - Each keeper is backed by a TPM. - - - UpsertPolicy calls LoadAllPolicies to find a policy by name, which is - O(n) and inefficient as the number of policies grows. Add a - LoadPolicyByName method to the backend interface that performs an - indexed lookup directly (SQLite can use an index on the name column). - This would make the lookup O(1) instead of O(n). - - Location: app/nexus/internal/state/base/policy.go (UpsertPolicy function) - Backend interface: app/nexus/internal/state/backend/interface.go - - - Note: this is non-trivial, but doable. - - Periodic rotation of the encryption keys is recommended, even in the - absence of compromise. Due to the nature of the AES-256-GCM - encryption - used, keys should be rotated before approximately 232 encryptions - have - been performed, following the guidelines of NIST publication - 800-38D. - - This can be achieved by having a separate encryption key protected - by - the root key and rotating the encryption key, and maybe maintaining - a - keyring. This way, we won't have to rotate shards to rotate the - encryption - key and won't need to change the shards -- this will also allow the - encryption key to be rotated behind-the-scenes automatically as per - NIST guidance. - - - spike dev mode: - - it will not require SPIFFE - - it will be in memory - - it will be a single binary - - it will present a SPIKE Nexus API in that binary. - - regular unsafe `curl` would work. - - would be SDK-compatible. - - ^ not sure it's worth the effort, but it will be nice-to-have. - - - attribute-based policy control - - path "secret/restricted" { - capabilities = ["create"] - allowed_parameters = { - "foo" = [] - "bar" = ["zip", "zap"] - } - } - - - maybe a default auditor SPIFFEID that can only read stuff (for - Pilot; - not for named admins; named admins will use the policy system - instead) - - - audit targets: - - file - - syslog - - socket - (if audit targets are enabled then command will not execute unless - an - audit trail is started) - - - maybe ha mode - - HA Mode in OpenBao: In HA mode, OpenBao operates with one active server - and multiple standby servers. The active server processes all requests, - while standby servers redirect requests to the active instance. If the - active server fails, one of the standby servers takes over as the new - active instance. This mechanism relies on PostgreSQL's ability to manage - locks and ensure consistency across nodes35. - Limitations: - The PostgreSQL backend for OpenBao is community-supported and considered - in an early preview stage, meaning it may have breaking changes or limited - testing in production environments2. - While PostgreSQL supports replication and failover mechanisms for its own - HA, these features operate independently of OpenBao's HA mode. Proper - configuration and monitoring of the PostgreSQL cluster are essential to - ensure database-level resilience - - - We need use cases in the website - - Policy-based access control for workloads - - Secret CRUD operations - - etc - - - a way to factory-reset SPIKE: reset db; recreate rootkey; delete - etc. - - spike operator reset: - deletes and recreates the ~/.spike folder - restarts the initialization flow to rekey keepers. - - volkan@spike:~/Desktop/WORKSPACE/spike$ spike secret get /db - Error reading secret: post: Problem connecting to peer - - ^ I get an error instead of a "secret not found" message. - - - {"time":"2025-04-25T13:24:52.652299515-07:00","level":"INFO","m":"HydrateMemoryFromBackingStore","m":"HydrateMemoryFromBackingStore: - secrets loaded"} - {"time":"2025-04-25T13:24:52.652368182-07:00","level":"INFO","m":"HydrateMemoryFromBackingStore","m":"HydrateMemoryFromBackingStore: - policies loaded"} - ^ - how can we know that this data has already been pushed. - a brute force way is to hash the last payloads and compare with the hashes of the current payloads. - if hydrated, no need to re-hydrate then. - but that requires two full table scans, json serialization, and hashing. - could there be a better way? - - - Test with different shamir ratios - - * 5/3 -- 5 keepers, out of 3 should be alive. - * 1/1 -- A single keeper - * 0/0 -- edge case; not sure how it should behave. - * in-memory -- in-memory mode should disregard any keeper config. - - - app/nexus/internal/state/backend/sqlite/persist/crypto.go - these are generic enough to move to the SDK. - maybe go through entire internal folders and see what can be ported. - - - validateContext(ctx, fName) - this can be an SDK utility function and can be used across the codebase instead of just this package. - - - - ability to lock nexus programmatically. - `spike operator lock/unlock` => will need the right clusterspiffeid for - the command to work. - - ^ instead of that, you can run a script that removes all SVID - registrations. That will effectively result in the same thing. - - also document this - also create a demo video - - - - TODO optimize sqlite default params and also make sure we retry - database operations -- at least check that we have sane defaults. - - ref: https://tenthousandmeters.com/blog/sqlite-concurrent-writes-and-database-is-locked-errors/ - - - pattern-based random secret generation - VSecM already does it; leverage it from it. - Or, alternatively, move the code to SPIKE SDK and let VSecM use it - from SPIKE. - - - dev mode with "zero" keepers. - this exists, but we need a demo video for it. - and maybe documentation update. - - - Integration test: Ensure SPIKE Pilot denies any operation when SPIKE - Nexus is not initialized. - - - Integration test: Ensure SPIKE Pilot warns the user if SPIKE Nexus is - unreachable. - - - Integration tests: Ensure we can create and read secrets. - - - Integration test: Ensure we can create and read policies. - - - // this pattern is repeated a lot; move to a helper function. - _ = os.Setenv("SPIKE_NEXUS_BACKEND_STORE", "memory") - defer func() { - if original != "" { - _ = os.Setenv("SPIKE_NEXUS_BACKEND_STORE", original) - } else { - _ = os.Unsetenv("SPIKE_NEXUS_BACKEND_STORE") - } - }() - - - Test SPIKE Lite setup. - And maybe create a demo video. - - - goes to spike sdk go - func Id() string { - id, err := crypto.RandomString(8) - if err != nil { - id = fmt.Sprintf("CRYPTO-ERR: %s", err.Error()) - } - return id - } - - - A `--dry run` feature for spike commands: - it will not create policies, secrets, etc, but just pass validations - and return success/failure responses instead. - useful for integration tests. - - - make sure we check Spike.LiteWorkload spiffe id in policies. - also make sure the encryption as a service works. - - - Notes: not for VSecM but for SPIKE directly maybe; - also create an ADR first, maybe. - - Need to think about this; I'm not sure we need a UI at all. - I am getting more fond of Claude Code model of triggering a web-device-PKCE - cfg flow and then doing everything through the CLI. - - For VSecM: create a UI that directly talks to SPIKE; depracate all VSecM - secret model. also delete most of the folders. start with a clean slate - - - You have OIDC-related drafts; polish them and move them somewhere visible - as an actionable roadmap. - - - introduce SPIKE to VSecM - https://github.com/vmware/secrets-manager/issues/1275 - - - consider using modernize: - https://pkg.go.dev/golang.org/x/tools/gopls/internal/analysis/modernize - - - for spike consider some of these if not there yet already - https://fluxcd.io/flux/security/ - - - const KeeperKeep APIURL = "/v1/store/keep" - const KeeperContribute APIURL = "/v1/store/contribute" - const KeeperShard APIURL = "/v1/store/shard" - - ^ these are unused in the SDK, which may hint that whatever uses them - here may need to be SDK methods maybe. - - - make tests concurrent again. - - - Go over entire documentation: There are places to update since - we have helm charts updates. - - - Sign generated binaries: - #!/usr/bin/env bash - set -euo pipefail - - ART_DIR="${1:-dist}" - MODE="${MODE:-kms}" # kms|keyless|file - KEY="${KEY:-awskms://arn:aws:kms:us-west-2:123456789012:key/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX}" - PUB="${PUB:-spike-cosign.pub}" - - cd "$ART_DIR" - - # 1) checksums - sha256sum * > SHA256SUMS - - signfile () { - local f="$1" - case "$MODE" in - kms) - cosign sign-blob --key "$KEY" --output-signature "$f.sig" "$f" - ;; - file) - cosign sign-blob --key cosign.key --output-signature "$f.sig" "$f" - ;; - keyless) - COSIGNEXPERIMENTAL=1 cosign sign-blob --yes \ - --output-signature "$f.sig" \ - --output-certificate "$f.pem" \ - --bundle "$f.bundle" \ - "$f" - ;; - *) - echo "Unknown MODE=$MODE" >&2; exit 1 - ;; - esac - } - - # 2) sign all artifacts + checksum file - for f in *; do - [[ "$f" =~ .(sig|pem|bundle)$ ]] && continue - sign_file "$f" - done - - # 3) export public key if using KMS - if [[ "$MODE" == "kms" ]]; then - cosign public-key --key "$KEY" > "../$PUB" - fi - - echo "Signed. Publish signatures + $PUB (if present)." - - - create a list for what needs for 1.0 version; i.e. to move SPIKE - out of alpha and make it "ready for production use with a hint of caution" - - SPIFFE Org already has standard requirements and a process for that. - Check it out. - - - after helm-charts changes are merged, verify that the quickstart - guide still works. - - - audit: right now we send audit logs to std err. - we would need different audit targets later. - - - this function is a temporary fix. - conxtext-based handling is already in the SDK - update it - func contributeWithContext( - ctx context.Context, api *spike.API, - share secretsharing.Share, keeperID string, - ) error { - - - // Serve the app. - sdkNet.ServeWithRoute( - appName, - source, - http.Route, - // AllowAll, because any workload can talk to SPIKE Nexus if they - // have a legitimate SPIFFE ID registration entry. - // we might want to further restrict this based on environment - // configuration maybe (for example, a predicate that checks regex - // matching on workload SPIFFFE IDs before granting access, - // if the matcher is not provided, AllowAll will be assumed. - predicate.AllowAll, - env.NexusTLSPortVal(), - ) - - - CLI debug logging to file. - - Currently, the spike CLI (spike policy, spike secret, etc.) does not - have any structured logging for debugging purposes. When errors occur, - only user-friendly messages are shown. - - Consider adding an optional file-based logging mechanism for the CLI - that can be enabled via an environment variable (e.g., SPIKE_CLI_LOG) - or a flag (e.g., --debug-log=/path/to/file). This would help with - troubleshooting without cluttering the terminal output. - - The logs should be structured (JSON) and include error codes, timestamps, - and context information. - - - also check out: https://developer.hashicorp.com/vault/docs/concepts/policies - to see if we can amend any updates to the policy rules - (one such update, for example, is limiting what kind of attributes are - allowed, but we should discuss whether that much granularity is worth the - hassle) - - - ADR: - * Added the ability to optionally skip database schema creation during SPIKE - initialization. This can be useful if the operator does not want to give - db schema modification privileges to SPIKE to adhere to the principle of - least privilege. The default behavior is to allow automatic schema creation. - Since SPIKE is assumed to own its backing store, limiting its access - does not provide a significant security benefit. Letting SPIKE manage - its own database schema provides operational convenience. - - - - these may go to the sdk - - func GenerateCustomNonce(s *DataStore) ([]byte, error) { - nonce := make([]byte, s.Cipher.NonceSize()) - if _, err := io.ReadFull(rand.Reader, nonce); err != nil { - return nil, err - } - return nonce, nil - } - - func EncryptWithCustomNonce(s *DataStore, nonce []byte, data []byte) ([]byte, error) { - if len(nonce) != s.Cipher.NonceSize() { - return nil, fmt.Errorf("invalid nonce size: got %d, want %d", len(nonce), s.Cipher.NonceSize()) - } - ciphertext := s.Cipher.Seal(nil, nonce, data, nil) - return ciphertext, nil - } - - - - these serialization and deserialization functions can be - extracted. --- They can even be part of the Go SDK. - - It can error, or swallow unknown permissions. - - // Deserialize permissions from comma-separated string - if permissionsStr != "" { - permissionStrs := strings.Split(permissionsStr, ",") - policy.Permissions = make([]data.PolicyPermission, len(permissionStrs)) - for i, permStr := range permissionStrs { - policy.Permissions[i] = data.PolicyPermission(strings.TrimSpace(permStr)) - } - } - - - missingFlags = append(missingFlags, "name") - } - if pathPattern == "" { - missingFlags = append(missingFlags, "path-pattern") - } - if SPIFFEIDPattern == "" { - missingFlags = append(missingFlags, "spiffeid-pattern") - } - if permsStr == "" { - missingFlags = append(missingFlags, "permissions") - - have these flag names as constants maybe. - - - go-spiffe uses context.Context so that we can time out SVID lookup instead - of waiting forever; create an env variable for that and implement it - wherever it makes sense. - - - func contains(permissions []data.PolicyPermission, - func hasAllPermissions( - these can be generic helper functions in the sdk. - - - u, err := url.JoinPath( - keeperAPIRoot, string(apiUrl.KeeperContribute), - ) - - these should be methods of SDK instead. - - - implement this sometime: - ci/wip-draft.txt - and find a place to run it every commit; could be github actions; but also - can take a long time, so could be a dedicated toy server too. - - maybe we can add a configurable "bootsrap time out" as an env var later. For now, the bootsrap app will try to - bootstrap the thing in an exponentially-backing-off loop until it succeeds. - - - update helm-charts-hardened for SPIKE HA setup - based on future/001-spike-ha.md, it should be trivial. - - also add a section in the docs about SPIKE HA setup. - - - go through internal files, some of them are generic enough to graduate - to the SDK. - - - Refactor app/spike/internal/cmd/secret/get.go to extract repeated - marshaling logic (yaml.Marshal, json.Marshal, json.MarshalIndent) into - helper functions. The same marshal-and-print pattern is duplicated - multiple times for different format types. - - - func VerifyShamirReconstruction(secret group.Scalar, shares []shamir.Share) { - this is generic enough to go to the sdk. - - Check if there are other similar reusable functions that can go. - - - validPermsList := "read, write, list, super" - SDK should define these instead. - - - these can be made generic and added to the SDK: - app/nexus/internal/state/base/validation.go - - - CLI Command Testing Guidance - - This issue documents strategies for testing SPIKE CLI commands with - varying levels of complexity. - - == UNIT TESTING (No Mocking Required) == - - The following helper functions can be tested directly without mocking: - - 1. Policy Commands (app/spike/internal/cmd/policy/): - - filter.go: validUUID() - UUID validation - - fotmat.go: formatPoliciesOutput(), formatPolicy() - output formatting - - validation.go: validatePolicySpec() - policy spec validation - - create.go: readPolicyFromFile(), getPolicyFromFlags() - input parsing - - 2. Secret Commands (app/spike/internal/cmd/secret/): - - validation.go: validSecretPath() - path validation - - print.go: formatTime(), printSecretResponse() - output formatting - - 3. Cipher Commands (app/spike/internal/cmd/cipher/): - - io.go: openInput(), openOutput() - file I/O handling - - Example test pattern for pure functions: - ```go - func TestValidUUID(t *testing.T) { - tests := []struct { - name string - uuid string - expected bool - }{ - {"valid UUID", "123e4567-e89b-12d3-a456-426614174000", true}, - {"invalid", "not-a-uuid", false}, - } - for _, tt := range tests { - t.Run(tt.name, func(t *testing.T) { - if validUUID(tt.uuid) != tt.expected { - t.Errorf("validUUID(%q) = %v, want %v", - tt.uuid, !tt.expected, tt.expected) - } - }) - } - } - ``` - - == HTTP-LEVEL MOCKING (For API Calls) == - - To test command Run functions that call the SDK API, use httptest.Server - to mock SPIKE Nexus responses: - - ```go - func TestSecretGetCommand_Success(t *testing.T) { - // Create mock server - server := httptest.NewTLSServer(http.HandlerFunc( - func(w http.ResponseWriter, r *http.Request) { - // Verify request - if r.URL.Path != "/v1/store/secret" { - t.Errorf("unexpected path: %s", r.URL.Path) - } - - // Return mock response - resp := reqres.SecretReadResponse{ - Data: map[string]string{"key": "value"}, - } - json.NewEncoder(w).Encode(resp) - }, - )) - defer server.Close() - - // Configure SDK to use mock server URL - // (requires setting SPIKE_NEXUS_API_URL env var or similar) - t.Setenv("SPIKE_NEXUS_API_URL", server.URL) - - // Execute command and verify output - // ... - } - ``` - - Challenges with HTTP mocking: - - SDK uses mTLS, so mock server needs proper TLS config - - May need to mock X509Source or bypass SPIFFE validation - - Consider creating test helpers for common mock scenarios - - == INTEGRATION TESTING (With Real SPIKE Nexus) == - - For full end-to-end testing, use the existing start.sh to spin up - a real SPIKE environment. The integration_test.go pattern in the - policy package shows this approach: - - 1. Start SPIKE via start.sh (SPIRE + Nexus + Keeper) - 2. Wait for initialization - 3. Run actual CLI commands - 4. Verify results - - ```go - // +build integration - - func TestPolicyCreateIntegration(t *testing.T) { - if os.Getenv("SPIKE_INTEGRATION_TEST") == "" { - t.Skip("Skipping integration test") - } - - // Use actual spike CLI binary or SDK - api, err := spike.New() - if err != nil { - t.Fatalf("Failed to create API client: %v", err) - } - defer api.Close() - - // Create policy - err = api.CreatePolicy(data.PolicySpec{ - Name: "test-policy", - // ... - }) - if err != nil { - t.Fatalf("CreatePolicy failed: %v", err) - } - - // Verify policy exists - policies, _ := api.ListPolicies("", "") - // Assert... - } - ``` - - Run integration tests with: - ```bash - # Start SPIKE environment first - ./hack/start.sh - - # Run integration tests - SPIKE_INTEGRATION_TEST=1 go test -tags=integration ./... - ``` - - == TESTING log.FatalErr CALLS == - - Functions that call log.FatalErr can be tested using the panic - recovery pattern: - - ```go - func TestAuthenticateForPilot_InvalidID(t *testing.T) { - // Enable panic mode for FatalErr - t.Setenv("SPIKE_STACK_TRACES_ON_LOG_FATAL", "true") - - defer func() { - if r := recover(); r == nil { - t.Error("Expected panic for invalid ID") - } - }() - - AuthenticateForPilot("invalid-spiffe-id") - t.Error("Should have panicked") - } - ``` - - == RECOMMENDED APPROACH == - - 1. Start with unit tests for pure helper functions (high value, low effort) - 2. Add HTTP-level mocks for critical API paths - 3. Use integration tests for end-to-end validation in CI - - Current coverage after unit tests: - - policy commands: ~48% - - secret commands: ~10% - - cipher commands: ~18% - - Target: 60%+ coverage for command packages through unit + mock tests. - - - env using log causes circular dependencies which we fixed by copying - a few functions; but instead these env value functions should return - errors and logging should be based on the returned error values. - - those errors also should be sentinel errors that indicate fatal - failures. - - func KeepersVal() map[string]string { - const fName = "KeepersVal" - - p := os.Getenv(NexusKeeperPeers) - - if p == "" { - log.FatalLn( - fName, - "message", - "SPIKE_NEXUS_KEEPER_PEERS must be configured in the environment", - ) - } - - - internal/net/request.go - ^ can be SDK candidates. - - func VerifyShamirReconstruction(secret group.Scalar, shares []shamir.Share) { - can go to sdk. - - - moved relatively complex items into the ./future folder; - they need some love. - - - - - create a list for what needs for 1.0 version; i.e. to move SPIKE - out of alpha and make it "ready for production use with a hint of caution" - - SPIFFE Org already has standard requirements and a process for that. - Check it out. - - - YAML-based secret creation from SPIKE CLI. - - Follow up: for k8s clusters, a custom resource that manages these secrets securely - (i.e., values will be encrypted, and can only be decrypted through SPIKE Pilot; we can create - a cli helper for that (`kubectl get --raw /apis/example.com/v1/namespaces/default/spikesecrets/demo-db/reveal`) - (as a custom resource) - or - a kubectl plugin - `curl -H "Authorization: Bearer $(kubectl config view --raw -o jsonpath='{.users[0].user.token}')" \ - "https://spike-controller.default.svc/reveal?name=$1"` - or a CSI Driver that mounts secrets to a volume. - or something like that: - - # Direct API call for cluster admins - kubectl get --raw /apis/spike.io/v1/namespaces/default/spikesecrets/demo-db/reveal - # Or with a simple kubectl plugin wrapper - kubectl spike reveal demo-db - # Or - kubectl spike reveal demo-db -o yaml - - cid:6062f33b-4a94-4b0f-a306-c13c4dd3d777 - - - - - Update: - This needs to be re-evaluated. As long as we make SPIKE log formatting - SOC-compliant; and streamable to a log aggregator, we don't necesarrily - need to implement an opinionated logging infra, as users will likely - be using their own secure log aggregation solutions anyway. - - It is possible to have a reference infrastructure somewhere, but I think - it's out of SPIKE (and also VSecM's) scope. - - --- - - SOC-2 Compliance Checklist for SPIKE Logging System - - ## Overall Plan - - * Make SPIKE Audit logs streamable, structured, and have mandatory SOC-2 - fields and actions. - * Position VSecM as a centralized log manager that does the heavy-lifting. - - ## Overview - - This checklist helps ensure the SPIKE's and VSecM's logging infrastructure - meets SOC-2 compliance requirements under the Trust Services Criteria, - with a focus on Security (mandatory), Availability, Processing Integrity, - Confidentiality, and Privacy. - - ## 1. Logging Infrastructure Setup - - ### Core Logging Requirements - - - [ ] **Centralized Log Management** (VSecM) - - Implement a centralized log repository for all SPIKE components - (Nexus, Keeper, Pilot, Bootstrap) - - Use structured logging format (JSON preferred) - - Ensure logs from all environments are aggregated - - - [ ] **Log Collection Scope** - - Application logs (errors, warnings, info, debug) - - Authentication and authorization events - - System access logs (successful and failed attempts) - - API request/response logs - - Security events and anomalies - - Configuration changes - - Administrative actions - - Data access and modification events - - Network traffic logs - - - [ ] **Real-time Monitoring** - - Implement continuous, real-time log collection - - Set up automated log analysis - - (potential future item) Configure SIEM integration. - - ## 2. Log Content Requirements - - ### Mandatory Fields for Each Log Entry - - [ ] **Timestamp** (ISO 8601 format with timezone) - - [ ] **Event Type/Category** (authentication, authorization, error, etc.) - - [ ] **User Identity** (user ID, service account, or system) - - [ ] **Source IP Address** - - [ ] **Component/Service Name** (Nexus, Keeper, Pilot) - - [ ] **Action Performed** - - [ ] **Result** (success/failure) - - [ ] **Session ID** (for correlation) - - [ ] **Unique Log ID** - - ### Security-Specific Events to Log - - - [ ] **Authentication Events** - - Login attempts (successful/failed) - - Logout events - - Password changes - - MFA events - - Session timeouts - - - [ ] **Authorization Events** - - Access grants/denials - - Privilege escalations - - Role changes - - Permission modifications - - - [ ] **Secret Management Events** - - Secret creation/deletion - - Secret access requests - - Secret modifications - - Key rotation events (feat: implement automatic root key rotation) - - Root key management operations - - - [ ] **System Changes** - - Configuration modifications - - Software updates/patches - - Security policy changes - - Certificate updates - - ## 3. Log Protection and Integrity - - ### Security Controls - - [ ] **Encryption** - - Encrypt logs in transit (TLS 1.2+) - - Encrypt logs at rest (AES-256 or stronger) - - - [ ] **Access Controls** - - Implement role-based access control (RBAC) for log access - - Principle of least privilege for log viewers - - Separate duties for log administration - - Multi-factor authentication for log access - - - [ ] **Integrity Protection** - - Implement log tampering detection - - Use write-once storage where possible - - Generate checksums/hashes for log files - - Implement log signing for critical events - - - [ ] **Segregation** - - Separate production logs from development/testing - - Isolate sensitive data logs with additional controls - - ## 4. Log Retention and Availability - - ### Retention Policies - - - [ ] **Define Retention Periods** (all configurable) - - Security event logs: minimum 12 months - - Access logs: minimum 90 days - - System logs: minimum 6 months - - Audit logs: minimum 7 years (or per regulatory requirements) - - - [ ] **Backup and Recovery** - - Regular automated backups of log data - - Test restore procedures quarterly - - Off-site/cloud backup storage - - Document recovery time objectives (RTO) - - - [ ] **Availability Requirements** - - Ensure 99.9% availability of logging system - - Implement redundancy for log collectors - - Set up failover mechanisms - - Monitor logging system health - - - ## 5. Monitoring and Alerting - - ### Alert Configuration - - - [ ] **Security Alerts** - - Multiple failed authentication attempts - - Unauthorized access attempts - - Privilege escalation events - - Configuration changes - - Suspicious API usage patterns - - Log system failures - - - [ ] **Threshold Settings** - - Define baselines for normal activity - - Set anomaly detection thresholds - - Configure rate-limiting alerts - - Establish escalation procedures - - - [ ] **Response Procedures** - - Document incident response procedures - - Define alert priority levels - - Establish notification chains - - Set response time SLAs - - ## 6. Compliance and Audit Support - - ### Documentation - - - [ ] **Logging Policy** - - Create a formal logging and monitoring policy - - Define what must be logged - - Specify retention requirements - - Document access procedures - - - [ ] **Procedures Documentation** - - Log review procedures - - Incident investigation procedures - - Log export/reporting procedures - - Evidence collection procedures - - ### Audit Trail Features - - - [ ] **Evidence Collection** - - Automated evidence gathering capabilities - - Log export in standard formats (CSV, JSON) - - Chain of custody documentation - - Audit report generation - - - [ ] **Compliance Mapping** - - Map log events to SOC-2 controls - - Create compliance dashboards - - Generate compliance reports - - Track control effectiveness - - ## 7. Privacy and Data Protection - - ### PII Handling - - - [ ] **Data Minimization** - - Avoid logging sensitive personal data - - Mask/redact PII in logs (SSN, credit cards, etc.) - - Implement field-level encryption for sensitive data - - - [ ] **Privacy Controls** - - Implement data classification - - Tag logs containing personal data - - Apply appropriate retention policies - - Enable right-to-erasure capabilities - - ## 8. Implementation Checklist - - ### Technical Implementation - - [ ] **Logging Framework** - - Choose an appropriate logging library (e.g., structured logging) - - Implement correlation IDs across services - - Add context to all log entries - - Standardize log formats across components - - - [ ] **Infrastructure Setup** - - Deploy log aggregation system (preferably open source with a non-evil license) - - Configure log forwarders/agents - - Set up log storage infrastructure - - Implement log rotation policies - - - [ ] **Integration Points** - - SPIFFE/SPIRE integration for identity - - mTLS logging for Pilot API calls - - Keeper redundancy event logging - - Nexus encryption/decryption audit logs - - ## 9. Testing and Validation - - ### Testing Requirements - - [ ] **Log Generation Testing** - - Verify all required events are logged - - Test log format consistency - - Validate timestamp accuracy - - Check correlation ID propagation - - - [ ] **Security Testing** - - Test access controls - - Verify encryption implementation - - Attempt log tampering (should fail) - - Test log injection prevention - - - [ ] **Performance Testing** - - Measure logging overhead - - Test high-volume scenarios - - Verify no log loss under load - - Monitor storage growth rates - - ## 10. Ongoing Maintenance - - ### Regular Activities - - - [ ] **Monthly Tasks** - - Review log storage capacity - - Analyze security alerts - - Update alert thresholds - - Review access logs - - - [ ] **Quarterly Tasks** - - Test backup/restore procedures - - Review and update logging policy - - Conduct log retention cleanup - - Perform access reviews - - - [ ] **Annual Tasks** - - Complete logging policy review - - Update risk assessment - - Conduct penetration testing - - Review with external auditor - - ## 11. Metrics and KPIs - - ### Monitoring Metrics - - [ ] **Operational Metrics** - - Log ingestion rate - - Storage utilization - - Query response times - - System availability percentage - - - [ ] **Security Metrics** - - Failed authentication attempts - - Unauthorized access attempts - - Time to detect incidents - - Mean time to respond (MTTR) - - - [ ] **Compliance Metrics** - - Percentage of systems with logging enabled - - Log retention compliance rate - - Audit finding closure rate - - Evidence collection time - - ## 12. Tools and Technologies to Consider - - ### Logging Solutions - - - **Open Source Options:** - - Fluentd + Prometheus + Grafana - - Graylog - - Apache Kafka for log streaming - - ### SIEM Integration - - - Configure integration with Security Information and Event Management systems - - Enable correlation rules - - Set up automated threat detection - - ## Priority Actions for SPIKE - - Given SPIKE's architecture as a secrets management system using SPIFFE: - - 1. **Immediate Priorities:** - - Implement comprehensive audit logging for all secret operations - - Add mTLS certificate validation logging - - Log all SPIFFE identity verification events - - Capture root key access and management events - - 2. **Critical Security Events:** - - Log every secret encryption/decryption operation - - Track Keeper failover events - - Monitor Pilot CLI authentication and commands - - Record all administrative access to Nexus - - 3. **Integration Requirements:** - - Ensure SPIRE integration includes workload attestation logs - - Log SVID issuance and rotation - - Capture trust bundle updates - - Monitor federation events if applicable - - - ## Notes - - - This checklist should be reviewed and updated quarterly - - SOC-2 Type II requires demonstrating control effectiveness - over 3-12 months - - - - - fix: - func TestInitialize_MemoryBackend_ValidKey(t *testing.T) { - - - Fix LogFatalXXX-related tests. - * func TestPostHTTPInteraction(t *testing.T) { - - Fix: - //func TestNew_CipherCreationFailure(t *testing.T) { - - - fix func TestInitializeBackend_UnknownType_DefaultsToMemory(t *testing.T) { - it is a logic error, so better to fix this release rather than waiting for - the next one. - - same here: - func TestInitializeBackend_NoEnvironmentVariable_DefaultsToMemory(t *testing.T) { - - - Fix: - func TestNew_InvalidKey(t *testing.T) { - - - Fix: - func TestUnmarshalShardResponse_InvalidInput(t *testing.T) { - - Fix: - func TestURLJoinPath(t *testing.T) { - - Fix: - func TestURLJoinPathForKeepers(t *testing.T) { - - - Make this testable: - - fmt.Println("") - fmt.Println("Usage: bootstrap -init") - fmt.Println("") - os.Exit(1) // define a global osExit function or maybe in SDK, to be able to test stuff. - - - func TestKeeperIDConversion(t *testing.T) { - keeper ids need to be stricter. add validation logic to the code. - - - func TestShardURL_InvalidInput(t *testing.T) { - Keeper API root shall not be empty. - - - - - - make the CI folder work again. - there is a wip-draft.txt there to think about. - we can run the tests in gh actions inside a container probably. - - - update SPIKE pilot to be able to delete multiple policies or secrets - - - This how policy list works now: - When using filters, you must provide **the exact regular expression pattern** as - defined in the policies you want to match. For example, if a policy is defined - with pattern `^secrets/database/production$`, you must use exactly that pattern - to find it---no partial matches or simpler patterns will work. - - ^ it would be nice for the list command to have substring matches. - - ^ we can then use the same substring match logic to list secrets - (i.e. list secrets that have db/creds anywhere in their path. - - - consider adding an "s3" backing store. - - this will be different from the Lite option. It will support policies, and other utilities, - it will essentially act as a file-based databased stored in an object store. - - policy management will be done via standard SPIKE policies. - - will be able to connect s3 and anything s3-compatible. - - s3 connection can be established by SPIFFE OIDC, or some other way. - - - nexus should have a status endpoint and pilot should warn the user if - nexus is not ready - - Bootstrap completed successfully! - ➜ spike git:(feature/no-cache) ✗ spike secret list - Error listing secret keys: post: Problem connecting to peer - Post "https://localhost:8553/v1/store/secrets?action=list": dial tcp 127.0.0.1:8553: connect: connection refused - ➜ spike git:(feature/no-cache) ✗ spike secret list - - - - fix all `t.Skip()` skipped tests. - - - - create a /status endpoint for SPIKE Nexus and use that for the - Bootstrap job instead of querying the jobs k8s api. - that will also mean, the Bootstrap job will be more secure since - its ServiceAccount will not need kube api access. - - - maybe for vsecm: plugin-based dynamic secrets to access third-party - services: - https://developer.hashicorp.com/vault/tutorials/get-started/understand-static-dynamic-secrets - - - if spike nexus had a prometheus endpoint, what kinds of metrics - would it expose? - - - vsecm: - Integration with Key Vault providers - - e.g. AWS Secrets Manager, CyberArk - - - SPIKE OPA Integration - --- - webhooks for external policies, and maybe ValidatingAdmissionPolicies (CEL based) - - - Kubernetes secrets as a backing store for SPIKE. - The secrets are stored in k8s secrets in encrypted form. - the workloads have to decrypt the secrets using the lite API. - - - Audit Logging: SOC 2 Type II compliance - Records of creation, modification, and deletion, including the user who performed the action and a timestamp. - - - ability for nexus to return encrypted secrets. - vsecm wants that. - - - VSecM should use spiffe.source too from SDK. - Also spiffe.source should have a timeout and err out if it cannot - acquire the source in a timely manner. The timeout should be - configurable from the environment options. - - - verify lite mode. - - - verify kind scripts. - - - vsecm: cleanup experimental parts; - switch to github registry; switch to %100 helm; - directly consume spire charts from upstreams - - - create a federated spike doc; and also a video. - - - - build-local.sh and build-push-sign.sh have a lot of commonalities; - maybe refactor/merge them? - - - VSecM ADR: - SPIKE Integration Plans - Create an ADR about the near/mid future plans wrt VSecM and SPIKE. - - - SPIKE Documentation - Generating Protocol Buffers should be before the "build the project" section - also, needed to do "go mod vendor" - https://vsecm.com/documentation/development/use-the-source/ - - - Make vSecM use helm only (you can override images if needed, - to test with local images. -- that will be better than - generating local manifests and maintaining them separately) - - - - Postgres support as a backing store. - - Postgres should be a plugin (similar to SQLite) - - - - Ability to channel audit logs to a log aggregator. - - NOTE: This feature can be delegated to VSecM instead. - - - - OIDC integration: Ability to connect to an identity provider. - - NOTE: This feature can be delegated to VSecM instead. - VSecM can be an identity broker; and SPIKE can be a client. - - - - ESO (External Secrets Operator) integration - - NOTE: This feature can be delegated to VSecM to. - - - - An ADMIN UI (linked to OIDC probably) - - NOTE: This feature can be delegated to VSecM instead. - VSecM can provide a UI for SPIKE Nexus. - - - - Ability to use the RESTful API without needing an SDK. - That could be hard though since we rely on SPIFFE authentication and - SPIFFE workload API to gather certs: We can use a tool to automate - that - part. But it's not that hard either if I know where my certs are: - `curl --cert /path/to/svid_cert.pem --key /path/to/svid_key.pem - https://mtls.example.com/resource` - - - > 80% unit test coverage - - - Fuzzing for the user-facing API - - - 100% Integration test (all features will have automated integration - tests - in all possible environments) - - - Ability to add custom metadata to secrets. - - - - - thing like - systemctl start spike-nexus - systemctl start spike-keeper - systemctl status spike-nexus - systemctl status spike-keeper - -- - also `spike status` from SPIKE Pilot should give a brief status of the - system... how many keepers, what are the health of the keepers, - health of nexus, how many secrets, resource usage, etc. - - - # to be added to docs. - # how to expose things to other clusters: - # kubectl port-forward --address 0.0.0.0 svc/nginx-lb 8080:80 - # no need for ingress - # no need for `kubectl port-forward` - # great for demo/development setups. - # Also forward registry to docker to work - # kubectl port-forward -n kube-system svc/registry 5000:80 - # You don't need to `eval $(minikube -p minikube docker-env)` - # Again, this is simpler. - - - VSecM: Use GCR. - - - VSecM fix: - func log(message string) { - conn, err := grpc.Dial( - SentinelLoggerUrl(), - grpc.WithTransportCredentials(insecure.NewCredentials()), - grpc.WithBlock(), - ) - Dial is deprecated - WithBlock is deprecated - - // Create a gRPC client - conn, err := grpc.Dial(lis.Addr().String(), grpc.WithInsecure(), grpc.WithBlock()) - if err != nil { - t.Fatalf("failed to dial server: %v", err) - } - WithInsecure/WithBlock are deprecated at rpc_test.go - - - VSecM fix: - http_test - nopcloser is deprecated - func TestReadBody_Success(t *testing.T) { - // Prepare the test data - cid := "test-cid" - expectedBody := []byte("test body content") - r := &http.Request{ - Body: ioutil.NopCloser(bytes.NewBuffer(expectedBody)), - } - - - VSecM Fix: - secret-server/main.go - ReadAll is deprected - - - VSecM Fix - // Read the request body - body, err := ioutil.ReadAll(r.Body) - if err != nil { - http.Error(w, "Cannot read body", http.StatusBadRequest) - return - } - defer r.Body.Close() - and there are unhandled errors in r.Body.Close() s. - - - VSecM Fix - - postgres.go +> remove postgres support; it's better to add it to SPIKE instead. - - - VSecM Fix: - move backoff/retry code to the go sdk. - - - VSecM Fix: - stream := cipher.NewCFBDecrypter(block, iv) - NewCFBDecrypter is deprecated. - decrypt.go - - same: - stream := cipher.NewCFBEncrypter(block, iv) - stream.XORKeyStream(ciphertext[aes.BlockSize:], []byte(data)) - NewCFBDecrypter is deprecated. - - - VSecM Fix: - remove relay client and relay server-related code. - - - VSecM: - move docs to public and update CloudFlare worker to automatically - consume it. - - - VSecM: need to automate documentation - right now, we create manual deployments on CloudFlare and that does - not scale. - - create a public folder in the repo - - let cloudflare update the website from the "public" folder. - - "older versions" are still broken; but since documentation is markdown - we can always point the tagged version of it as former docs we - don't need an entire browsable website for it. At least, that's how we - do it with SPIKE and it saves effort. - - - There is an ongoing work on HTML...ization of the Turtle Book. - We also need to start a work on updating spiffe.io for the - new book. - - - in development mode, nexus shall act as a single binary: - - you can create secrets and policies via `nexus create policy` etc - - that can be done by sharing - "github.com/spiffe/spike/app/spike/internal/cmd" - between nexus and pilot - - this can even be an optional flag on nexus - (i.e. SPIKE_NEXUS_ENABLE_PILOT_CLI) - running ./nexus will start a server - but run - ning nexus with args will register secrets and policies. - - - Consider using OSS Security Scorecard: - https://github.com/vmware-tanzu/secrets-manager/security/code-scanning/tools/Scorecard/status - - - SPIKE automatic rotation of encryption key. - the shards will create a root key and the root key will encrypt the - encryption key. - so SPIKE can rotate the encryption key in the background and encrypt - it with the new root key. - this way, we won't have to rotate the shards to rotate the - encryption key. - - - SPIKE CSI Driver - - the CSI Secrets Store driver enables users to create - `SecretProviderClass` objects. These objects define which secret - provider - to use and what secrets to retrieve. When pods requesting CSI - volumes are - made, the CSI Secrets Store driver sends the request to the OpenBao - CSI - provider if the provider is `vault`. The CSI provider then uses the - specified `SecretProviderClass` and the pod’s service account to - retrieve - the secrets from OpenBao and mount them into the pod’s CSI volume. - Note - that the secret is retrieved from SPIKE Nexus and populated to the - CSI - secrets store volume during the `ContainerCreation` phase. - Therefore, pods - are blocked from starting until the secrets are read from SPIKE and - written to the volume. - - - shall we implement rate limiting; or should that be out of scope - (i.e. to be implemented by the user. - - - more fine grained policy management - - 1. an explicit deny will override allows - 2. have allowed/disallowed/required parameters - 3. etc. - - # This section grants all access on "secret/*". further restrictions - can be - # applied to this broad policy, as shown below. - path "secret/*" { - capabilities = ["create", "read", "update", "patch", "delete", - "list", "scan"] - } - - # Even though we allowed secret/*, this line explicitly denies - # secret/super-secret. this takes precedence. - path "secret/super-secret" { - capabilities = ["deny"] - } - - # Policies can also specify allowed, disallowed, and required - parameters. here - # the key "secret/restricted" can only contain "foo" (any value) and - "bar" (one - # of "zip" or "zap"). - path "secret/restricted" { - capabilities = ["create"] - allowed_parameters = { - "foo" = [] - "bar" = ["zip", "zap"] - } - - but also, instead of going deep down into the policy rabbit hole, - maybe - it's better to rely on well-established policy engines like OPA. - - A rego-based evaluation will give allow/deny decisions, which SPIKE - Nexus - can then honor. - - Think about pros/cons of each approach. -- SPIKE can have a - good-enough - default policy engine, and for more sophisticated functionality we - can - leverage OPA. - - - key rotation - - NIST rotation guidance - - Periodic rotation of the encryption keys is recommended, even in the - absence of compromise. Due to the nature of the AES-256-GCM - encryption - used, keys should be rotated before approximately 232 - encryptions have been performed, following the guidelines of NIST - publication 800-38D. - - SPIKE will automatically rotate the backend encryption key prior to - reaching - 232 encryption operations by default. - - also support manual key rotation - - - Do an internal security analysis / threat model for spike. - - - dynamic secrets - - - use case: - one time access to an extremely limited subset of secrets - (maybe using a one time, or time-bound token) - but also consider if SPIKE needs tokens at all; I think we can - piggyback - most of the authentication to SPIFFE and/or JWT -- having to convert - various kinds of tokens into internal secrets store tokens is not - that much needed. - - - - TDO Telemetry - - core system metrics - - audit log metrics - - authentication metrics - - database metrics - - policy metrics - - secrets metrics - - - "token" secret type - - will be secure random - - will have expiration - - - double-encryption of nexus-keeper comms (in case mTLS gets - compromised, or - SPIRE is configured to use an upstream authority that is - compromised, this - will provide end-to-end encryption and an additional layer of - security - over - the existing PKI) - - - * Implement strict API access controls: - * Use mTLS for all API connections - * Enforce SPIFFE-based authentication - * Implement rate limiting to prevent brute force attacks - * Configure request validation: - * Validate all input parameters - * Implement request size limits - * Set appropriate timeout values - * Audit API usage: - * Log all API requests - * Monitor for suspicious patterns - * Regular review of API access logs - ---- - * Enable comprehensive auditing: - * Log all secret access attempts - * Track configuration changes - * Monitor system events - * Implement compliance controls: - * Regular compliance checks - * Documentation of security controls - * Periodic security assessments - --- - * Tune for security and performance: - * Optimize TLS session handling - * Configure appropriate connection pools - * Set proper cache sizes - * Monitor performance metrics: - * Track response times - * Monitor error rates - * Alert on performance degradation - - - ability to clone a keeper cluster to another standby keeper cluster - (for redundancy). - this way, if the set of keepers become not operational, we can - hot-switch to the other keeper cluster. - the assumption here is the redundant keeper cluster either remains - healthy, or is somehow snapshotted -- since the shards are in - memory, snapshotting will be hard. -- but stil it's worth thinking - about. - an alternative option would be to simplyh increase the number of - keepers. - - - work on the "named admin" feature (using Keycloak as an OIDC - provider) - This is required for "named admin" feature. - - - Consider using google kms, azure keyvault, and other providers - (including an external SPIKE deployment) for root key recovery. - question to consider is whether it's really needed - second question to consider is what to link kms to (keepers or - nexus?) - keepers would be better because we'll back up the shards only then. - or google kms can be used as an alternative to keepers - (i.e., store encrypted dek, with the encrypted root key on nexus; - only kms can decrypt it -- but, to me, it does not provide any - additional advantage since if you are on the machine, you can talk - to - google kms anyway) - - - etcd-like watch feature for SPIKE - - one functionality that would be really cool and kind of a game changer I - think, but would be hard to do with sql, would be something similar to - etcd/k8s's watches. - - spike watch /foo keyhere -o /tmp/somefile - - would update /tmp/somefile whenever it changes on the server - - - idea: custom resources for policies and secrets. - - - adding a timeout or circuit breaker to the infinite loop in - BootstrapBackingStoreWithNewRootKey, I was suggesting a way to prevent - the function from running indefinitely if something goes wrong with keeper - initialization. - The current implementation uses: - for { - // Ensure to get a success response from ALL keepers eventually. - exit := iterateKeepersToBootstrap( - keepers, rootShares, successfulKeepers, source, - ) - if exit { - return - } - - log.Log().Info(fName, "message", "Waiting for keepers to initialize") - // TODO: make the time configurable. - time.Sleep(5 * time.Second) - } - This loop will continue forever until all keepers are successfully - initialized. While this makes sense for normal operation, there are - scenarios where this could become problematic: - - If one or more keepers are permanently unavailable or unreachable - If there's a persistent network issue preventing communication - If there's a configuration issue that makes successful initialization - impossible - - A potential improvement would be to add: - maxAttempts := env.GetMaxBootstrapAttempts() // Could be configurable - attempts := 0 - - ctx, cancel := context.WithTimeout(context.Background(), - env.GetBootstrapTimeout()) - defer cancel() - - for { - select { - case lt-ctx.Done(): - log.Log().Warn(fName, "message", "Bootstrap timed out after waiting threshold") - // Implement fallback strategy or escalate the issue - return - default: - attempts++ - if maxAttempts > 0 ++ attempts > maxAttempts { - log.Log().Warn(fName, "message", "Exceeded maximum bootstrap attempts") - // Implement fallback strategy or escalate the issue - return - } - - exit := iterateKeepersToBootstrap( - keepers, rootShares, successfulKeepers, source, - ) - if exit { - return - } - - log.Log().Info(fName, "message", "Waiting for keepers to initialize", - "attempt", attempts, "maxAttempts", maxAttempts) - time.Sleep(5 * time.Second) - } - } - This approach provides: - - A maximum number of attempts (configurable) - A total timeout for the entire operation (configurable) - Better observability of progress through attempt counting - - In highly reliable systems, you might want the bootstrap to keep trying - forever, but even then, it's valuable to have observability into how - long it's been trying and an option to break the loop if needed. - - - - - Make vSecM uses helm only (you can override images if needed, - to test with local images. -- that will be better than - generating local manifests and maintaining them separately) - - - Create VSecM ADR: - - VSecM-SPIKE Integration Strategy - - SPIKE: SPIFFE-native secrets manager - VSecM: Secrets Manager and Orchestrator - - SPIKE: mandatory SPIFFE use - VSecM: SPIFFE is a core feature but can work/integrate with others - - SPIKE: minimal CLI as UX structure (and API) - VSecM: policy engine, audits, secret LCM, automation, UI-ready. - - SPIKE: embedded, edge, stateless - VSecM: stateful, policy-aware, multi-tenant - - Helm: - SPIKE: subchart under SPIFFE - VSecM: consumes SPIRE helm chart; can enable SPIKE - * SPIKE is set up as the "default" secrets manager and first-class - integration; but it can integrate with other secrets stores too. - - - VSecM: - For Ubuntu users; do not use snap to install docker as it can create permission issues when working with minikube. - */ - ^ - rendering on the page has error. - - - VSecM: - consume SPIRE from upstream helm instead of our custom fork. - - - document limits and maximums of SPIKE (such as key length, path - length, policy size etc) - - also ensure that in the source code. - - - get an OpenSSF badge sometime. - - - OIDC authentication for named admins. - - - SPIKE Dynamic secret sidecar injector - - - v.1.0.0 Requirements: - - Having S3 as a backing store - - This is different than SPIKE Lite - - - Consider a health check / heartbeat between Nexus and Keeper. - This can be more frequent than the root key sync interval. - - - double encryption when passing secrets around - (can be optional for client-nexus interaction; and can be mandatory - for - tools that transcend trust boundaries (as in a relay / message queue - that - may be used for secrets federation) - - - admin ui - - - - AWS KMS support for keepers - - Azure keyvault support for keepers - - GCP kms support for keepers - - HSM support for keepers - - OCI kms support for keepers - - keepers storing their shards in a separate SPIKE deployment - (i.e. SPIKE using another SPIKE to restore root token) - - - better play with OIDC and keycloak sometime. - - - wrt: secure erasing shards and the root key >> - It would be interesting to try and chat with some of the folks under - the cncf - (That's a good idea indeed; I'm noting it down.) - - - over the break, I dusted off - https://github.com/spiffe/helm-charts-hardened/pull/166 and started - playing with the new k8s built in cel based mutation functionality. - the k8s cel support is a little rough, but I was able to do a whole - lot in it, and think I can probably get it to work for everything. - once 1.33 hits, I think it will be even easier. - I mention this, as I think spike may want similar functionality? - csi driver, specify secrets to fetch to volume automatically, keep - it up to date, and maybe poke the process once refreshed - - - consider using NATS for cross trust boundary (or nor) secret - federation - note: NATS had licensing issues recently. - - - spike, but as an http proxy? - like, point http(s)_proxy at this thing, connect to it using your spiffe id, the proxy can use policies and the - client spiffeid to allow/disallow an http proxy request - No; but not a bad idea indeed. -- will look around; I'll share if I find something related. - - - Check if we need traversal resistant file api needs anywhere. - SPIKE: traversal-resistant file apis: https://go.dev/blog/osroot - - - all components shall have - liveness and readiness endpoints - (or maybe we can design it once we k8s...ify things. - - - if a keeper crashes it has to wait for the next nexus cycle which is - suboptimal. Instead, nexus can send a ping that returns an overall - status - of keeper (i.e. if it's populated or not) - this can be more frequent than hydration; and once nexus realizes - keeper - is down, it can rehydrate it. - - in addition; nexus can first check the sha hash of the keeper's shard. - before resending; if the hashes match, it won't restransmit the shard. - - - A /stats endpoint. - - A dedicated /stats endpoint will be implemented to provide real-time - metrics about: - Total number of secrets managed. - Status of the key-value store. - Resource utilization metrics (e.g., CPU, memory). - This endpoint will support integration with monitoring tools for enhanced - observability. - These measures will ensure comprehensive monitoring and troubleshooting. - - - configure SPIKE to rekey itself as per NIST guidelines. - Also maybe `spike operator rekey` to manually initiate that. - `spike operator rekey` will also change the shamir shares, wheras the - internal rekey will just change the encryption key, leaving the shamir - shares intact. - - - Fleet management: - - There is a management plane cluster - - There is a control plane cluster - - There are workload clusters connected to the control plane - - All of those are their own trust domains. - - There is MP-CP connectivity - - There is CP-WL connectivity - - MP has a central secrets store - - WL and CP need secrets - - Securely dispatch them without "ever" using Kubernetes secrets. - - Have an alternative that uses ESO and a restricted secrets - namespace - that no one other than SPIKE components can see into. - - - aes256-cbc - jay:U2FsdGVkX1+VhdGia1yk+JAUSraXj60ZA2ydT9TuHmQBE1sWLcMLb5z0B76sCqpJVGi1GQCl8BnnoV5kznYneQ== - - - TDO it's early but have a deprecation policy for SPIKE - https://external-secrets.io/latest/introduction/deprecation-policy/ - - - TOO update vsecm documentation about cloudflare changes 1. manually upload 2. point to github for older versions. - - - Consider metrics collector integration. - - This is from SPIRE: - The metrics collectors that are currently supported are Prometheus, Statsd, - DogStatsd, and M3. Multiple collectors can be configured simultaneously, both in - the servers and the agents. - - Think about what telemetry SPIKE can create. - - create telemetry data and support certain metrics collectors. - - create a monitoring epic - - - [vmware-tanzu/secrets-manager] Scorecard supply-chain security workflow run - ^ - this is constantly failing. - maybe disable it. - - also it's about time to cut a VSecM release since we had quite a few - security patches in already. - - - VSecM: Maybe use SPIFFE Helper instead of VSecM sidecar since it - essentially does the same thing. Or maybe have an alternative - implementation that uses spiffe helper instead of vsecm sidecar. - - - ideation: - - state is expensive to maintain. - thats one of the reasons cloud services try and decouple/minimize - state - as such, the fewest number of state stores I can get away with - reasonably the better - and the state stores that are light weight are much better then the - state stores that are heavy weight. - there is no more heavyyweight state store than a network attached - sql sever. - - - It makes sense to argue that "we already have paid the expensive - cost of a postgresql, - so we just want to use that rather then add anything else". That, - can make sense. - - but for those of us not carrying a postgresql already, its better - not to have to have one added. - - so... it makes sense to make the backing store "plugin-based" - - 3 backends that people might want for different reasons: - * s3 - be stateless for your own admin needs, state managed by - someone else - * k8s crds - you are already maintaining an etcd cluster. Might as - well reuse it - * postgresql - you maintain a postgresql and want to reuse that - - the first two initially feel different then postgresql code wise... - they are document stores. - But posgres is pretty much json-compatible; besides SPIKE does not - have a complicated ata model. - So, we can find a common ground and treat all databases that are - plugged-in as forms of document stores. - - It could keep the code to talk to the db to a real minimum. - - The files should be encrypted by the spike key, so should be fine - just putting in a k8s crd or something without a second layer of - encryption - that can be a big selling point. already have a small k8s cluster? - just add this component and now you have a secret store too. no - hassle. - - - use custom resources as backing store; - since everything is encrypted and not many people want a fast - secrets creation throughtput it woudl be useful. - because then you can do `helm install spiffe/spire` and use it - without any state tracking. - - - for k8s instructions (docs) - Might recommend deploying with the chart rather then from scratch, - which has a lot of those settings. then we can call out the settings - in the chart on how to do it - - - An external secrets store (such as Hashi Vault) can use SPIKE Nexus - to - auto-unseal itself. - - - multiple keeper clusters: - - keepers: - - nodes: [n1, n2, n3, n4, n5] - - nodes: [dr1, dr2] - - if it cant assemble back from the first pool, it could try the next - pool, which could be stood up only during disaster recovery. - - - a tool to read from one cluster of keepers to hydrate a different - cluster of keepers. - - - since OPA knows REST, can we expose a policy evaluation endpoint to - help OPA augment/extend SPIKE policy decisions? - - - maybe create an interface for kv, so we can have thread-safe - variants too. - - - maybe create a password manager tool as an example use case - - - A `stats` endpoint to show the overall - system utilization - (how many secrets; how much memory, etc) - - - maybe inspire admin UI from keybase - https://keybase.io/v0lk4n/devices - for that, we need an admin ui first :) - for that we need keycloak to experiment with first. - - - wrt ADR-0014: - Maybe we should use something S3-compatible as primary storage - instead of sqlite. - But that can wait until we implement other features. - - Besides, Postgres support will be something that some of the - stakeholders - want to see too. - - - SPIKE Dev Mode: - - * Single binary - * `keeper` functionality runs in memory - * `nexus` uses an in-memory store, and its functionality is in the - single - binary too. - * only networking is between the binary and SPIRE Agent. - * For development only. - - The design should be maintainable with code reuse and should not - turn into - maintaining two separate projects. - - - rate limiting to api endpoints. - - - - S3 (or compatible) backing store - - - File-based backing store - - - In memory backing store - - - Kubernetes Deployment - - - - - Security Measures (SPIKE Nexus) - - Encrypting the root key with admin password is good - Consider adding salt to the password encryption - - Maybe add a key rotation mechanism for the future - - - - Error Handling - - Good use of exponential retries - - Consider adding specific error types/codes for different failure - scenarios - - Might want to add cleanup steps for partial initialization - failures - - - Ability to stream logs and audit trails outside of std out. - - - Audit logs should write to a separate location. - - - Create a dedicated OIDC resource server (that acts like Pilot but - exposes - a - restful API for things like CI/CD integration. - - - HSM integration (i.e. root key is managed/provided by an HSM, and - the key - ever leaves the trust boundary of the HSM. - - - Ability to rotate the root key (automatic via Nexus). - - - Ability to rotate the admin token (manual). - - - Encourage to create users instead of relying on the system user. - - -