[V2] Optional observability provider (New Relic first)

[perki]

Pluggable observability façade under components/business/src/observability/. New Relic ships as the first concrete provider. Opt-in: master reads PlatformDB-encrypted secrets, spreads the relevant env to every worker, and a line-1 boot shim loads the agent before http/express/pg. Admin CLI bin/observability.js for enable/disable/configure. Sensitive headers and request bodies stripped by default.

[perki]

Decisions taken

• Provider abstraction so additional APMs can be added later without core depending on a specific vendor.
• Secrets encrypted at rest in PlatformDB using a key derived from auth.adminAccessKey.
• The newrelic npm package is optionalDependencies, so operators not using it pay no install cost.

[perki]

Reopening to finish the sensitive-data handling side of this feature.

The façade, the New Relic adapter, the PlatformDB-encrypted config and the admin CLI all shipped and are in master. What was never completed is the validation gate that this feature was supposed to pass before an operator turns it on in production: proving that no personal data reaches the vendor's transaction traces. That gate was parked in April waiting on an operator-provided license key, and two gaps have since been identified in review.

Remaining scope

1. High Security Mode default and its opt-in path. The adapter defaults high_security to off, because account-side HSM is irreversible and a client/account mismatch makes the agent's connect handshake fail. The intent was that operators who want HSM could opt in via configuration, but that knob is not actually wired: the config loader never reads a high-security value, so the environment variable is always false. Either build the opt-in or document plainly that there is none.

2. Request URLs and transaction names are not covered by the attribute filter. The current filter excludes sensitive headers and the request body, but not the request URI. On this API a path or query string can carry a username, event ids, attachment ids, and in some flows an access token (?auth=, ?readToken=). The Host header, which is captured by default, carries <username> as a subdomain in DNS-ful topologies. Route shape and timings are the useful APM signal here; subject-linking identifiers should not leave the deployment.

3. Staging validation. Enable on a staging cluster, generate representative traffic, and produce positive evidence (named queries with pass/fail criteria) that identifiers are absent from the collected attributes.

4. Operator documentation and data-flow documentation refresh, including a stale "high-security mode is on" claim in the operator guide.

This is not a live exposure. Observability is opt-in and ships disabled; a deployment that has not configured a provider sends nothing anywhere. The gaps above matter for operators who enable it, in particular those handling health data, who need a precise answer to "what identifiers does my APM vendor hold?".

Progress will be reported here so dependent work can track it from this issue.

[perki]

The data-protection half of this work is now tracked separately in #116 (scrubbing personal data from observability telemetry), so that work depending specifically on the scrubbing posture can follow it without watching the whole provider feature. #116 also covers a second data path this issue's earlier comment did not mention: forwarded application logs, which the agent enables by default and which attribute exclusion does not reach.

This issue stays open for the provider feature itself.