[FEATURE] #2346
Description
WARNING: Model outputs or model “verification” must never be used as the sole or final evidence.
Any decision that relies on a model as the only verification mechanism is considered a waiver of responsibility and must not be deployed or used for compliance.
CORE STATEMENT
Models are not oracles. Delegating “verification” to a model transfers uncertainty, unpredictability, and adversarial risk outward while obscuring responsibility.
When incidents occur, the name written in documentation does not bear liability. Legal and operational responsibility rests with the organization and individuals who deploy and operate the system.
HOW THIS FAILS (REAL RISKS)
- Legal / Compliance Risk: Model error → incorrect decision → fines, lawsuits, regulatory action
- Security Risk: Prompt injection or adversarial inputs cause approval of sensitive data or unsafe actions
- Availability Risk: False negatives block critical workflows → service outages
- Financial Risk: Outsourced verification leads to uncontrolled cost and liability
- Audit Risk: No chain-of-evidence → impossible RCA and accountability
- Social / Trust Risk: Ethical judgment delegated to a black box → institutional credibility collapse
COMMON FAILURE MODES
- “The model says the document is valid” → Legal signs → invalid contract submitted
- “The model confirms discharge eligibility” → clinical error
- “The model approved payment” → fraud or mispayment
If model “verification” is used alone—with no human signature, no proof package, and no review—the outcome is disaster.
MANDATORY POLICY REQUIREMENTS (NON-NEGOTIABLE)
- Model Is Assistive Only:
Model outputs must never be the sole or final evidence. All final decisions require explicit approval by a named human Owner (signed or logged). - Proof Package (Required Per Decision, Non-Removable):
Required fields:
request_id, caller_id, purpose, input_raw, model_version, model_output_raw,
confidence_signals, rule_version, matched_patterns, timestamp,
human_owner_id, human_decision, human_reason, audit_blob_sha
Retention: minimum 365 days (or longer if required by regulation)
Storage: append-only, immutable audit store (WORM or equivalent) - Deterministic Checks Are Mandatory:
Model verification must not replace deterministic pre-checks.
For any critical decision, a verifiable deterministic admission gate must pass before model output can be considered. - Never-Allowed Scenarios (Model-Only Decisions Prohibited):
- Medical emergency or final diagnosis decisions
- Financial payment or funds authorization
- Legal contract execution
- Control-plane actions (deploys, secret rotation)
- Government or regulatory responses
- Incident forensics or security triage
- Shadow-Mode & Sampling Validation (Required):
Any new model-verification flow must run in shadow-mode for at least 30 days,
recording decisions without affecting production, producing FN/FP reports.
Failure to meet predefined SLOs (e.g., FN ≤ 0.5%) blocks production release. - Owners & SLA (Required and Reachable):
Each rule and pipeline must define:
Policy Owner (Governance), Runtime Owner (SRE), Product Owner, Legal Owner.
Incident escalation must allow human review assignment within 15 minutes. - Rule Changes Are Controlled (Git-Governed):
All rule changes require PRs, test vectors, CI regression, and code-owner approval.
Hotfixes must be logged and followed by backfilled tests. - Vendor Contracts Must Assign Responsibility:
Vendor terms must not claim model outputs have legal authority or replace human judgment.
No liability disclaimers based solely on “model output.”
DECISION-PROOF JSON SCHEMA (REQUIRED)
{
"request_id": "string",
"caller_id": "string",
"purpose": "string",
"input_raw": "string",
"model_version": "string",
"model_output_raw": "string",
"confidence_signals": {
"risk_score": "int",
"complexity_score": "int",
"template_score": "int"
},
"rule_version": "string",
"matched_patterns": ["string"],
"timestamp": "ISO8601",
"human_owner_id": "string|null",
"human_decision": "approved|rejected|n/a",
"human_reason": "string|null",
"audit_blob_sha": "sha256"
}
PRE-LAUNCH MINIMUM CHECKLIST
- Decision proofs stored in append-only audit storage (≥365 days)
- Shadow-mode active for ≥30 days with FN/FP reports meeting SLOs
- Never-allowed flows hard-blocked from model-only verification
- Owners (Policy/Runtime/Product/Legal) defined and on-call
- CI regression with ≥200 test vectors (including FN/FP cases)
- Cost-vs-benefit validated: Net compute savings > 0 (including gate overhead)
- Vendor contracts explicitly deny legal authority of model outputs
MINIMAL INCIDENT PLAYBOOK
- Detect: FN-rate threshold breach or DLP alert → page Incident Commander (0–15 min)
- Isolate: switch to shadow-only or human-only (15–60 min)
- Evidence Capture: freeze audit sink, export proof packages (immediate)
- Triage: Policy Owner + Security + Product review samples (≤2 hours)
- Fix: rule error → Policy Owner hotfix PR; infra issue → Runtime hotfix (24–72 hours)
- RCA: publish RCA within 7 days; merge preventive tests within 30 days
ONE-LINE POLICY TEXT
This organization prohibits using any model output as the sole or final basis for decisions.
Models are assistive only. All final decisions must be explicitly approved and recorded by a named human Owner with a complete Proof Package.
Any violation assigns legal and operational liability to the executing organization and individuals.
CRUEL SUMMARY
Letting a model “verify” your work is a convenient excuse to avoid evidence gathering, responsibility, and compliance.
Without proof packages, human owners, never-allowed critical lists, and shadow-mode validation, you are not using AI for verification—you are gambling with future lawsuits and losses.