[POC][PND] AlertZero × recommended actions: two AD workers + per-action containment approval - #288196
Draft
opauloh wants to merge 34 commits into
Draft
[POC][PND] AlertZero × recommended actions: two AD workers + per-action containment approval#288196opauloh wants to merge 34 commits into
opauloh wants to merge 34 commits into
Conversation
…ice) Add a thin end-to-end Attack Discovery worker driven by the Watch Floor orchestrator on a schedule. - New worker system-security-watch-attack-discovery: one security.attack-discovery.run step (retrieve -> generate -> validate -> persist to the shared ad-hoc AD index) -> ai.agent investigation conversation -> workflow.output; concurrency drop/max 1. - watch_floor.yaml: add scheduled trigger (every 15m); route steps by execution.triggeredBy (alert -> triage, scheduled -> AD worker); pass AD inputs from consts.watch_policy.config. - Register the worker and bump PND_WATCH_FLOOR_WORKFLOW version 4 -> 5. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Managed static install writes the workflow document but never wires Task Manager, so Watch Floor's scheduled trigger never ran. Add a route-driven ensure-schedule that re-asserts enabled=true through the Workflows management API for the caller's space, which reaches syncSchedulerAfterSave and programs the task.
- New POST /internal/pnd/watches/{watchId}/_schedule -> WatchesService.ensureSchedule().
- Add PND_WATCH_SCHEDULE_URL_TEMPLATE / buildWatchScheduleUrl to kbn-pnd-common.
- Cover ensureSchedule (scheduled / not-found / unavailable) in watches_service.test.ts.
- Single-space: the task id workflow:<id>:scheduled is not space-scoped, so one global doc schedules one space; multi-space (per-space docs) is a follow-up.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…dence
getEsWorkflowForScheduler re-read the workflow with an exact-spaceId filter, so it never matched a global (spaceId '*') managed workflow like the PND watches. syncSchedulerAfterSave then bailed ('not found after save') and no Task Manager task was created, even though updateWorkflow succeeded. Pass includeGlobal so the scheduler resolves global docs and programs their scheduled triggers.
- workflow_crud_service.getEsWorkflowForScheduler: buildWorkflowFilters space includeGlobal: true.
- watch_floor.yaml: scheduled trigger every 15m -> 1m; PND_WATCH_FLOOR_WORKFLOW version 5 -> 6.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Revert the 1m test cadence to 15m; PND_WATCH_FLOOR_WORKFLOW version 6 -> 7 so versionStrategy:auto reinstalls. Takes effect after a Kibana restart + re-run of the _schedule endpoint (bulkUpdateSchedules updates the task interval). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… the AD worker
The run step returns { execution_uuid, status: 'pending' } once generation exceeds its 90s soft deadline, so the AD worker never saw the discoveries inline. Add an event-log-backed get_status step (reusing getWorkflowExecutionsTracking + extractPipelineValidationData) that resolves discoveries by execution_uuid, and rewire the worker to run async then while-poll get_status until terminal before writing the investigation.
- New step security.attack-discovery.get_status: common schema, server handler (+ unit test), public definition; registered in both server and public registries.
- watch_attack_discovery.yaml: run (mode: async) -> while-poll get_status (15s x40) -> final_status -> write_investigation -> output; worker version 1 -> 2.
- register_workflow_steps.test.ts updated for the 6th step.
- NOTE: the platform-wide Scout step_definitions_approval snapshot must be regenerated on a stack to include the new step.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…_ stutter) The step was named get_status, so the get<Name>StepDefinition factory produced getGetStatusStepDefinition / get_get_status_step_definition.ts. Rename the step to 'security.attack-discovery.status' so every symbol reads naturally (StatusStep*, getStatusStepDefinition) with a single get. The Agent Builder get_status tool is a separate registry and keeps its id. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a PND-owned flyout that renders the AD Worker config controls (alert retrieval, generation, validation) mirroring the Security Solution schedule form, without depending on triggersActionsUi or the Assistant context. The connector dropdown is the one real reuse: the shared @kbn/inference-connectors useLoadConnectors hook (http-only). Config is local state only (no persistence) and its shape equals the security.attack-discovery.run step inputs, shown in a live JSON preview. - New public/components/ad_worker_config/* module + OpenAdWorkerConfigButton hosted on the Settings page. - Only new dependency: @kbn/inference-connectors (platform/shared) in tsconfig kbn_references. - RTL test covers render, section presence, connector loading (featureId attack_discovery), ES|QL toggle, live preview, connector listing, and close. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…tabs (Option A spike) Feasibility spike for Option A (reuse the AD flyout's workflow_settings_view components). Vendors the self-contained security_solution controls into PND to prove they render outside that plugin and to surface the remaining blockers. - Vendored (copied) StepAccordion, PipelineIndicator, QueryModeSelector — EUI/Emotion-only, drop in unchanged; provenance noted in each file. - Recomposed the flyout as a numbered steps timeline (Alert retrieval -> Generation -> Validation) with the ES|QL <-> Query-builder mode toggle; Max alerts / time range / filter now live under the Query-builder tab; ES|QL editor under the ES|QL tab. - Real workflow lists via http (GET /api/workflows): Alert-retrieval workflows multi-select + Validation workflow list. Added 'Run every' (maps to the orchestrator schedule). - BLOCKERS deferred (documented in tooltip/report): connector '+ Add model' needs the elastic-assistant selector (triggersActionsUi + AssistantProvider); the KQL Query-builder needs data/data-view-manager. Production = shared @kbn/attack-discovery-workflow-config package (this vendors instead of extracting). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reverted the shared-package extraction spike (kept the components rebuilt inside PND) and refined per review: Alert retrieval is now ES|QL-only with a pre-populated ES|QL query, behind the mode switch buttons + an information icon (Query-builder mode shows a placeholder). Vendored steps timeline (StepAccordion/PipelineIndicator), Generation > Run every, and the Validation workflow list (built-in + validation workflows) are retained. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Use the valid EUI 'info' icon (questionInCircle no longer exists) for all info tooltips. - 'Alert retrieval method' and 'Alert retrieval workflows' are now EuiSwitch toggles (enable/disable) with info icons, matching the AD flyout; the retrieval-workflows selector is revealed when its switch is on. - Alert retrieval stays ES|QL-only with a pre-populated query (behind the mode switch buttons). - Connector dropdown now includes a '+ Add model' option that navigates to the connectors management page (no triggersActionsUi / AssistantProvider needed). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ed fields from output - Rename the retrieval switch to 'ES|QL query' and drop the ES|QL/Query-builder mode tab; the switch just shows the ES|QL query field. - ES|QL field now uses the Monaco CodeEditor (@kbn/code-editor + @kbn/monaco ESQL_LANG_ID) for syntax highlighting instead of a plain textarea. - Add toWorkerInputs(): when the ES|QL switch is off, alert_retrieval_mode and esql_query are omitted from the resulting worker-inputs object (shown in the preview). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Watch AD worker uses security.attack-discovery.run in sync mode (default) with a step timeout; removes the async run + while-poll machinery and consumes the run output inline. - Draft one investigation per discovered attack (foreach); create a single tracking investigation when a run completes with no attacks. - Remove the security.attack-discovery.status step type (no longer needed): server/public registrations, barrels, definition + tests. - Update AD worker flow diagram (png + mmd) to the simplified sync flow. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ine)
The run step's sync mode raced a hard-coded 90s soft deadline and returned
{ execution_uuid, status: pending } for slow generations, so workflow callers
(e.g. the Watch Floor AD worker) never received discoveries and downstream
steps were skipped.
The 90s soft deadline exists to keep the Agent Builder run *tool* under its
120s wrapper ceiling; the AB tool is a separate implementation that applies
its own soft deadline and does not use this step. Sync mode in the run step
now awaits the pipeline to completion, bounded by the step's own timeout, and
returns discoveries inline. Async mode is unchanged.
Updates run-step tests and README timing table / ADR-012 accordingly.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The legacy `name`/`type` array output format constrains array items to string/number/boolean, so the `recommended_actions` array of objects was rejected at `emit_result` with "Output validation failed: recommended_actions: Invalid input". Declare `outputs` in JSON Schema object format so the array can hold objects, and bump the managed workflow version 5 -> 6.
Resolve two additive conflicts against main: - managed workflow registry: keep both PND_WATCH_ATTACK_DISCOVERY_WORKFLOW and the new THREAT_INTEL_* workflows. - security skills test: keep both recommendedActionsSkill and the siem_migration skills in imports and ALL_SKILLS.
Introduce the capability for the Watch Attack Discovery workflow to execute recommended Kibana API actions. This change adds: - A global `execute_actions` kill-switch, disabled by default. - A `waitForInput` step for analysts to review and approve/disapprove individual actions. - Logic to call Kibana APIs for creating cases, setting asset criticality, isolating hosts, killing processes, and hunting process persistence for approved actions. - Robust logging of execution results (success, failure, skipped) for each attempt. - Updates to the AI skill to guide the model on when and how to include `execution_params` for executable actions based on concrete evidence.
feat(pnd): Enhance AD worker robustness and action lifecycle tracking Harden the AI `recommend_actions` step for robustness against failures and tool hallucination, safeguarding downstream logic. Enhance case integration by attaching source alerts and host/IP observables. Migrate asset criticality updates to use Entity Store v2. Provide comprehensive accounting of all recommended actions, including reasons for non-execution. ```
…sion of AlertZero, populated by Attack discovery 2.0.  _Above: The AlertZero home page populated by running Attack discovery_ This POC also prototypes: - Workflows use `waitForInput`, not branching logic, to _always_ pause for Human In the Loop (HITL) input - An auto-approver that runs at higher levels of autonomy - An additional flyout to show potential _future_ actions and link to previous worklfow executions The screenshot below illustrates the first few workflow steps that run when Attack discoveries are generated:  _Above: The first workflow steps when Attack discoveries are generated_ When Attack discoveries are generated: - A new `security.attackDiscoveryCreated` workflow trigger is invoked for each discovery - Conversation IDs are derived for each discovery (to prevent duplicate conversations) - An _investigation_ agent builder conversation is created via `/api/agent_builder/conversations` for each discovery - A `type: 'text'` attachment for the attack discovery is added to the conversation - Another agent builder conversation, the sub-conversation, is created for the `Open an investigation` HITL approval - A `waitForInput` workflow step, which is paired with a sub-conversation, appears on the AlertZero page  _Above: Each `waitForInput` HITL step is paired with a sub-conversation_ At the `manual` autonomy level, a human must approve the investigation with human in the loop (HITL) input, as illustrated by the following screenshot:  _Above: Human in the loop (HITL) approval of an investigation_ - After a human or higher autonomy-level approves, a `workflow.execute` step kicks off (the separate) `system-security-watch-deep` workflow, with the attack discovery ID as input, to investigate:  _Above: The `Investigate` step executes another workflow_ - The investigation workflow uses the `ai.agent` step to triage attack discovery alerts via the `alert-analysis` agent builder skill to determine if the discovery is a true positive - If the investigation indicated the attack was real, a `waitForInput` step displays a HITL _Escalate_ entry on the AlertZero page:  _Above: Users are prompted to `Escalate to an incident`_ - A new _Incident_ agent builder conversation is created when users escalate - The incident conversation includes references to the original investigation sub conversations - Users conform the incident is contained via the _Contain_ HITL step:  _Above: Users are prompted to `Confirm containment`_ - This branch does _NOT_ prototype real containment - The `await_incident_contained` step uses `waitForInput` to confirm the incident was contained - After containment, the `Post-Incident Watch` workflow runs to suggest tuning / improvements to reduce detection gaps - This POC does NOT impement real detection rule tuning e2e - A (stubbed) HITL entry is displayed on the AlertZero page, representing the proposed tuning:  _Above: A proposed detection rule tuning_ Each watch has three levels of automony: | Level | HITL Approvals | |---|---| | `manual` | All phases (e.g. opening an investigation) require HITL approval | | `assisted` | Routine, reversible actions are automatically approved. Anything else requires HITL approval. | | `supervised` | Most actions are auto-approved via an allow-list. Some still require HITL. | In the POC: - Workflows _always_ `waitForInput`, regardless of the currently configured autonomy level - At higher levels of autonomy, the `Watch Auto Approver` workflow uses an internal API to auto-approve workflows waiting for input The POC implements an additional flyout (with placeholder UX) to show potential _future_ actions and link to previous workflow executions, as illustrated by the following screenshot:  _Above: The Lifecycle (placeholder) flyout shows actions that may happen in the future_ The POC implemnts a primative `/chats` page that _simulates_ parent-child relationships between agent builder conversations, as a stopgap until Agent Builder supports them natively. The page has two separate groups of conversations: - Incidents - Investigations as illustrated by the following screenshot:  _Above: Incident chats with Attack discovery attachments_ Running the POC requires enabling the AlertZero feature flag (details below) and the following Kibana advanced settings:  1) Enable the AlertZero feature flag in `kibana.dev.yml`: ```yaml xpack.pnd: enabled: true ``` 2) In Stack Management → Advanced Settings, enable Attack discovery 2.0: - `securitySolution:enableAttackDiscoveryWorkflows: true` In Stack Management → Advanced Settings, optionally enable the following settings: a) `workflows:ui:showManagedWorkflows` b) `workflows:experimentalFeatures` See the AletZero [README](x-pack/solutions/security/plugins/pnd/README.md) for details. _Created with Claude Code, Cursor, Opus 5, and Grok 4.6_
…values. Catalog watches stay disabled until enabled in a space; autonomy lives on those template values instead of uiSettings, and boot only installs static helpers (including Detection and the auto-approver). Co-authored-by: Cursor <cursoragent@cursor.com>
…ended-actions Merges the AD recommended-actions HITL POC (elastic#287158) into the AlertZero runnable POC (elastic#287301). Resolution strategy: - AlertZero's four-phase Watch Floor orchestrator is kept as the foundation (watch_floor.yaml, definitions/pnd structure, settings page, watches service). - The recommended-actions skill, @kbn/pnd-common action contracts, discoveries run-step sync-await change, ensure-schedule route/service, AD worker config UI components, and the workflow_crud_service includeGlobal fix are carried in. - The 1478-line combined watch_attack_discovery worker is removed: its generation lane and its recommend/review/execute lane are re-landed next as a dedicated Attack Discovery Generation watch and a rebuilt Floor Phase 3. - ensureSchedule and the _schedule route are adapted to AlertZero's per-space managed-install model and RouteAuthz factories. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…Floor Implements the two-worker Attack Discovery split and rebuilds Phase 3 of the Watch Floor around recommended actions: - New Attack Discovery Generation watch (per-space catalog watch, system-security-watch-attack-discovery-generation): scheduled trigger with a configurable cadence, alert size, lookback and connector persisted as managed template values, concurrency drop max 1. Each net-new persisted discovery emits security.attackDiscoveryCreated and wakes one Watch Floor run — keeping generation and the per-discovery HITL lifecycle in separate workflows so a parked gate can never stall a generation tick and the engine's avoid-loop guard passes. - Watch Floor v21 Phase 3: recommend_actions (recommended-actions Agent Builder skill, structured output) stages containment actions; the always-human incident_contained gate now carries a per-action approval (approved_actions, embedded in the card via a label-anchored JSON contract); approved Kibana-executable actions execute post-gate as the approving analyst (create case + alert attachments + observables, Entity Store asset criticality, isolate host, kill process, hunt process persistence) with a complete per-action ledger; incident_closed and the post-incident tuning lane are unchanged. - Queue UI: RecommendedActionsDecisionForm renders one toggle per staged action (manual actions listed read-only; surfaced-only actions flagged), defaulting everything off; dismiss disables and empties the approval. - Lifecycle flyout: new Containment actions section + step-3-6 evidence badges render the execution ledger via the executions route's new containmentActions projection; phase catalog gains Recommend/Execute rows. - Watch detail: Frequency select is live for the generation watch, plus a Generation section (alerts per run, alert window, connector); settings writes re-assert enablement so Task Manager reprograms the cadence. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…nt gate - watch_floor v22: open_incident continues on failure (same rationale as draft_tuning — a tool-call hallucination or empty structured response from the incident agent must not kill a run a human already escalated; no retry because the turn creates a conversation at a fixed id). The containment card says plainly when the incident brief is absent; recommend_actions already stages actions from the attack discovery and investigation assessment alone. - ensureSchedule resolves the watch registration before the managed-workflows status read (type-correct id narrowing). - Queue/watch/chat surfaces updated to the containment gate's new 'Review containment actions' label; sixth watch labeled on the queue rows. - PND README, discoveries README and the ad-worker-flow diagram updated to the merged two-worker + recommended-actions architecture. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…b-action errors
Watch Floor v23: the case-observable attachments now POST
/internal/cases/{case_id}/observables (the public versioned route no longer
exists on main and the cap is 10 per case), and the attach/observable ledger
records carry the failing error so a failed row explains itself. The
executions projection also fetches step outputs (includeOutput) so the
containment ledger actually reaches the lifecycle flyout.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…actions JSON The chrome-less approval modal had no height bound and the card no internal scroll, so a containment gate with a long staged-action list pushed its own toggle form and the footer buttons off screen. The modal now caps at the viewport, the card is a flex column whose middle region scrolls while the header and Approve/Cancel footer stay put, and the machine-readable `Staged containment actions JSON:` block is stripped from the displayed reasoning on exactly the rows where the toggle form renders those actions — the prose on either side of the anchor is kept. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…e agent hunt analyze_exfiltration_ips was surfaced-only because no single Kibana endpoint sits behind threat_hunting.exfil_ips. Watch Floor v24 gives an approved toggle a real execution lane that fits what the action is: a scoped, strictly read-only ai.agent hunt (ES|QL over the space's alert/event indices, bounded to 7 days) whose findings are appended to the incident conversation — the audit thread — and whose bounded structured result (summary, communicating hosts, exfiltration_suspected) lands in the per-action ledger as succeeded/failed. The accounting drops the surfaced-only carve-outs (a dismissed or untoggled analysis now ledgers like any other Kibana action), and the toggle form's badge now reads "Read-only agent hunt — findings post to the incident chat" so the toggle states exactly what approval authorizes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ary bound
A staged-action array with full target lists can exceed the reasoning
summary's 8192-character projection bound on its own — a live run truncated
the JSON mid-string, and the toggle form correctly failed closed to the fixed
controls, losing the per-action approval. The machine-readable list now rides
the UNtruncated reasoning sections entry ('Staged containment actions', title
pinned on both sides): the server projects it verbatim onto a new bounded
PndProposalRow.stagedActions field for the containment gate only, the toggle
form prefers that field (the summary label anchor stays as the fallback for
already-parked rows), and the Floor's summary is prose-only again (v25). The
recommend_actions schema also bounds the target arrays as defense in depth.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
|
@opauloh, this PR increases one or more page-load bundle sizes by 15% or more:
Large bundle size increases can affect page load performance. Consider whether dependencies can be lazy-loaded or code split to reduce the bundle. See the bundle optimization guide for tips. |
|
🤖 Jobs for this PR can be triggered through checkboxes. 🚧
ℹ️ To trigger the CI, please tick the checkbox below 👇
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Merges the AD recommended-actions HITL POC (#287158) into the AlertZero runnable POC (#287301), and implements the two-worker Attack Discovery split from elastic/security-team#18972 on top.
Relates to elastic/security-team#18972, #287158, #287301.
Two AD workers
Attack Discovery Generationcatalog watch (system-security-watch-attack-discovery-generation): per-space, scheduled trigger (concurrency: drop, max: 1), singlesecurity.attack-discovery.runstep. Cadence (5m/15m/30m/hourly), alerts-per-run, lookback and connector are managed template values, editable from the watch's settings page in Watch Floor; a settings write re-asserts enablement so Task Manager reprograms the schedule.security.attackDiscoveryCreatedand wakes one Watch Floor run per discovery. Keeping generation in its own workflow means a parked HITL gate can never stall or drop a generation tick, and the engine's avoid-loop guard passes.Watch Floor Phase 3: recommended actions replace the containment stub (floor v25)
recommended-actionsAgent Builder skill (structured output, bounded), and the always-humanincident_containedgate carries a per-action approval (approved_actions; per Nir's per-action-veto feedback). The queue card renders one toggle per Kibana-executable action (manual actions read-only; toggles default off; dismiss disables and empties).analyze_exfiltration_ipsruns as a strictly read-onlyai.agenthunt whose findings post to the incident conversation.succeeded/submitted/failed/skipped/not_executed+ reason), projected throughGET /internal/pnd/executions/{correlationId}into a new Containment actions section and per-step badges in the four-phase lifecycle flyout.incident_closedand the phase-4 post-incident tuning lane are untouched.Contracts & hardening
PndProposalRow.stagedActionsfield (the 8192-char summary bound truncated a real run's array mid-string; the summary label anchor stays as a fallback for already-parked rows)._respondaccepts and forwardsapproved_actions; gate registry label → "Review containment actions"; phase catalog gainsRecommend containment actions/Execute approved actionsrows (16-row skeleton).open_incidentcontinues on failure (the incident agent intermittently returns an empty structured response; a human-approved escalation must still reach the containment gate — same rationale asdraft_tuning).recommended-actionsskill (+recommendedActionsSkillflag),@kbn/pnd-commonaction contracts, discoveries run-step sync-await,_scheduleroute (adapted to the per-space install model),workflow_crud_serviceincludeGlobal fix.Verified end to end
On a live stack (EIS connectors, real + synthetic alerts): scheduled generation → per-discovery floor runs → both HITL prompts → per-action containment approval → real case created, real Entity Store criticality write, endpoint actions honestly
skipped(no agents), unapproved action ledgerednot approved by analyst, read-only exfil hunt posting findings to the incident thread →incident_closed→ post-incident tuning drafted, backtested and parked at its gate. Schedule reprogramming, concurrency drop, and settings persistence exercised live.~6,300 tests green across the touched suites (managed workflows, kbn-pnd-common, pnd server/public, security_solution skills, discoveries); scoped type checks green. PND README, diagram and discoveries README updated.
Screenshot
ad-poc-recommended-actions-compressed.mp4
WIP / POC
Draft on the fork for review of the merged direction. Known follow-ups: AD dedup is fuzzy across LLM-reworded duplicates of the same attack (queue fills up on recurring scheduled runs); the threads
_ensuretitle still says "Decision on confirming this incident is contained".