docs(rfc): propose persistent flow queue and completion webhooks

[mason5052]

Summary

Adds a design RFC at examples/proposals/flow_concurrency.md describing a possible direction for native flow concurrency control and completion webhooks, as raised in issue #298. The RFC is documentation only; no runtime, schema, GraphQL, REST, or UI behavior changes here.

Problem

Issue #298 covers two real pain points for external orchestrators that drive PentAGI from a pipeline:

• createFlow accepts unlimited concurrent calls, so calling it N times immediately spins up N Kali containers and saturates the host.
• There is no native completion notification, so external schedulers maintain a polling loop on GET /api/v1/flows/{id}.

A naive fix can easily slip back into the anti-pattern PR #268 was rejected for: a hidden in-memory queue with no persistence, no UI/API/DB visibility, and no cancelability. PentAGI maintainers have asked, both in PR #268 and in the assistant flow management direction shipped in PR #292, for explicit, inspectable, manageable flow control instead of implicit background lifecycle.

Solution

The RFC follows the maintainer's relocated proposal pattern at examples/proposals/<topic>.md (commit 47de4e4 moved backend/docs/evidence_chain_rfc.md to examples/proposals/evidence_chain.md). It deliberately mirrors the structure of examples/proposals/evidence_chain.md (PR #277) and includes:

• Goals limited to capping concurrent flows, making queued flows first-class, replacing external polling with at-least-once webhooks, and preserving the existing createFlow contract.
• Non-Goals that forbid hidden in-memory queues, multi-tenant scheduling/SLA, a generic event bus, and any change to what finished means for tasks/subtasks/toolcalls.
• Design Principles centered on persistence, visibility, manageability, explicit promotion, clear finished semantics, and at-least-once delivery.
• Proposed concurrency model: a new persisted queued status, a single MAX_CONCURRENT_FLOWS env var, an explicit promoter, full UI/API visibility (listings, filters, detail, assistant view), and a user-driven cancel path with a documented terminal status.
• Proposed completion webhook model: per-flow webhookUrl plus a global FLOW_WEBHOOK_URL fallback, HMAC-SHA256 signed payloads via X-PentAGI-Signature, stable delivery_id for receiver dedup, persisted flow_webhook_deliveries rows, bounded exponential backoff, and SSRF mitigations (config-time and delivery-time DNS validation, redacted secrets in logs, no retry on most 4xx).
• Storage and API surface sketches that intentionally do not commit to a final migration shape.
• Open Questions covering per-user limits, createFlow blocking semantics, alignment of the signature scheme with the issue RFC: Cryptographic evidence chain for PentAGI pentest operations #235 receipt direction, and queued-flow handling for resources/uploads/messages.
• Suggested First Milestone that lands the queue end-to-end before webhooks, so each lifecycle change can be reviewed in isolation and PR sizes stay small.

The RFC explicitly rejects in-memory queues, hidden lifecycle state, and implicit promotion semantics, and references the PR #268 lesson directly.

User Impact

• Maintainers and contributors get a written design surface to push back on before any code lands. The RFC is opinionated about what is out of scope so a future implementation PR cannot quietly grow into hidden lifecycle work.
• Operators evaluating PentAGI for batch pipelines see the intended direction (a single MAX_CONCURRENT_FLOWS knob plus signed completion webhooks) and the staged milestones, so they can plan integration work even before any feature ships.
• External orchestrators learn what to expect long-term: a queueable flow status, signed webhooks with a documented payload, and idempotent delivery semantics, instead of building yet another polling loop.

Test Plan

• git diff --stat shows only examples/proposals/flow_concurrency.md changed.
• Markdown renders cleanly (no broken anchor, list, table, or code-fence syntax).
• Section structure mirrors examples/proposals/evidence_chain.md (Goals / Non-Goals / Proposed model / Open Questions / Suggested First Milestone) and adds an explicit Safety and Security section.
• No code, schema, GraphQL, REST, or UI changes.

Refs #298

[Copilot]

Pull request overview

Adds an RFC proposal documenting a persisted flow queue (with a global concurrency cap) and at-least-once completion webhooks, providing a design direction for addressing orchestration pain points raised in #298 without introducing runtime changes.

Changes:

• Introduces a new design RFC covering a persisted queued flow status and explicit promotion model.
• Proposes a completion webhook design (configuration, payload, signing, retries, SSRF mitigations).
• Captures open questions and a staged milestone plan for implementation.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.