Skip to content

feat: add multimodal collection for qoder - #257

Open
xiongyunn wants to merge 9 commits into
alibaba:mainfrom
xiongyunn:xy/multimodal_qoder
Open

feat: add multimodal collection for qoder#257
xiongyunn wants to merge 9 commits into
alibaba:mainfrom
xiongyunn:xy/multimodal_qoder

Conversation

@xiongyunn

Copy link
Copy Markdown
Contributor

Summary

This PR adds multimodal media collection for Qoder IDE and Qoder CLI to Pilot. Local image paths are read at write time, converted to storage uri parts via the shared MultimodalProcessor.pathToUri path, uploaded asynchronously to OSS or SLS PutObject, and attached as structured GenAI metadata without changing Pilot’s existing event wire contract.

  • Reuses the existing src/multimodal stack (Processor, Uploader, OSS/SLS, retry, queue backpressure). Qoder does not add a new output surface or storage protocol.
  • Qoder IDE: SQLite attachedImagePaths (input), Image file: / ImageGen paths (tool), and ![...](path) in assistant text (output).
  • Qoder CLI: paste [Image: source: <path>] and @path (input); Read image: / Image file: / ImageGen paths (tool). No output surface — CLI assistant text does not embed images.
  • Hook processor keeps isMeta paste-source rows on the same turn as the real user prompt so CLI local paths reach messages_delta.
  • Fail-open: missing files, oversized images, and pathToUri throws do not drop the original text events; skips are logged.
  • Adds unit coverage for IDE/CLI extractors, uploadMode gates, part cap, fail-open, and hook turn folding.
  • Updates English and Chinese multimodal / agents docs for Qoder IDE + CLI coverage.

What changed

Qoder IDE (qoder / qoder-idea)

  • After IDE token enrich, enrichIdeMultimodal walks the session and converts local image paths with pathToUri.
  • uploadMode surfaces:
    • input — SQLite chat_record.extra.attachedImagePaths (and context images) onto the matching llm.request / user messages_delta;
    • tooltool.result text Image file: <path> and ImageGen The absolute path of the image is: <path> rewritten to text + uri parts;
    • outputllm.response markdown ![...](path) with an image extension;
    • both — all of the above; none skips.
  • Glob listings that were never visually read are not collected.
  • Process-local request_id → attached-path LRU plus processor path/sha256 cache avoid repeat disk reads and duplicate uploads.

Qoder CLI (qoder-cli)

  • Same qoder-trace collect path, after CLI token enrich. No SQLite. Config key remains agents.qoder.multimodal.
  • uploadMode surfaces:
    • input[Image: source: <path>] (strip , original … metadata) and @path (relative paths join agent.qoder.cwd);
    • toolRead image: <path>, Image file: <path>, ImageGen absolute path;
    • output — unused (both = input + tool).
  • Remote OSS / presigned URLs in transcript image blocks are not used as the upload source; local pathToUri is.
  • Prompt text like picture/pipeline.jpg这个是什么? (no @) is not parsed as a path.
  • Per-path convert is unique’d at extract time; other and llm.request that both carry the source text each get a uri part (upload still deduped by cache/sha256).
  • Caps converted images per carrier at MAX_MULTIMODAL_PARTS (10).

Hook processor (CLI paste path)

  • Transcript paste is two user rows: the human prompt ([Image #0]… + remote image block) and an isMeta sidecar [Image: source: <local path>].
  • isRealUserPrompt is type === user && !tool_result && !isMeta. isMeta no longer starts a new turn (same as tool results).
  • Used for turn split, incremental/triggered windows, and userRow selection (fallback to the old user-and-not-tool-result find).
  • buildUserMessageParts (CLI only) keeps the primary prompt and appends other same-turn user texts (the Image:source sidecar) onto other / llm.request messages_delta. IDE parts stay a single primary text.

Wiring

  • QoderTraceInput.collect: IDE enrich per session; CLI enrich per turn; both inject pathToUri from the process-scoped processor when infra + agents.qoder.multimodal.uploadMode enable it.
  • Supported agent ids: codex, qoder (MULTIMODAL_SUPPORTED_AGENT_IDS). qoder-cli / qoder-idea resolve to qoder via the existing agent alias map.

Implementation overview

Qoder transcript / hook history
  -> hook processor (CLI: fold isMeta Image:source into the same turn parts)
  -> QoderTraceInput.collect
       IDE: enrichIdeMultimodal (SQLite attach + tool/output paths)
       CLI: enrichCliMultimodal (Image:source / @path + Read/ImageGen)
  -> MultimodalProcessor.pathToUri
       stat / read bytes / hash / joinStorageUri
       enqueue Uploader.upload (async)
  -> message / tool.result parts: text + uri (no base64 in output)
  -> optional gen_ai.input.multimodal_metadata from uri parts
  -> Pilot MultiFlusher emits the existing wire protocol

Configuration split (unchanged from the Codex multimodal PR):

  • Global infra: config.multimodal (uploader, credentials, storageBasePath for OSS; SLS derives sls://{project}/{logstore}).
  • Per-agent policy: agents.qoder.multimodal.uploadMode (none default; input / tool / output / both).

Quality results

Automated checks

  • tests/unit/inputs/qoder-cli-multimodal.test.ts — paste / @ extract, tool prefixes, glob skip, uploadMode gates, part cap, fail-open, both carriers get uri
  • tests/unit/inputs/qoder-trace-input.test.ts — IDE + CLI collect convert, Image file / ImageGen, part cap
  • tests/unit/hooks/qoder-hook-processor-retry.test.mjs — CLI Image:source appended on the primary prompt; isMeta does not start a new turn; IDE does not append Image:source parts

@ralf0131 ralf0131 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

This PR adds multimodal media collection for Qoder IDE and CLI, reusing the existing src/multimodal stack (Processor, Uploader, OSS/SLS, retry, queue backpressure). The implementation is clean, well-structured, and follows established patterns.

Key observations:

  • IDE multimodal (qoder-ide-multimodal.ts): Correctly walks session entries, converts local image paths with pathToUri, handles input/tool/output surfaces with proper uploadMode gating. LRU cache avoids duplicate uploads.
  • CLI multimodal (qoder-cli-multimodal.ts): Parses [Image: source:] paste and @path patterns, handles tool result images. No output surface (correct for CLI).
  • SQLite reader (sqlite-token-reader.ts): Batch query with parameterized SQL, fail-open, proper JSON parsing of extra.attachedImagePaths with context fallback.
  • Hook processor (qoder-hook-processor.mjs): isRealUserPrompt() correctly adds isMeta filtering to exclude paste-source rows from turn boundaries. buildUserMessageParts() attaches local image URIs to CLI user prompts.
  • Orchestrator wiring: Clean integration with existing multimodal infrastructure, proper config gating.

Design strengths:

  • Fail-open throughout — missing files, oversized images, and pathToUri throws do not drop original events
  • Process-local LRU + sha256 cache avoid repeat disk reads and duplicate uploads
  • uploadMode surfaces (input/tool/output/both/none) provide fine-grained control
  • Remote URLs are not used as upload sources — only local pathToUri

LGTM ✅


Automated review by github-manager-bot

@xiongyunn xiongyunn changed the title (feat) add multimodal collection for qoder feat: add multimodal collection for qoder Aug 14, 2026

@ralf0131 ralf0131 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

Re-review after new commit f5ebc6f — adds remote/multi-profile SQLite path lookup for Qoder. Clean refactoring: extracts QODER_DB_TAIL constant, adds listHashedProfileDbPaths() for hashed profile directories, and deduplicates candidates via Set.

LGTM — extends path resolution correctly for Linux remote and multi-profile scenarios.


Automated review by github-manager-bot

@ralf0131 ralf0131 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

Re-review after new commit — adds remote/multi-profile SQLite path lookup for Qoder. Clean refactoring with hashed profile directory scanning and Set-based deduplication.

LGTM — extends path resolution correctly for Linux remote and multi-profile scenarios.


Automated review by github-manager-bot

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants