feat: add multimodal collection for qoder - #257
Conversation
ralf0131
left a comment
There was a problem hiding this comment.
Summary
This PR adds multimodal media collection for Qoder IDE and CLI, reusing the existing src/multimodal stack (Processor, Uploader, OSS/SLS, retry, queue backpressure). The implementation is clean, well-structured, and follows established patterns.
Key observations:
- IDE multimodal (
qoder-ide-multimodal.ts): Correctly walks session entries, converts local image paths withpathToUri, handles input/tool/output surfaces with properuploadModegating. LRU cache avoids duplicate uploads. - CLI multimodal (
qoder-cli-multimodal.ts): Parses[Image: source:]paste and@pathpatterns, handles tool result images. No output surface (correct for CLI). - SQLite reader (
sqlite-token-reader.ts): Batch query with parameterized SQL, fail-open, proper JSON parsing ofextra.attachedImagePathswith context fallback. - Hook processor (
qoder-hook-processor.mjs):isRealUserPrompt()correctly addsisMetafiltering to exclude paste-source rows from turn boundaries.buildUserMessageParts()attaches local image URIs to CLI user prompts. - Orchestrator wiring: Clean integration with existing multimodal infrastructure, proper config gating.
Design strengths:
- Fail-open throughout — missing files, oversized images, and
pathToUrithrows do not drop original events - Process-local LRU + sha256 cache avoid repeat disk reads and duplicate uploads
uploadModesurfaces (input/tool/output/both/none) provide fine-grained control- Remote URLs are not used as upload sources — only local
pathToUri
LGTM ✅
Automated review by github-manager-bot
ralf0131
left a comment
There was a problem hiding this comment.
Summary
Re-review after new commit f5ebc6f — adds remote/multi-profile SQLite path lookup for Qoder. Clean refactoring: extracts QODER_DB_TAIL constant, adds listHashedProfileDbPaths() for hashed profile directories, and deduplicates candidates via Set.
LGTM — extends path resolution correctly for Linux remote and multi-profile scenarios.
Automated review by github-manager-bot
ralf0131
left a comment
There was a problem hiding this comment.
Summary
Re-review after new commit — adds remote/multi-profile SQLite path lookup for Qoder. Clean refactoring with hashed profile directory scanning and Set-based deduplication.
LGTM — extends path resolution correctly for Linux remote and multi-profile scenarios.
Automated review by github-manager-bot
Summary
This PR adds multimodal media collection for Qoder IDE and Qoder CLI to Pilot. Local image paths are read at write time, converted to storage
uriparts via the sharedMultimodalProcessor.pathToUripath, uploaded asynchronously to OSS or SLS PutObject, and attached as structured GenAI metadata without changing Pilot’s existing event wire contract.src/multimodalstack (Processor, Uploader, OSS/SLS, retry, queue backpressure). Qoder does not add a new output surface or storage protocol.attachedImagePaths(input),Image file:/ ImageGen paths (tool), andin assistant text (output).[Image: source: <path>]and@path(input);Read image:/Image file:/ ImageGen paths (tool). No output surface — CLI assistant text does not embed images.isMetapaste-source rows on the same turn as the real user prompt so CLI local paths reachmessages_delta.pathToUrithrows do not drop the original text events; skips are logged.uploadModegates, part cap, fail-open, and hook turn folding.What changed
Qoder IDE (
qoder/qoder-idea)enrichIdeMultimodalwalks the session and converts local image paths withpathToUri.uploadModesurfaces:input— SQLitechat_record.extra.attachedImagePaths(and context images) onto the matchingllm.request/ usermessages_delta;tool—tool.resulttextImage file: <path>and ImageGenThe absolute path of the image is: <path>rewritten to text +uriparts;output—llm.responsemarkdownwith an image extension;both— all of the above;noneskips.Qoder CLI (
qoder-cli)qoder-tracecollect path, after CLI token enrich. No SQLite. Config key remainsagents.qoder.multimodal.uploadModesurfaces:input—[Image: source: <path>](strip, original …metadata) and@path(relative paths joinagent.qoder.cwd);tool—Read image: <path>,Image file: <path>, ImageGen absolute path;output— unused (both= input + tool).pathToUriis.picture/pipeline.jpg这个是什么?(no@) is not parsed as a path.otherandllm.requestthat both carry the source text each get auripart (upload still deduped by cache/sha256).MAX_MULTIMODAL_PARTS(10).Hook processor (CLI paste path)
[Image #0]…+ remote image block) and anisMetasidecar[Image: source: <local path>].isRealUserPromptistype === user && !tool_result && !isMeta.isMetano longer starts a new turn (same as tool results).userRowselection (fallback to the old user-and-not-tool-result find).buildUserMessageParts(CLI only) keeps the primary prompt and appends other same-turn user texts (the Image:source sidecar) ontoother/llm.requestmessages_delta. IDE parts stay a single primary text.Wiring
QoderTraceInput.collect: IDE enrich per session; CLI enrich per turn; both injectpathToUrifrom the process-scoped processor when infra +agents.qoder.multimodal.uploadModeenable it.codex,qoder(MULTIMODAL_SUPPORTED_AGENT_IDS).qoder-cli/qoder-idearesolve toqodervia the existing agent alias map.Implementation overview
Configuration split (unchanged from the Codex multimodal PR):
config.multimodal(uploader, credentials,storageBasePathfor OSS; SLS derivessls://{project}/{logstore}).agents.qoder.multimodal.uploadMode(nonedefault;input/tool/output/both).Quality results
Automated checks
tests/unit/inputs/qoder-cli-multimodal.test.ts— paste /@extract, tool prefixes, glob skip,uploadModegates, part cap, fail-open, both carriers get uritests/unit/inputs/qoder-trace-input.test.ts— IDE + CLI collect convert, Image file / ImageGen, part captests/unit/hooks/qoder-hook-processor-retry.test.mjs— CLI Image:source appended on the primary prompt; isMeta does not start a new turn; IDE does not append Image:source parts