Skip to content

fix(responses): preserve tool images and bound inline media - #3365

Merged
lbjlaq merged 2 commits into
lbjlaq:mainfrom
buluw:codex/responses-media-bounds
Aug 30, 2026
Merged

fix(responses): preserve tool images and bound inline media#3365
lbjlaq merged 2 commits into
lbjlaq:mainfrom
buluw:codex/responses-media-bounds

Conversation

@buluw

@buluw buluw commented Aug 30, 2026

Copy link
Copy Markdown
Contributor

Summary

Preserve images returned by Responses tool outputs as multimodal model input while bounding inline image memory and removing historical inline media before replay or session caching.

Closes #3354.
Closes #3355.

Depends on #3362. This is a stacked PR; its new behavior is contained in one independent commit and the displayed diff will narrow after the prerequisite PR merges.

Changes

  • Parse text and image parts from string, array, typed-object, and { "content": ... } tool outputs.
  • Keep text-only tool output as a string, but emit mixed text/media as OpenAI content blocks so the existing mapper produces Gemini inlineData parts.
  • Apply the same tool-output conversion to HTTP Responses and the WebSocket compatibility path.
  • Limit Responses inputs to 16 data-URL images, 20 MiB decoded per image, and 32 MiB decoded in aggregate.
  • Measure decoded base64 size without allocating a second decoded image buffer.
  • Remove inline image/audio data from cached request history, completed output, debug snapshots, and stored HTTP session input while retaining small placeholders.
  • Preserve media in the latest user turn, but replace media before that turn before enforcing the current request limits.
  • Move owned JSON values through conversion and history paths to avoid repeatedly cloning large base64 payloads.

Files

  • src-tauri/src/proxy/handlers/openai.rs
  • src-tauri/src/proxy/http_session_store.rs
  • src-tauri/src/proxy/mappers/openai/request.rs
  • src-tauri/src/proxy/adapters/apply_patch_preflight.rs

Tests

  • cargo test --manifest-path src-tauri/Cargo.toml --offline --lib proxy::handlers::openai::stream_peek_tests
  • Result: 19 passed, 0 failed.
  • Coverage includes tool-output images reaching Gemini inlineData, text-only compatibility, pure-image history placeholders, case-insensitive data-URL metadata, per-image and aggregate byte limits, 16-image limits, and pruning historical images before validating the latest user turn.
  • git diff --check passed.

Compatibility

  • Text-only Responses tool outputs retain their existing string representation.
  • Current-turn images remain available to the model; only older inline media is replaced by a small textual placeholder.
  • The hard byte and count limits apply to inline data: image URLs. Remote image URL behavior is unchanged and is not claimed as directly tested here.
  • Existing HTTP session storage remains compatible; the session graph and response-ID semantics are intentionally left to a follow-up PR.
  • This PR does not change image generation account scheduling, retry policy, or HTTP disconnect handling.

@lbjlaq
lbjlaq merged commit 6529f4d into lbjlaq:main Aug 30, 2026
4 of 7 checks passed
@buluw
buluw deleted the codex/responses-media-bounds branch August 31, 2026 03:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Bound Responses image inputs and discard historical inline media [Bug] Preserve images returned by Responses tool outputs as multimodal input

2 participants