Repository navigation
feat: port apify-client-js content encoding, log bytes and dataset stream to the Rust client - #30
Merged
Merged
Conversation
…ient Ports apify-client-js #1140 (setRecord contentEncoding), #1141 (log.get buffer), and #1152 (downloadItems stream, superseding #1139's streamItems). No OpenAPI spec change. Factors the raw-streaming-GET logic LogClient::stream already had into a shared base::get_byte_stream helper, now reused by DatasetClient::download_items_stream. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G3FfQf9r45KVCa5VAYAjkd
Addresses review iteration 1: trims the ApifyClientError and get_byte_stream rustdoc to drop cross-client narration/caller-list padding, and wraps get_byte_stream's send() in a DEFAULT_REQUEST_TIMEOUT response-start timeout so a dead connection surfaces ApifyClientError::Timeout instead of hanging forever. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G3FfQf9r45KVCa5VAYAjkd
Adds HttpBackend::send_streaming (default: buffer via send(), so existing backends keep compiling) and HttpClient::call_streaming, which applies auth/user-agent like call() and bounds the response-start wait plus per-chunk inactivity without capping total stream duration. base::get_byte_stream now delegates to it instead of opening its own unconfigured reqwest::Client, so log().stream() and download_items_stream() honor a custom or proxy/TLS-configured backend like every buffered endpoint. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G3FfQf9r45KVCa5VAYAjkd
call_streaming now uses the caller-supplied request.timeout (the per-endpoint base every buffered call also uses) instead of the client-wide retry budget, so the field base.rs populates is no longer silently ignored; the reuse as both response-start and per-chunk inactivity window is documented as matching the reference clients' single-timeoutSecs streaming semantics. with_inactivity_timeout now ends the stream after one Err(Timeout) instead of retrying the stalled read forever. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G3FfQf9r45KVCa5VAYAjkd
LogClient::stream_with_options and DatasetClient::download_items_stream still described streaming as going "through reqwest directly" / "bypassing the buffered backend", which was true before commit 7749f74 routed it through the configured HttpBackend. Updated both comments to describe the current behavior. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G3FfQf9r45KVCa5VAYAjkd
Both were pub(crate) but, after this branch's earlier streaming refactor consolidated their only call sites into http_client.rs itself, had no callers outside the module. Dead cross-module reach; no behavior change. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G3FfQf9r45KVCa5VAYAjkd
LogClient::stream and DatasetClient::download_items_stream promised unconditional completion/no corruption without mentioning the 6-minute per-chunk inactivity timeout call_streaming now applies, contradicting documented behavior for a quiet-but-still-running stream. Also softened the call_streaming budget-reuse comment, which claimed closer mechanism-level parity with the JS reference's axios/Node socket-level idle timeout than is accurate (same outcome, different mechanism). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G3FfQf9r45KVCa5VAYAjkd
README.md (x2) and docs/README.md pinned an install version from long before the crate's current 0.13.0, incompatible with every feature those same pages document (streaming, get_bytes, set_record_raw_with_content_encoding, wait_for_resources, HttpBackend, ...). Bumped all three to "0.13". Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G3FfQf9r45KVCa5VAYAjkd
Pijukatel
added
adhoc
Claude
apify-service-account
added
the
tested
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Ports the Rust-relevant features from the reference
apify-client-jsbatch (apify/apify-client-js#1140, #1141, #1152, #1139, #1145). No OpenAPI spec change: the client stays onv2-2026-10-08T073338Z, which matches the live spec.KeyValueStoreClient::set_record_raw_with_content_encodinguploads a value that is already compressed, and the client skips its own compression (apify-client-js#1140).LogClient::get_bytes/get_bytes_with_optionsreturn the raw log bytes (apify-client-js#1141).DatasetClient::download_items_streamstreams a dataset export with constant memory use (apify-client-js#1152, which supersedes #1139).log().stream()anddownload_items_stream()) now go through the configuredHttpBackendvia a new default method,HttpBackend::send_streaming, which doesn't break existing backends. Before this, they used an unconfiguredreqwest::Client. They also get a response-start timeout and a per-chunk inactivity timeout, so a stalled stream ends instead of hanging."0.6"install-version pins inREADME.mdanddocs/are corrected.