Skip to content

Conversation

hssyoo
Copy link
Contributor

@hssyoo hssyoo commented Aug 27, 2025

This PR exposes an existing S3 transfer config io_chunksize so that it's configurable via the shared config file.

Copied from doc update in the PR:

When a GET request is called for downloads, the response contains a file-like
object that streams data fetched from S3. Chunks are read from the stream and
queued in-memory for writes. io_chunksize configures the maximum size of
elements in the IO queue. This value can be specified using the same semantics
as multipart_threshold, that is either as the number of bytes as an
integer, or using a size suffix.
Increasing this value may result in higher overall throughput by preventing
blocking in cases where large objects are downloaded in environments where
network speed exceeds disk write speed.

@hssyoo hssyoo force-pushed the expose-io-chunksize branch from 457f3ce to efc1154 Compare August 27, 2025 15:02
@hssyoo hssyoo marked this pull request as ready for review August 27, 2025 17:21
@kdaily
Copy link
Member

kdaily commented Aug 28, 2025

Question: should this get added to the TransferConfig class as well?

class TransferConfig:

I ask because I don't think sending it through kwargs is going to persist it on the object created here:

https://github.com/aws/aws-cli/pull/9678/files#diff-280c24f20aa46551fbfd53f1910ece8ac396fae5de0956cf3555015592d45161R217

@hssyoo
Copy link
Contributor Author

hssyoo commented Sep 9, 2025

Question: should this get added to the TransferConfig class as well?

class TransferConfig:

I ask because I don't think sending it through kwargs is going to persist it on the object created here:

https://github.com/aws/aws-cli/pull/9678/files#diff-280c24f20aa46551fbfd53f1910ece8ac396fae5de0956cf3555015592d45161R217

The referenced code is effectively dead. It's only used for a GameLift customization [source]. The TransferConfig object used everywhere else is here:

class TransferConfig:

Are you actually seeing the value not being persisted when configuring io_chunksize?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants