Skip to content

Commit eee98af

Browse files
Document the onerror workaround with its removal condition
fsspec's AbstractFileSystem.mv() passed the typo'd "onerror" keyword (instead of "on_error", which copy() consumes) until it was fixed in fsspec 2026.6.0, so it leaks through copy(**kwargs) into cp_file on older releases and must not reach the S3 API. Replace the stale TODO with the actual mechanism and the fsspec version at which the pop can be removed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1 parent 1732653 commit eee98af

2 files changed

Lines changed: 7 additions & 3 deletions

File tree

pyathena/filesystem/s3.py

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1011,10 +1011,12 @@ def cp_file(
10111011
to optimize performance for large files. The copy operation is
10121012
performed entirely on the S3 service without data transfer.
10131013
"""
1014-
# TODO: Delete the value that seems to be a typo, onerror=false.
1014+
# fsspec < 2026.6.0: AbstractFileSystem.mv() passed the typo'd
1015+
# "onerror" keyword (instead of "on_error", which copy() consumes),
1016+
# so it leaked through copy(**kwargs) into cp_file and must not
1017+
# reach the S3 API. Remove this once the fsspec requirement is
1018+
# >= 2026.6.0, where mv() passes on_error correctly.
10151019
# https://github.com/fsspec/filesystem_spec/commit/346a589fef9308550ffa3d0d510f2db67281bb05
1016-
# https://github.com/fsspec/filesystem_spec/blob/2024.10.0/fsspec/spec.py#L1185
1017-
# https://github.com/fsspec/filesystem_spec/blob/2024.10.0/fsspec/spec.py#L1077
10181020
kwargs.pop("onerror", None)
10191021
bucket1, key1, version_id1 = self.parse_path(path1)
10201022
bucket2, key2, version_id2 = self.parse_path(path2)

pyathena/filesystem/s3_async.py

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -191,6 +191,8 @@ async def _delete_chunk(chunk: list[dict[str, Any]]) -> None:
191191

192192
async def _cp_file(self, path1: str, path2: str, **kwargs) -> None:
193193
"""Copy an S3 object, using async parallel multipart upload for large files."""
194+
# fsspec < 2026.6.0 leaks the typo'd "onerror" keyword from mv();
195+
# see S3FileSystem.cp_file.
194196
kwargs.pop("onerror", None)
195197
bucket1, key1, version_id1 = self.parse_path(path1)
196198
bucket2, key2, version_id2 = self.parse_path(path2)

0 commit comments

Comments
 (0)