help: worker hangs while downloading many files (multi GB) in deadline cloud CMF #546
Description
Describe the bug
Using this simple code to download data from S3 during single step (deadline job bundle)
command = [
"aws", "s3", "cp",
s3_path,
target_dir, "--recursive"
]
try:
print_log(f"Downloading data from s3 {s3_path} to {target_dir}")
result = subprocess.run(
command,
check=True,
shell=True,
capture_output=True,
text=True
)
print_log(f"Command output:\n{result.stdout}")
if result.stderr:
print_log(f"Command errors:\n{result.stderr}")
except subprocess.CalledProcessError as e:
print_log(f"Error running command: {e}")
print_log(f"Command output:\n{e.stdout}")
print_log(f"Command errors:\n{e.stderr}")
return False
except FileNotFoundError as fne:
print_log(f"Failed to download data from s3. Check the path: {s3_path}")
print_log(f"Error: {fne}")
return False
Command execution hangs at:
2025/02/13 16:45:14+00:00 Completed 1.1 GiB/1.1 GiB (100.5 MiB/s) with 33 file(s) remaining
2025/02/13 16:45:14+00:00 Completed 1.1 GiB/1.1 GiB (100.5 MiB/s) with 33 file(s) remaining
Expected Behaviour
Command finishes successfully same command but when run from Powershell without deadline cloud worker context simply doesn't hang.
Current Behaviour
The following command hangs when run in under deadline cloud worker context (python script as step in deadline job bundle)
aws s3 cp <A> <B> --recursive
Command execution hangs:
2025/02/13 16:45:14+00:00 Completed 1.1 GiB/1.1 GiB (100.6 MiB/s) with 33 file(s) remaining
2025/02/13 16:45:14+00:00 Completed 1.1 GiB/1.1 GiB (100.5 MiB/s) with 33 file(s) remaining
2025/02/13 16:45:14+00:00 Completed 1.1 GiB/1.1 GiB (100.5 MiB/s) with 33 file(s) remaining
Reproduction Steps
aws s3 cp <A> <B> --recursive + use s3 bucket with at least 2GB of data in it + many many small files (like 1000 or 2000 files).
Environment
At minimum:
- Operating system - Windows
- Python
- Latest version of deadline cloud worker (python library)
Job bundle template:
steps:
- name: Render Scene
script:
actions:
onRun:
command: python
args: ['{{Param.LauncherFile}}']