-
Notifications
You must be signed in to change notification settings - Fork 1.2k
change: When rootlessDocker is enabled, return a fixed SageMaker IP #5236
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 5 commits
5a00e91
d16da72
06ae774
fb69aaf
b1fd26e
6c1f8e6
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -153,14 +153,36 @@ def get_child_process_ids(pid): | |
def get_docker_host(): | ||
"""Discover remote docker host address (if applicable) or use "localhost" | ||
|
||
Use "docker context inspect" to read current docker host endpoint url, | ||
When rootlessDocker is enabled (Cgroup Driver: none), use fixed SageMaker IP. | ||
Otherwise, Use "docker context inspect" to read current docker host endpoint url, | ||
url must start with "tcp://" | ||
|
||
Args: | ||
|
||
Returns: | ||
docker_host (str): Docker host DNS or IP address | ||
""" | ||
# Check if using SageMaker rootless Docker by examining storage driver | ||
try: | ||
cmd = ["docker", "info"] | ||
process = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE) | ||
output, err = process.communicate() | ||
if process.returncode == 0: # Check return code instead of stderr | ||
output_text = output.decode("utf-8") | ||
# Check for rootless Docker by looking at Cgroup Driver | ||
if "Cgroup Driver: none" in output_text: | ||
# log the result of check | ||
logger.warning("RootlessDocker detected (Cgroup Driver: none), returning fixed IP.") | ||
# SageMaker rootless Docker detected - return fixed IP | ||
return "172.17.0.1" | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Is this a general IP address used throughout docker ? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The IP is from the LLC document. This is supposed to be configured by SageMaker. Sravya told me that we will use this specific IP. |
||
else: | ||
logger.warning( | ||
"RootlessDocker not detected, falling back to remote host IP or localhost." | ||
) | ||
except subprocess.SubprocessError as e: | ||
logger.warning("Failed to run 'docker info' command when checking rootlessDocker: %s.", e) | ||
|
||
# Fallback to existing logic for remote Docker hosts | ||
cmd = "docker context inspect".split() | ||
process = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE) | ||
output, err = process.communicate() | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What is this value for rootful docker?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For rootful docker right now it is
cgroupfs
.