Skip to content

Commit 492e9fb

Browse files
ivan-khvostishkovivan-khvostishkov
committed
#67 - Updating to new SageMaker Studio and deprecating Studio Classic
1 parent ec0c80b commit 492e9fb

24 files changed

+251
-54
lines changed

‎.gitignore

+2
Original file line numberDiff line numberDiff line change
@@ -25,3 +25,5 @@ __pycache__
2525
/tests/.sm-ssh-owner
2626
/tests/.pymon
2727
/.pymon
28+
/workspace.xml
29+
/shelf/Changes/shelved.patch

‎IAM_SSM_Setup.md

+1-1
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@ SageMaker SSH Helper relies on the AWS Systems Manager service to create SSH tun
1212

1313
### Automated setup with CDK and Cloud9
1414

15-
a. From AWS Console, pop up [CloudShell](https://aws.amazon.com/cloudshell/) environment (use the button located at the bottom left corner of the browser window). Alternatively, you can the commands run in your local terminal. In this case, make sure you've installed Node.js and CDK and fulfilled [all other CDK prerequisites](https://docs.aws.amazon.com/cdk/v2/guide/getting_started.html#getting_started_prerequisites). In both cases you need to have an admin role.
15+
a. From AWS Console, pop up [CloudShell](https://aws.amazon.com/cloudshell/) environment. Alternatively, you can the commands run in your local terminal. In this case, make sure you've installed Node.js and CDK and fulfilled [all other CDK prerequisites](https://docs.aws.amazon.com/cdk/v2/guide/getting_started.html#getting_started_prerequisites). In both cases you need to have an admin role.
1616

1717
b. Define your SageMaker role, local user role, AWS account ID and AWS Region as variables by executing the following commands in the terminal line by line:
1818

‎README.md

+23-8
Original file line numberDiff line numberDiff line change
@@ -90,7 +90,6 @@ Install the latest stable version of library from the [PyPI repository](https://
9090
```shell
9191
pip install sagemaker-ssh-helper
9292
```
93-
9493
**Caution:** It's always recommended to install the library into a Python venv, not into the system env. If you want to use later the SSH plugins of your IDE that will use the system env and system Python, you should add the venv into the system PATH, as described in the section [Remote code execution with PyCharm / VSCode over SSH](#remote-interpreter).
9594

9695
If you're working on Windows, see [FAQ](FAQ.md#is-windows-supported).
@@ -552,12 +551,21 @@ sm-ssh connect ssh-training-example-2023-07-25-03-18-04-490.training.sagemaker -
552551
Alternatively, instead of using `sm-ssh connect` command, you can use the native `ssh` command, but it will require you to update your [ssh config](https://linux.die.net/man/5/ssh_config), typically `~/.ssh/config`, with `sm-ssh start-proxy` command as follows:
553552

554553
```bash
554+
Host *.studio.sagemaker
555+
IdentityFile ~/.ssh/%h
556+
PasswordAuthentication no
557+
ConnectTimeout 120
558+
ServerAliveInterval 15
559+
ServerAliveCountMax 8
560+
ProxyCommand sm-ssh start-proxy %h
561+
User sagemaker-user
562+
555563
Host *.*.sagemaker
556564
IdentityFile ~/.ssh/%h
557565
PasswordAuthentication no
558-
ConnectTimeout 90
566+
ConnectTimeout 120
559567
ServerAliveInterval 15
560-
ServerAliveCountMax 6
568+
ServerAliveCountMax 8
561569
ProxyCommand sm-ssh start-proxy %h
562570
User root
563571
```
@@ -615,7 +623,8 @@ ssh sh-training-manual-2023-10-02-14-38-56-744.training.sagemaker
615623

616624
A. Follow the [instructions in the PyCharm docs](https://www.jetbrains.com/help/pycharm/remote-debugging-with-product.html#remote-interpreter), to configure the remote interpreter in PyCharm.
617625

618-
In the field for host name, put the same value as for `fqdn` in the [`sm-ssh` command](#sm-ssh), e.g., `ssh-training-manual-2023-10-02-14-38-56-744.training.sagemaker`, and use `root` as the username.
626+
In the field for host name, put the same value as for `fqdn` in the [`sm-ssh` command](#sm-ssh), e.g., `ssh-training-manual-2023-10-02-14-38-56-744.training.sagemaker`, and use `root` as the username.
627+
For SageMaker Studio you might want to use `sagemaker-user` instead of `root`.
619628

620629
![](images/pycharm_training.png)
621630

@@ -625,16 +634,16 @@ If PyCharm asks for the SSH key, point to the `~/.ssh/<fqdn>` private key file t
625634

626635
*Note:* If PyCharm says connection refused, it can be due to timeout. Check that you can connect to this host from your system terminal with `ssh` and `sm-ssh` and try configuring the remote interpreter again.
627636

628-
*Tip:* When you configure Python interpreter in PyCharm, it's recommended to configure [the deployment path mapping](https://www.jetbrains.com/help/pycharm/creating-local-server-configuration.html#mapping) for you project to point into `/root/project_name` instead of default `/tmp/pycharm_project_123`. This is how you will be able to see your project in SageMaker Studio and PyCharm will automatically sync your local dir to the remote dir.
629-
630-
*Tip:* Also instead of creating a new venv, point the Python interpreter to the existing location.
637+
*Tip:* Instead of creating a new venv, point the Python interpreter to the existing location.
631638
You can find this location by running a cell with `import sys; sys.executable` command in a SageMaker Studio notebook. You will get something like `/opt/conda/bin/python`.
632639

633640
*Tip:* Now you also can [upload and download files from remote](https://www.jetbrains.com/help/pycharm/uploading-and-downloading-files.html) and [synchronize files with remote](https://www.jetbrains.com/help/pycharm/comparing-deployed-files-and-folders-with-their-local-versions.html).
634641

635642
B. Follow the [instructions for VSCode](https://code.visualstudio.com/docs/remote/ssh), to configure local Visual Studio Code app
636643

637-
Put the `root@fqdn` as the hostname to connect to, e.g., `root@ssh-training-example-2023-07-25-03-18-04-490.training.sagemaker` .
644+
Put the `root@fqdn` as the hostname to connect to, e.g., `root@ssh-training-example-2023-07-25-03-18-04-490.training.sagemaker` .
645+
For SageMaker Studio you might want to use `sagemaker-user` instead of `root`,
646+
e.g. `[email protected]`.
638647

639648
![](images/vscode_training.png)
640649

@@ -787,15 +796,21 @@ You might want to change the `LOCAL_USER_ID` variable upon the first run, to pre
787796
2. Configure remote interpreter in PyCharm / VS Code to connect to SageMaker Studio
788797

789798
Use `app_name.app_space_name.domain_id.studio.sagemaker` or `app_name.studio.sagemaker` as the `fqdn` to connect.
799+
Use `sagemaker-user` as user.
800+
See more details on how to configure the interpreter for the IDE in the [Remote code execution](#remote-interpreter) section,
801+
step "3. Configure the remote interpreter in your IDE".
790802

791803
To see available apps to connect to, you may run the `list` command:
792804

793805
```
794806
sm-ssh list studio.sagemaker
795807
```
808+
*Tip:* When you configure Python interpreter in PyCharm with SageMaker Studio, it's recommended to configure [the deployment path mapping](https://www.jetbrains.com/help/pycharm/creating-local-server-configuration.html#mapping) for you project to point into `/home/sagemaker-user/user-default-efs/project_name` instead of default `/tmp/pycharm_project_123`.
809+
This is how you will be able to see your project in SageMaker Studio and PyCharm will automatically sync your local dir to the remote dir.
796810

797811
*Note:* If you're using Windows, see [the FAQ](FAQ.md#is-windows-supported).
798812

813+
799814
3. Using the remote Jupyter Notebook
800815

801816
In recent versions of PyCharm, Jupyter Notebook is tunnelled automatically through remote interpreter connection. You might need to add `--allow-root` argument to the command line, when your remote interpreter runs under root:

‎SageMaker_SSH_IDE.ipynb

+4-17
Original file line numberDiff line numberDiff line change
@@ -66,16 +66,6 @@
6666
"The next command will install the software for remote access, including SSH daemon and VNC server and XFCE4 desktop. If you don't want to run a VNC server or Jupyter notebook, pass the flag `--ssh-only` (comment the first line and uncomment the second)."
6767
]
6868
},
69-
{
70-
"cell_type": "code",
71-
"execution_count": null,
72-
"metadata": {},
73-
"outputs": [],
74-
"source": [
75-
"%%sh\n",
76-
"cd /opt/"
77-
]
78-
},
7969
{
8070
"cell_type": "code",
8171
"execution_count": null,
@@ -120,10 +110,7 @@
120110
"cell_type": "code",
121111
"execution_count": null,
122112
"metadata": {
123-
"collapsed": false,
124-
"jupyter": {
125-
"outputs_hidden": false
126-
}
113+
"collapsed": false
127114
},
128115
"outputs": [],
129116
"source": [
@@ -864,9 +851,9 @@
864851
],
865852
"instance_type": "ml.m5.large",
866853
"kernelspec": {
867-
"display_name": "Python 3 (ipykernel)",
854+
"display_name": "Python 3 (Data Science 2.0)",
868855
"language": "python",
869-
"name": "python3"
856+
"name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:eu-west-1:470317259841:image/sagemaker-data-science-38"
870857
},
871858
"language_info": {
872859
"codemirror_mode": {
@@ -878,7 +865,7 @@
878865
"name": "python",
879866
"nbconvert_exporter": "python",
880867
"pygments_lexer": "ipython3",
881-
"version": "3.11.11"
868+
"version": "3.8.13"
882869
}
883870
},
884871
"nbformat": 4,

‎aws.xml

+30
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
<?xml version="1.0" encoding="UTF-8"?>
2+
<project version="4">
3+
<component name="accountSettings">
4+
<option name="activeProfile" value="profile:default" />
5+
<option name="activeRegion" value="eu-west-1" />
6+
<option name="recentlyUsedProfiles">
7+
<list>
8+
<option value="profile:default" />
9+
<option value="profile:ssh" />
10+
</list>
11+
</option>
12+
<option name="recentlyUsedRegions">
13+
<list>
14+
<option value="eu-west-1" />
15+
<option value="eu-west-2" />
16+
</list>
17+
</option>
18+
</component>
19+
<component name="connectionPinningManager">
20+
<option name="pinnedConnections">
21+
<map>
22+
<entry key="aws.codecatalyst" value="sso;us-east-1;https://view.awsapps.com/start" />
23+
<entry key="aws.codewhisperer" value="sso;us-east-1;https://view.awsapps.com/start" />
24+
</map>
25+
</option>
26+
</component>
27+
<component name="explorerToolWindow">
28+
<option name="selectedTab" value="Explorer" />
29+
</component>
30+
</project>

‎dataSources.local.xml

+10
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
<?xml version="1.0" encoding="UTF-8"?>
2+
<project version="4">
3+
<component name="dataSourceStorageLocal" created-in="PY-243.22562.220">
4+
<data-source name=".coverage" uuid="25558db9-f61d-4d35-ae95-03e23b2d29ee">
5+
<database-info product="" version="" jdbc-version="" driver-name="" driver-version="" dbms="SQLITE" exact-version="0" />
6+
<secret-storage>forget</secret-storage>
7+
<schema-mapping />
8+
</data-source>
9+
</component>
10+
</project>

‎dataSources.xml

+12
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
<?xml version="1.0" encoding="UTF-8"?>
2+
<project version="4">
3+
<component name="DataSourceManagerImpl" format="xml" multifile-model="true">
4+
<data-source source="LOCAL" name=".coverage" uuid="25558db9-f61d-4d35-ae95-03e23b2d29ee">
5+
<driver-ref>sqlite.xerial</driver-ref>
6+
<synchronize>true</synchronize>
7+
<jdbc-driver>org.sqlite.JDBC</jdbc-driver>
8+
<jdbc-url>jdbc:sqlite:$PROJECT_DIR$/tests/.coverage</jdbc-url>
9+
<working-dir>$ProjectFileDir$</working-dir>
10+
</data-source>
11+
</component>
12+
</project>

‎deployment.xml

+22
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
<?xml version="1.0" encoding="UTF-8"?>
2+
<project version="4">
3+
<component name="PublishConfigData" autoUpload="Always" serverName="[email protected]:22 agent" remoteFilesAllowedToDisappearOnAutoupload="false">
4+
<serverData>
5+
<paths name="root@sagemaker-data-science-ml-m5-large-6590da95dc67eec021b14bedc036.default-20240514t204234.d-j53oaxv4za10.studio.sagemaker:22 agent">
6+
<serverdata>
7+
<mappings>
8+
<mapping deploy="/root/sagemaker-ssh-helper-dev" local="$PROJECT_DIR$" />
9+
</mappings>
10+
</serverdata>
11+
</paths>
12+
<paths name="[email protected]:22 agent">
13+
<serverdata>
14+
<mappings>
15+
<mapping deploy="/home/sagemaker-user/user-default-efs/sagemaker-ssh-helper-dev" local="$PROJECT_DIR$" />
16+
</mappings>
17+
</serverdata>
18+
</paths>
19+
</serverData>
20+
<option name="myAutoUpload" value="ALWAYS" />
21+
</component>
22+
</project>

‎inspectionProfiles/Project_Default.xml

+21
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
<component name="InspectionProjectProfileManager">
2+
<profile version="1.0">
3+
<option name="myName" value="Project Default" />
4+
<inspection_tool class="PyCompatibilityInspection" enabled="true" level="WARNING" enabled_by_default="true">
5+
<option name="ourVersions">
6+
<value>
7+
<list size="1">
8+
<item index="0" class="java.lang.String" itemvalue="3.10" />
9+
</list>
10+
</value>
11+
</option>
12+
</inspection_tool>
13+
<inspection_tool class="PyUnresolvedReferencesInspection" enabled="true" level="WARNING" enabled_by_default="true">
14+
<option name="ignoredIdentifiers">
15+
<list>
16+
<option value="pytorch_lightning.utilities.seed" />
17+
</list>
18+
</option>
19+
</inspection_tool>
20+
</profile>
21+
</component>

‎inspectionProfiles/profiles_settings.xml

+7
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
<component name="InspectionProjectProfileManager">
2+
<settings>
3+
<option name="PROJECT_PROFILE" value="Default" />
4+
<option name="USE_PROJECT_PROFILE" value="false" />
5+
<version value="1.0" />
6+
</settings>
7+
</component>

‎kernel-lc-config.sh

+22-10
Original file line numberDiff line numberDiff line change
@@ -18,25 +18,37 @@ LOCAL_USER_ID="AIDACKCEVSQ6C2EXAMPLE:terry@SSO"
1818

1919
set -e
2020

21+
# If not root, execute as root via sudo
22+
if [ "$EUID" -ne 0 ]; then
23+
SUDO="sudo"
24+
$SUDO true
25+
echo 'kernel-lc-config.sh: INFO - Executing as root via sudo'
26+
exec $SUDO -E HOME=/root "$0" "$@"
27+
exit 0
28+
fi
29+
30+
PYTHON_BIN=$(which python3 || which python)
31+
2132
if [ -f /opt/sagemaker-ssh-helper/.ssh-ide-configured ]; then
2233
echo 'kernel-lc-config.sh: INFO - SageMaker SSH Helper is already installed, remove /opt/sagemaker-ssh-helper/.ssh-ide-configured to reinstall'
2334
else
24-
pip3 uninstall -y -q awscli
25-
pip3 install -q sagemaker-ssh-helper
35+
$PYTHON_BIN -m pip uninstall -y -q awscli
36+
$PYTHON_BIN -m pip install -q sagemaker-ssh-helper
2637

2738
# Uncomment two lines below to update SageMaker SSH Helper to the latest dev version from the main branch
28-
#git clone https://github.com/aws-samples/sagemaker-ssh-helper.git ./sagemaker-ssh-helper/ || echo 'Already cloned'
29-
#cd ./sagemaker-ssh-helper/ && git pull --no-rebase && git clean -f && pip install . && cd ..
39+
#git clone https://github.com/aws-samples/sagemaker-ssh-helper.git /tmp/sagemaker-ssh-helper/ || echo 'Already cloned'
40+
#cd /tmp/sagemaker-ssh-helper/ && git pull --no-rebase && git clean -f && $PYTHON_BIN -m pip install . && cd -
3041
fi
3142

32-
sm-ssh-ide get-metadata
33-
34-
which python3
35-
3643
# We assume that the kernels are is installed into the sys prefix, e.g. with ipykernel install --sys-prefix command
37-
SYSTEM_PYTHON_PREFIX=$(python3 -c "from __future__ import print_function;import sys; print(sys.prefix)")
44+
SYSTEM_PYTHON_PREFIX=$($PYTHON_BIN -c "from __future__ import print_function;import sys; print(sys.prefix)")
3845
export JUPYTER_PATH="$SYSTEM_PYTHON_PREFIX/share/jupyter/"
3946

47+
export PATH="$PATH:$SYSTEM_PYTHON_PREFIX/bin"
48+
49+
sm-ssh-ide get-metadata
50+
sm-ssh-ide env-diagnostics
51+
4052
# If already configured in the container, it will not take any effect:
4153
sm-ssh-ide configure
4254
#sm-ssh-ide configure --ssh-only
@@ -59,4 +71,4 @@ sm-ssh-ide init-ssm
5971

6072
nohup sm-ssh-ide start &
6173

62-
nohup sm-ssh-ide ssm-agent &
74+
nohup sm-ssh-ide ssm-agent &

‎misc.xml

+10
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
<?xml version="1.0" encoding="UTF-8"?>
2+
<project version="4">
3+
<component name="Black">
4+
<option name="sdkName" value="Python 3.9 (sagemaker-ssh-helper-dev)" />
5+
</component>
6+
<component name="ProjectRootManager" version="2" project-jdk-name="Remote Python 3.11.11 (sftp://[email protected]:22/opt/conda/bin/python3)" project-jdk-type="Python SDK" />
7+
<component name="PythonCompatibilityInspectionAdvertiser">
8+
<option name="version" value="3" />
9+
</component>
10+
</project>

‎modules.xml

+8
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
<?xml version="1.0" encoding="UTF-8"?>
2+
<project version="4">
3+
<component name="ProjectModuleManager">
4+
<modules>
5+
<module fileurl="file://$PROJECT_DIR$/.idea/sagemaker-ssh-helper.iml" filepath="$PROJECT_DIR$/.idea/sagemaker-ssh-helper.iml" />
6+
</modules>
7+
</component>
8+
</project>

‎other.xml

+6
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
<?xml version="1.0" encoding="UTF-8"?>
2+
<project version="4">
3+
<component name="PySciProjectComponent">
4+
<option name="PY_SCI_VIEW_SUGGESTED" value="true" />
5+
</component>
6+
</project>

‎runConfigurations/SageMaker_Debug_Server.xml

+15
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
<component name="ProjectRunConfigurationManager">
2+
<configuration default="false" name="SageMaker Debug Server" type="PyRemoteDebugConfigurationType" factoryName="Python Remote Debug">
3+
<module name="sagemaker-ssh-helper" />
4+
<option name="PORT" value="12345" />
5+
<option name="HOST" value="localhost" />
6+
<PathMappingSettings>
7+
<option name="pathMappings">
8+
<list />
9+
</option>
10+
</PathMappingSettings>
11+
<option name="REDIRECT_OUTPUT" value="true" />
12+
<option name="SUSPEND_AFTER_CONNECT" value="true" />
13+
<method v="2" />
14+
</configuration>
15+
</component>

‎sagemaker-ssh-helper.iml

+28
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
<?xml version="1.0" encoding="UTF-8"?>
2+
<module type="PYTHON_MODULE" version="4">
3+
<component name="NewModuleRootManager">
4+
<content url="file://$MODULE_DIR$">
5+
<excludeFolder url="file://$MODULE_DIR$/venv" />
6+
<excludeFolder url="file://$MODULE_DIR$/build" />
7+
<excludeFolder url="file://$MODULE_DIR$/dist" />
8+
<excludeFolder url="file://$MODULE_DIR$/tests/opt_ml/code" />
9+
<excludeFolder url="file://$MODULE_DIR$/tests/opt_ml/model" />
10+
<excludeFolder url="file://$MODULE_DIR$/tests/opt_ml/output" />
11+
<excludeFolder url="file://$MODULE_DIR$/tests/.pytest_cache" />
12+
<excludeFolder url="file://$MODULE_DIR$/tests/.pytest_cache/v" />
13+
<excludeFolder url="file://$MODULE_DIR$/tests/output" />
14+
<excludeFolder url="file://$MODULE_DIR$/sagemaker_ssh_helper.egg-info" />
15+
<excludeFolder url="file://$MODULE_DIR$/.idea/dataSources" />
16+
<excludeFolder url="file://$MODULE_DIR$/src_diff" />
17+
<excludeFolder url="file://$MODULE_DIR$/.pytest_cache/v" />
18+
</content>
19+
<orderEntry type="jdk" jdkName="Remote Python 3.11.11 (sftp://[email protected]:22/opt/conda/bin/python3)" jdkType="Python SDK" />
20+
<orderEntry type="sourceFolder" forTests="false" />
21+
</component>
22+
<component name="PackageRequirementsSettings">
23+
<option name="requirementsPath" value="" />
24+
</component>
25+
<component name="TestRunnerService">
26+
<option name="PROJECT_TEST_RUNNER" value="py.test" />
27+
</component>
28+
</module>

0 commit comments

Comments
 (0)