-
Notifications
You must be signed in to change notification settings - Fork 4
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add documentation on usage and custom images
- Loading branch information
Showing
2 changed files
with
146 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,60 @@ | ||
# Creating Custom Images | ||
|
||
1. Clone this repo | ||
|
||
2. Create a new branch `git checkout -b feature/update_package_<xyz>` | ||
|
||
3. and open `environment.yml`. It might look something like this: | ||
|
||
```yaml | ||
channels: | ||
- conda-forge | ||
- nodefaults | ||
dependencies: | ||
- ipykernel | ||
# Dependencies for VS Code IDE | ||
- code-server>=3.2 | ||
- jupyter-vscode-proxy | ||
- openssh | ||
- pre_commit | ||
- pip | ||
- pip: | ||
- git+https://github.com/MAAP-Project/[email protected] | ||
- jupyter-sshd-proxy | ||
- jupyterlab-bxplorer | ||
variables: | ||
TITILER_STAC_ENDPOINT: 'https://openveda.cloud/api/stac' | ||
TITILER_ENDPOINT: 'https://openveda.cloud/api/raster' | ||
STAC_CATALOG_NAME: 'VEDA STAC' | ||
STAC_CATALOG_URL: 'https://openveda.cloud/api/stac' | ||
STAC_BROWSER_URL: 'https://openveda.cloud/' | ||
``` | ||
4. Add the pip packages or conda packages you need installed in the `dependencies` block. Or any other customizations you need. | ||
|
||
5. commit and push changes to a remote branch on **this repo**. | ||
|
||
6. Once the changes are pushed to a remote branch on this repo, GH actions will automatically run the image building pipeline. You can watch this in the [GH actions tab](https://github.com/NASA-IMPACT/veda-jh-environments/actions) | ||
|
||
7. Look for the workflow named `Build and push container image` for the most recent commit to your branch. Click into the workflow run to see more details on what is happening. | ||
|
||
8. Once the workflow run completes you should see something like the following in the `Tag and push image` step which indicates the image was built successfully. | ||
|
||
```bash | ||
Run FULL_IMAGE_NAME=public.ecr.aws/nasa-veda/pangeo-notebook-veda-image | ||
The push refers to repository [public.ecr.aws/nasa-veda/pangeo-notebook-veda-image] | ||
... | ||
... | ||
f521e8d7b8b9: Pushed | ||
18b1f1fbf84a: Pushed | ||
b98733c8ba26: digest: sha256:51d2a3c6e089006b4b9246251a5e046f490f78758796f7bf4345d4777bef16f2 size: 4500 | ||
Pushed image: public.ecr.aws/nasa-veda/pangeo-notebook-veda-image:b98733c8ba26 | ||
``` | ||
|
||
Note the full image URI will be `public.ecr.aws/nasa-veda/pangeo-notebook-veda-image:<imageTag>`. The `imageTag` should match the latest commit hash of your branch. | ||
|
||
10. Copy the image URI from the logs of the GH action run that looks like `public.ecr.aws/nasa-veda/pangeo-notebook-veda-image:<commit-hash>`. We can test this image on staging by going to https://staging.nasa-veda.2i2c.cloud/hub/spawn and selecting "Other" from the "Image" dropdown. | ||
|
||
11. In the "Custom image" input box, paste the image URI from the logs of the GH action run and click "Start". | ||
|
||
12. If all goes well, you'll see the Jupyter server start up with your custom changes. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,86 @@ | ||
|
||
# Updating 2i2c's Singleuser Image for VEDA JupyterHubs | ||
|
||
JupyterHub configuration has the concept of a `singleuser` image that is the | ||
[default image for spinning up user pods](https://z2jh.jupyter.org/en/stable/jupyterhub/customizing/user-environment.html#choose-and-use-an-existing-docker-image). | ||
|
||
Currently https://staging.nasa-veda.2i2c.cloud/ and https://nasa-veda.2i2c.cloud/ both use the custom image from this repository as the default. | ||
Below we walk through how to update this image and get it in these VEDA JH instances. This allows us to add | ||
custom packages without us needing to request these custom packages upstream in the `pangeo-notebook` image. | ||
|
||
## Update the Conda Environment | ||
|
||
1. Clone this repo | ||
|
||
2. Create a new branch `git checkout -b feature/update_package_<xyz>` | ||
|
||
3. and open `environment.yml`. It might look something like this: | ||
|
||
```yaml | ||
channels: | ||
- conda-forge | ||
- nodefaults | ||
dependencies: | ||
- ipykernel | ||
# Dependencies for VS Code IDE | ||
- code-server>=3.2 | ||
- jupyter-vscode-proxy | ||
- openssh | ||
- pre_commit | ||
- pip | ||
- pip: | ||
- git+https://github.com/MAAP-Project/[email protected] | ||
- jupyter-sshd-proxy | ||
- jupyterlab-bxplorer | ||
variables: | ||
TITILER_STAC_ENDPOINT: 'https://openveda.cloud/api/stac' | ||
TITILER_ENDPOINT: 'https://openveda.cloud/api/raster' | ||
STAC_CATALOG_NAME: 'VEDA STAC' | ||
STAC_CATALOG_URL: 'https://openveda.cloud/api/stac' | ||
STAC_BROWSER_URL: 'https://openveda.cloud/' | ||
``` | ||
4. Add the pip packages or conda packages you need installed in the `dependencies` block | ||
|
||
5. commit and push changes to the remote feature branch and create a PR | ||
|
||
## Test the Image on staging | ||
|
||
1. Once the changes are pushed to a remote branch on this repo, GH actions will automatically run the image building pipeline. You can watch this in the [GH actions tab](https://github.com/NASA-IMPACT/veda-jh-environments/actions) | ||
|
||
2. Look for the workflow named `Build and push container image` for the most recent commit to your branch. Click into the workflow run to see more details on what is happening. | ||
|
||
3. Once the workflow run completes you should see something like the following in the `Tag and push image` step which indicates the image was built successfully. | ||
|
||
```bash | ||
Run FULL_IMAGE_NAME=public.ecr.aws/nasa-veda/pangeo-notebook-veda-image | ||
The push refers to repository [public.ecr.aws/nasa-veda/pangeo-notebook-veda-image] | ||
... | ||
... | ||
f521e8d7b8b9: Pushed | ||
18b1f1fbf84a: Pushed | ||
b98733c8ba26: digest: sha256:51d2a3c6e089006b4b9246251a5e046f490f78758796f7bf4345d4777bef16f2 size: 4500 | ||
Pushed image: public.ecr.aws/nasa-veda/pangeo-notebook-veda-image:b98733c8ba26 | ||
``` | ||
|
||
Note the full image URI will be `public.ecr.aws/nasa-veda/pangeo-notebook-veda-image:<imageTag>`. The `imageTag` should match the latest commit hash of your branch. | ||
|
||
4. Copy the image URI from the logs of the GH action run that looks like `public.ecr.aws/nasa-veda/pangeo-notebook-veda-image:<commit-hash>`. We can test this image on staging by going to https://staging.nasa-veda.2i2c.cloud/hub/spawn and selecting "Other" from the "Image" dropdown. | ||
|
||
5. In the "Custom image" input box, paste the image URI from the logs of the GH action run and click "Start". | ||
|
||
6. If all goes well, you'll see the Jupyter server start up with your custom packages. | ||
|
||
7. If everything works as expected, you can go back to the PR and request a review or merge it into `main` as appropriate. | ||
|
||
8. Once the PR is merged, it should trigger a new GH action to build the image and push it to ECR. Once the image is pushed, we can get the image tag from GH actions by following the steps outlines above in point 3. | ||
|
||
9. Now we need to update the singleuser image in 2i2c's infrastructure repo. | ||
|
||
## Update the Singleuser Image for VEDA JupyterHubs in 2i2c's Infrastructure Repo | ||
|
||
1. If you haven't already [clone DS's fork](https://github.com/developmentseed/infrastructure/) of [2i2c's insfrastructure repo](https://github.com/2i2c-org/infrastructure) | ||
|
||
2. In `config/clusters/nasa-veda/common.values.yaml`, update the profile choice correspoding to the Pangeo image in the `singleuser` image block with the new image URI from the GH action run we got in the previous step. We can refer to the past [PR over here](https://github.com/2i2c-org/infrastructure/commit/1f39db71972fa420a24b8dc2dab0eec7d3e05d25) to see how this looks. It's recommended to update the staging hub's image first. And then once that's verified to work, update the production hub's image. | ||
|
||
3. After the PR is created, reviewed and merged, 2i2c's CI/CD process will automatically deploy the image to staging and production hubs as needed. |