Skip to content

Commit 7df527a

Browse files
Update tutorials to include memory benchmarks (#163)
* Updated mem usage + copy edits to CryoCloud_demo * updated GrIMP tutorial memory requirements (not measured any Dask tasks). Copy+edits. Flagging dependency issue with Numba library req older numpy. * updated GrIMP tutorial memory requirements (not measured any Dask tasks). Copy+edits. Flagging dependency issue with Numba library req older numpy. * added memory benchmark. Tutorial runs again after re-release of ATL15 data that caused earthdata to return null * added memory benchmark and patch dependency issue so tutorial runs * added memory benchark and learning objectives * added memory benchmark and fixed icepyx version issue * added memory benchmark and fixed icepyx .vars -> .variables issue * updated sliderule_applications to include memory benchmark. Runs fine on cryo-geo environment * Renamed and split SWOT tutorial files: (0) download/access (1) plot coverage (2) plot comparison with IS2 * fix:typo, date * More tutorial updates Minor file structure org, memory benchmarks, etc. Note that some tutorials are still broken due to mismatched dependencies. This is a task for later. * cleanup old CC files --------- Co-authored-by: Tasha Snow <tsnow@mines.edu>
1 parent 439c495 commit 7df527a

42 files changed

Lines changed: 19886 additions & 69923 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎book/external/ICESat2_ATL10-h5coro_large_scale_time_series.ipynb‎

Lines changed: 57 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -38,7 +38,51 @@
3838
"tags": [],
3939
"user_expressions": []
4040
},
41-
"source": "## **1. Tutorial Overview**\n\nThis tutorial is designed for the \"DAAC data access in the cloud hands-on experience\" session at the 2023 NSIDC DAAC [User Working Group (UWG)](https://nsidc.org/data/data-programs/nsidc-daac/about-daac#anchor-2) Meeting. \n\nThe NSIDC DAAC archives and distributes Daily and Monthly Gridded [Sea Ice Freeboard (ATL20)](https://nsidc.org/data/atl20) and [Polar Sea Surface Height Anomaly (ATL21)](https://nsidc.org/data/atl21) data sets from the ICESat-2 Mission, derived from the lower level [ATL10](https://nsidc.org/data/atl10) data set. However, we may want these lower level point data to be gridded and averaged at a weekly cadence, or using a different projection or other gridding parameters. \n\nThis tutorial session is in two parts: \n* We will first guide you through this Jupyter Notebook running in the AWS `us-west-2` region, where data are hosted in the NASA Earthdata Cloud. The notebook utilizes several libraries to performantly search, access, read, and grid the data including `earthaccess`, `h5coro`, and `geopandas`.\n\n* This notebook will focus on the Ross Sea, Antarctica. But let's say we want to scale this analysis to the entire continent. In the second portion, we will present how to scale and run this same workflow from a script (see [workflow.py](./h5cloud/workflow.py) in the `h5cloud` folder within this notebook's directory) that can be run from your laptop, using [Coiled](https://www.coiled.io/). \n\n### **Credits**\n\nThe notebook was created by Andy Barrett and Luis Lopez of NSIDC.\n\nFor questions regarding the notebook, or to report problems, please create a new issue in the [NSIDC-Data-Tutorials repo](https://github.com/nsidc/NSIDC-Data-Tutorials/issues).\n\n### **Learning Objectives**\n\nBy the end of this demonstration you will be able to: \n1. Use `earthaccess` to authenticate with Earthdata Login, search for ICESat-2 data using spatial and temporal filters, and directly access files in the cloud.\n2. Open data granules using `h5coro` to efficiently read HDF5 data from the NSIDC DAAC S3 bucket.\n3. Load data into a geopandas.DataFrame containing geodetic coordinates, ancillary variables, and date/time converted from ATLAS Epoch.\n4. Grid track data to EASE-Grid v2 6.25 km projected grid using drop-in-the-bucket resampling. \n5. Calculate mean statistics and assign aggregated data to grid cells. \n5. Visualize aggregated sea ice height data on a map.\n\n### **Prerequisites**\n\n1. We are running this notebook in the [CryoCloud](https://book.cryointhecloud.com/) JupyterHub. For more information, see the CryoCloud [Getting Started](https://book.cryointhecloud.com/getting-started) documentation.\n**It is advised that you use at least a 16GB instance for this notebook.** \n2. An Earthdata Login is required for data access. If you don't have one, you can register for one [here](https://urs.earthdata.nasa.gov/).\n3. It is recommended that you create a .netrc file that contains your Earthdata Login credentials, stored in your home directory. If you do not have a .netrc file, `earthaccess` will prompt you to enter your Earthdata Login username and password.\n\n### **Example of end product** \nAt the end of this tutorial, the following figure will be generated, demonstrating a year's worth of ATL10 Sea Ice Freeboard height data gridded over the Ross Sea, Antarctica:\n<center>\n<img src='./img/icesat2.atl10.gridded.count_segments.ross_sea.png'/>\n</center>\n\n### **Time requirement**\n\nAllow approximately 40 minutes to complete this tutorial."
41+
"source": [
42+
"## **1. Tutorial Overview**\n",
43+
"\n",
44+
"This tutorial is designed for the \"DAAC data access in the cloud hands-on experience\" session at the 2023 NSIDC DAAC [User Working Group (UWG)](https://nsidc.org/data/data-programs/nsidc-daac/about-daac#anchor-2) Meeting. \n",
45+
"\n",
46+
"The NSIDC DAAC archives and distributes Daily and Monthly Gridded [Sea Ice Freeboard (ATL20)](https://nsidc.org/data/atl20) and [Polar Sea Surface Height Anomaly (ATL21)](https://nsidc.org/data/atl21) data sets from the ICESat-2 Mission, derived from the lower level [ATL10](https://nsidc.org/data/atl10) data set. However, we may want these lower level point data to be gridded and averaged at a weekly cadence, or using a different projection or other gridding parameters. \n",
47+
"\n",
48+
"This tutorial session is in two parts: \n",
49+
"* We will first guide you through this Jupyter Notebook running in the AWS `us-west-2` region, where data are hosted in the NASA Earthdata Cloud. The notebook utilizes several libraries to performantly search, access, read, and grid the data including `earthaccess`, `h5coro`, and `geopandas`.\n",
50+
"\n",
51+
"* This notebook will focus on the Ross Sea, Antarctica. But let's say we want to scale this analysis to the entire continent. In the second portion, we will present how to scale and run this same workflow from a script (see [workflow.py](./h5cloud/workflow.py) in the `h5cloud` folder within this notebook's directory) that can be run from your laptop, using [Coiled](https://www.coiled.io/). \n",
52+
"\n",
53+
"### **Credits**\n",
54+
"\n",
55+
"The notebook was created by Andy Barrett and Luis Lopez of NSIDC.\n",
56+
"\n",
57+
"For questions regarding the notebook, or to report problems, please create a new issue in the [NSIDC-Data-Tutorials repo](https://github.com/nsidc/NSIDC-Data-Tutorials/issues).\n",
58+
"\n",
59+
"### **Learning Objectives**\n",
60+
"\n",
61+
"By the end of this demonstration you will be able to: \n",
62+
"1. Use `earthaccess` to authenticate with Earthdata Login, search for ICESat-2 data using spatial and temporal filters, and directly access files in the cloud.\n",
63+
"2. Open data granules using `h5coro` to efficiently read HDF5 data from the NSIDC DAAC S3 bucket.\n",
64+
"3. Load data into a geopandas.DataFrame containing geodetic coordinates, ancillary variables, and date/time converted from ATLAS Epoch.\n",
65+
"4. Grid track data to EASE-Grid v2 6.25 km projected grid using drop-in-the-bucket resampling. \n",
66+
"5. Calculate mean statistics and assign aggregated data to grid cells. \n",
67+
"5. Visualize aggregated sea ice height data on a map.\n",
68+
"\n",
69+
"### **Prerequisites**\n",
70+
"\n",
71+
"1. We are running this notebook in the [CryoCloud](https://book.cryointhecloud.com/) JupyterHub. For more information, see the CryoCloud [Getting Started](https://book.cryointhecloud.com/getting-started) documentation.\n",
72+
"**It is advised that you use at least a 16GB instance for this notebook.** \n",
73+
"2. An Earthdata Login is required for data access. If you don't have one, you can register for one [here](https://urs.earthdata.nasa.gov/).\n",
74+
"3. It is recommended that you create a .netrc file that contains your Earthdata Login credentials, stored in your home directory. If you do not have a .netrc file, `earthaccess` will prompt you to enter your Earthdata Login username and password.\n",
75+
"\n",
76+
"### **Example of end product** \n",
77+
"At the end of this tutorial, the following figure will be generated, demonstrating a year's worth of ATL10 Sea Ice Freeboard height data gridded over the Ross Sea, Antarctica:\n",
78+
"<center>\n",
79+
"<img src='./img/icesat2.atl10.gridded.count_segments.ross_sea.png'/>\n",
80+
"</center>\n",
81+
"\n",
82+
"### **Time requirement**\n",
83+
"\n",
84+
"Allow approximately 40 minutes to complete this tutorial."
85+
]
4286
},
4387
{
4488
"cell_type": "markdown",
@@ -67,29 +111,25 @@
67111
"%%capture\n",
68112
"# suppress install outputs\n",
69113
"\n",
70-
"!pip uninstall -y earthaccess h5coro\n",
71-
"!pip install earthaccess==0.6.1\n",
114+
"%pip uninstall -y earthaccess h5coro\n",
115+
"%pip install earthaccess==0.6.1\n",
72116
"\n",
73117
"# h5coro has new features that we need that are not released\n",
74-
"!pip install git+https://github.com/ICESat2-SlideRule/h5coro.git@main"
118+
"%pip install git+https://github.com/ICESat2-SlideRule/h5coro.git@main"
75119
]
76120
},
77121
{
78122
"cell_type": "markdown",
79123
"id": "e7bdaa85-ac4e-4172-9154-1d0992414cc1",
80-
"metadata": {
81-
"user_expressions": []
82-
},
124+
"metadata": {},
83125
"source": [
84126
"**NOTE**: Restart the kernel and clean output after running the cell above."
85127
]
86128
},
87129
{
88130
"cell_type": "markdown",
89131
"id": "7820a737-33f0-4470-b9a4-03c5c4f0354c",
90-
"metadata": {
91-
"user_expressions": []
92-
},
132+
"metadata": {},
93133
"source": [
94134
"### **Import Packages**"
95135
]
@@ -122,17 +162,15 @@
122162
"import cartopy.crs as ccrs\n",
123163
"import cartopy.feature as cfeature\n",
124164
"\n",
125-
"from h5cloud.read_atl10 import read_atl10\n",
165+
"# from h5cloud.read_atl10 import read_atl10\n",
126166
"\n",
127167
"print(f\"earthaccess: {earthaccess.__version__}\")"
128168
]
129169
},
130170
{
131171
"cell_type": "markdown",
132172
"id": "1966ffa6-a5f2-4520-a8dc-f37678a2cf7a",
133-
"metadata": {
134-
"user_expressions": []
135-
},
173+
"metadata": {},
136174
"source": [
137175
"### Authenticate\n",
138176
"\n",
@@ -154,9 +192,7 @@
154192
{
155193
"cell_type": "markdown",
156194
"id": "da19f604-0288-4358-ab88-d18c986f7cc8",
157-
"metadata": {
158-
"user_expressions": []
159-
},
195+
"metadata": {},
160196
"source": [
161197
"### **Search for ICESat-2 ATL10 data**\n",
162198
"\n",
@@ -218,9 +254,7 @@
218254
{
219255
"cell_type": "markdown",
220256
"id": "b6887297-2b02-4e76-9bc5-5e804c54c5d7",
221-
"metadata": {
222-
"user_expressions": []
223-
},
257+
"metadata": {},
224258
"source": [
225259
"### Access the Granules\n",
226260
"\n",
@@ -233,8 +267,7 @@
233267
"cell_type": "markdown",
234268
"id": "80340da9-1b3b-458d-9d94-9492657f94bf",
235269
"metadata": {
236-
"tags": [],
237-
"user_expressions": []
270+
"tags": []
238271
},
239272
"source": [
240273
"#### Read data into `geopandas.DataFrame`\n",
@@ -855,9 +888,9 @@
855888
"name": "python",
856889
"nbconvert_exporter": "python",
857890
"pygments_lexer": "ipython3",
858-
"version": "3.10.12"
891+
"version": "3.14.7"
859892
}
860893
},
861894
"nbformat": 4,
862895
"nbformat_minor": 5
863-
}
896+
}

‎book/myst.yml‎

Lines changed: 9 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -49,8 +49,8 @@ project:
4949
- file: tutorials/index.md
5050
children:
5151
- file: tutorials/stratusgeo_demo/StratusGeo_demo.ipynb
52-
- file: tutorials/IS2_cloud_Landsat_integration.ipynb
53-
- file: tutorials/ICESat2_ATL15_dhdt/ICESat2_ATL15_dhdt.ipynb
52+
- file: tutorials/IS2_Landsat_integration/IS2_cloud_Landsat_integration.ipynb
53+
- file: tutorials/IS2_ATL15_dhdt/ICESat2_ATL15_dhdt.ipynb
5454
- file: tutorials/IS2_ATL15_surface_height_anomalies/IS2_ATL15_surface_height_anomalies.ipynb
5555
- file: external/ICESat2_ATL10-h5coro_large_scale_time_series.ipynb
5656
- file: tutorials/NASA-Earthdata-Cloud-Access/index.md
@@ -61,14 +61,15 @@ project:
6161
- file: tutorials/NASA-Earthdata-Cloud-Access/4.icepyx.ipynb
6262
- file: tutorials/NASA-Earthdata-Cloud-Access/4b.icepyx-ATL06.ipynb
6363
- file: tutorials/SlideRule_applications/SlideRule_applications.ipynb
64-
- file: tutorials/dask_for_geoscientists.ipynb
65-
- file: tutorials/ARCOdata_writingZarrs.ipynb
64+
- file: tutorials/Dask/dask_for_geoscientists.ipynb
65+
- file: tutorials/Zarr_with_ApRES/ARCOdata_writingZarrs.ipynb
6666
- file: tutorials/GrIMP/GrIMP_tutorial_AGU23.ipynb
67-
- file: tutorials/ITS_LIVE_plot_point_timeseries.ipynb
68-
- file: tutorials/SWOT_HR/index.md
67+
- file: tutorials/ITS_LIVE/ITS_LIVE_plot_point_timeseries.ipynb
68+
- file: tutorials/SWOT/index.md
6969
children:
70-
- file: tutorials/SWOT_HR/SWOT_HR_access.ipynb
71-
- file: tutorials/SWOT_HR/SWOT_HR_w_IS2.ipynb
70+
- file: tutorials/SWOT_HR/0_SWOT_LRHR_access.ipynb
71+
- file: tutorials/SWOT_HR/1_SWOT_HR_coverage.ipynb
72+
- file: tutorials/SWOT_HR/2_SWOT_HR_vs_ICESat2.ipynb
7273
- file: contributing/index.md
7374
children:
7475
- file: contributing/workflow.md

0 commit comments

Comments
 (0)