Merge pull request #20 from NCAR/fix_repo

Fix repo

Author: hrhampapura

README.md

@@ -1,7 +1,7 @@
 # OSDF-Examples
 
 ## About
-Contains Jupyter notebook workflows which access climate data from various [OSDF](https://osg-htc.org/services/osdf.html) origins using [PelicanFS](https://github.com/PelicanPlatform/pelicanfs). The examples include workflows where computations are executed on various platforms like NCAR's Casper, TACC's Stampede3 and cloud computing platforms like Jetstream2. If you use any workflow in this repository, please cite using [![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.16863133.svg)](https://doi.org/10.5281/zenodo.16863133)
+This repository contains Jupyter notebook workflows which access climate data from various [OSDF](https://osg-htc.org/services/osdf.html) origins using [PelicanFS](https://github.com/PelicanPlatform/pelicanfs). The examples include workflows where computations are executed on various platforms like NCAR's Casper, TACC's Stampede3 and cloud computing platforms like Jetstream2. If you use any workflow in this repository, please cite using [![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.16863133.svg)](https://doi.org/10.5281/zenodo.16863133)
 
 
 ## Example Workflows

docs/computational_platforms.md

@@ -0,0 +1,9 @@
+---
+title: Other Computational Platforms
+date: 2026-1-28
+author: Harsha R. Hampapura
+---
+
+
+## Overview 
+- This section contains notebooks that are executed on other computational platforms like TACC's Stampede3, Indiana University's Jetstream2, the Open Science Pool (OSPool) etc, demonstraing the capability of OSDF and the PelicanFS software to deliver data to compute.

docs/gdex_workflows.md

@@ -1,127 +1,15 @@
 ---
-title: Reanalysis
-date: 2025-12-05
-author: Chia-Wei Hsu
+title: GDEX workflows
+date: 2026-1-28
+author: Harsha R. Hampapura
 ---
 
 
-## Overview
+## Overview - Geoscience Data Exchange (GDEX)
+- This section contains notebooks that are executed on NCAR's HPC system Casper. It has been further subdivided into 3 sections: 
+    - NCAR Data Origin - Notebooks that illustrate data streaming from NCAR's OSDF origin
+    - Other Data Origins - Notebooks that illustrate data streaming from other OSDF origins like the AWS Open Data origin.
+    - ML Workflows - Machine Learning Workflows
 
-Reanalysis products are comprehensive datasets that combine historical observations with models simulation to create a consistent, spatially and temporally complete representation of the Earth's past climate and weather. They are one of the most valuable resources in Earth system science for understanding climate variability, trends, and dynamics.
 
-## Reanalysis vs. Analysis vs. Simulations vs. Observations
-
-Understanding the differences between these fundamental data types is crucial for Earth system science research.
-
-:::{important} Reanalysis vs. Analysis 👈
-:class: dropdown
-**Analysis** refers to the real-time, operational data assimilation product produced by weather forecasting centers:
-- Uses the **current** version of the forecast model and assimilation system
-- The model and assimilation methods **change over time** as improvements are made
-- Optimized for short-term weather forecasting
-- Creates discontinuities when the system is upgraded
-
-**Reanalysis** retrospectively processes historical observations:
-- Uses a **fixed** model and assimilation system for the entire time period
-- Ensures temporal **consistency** and homogeneity
-- Not updated in real-time; produced in multi-year projects
-- Better suited for climate studies and long-term trend analysis
-- More computationally expensive due to reprocessing decades of data
-
-**Key Distinction**: Analysis prioritizes current forecast skill; reanalysis prioritizes long-term consistency.
-:::
-
-:::{important} Reanalysis vs. Model Simulations 👈
-:class: dropdown
-**Model Simulations** (also called free-running simulations or climate projections):
-- Run forward in time without observational constraints
-- Initial conditions may come from observations, but the model evolves freely
-- Used for future climate projections and sensitivity experiments
-- Can drift from observed climate due to model biases
-- Examples: CMIP6 models, CESM simulations
-
-**Reanalysis**:
-- **Continuously constrained** by observations through data assimilation
-- Cannot drift far from observed atmospheric state
-- Represents the "best estimate" of what actually happened
-- Limited to historical periods where observations exist
-- Blends model physics with observational evidence
-
-**Key Distinction**: Simulations show what the model thinks should happen; reanalysis shows what actually happened (constrained by observations).
-:::
-
-
-:::{important} Reanalysis vs. Observations 👈
-:class: dropdown
-**Direct Observations** (in-situ and remote sensing):
-- Actual measurements from instruments
-- **Highest accuracy** at measurement location and time
-- Spatially and temporally **incomplete** (gaps between stations, satellite swaths)
-- Different instruments have different biases and uncertainties
-- No information about unobserved variables or locations
-- Examples: Weather station data, satellite retrievals, radiosonde profiles
-
-**Reanalysis**:
-- **Gridded, gap-filled** product combining observations with model physics
-- Provides estimates even where/when no observations exist
-- Spatially and temporally **complete**
-- Includes hundreds of variables, many not directly observed
-- **Less accurate** than direct observations at observed locations
-- Smooths out small-scale features
-- Subject to both observational and model uncertainties
-
-**Key Distinction**: Observations provide ground truth but are incomplete; reanalysis provides complete coverage but is less accurate than direct observations.
-:::
-
-
-
-## How Reanalysis Works
-
-### Data Assimilation Process
-
-Reanalysis uses a process called **data assimilation** to blend:
-
-1. **Observational Data**: Including satellite measurements, weather stations, radiosondes (weather balloons), ship and buoy observations, and aircraft reports
-2. **Numerical Models**: Physics-based models that simulate atmospheric, oceanic, and land surface processes
-
-The data assimilation system statistically combines these elements, weighing observations and model predictions based on their respective uncertainties to produce the "best estimate" of the atmospheric state at any given time.
-
-### Key Characteristics
-
-- **Temporal Consistency**: Uses a fixed, modern assimilation system and model throughout the entire reanalysis period.
-- **Spatial Completeness**: Fills gaps where observations are sparse or unavailable using model physics
-- **Regular Grid**: Produces data on uniform spatial and temporal grids, making it easier to analyze
-- **Multiple Variables**: Provides hundreds of atmospheric, oceanic, and land variables that are physically consistent with each other
-
-
-
-## Applications in Earth System Science
-
-Reanalysis products are used for:
-
-1. **Climate Monitoring**: Tracking long-term temperature, precipitation, and circulation patterns
-2. **Extreme Event Analysis**: Studying hurricanes, droughts, heat waves, and other extreme weather
-3. **Model Validation**: Evaluating climate and weather models against a consistent reference
-4. **Forcing Data**: Driving regional models, hydrological models, and impact assessments
-5. **Process Studies**: Understanding physical mechanisms and teleconnections
-6. **Trend Analysis**: Identifying climate change signals and natural variability
-7. **Deep Learning/Machine Learning**: Training and validating AI-based weather and climate models
-
-
-
-## Best Practices
-
-When using reanalysis data:
-
-1. **Choose the Right Product**: Consider temporal coverage, spatial resolution, and which variables are best represented
-2. **Validate Carefully**: Compare with independent observations when possible
-3. **Use Multiple Products**: Cross-validate findings across different reanalysis systems
-4. **Understand Uncertainty**: Be aware of which variables are observation-constrained vs. model-derived
-5. **Check Documentation**: Review known issues and dataset updates from producing centers
-
-## Resources
-
-Major reanalysis centers provide extensive documentation:
-- [ECMWF ERA5 Documentation](https://confluence.ecmwf.int/display/CKB/ERA5)
-- [JMA JRA-3Q Documentation](https://jra.kishou.go.jp/)
 

docs/introduction.md

@@ -1,32 +1,33 @@
 ---
 title: Introduction
+author:  Harsha R. Hampapura
 date: 2026-1-28
 ---
 
 # Introduction
 
-Welcome to the **OSDF Examples** documentation! This repository provides example notebooks and scripts that demonstrate how to access data from NCAR's GDEX for cool geoscience applications and visualizations.
+Welcome to the **OSDF Examples** repository! This repository provides example notebooks and scripts that demonstrate how to access data from via the Open Science Data Federation([OSDF](https://osg-htc.org/services/osdf.html)) using the using [PelicanFS](https://github.com/PelicanPlatform/pelicanfs) software. All the notebooks demonstrate how to stream geoscience data into your workflows and perform an interesting calculation/visualization. If you wish to learn more about OSDF and/or Pelican, please refer to the [OSDF cookbook](https://projectpythia.org/osdf-cookbook/).
 
 :::{warning} Important Notice
-This jupyter book is under active development and is intended primarily for use by users who have access to NCAR's HPC resources (**NCAR HPC users**). In all the examples, we assume that you have access to NCAR's jupyterhub. If you are an external user trying to stream data from NCAR's GDEX into your workflows, please see : [osdf_examples](https://ncar.github.io/osdf_examples/) (**OSDF users**)
+This jupyter book is under active development. You will need to set up your python enviroment using the requirements.txt file before running any of the notebooks. Please open an issue on the associated github repository to report any bugs or suggest improvements.
 ::: 
 
 ## How is the repository organized?
 
-This repository is organized by into various sections based on the type of dataset used:
+This repository is organized by into various sections (mostly) based on the data origins from which the data is accessed and the computational platforms used to execute the notebooks.
 
-- **Observations** (satellite, in-situ measurements)
-- **Analysis** (NCEP GFS analyses)
-- **Reanalysis** (ERA5, JRA-3Q, etc.)
-- **Model output/Simulations** (CESM, CMIP, etc.)
-- **Data Fusion** (Combination of two or more datasets)
-- **Contribution** (A guide on contributions)
+- **NCAR GDEX workflows** (Workflows that are executed on NCAR's HPC system Casper)
+    - NCAR Data Origin (Illustrate data streaming from NCAR's OSDF origin)
+    - Other Data Origins (Illustrate data streaming from other OSDF origins like AWS etc)
+    - ML Workflows (Machine Learning Workflows)
+- **Other Computational Platforms** (Workflows that are executed other HPC, cloud computing platforms)
+- **NDC Workflows** (Workflows that were developed as a part of the National Discovery Cloud (NDC) initiative )
+- **Scripts** (Python scripts and any content that is not a jupyter notebook)
 
-### 🌐 Multiple Access Methods
-Some of these notebooks use intake/intake-ESM catalogs that support multiple access patterns :
-- **POSIX**: Direct filesystem access for NCAR HPC users
-- **HTTPS**: Web-based access for remote users
+### 🌐 Access Methods
+Some of these notebooks use intake/intake-ESM catalogs in conjunction with the PelicanFS software to stream data. Others directly use the PelicanFS software to load data into xarray.
 
 ## Repository Structure
 - **`docs/`** - Introductory markdown files for each section
-- **`examples/`** - Python script examples for generating dataset catalog
+- **`notebooks/`** - All computational workflows/examples archived as jupyter notebooks
+- **`scripts/`** - Python scripts and any content that is not a jupyter notebook

docs/ndc_workflows.md

@@ -0,0 +1,10 @@
+---
+title: NDC workflows
+author:  Harsha R. Hampapura
+date: 2026-1-28
+---
+
+# NDC workflows
+
+This section contains workflows that were developed as a part of the National Discovery Cloud's (NDC) [pathfinder](https://ndc-pathfinders.org/) initiative. Most of these notebooks can be run on your laptop orpersonal device without the need for access to an HPC system.
+

myst.yml

@@ -17,10 +17,12 @@ project:
   # To autogenerate a Table of Contents, run "jupyter book init --write-toc"
   toc:
     # Auto-generated by `myst init --write-toc`
+    - file: docs/introduction.md
     - file: README.md
 
     - title: NCAR GDEX workflows
       children:
+        - file: docs/gdex_workflows.md
         - title: NCAR data origin
           children:
             - file: notebooks/cesm_bias.ipynb
@@ -50,13 +52,15 @@ project:
 
     - title: Other computational Platforms
       children:
+        - file: docs/computational_platforms.md
         - file: notebooks/cesm_osdf_stampede3.ipynb
         - file: notebooks/ndc_workflows/ncar_benchmark.ipynb
         - file: notebooks/ndc_workflows/ncar_benchmark_simple.ipynb
         - file: notebooks/ndc_workflows/ncar_benchmark_ap40.ipynb
 
     - title: NDC Workflows
       children:
+        - file: docs/ndc_workflows.md
         - file: notebooks/ndc_workflows/aws_benchmark.ipynb
         - file: notebooks/ndc_workflows/envistor_test_ap40.ipynb
         - file: notebooks/ndc_workflows/pycogss_spectral_change.ipynb