Foundation skills that Project Pythia will support

[brian-rose]

At today's meeting (1/21/2021) of the Education Working Group we discussed building content for teaching foundational skills that everyone needs to use the scientific python ecosystem effectively.

The idea is that we will host and maintain a set of high quality teaching material, using multiple formats, that will:

• serve as entry points for new users
• Be cross-referenced with each other as needed
• Serve as foundational reference material for more specialized / higher level tutorials and examples

It was said in the meeting that one goal of this initiative is to "beat google". There's lots of existing content on foundational skills but it is scattered, sometimes inconsistent, and not always tailored to geosciences.

To get started, we need to converge on a list of topics that we will cover. Let's use this discussion thread to develop the list.

[brian-rose]

I'll start! At the meeting today we identified the following:

• Plain Python in a terminal (possibly delivered through a JupyterLab): looping, conditionals, data structures, etc.
  • target audience would be people with some introduction to programming but no knowledge of Python
• Jupyter notebook / lab
  • some advocate for teaching lab from the beginning, because it's the future and feature set makes certain things easier to do
  • others suggest the opposite, traditional notebook is simpler and cleaner and easier to teach
• conda
  • E.g. best practices for creating environments
  • Valuable to the community, but will bring a lot of headaches. Do we want to support this?
• Git and GitHub
  • Like conda, this would be outside of notebook environments
  • Software Carpentries material is good and up to date
  • Essential for entraining new users into the open-source community
• scientific stack: numpy, matplotlib, pandas, xarray, etc.
  • Good material already in the Pangeo tutorials

What's missing from this list?

[ktyle]

Should we flesh out (i.e. enumerate) the "etc" portion of the scientific stack at this point?

[clyne]

I couldn't think of anything in the "etc" category. What did we miss?

[ktyle]

see below!

[ktyle]

I also think we should include some content related to how to use Markdown effectively (Jupyterlab has a great real-time Markdown renderer)

[brian-rose]

+1 for a simple Markdown tutorial. It shows up everywhere, including right here in this discussion engine!

This could then be cross-referenced from GitHub tutorials and elsewhere.

[clyne]

Great summary, @brian-rose . I can't think of anything that's missing. I will add that I think of first four as "basement" level that needs to be addressed before moving to the ground floor (scientific stack).

[ktyle]

Here's my take on what consists the scientific stack we would target. The order is intentional, but not etched in stone. A lot is in here, but figure the more the merrier at this point.

1. Numpy
2. Matplotlib
3. Cartopy
4. Working with date and time objects: datetime
5. Pandas
6. Meteorological data formats: NetCDF, GRIB, Zarr
7. Xarray
8. MetPy
9. GeoCAT
10. Raster and Vector (Shapefile) data: Rasterio, Rioxarray
11. Geotiffs, including cloud-optimized Geotiffs (COG): Contextily, Intake
12. Geopandas
13. Interactive visualizations: ipyleaflet, xarray-leaflet, Folium
14. Interactive visualizations: hvplot and Holoviews ecosystem
15. Interactive visualizations: Plotly
16. Dashboards: Panel, Voila, Streamlit
17. Cloud-hosted datasets and libraries: s3fs
18. Dask
19. Machine Learning: scikit-learn, pytorch, tensorflow, keras

[ktyle]

Yep, I agree ... I just wanted to try to dump as much as I could think of on there while it's occupying my single CPU in my brain :)

[ktyle]

I would venture to say that items 1-8 would be the core foundation of the scientific stack we would target.

[brian-rose]

Agreed! Although one could argue that this list is skewed toward meteorological applications, e.g. metpy and GRIB format are not relevant to oceanographers.

[clyne]

Agreed! Although one could argue that this list is skewed toward meteorological applications, e.g. metpy and GRIB format are not relevant to oceanographers.

With no disrespect to @dcamron and @dopplershift :-), I would opine that MetPy is domain specific and should not be on the same list as the other foundational topics needed for anyone to do science in the Python ecosystem.

[dopplershift]

😄 I definitely do not consider MetPy "foundational" to geoscience Python use. I'd also draw the line at 7 (xarray) on Kevin's list if we're talking about "foundational" things.

[brian-rose]

Great, so to summarize what we've got so far, a draft list of the "foundational" skills we seek to support is as follows:

• Basic Python syntax
• Jupyter environments
• markdown
• conda
• git and github
• core science packages:
  • numpy
  • matplotlib
  • cartopy
  • datetime
  • pandas
  • data formats
  • xarray

I would advocate for not getting too hung up on the specific ordering of topics. Much of the python package material will need to be cross-referenced, e.g. it's natural to discuss data formats, xarray, and mapping with cartopy together in the same example. Material that is presented as self-guided tutorials should not try to enforce too strict an ordering IMO.

[ktyle]

Sounds good to me; however, we mentioned Markdown as well yesterday ... do you want that included?

[brian-rose]

Oops, yes, I will edit my list!

[aaarendt]

Hi Everyone, I'm a data science fellow at the UW eScience Institute and I've helped design and host a series of hackweeks in the geosciences over the past few years. We're now exploring an expanded program where we assist other institutions to offer their own hackweeks for their communities. As part of that we are really interested in standardizing our tutorial content rather than reinventing the wheel for each event. I wondered if there might be opportunities to partner with your team in some capacity.

Our teams have wrestled over the years with how to provide foundational skills. Here is an example of the content we offered at our recent ICESat-2 hackweek. One thing we teach that I noticed was not yet on your list is basic Shell/command line skills.

We often categorize our content according to data type, for example vector and raster data. My colleague David Shean has designed a fantastic course Geospatial Data Analysis with Python that follows such a framework. Looking again at David's content, I wonder if GDAL should be considered a core tool.

We use Pangeo infrastructure for everything we teach and we usually find it's important to explain fundamental concepts of the cloud in our foundational tutorials.

[clyne]

Hi @aaarendt, Thanks for reaching out to us. We're still trying to grapple with the issue of how to provide foundational skills ourselves. I think we're reaching some agreement on what topics should be covered (though you are probably right that basic command line use is missing from our list), and we now need to figure out how we're going to cover them. I think we'll be discussing this topic at our next Educational Working Group meeting this Thursday. The meetings are open to anyone, and you can find meeting coordinates here. I encourage you to attend. Alternatively, we can followup afterward and possibly have a separate meeting.

[brian-rose]

Just noting here that we have also discussed "How to create a good Python package" (including tutorials on testing and CI) as part of our core skills, e.g. in #43

[brian-rose]

We now have a first draft of a JupyterBook site outlining the foundational material we intend to cover. Take a look here: https://projectpythia.org/pythia-foundations/

[brian-rose]

The source repo is https://github.com/ProjectPythia/pythia-foundations/

[ktyle]

Sweet!!

[clyne]

Looks like a great start, @brian-rose. A couple of discussion items:

1. Is the order of topics what we want? It starts with "Basic Python Syntax". Should the first topic be conda or jupyter notebooks?
2. The overall layout of the book seems reasonable to me. Maybe a good next step would be to provide a bare bones tutorial on one of the foundational topics? With the organization and style of this settled perhaps we can proceed in parallel with adding other foundation topics.

[brian-rose]

@clyne these are excellent discussion points, and I expect to devote much of our next EWG meeting to brainstorming on this.