Skip to content

Improve lidar data osm bridges #1787

Draft
AliForghani-NOAA wants to merge 11 commits intodevfrom
dev-lidar-bridge-upgrade
Draft

Improve lidar data osm bridges #1787
AliForghani-NOAA wants to merge 11 commits intodevfrom
dev-lidar-bridge-upgrade

Conversation

@AliForghani-NOAA
Copy link
Copy Markdown
Collaborator

@AliForghani-NOAA AliForghani-NOAA commented Mar 17, 2026

[Short description explaining the high-level reason for the pull request]

Adding pdal to docker image

Updated the lidar bridge workflow to run PDAL from the standard Docker image instead of requiring a separate conda environment. In data/bridges/make_rasters_using_lidar.py, the two PDAL execution points were changed from Python pdal bindings to the PDAL CLI by writing the pipeline JSON to a temporary file and invoking pdal pipeline with subprocess. The pipeline definitions themselves were unchanged.

Docker image changes:

  • Added a dedicated PDAL runtime inside the image using micromamba, installed at /opt/pdal-env.
  • Installed pdal=2.9.3 in that isolated environment, along with matching gdal, proj, and proj-data.
  • Kept the main container Python and geospatial stack unchanged to avoid interfering with existing GDAL/geopandas behavior.
  • Exposed the isolated PDAL binary through environment variables, using PDAL_CLI_PATH=/opt/pdal-env/bin/pdal and PDAL_ENV_ROOT=/opt/pdal-env.

How make_rasters_using_lidar.py uses that environment:

  • The script now runs PDAL through the CLI with subprocess.run(...) instead of relying on Python PDAL bindings.
  • Before invoking PDAL, the script sets PROJ_LIB, PROJ_DATA, and GDAL_DATA only for the PDAL subprocess so it uses the matching runtime data in /opt/pdal-env.
  • This allows the workflow to use the newer PDAL installation without affecting the rest of the container environment.

I also added laspy to Pipfile and Pipfile.lock so the image includes the Python dependency needed by the bridge raster workflow.

Additions

Changes

Removals


Testing

Generally, you do not copy this part into the ChangeLog. These are some quick notes on what you did test and/or notes for the reviewer to help with their review testing.


Deployment Plan (For FIM developers use)

  • Does the change impact inputs, docker or python packages?

    • Yes
    • No (f no.. skip the rest of the Deployment Plan section)
  • If you are not a FIM dev team member: Please let us know what you need and we can help with it.

  • If you are a FIM Dev team member:

    • Please work with the DevOps team and do not just go ahead and do it without some co-ordination.

    • Copy where you can, assign where you can not, and it is your responsibility to ensure it is done. Please ensure it is completed before the PR is merged.

    • Has new or updated python packages, PipFile, Pipefile.lock or Dockerfile changes? DevOps can help or take care of it if you want. Just need to know if it is required.

      • Yes
      • No
    • Require new or adjusted data inputs? Does it have a way to version (folder or file dates)?

      • No
      • Yes
        • Require new pre-clip set or any other data reloads, such as DEMS, osm, etc. ie.. pre-requisite re-data upstream of your input changes.
          • Yes
          • No
        • Has the inputs been copied/exist in all four enviros:
          • FIM EFS
          • FIM S3
          • ESIP
          • Dev1
  • Please use caution in removing older version unless it is at least two versions ago. Confirm with DevOps if cleanup might be involved.

  • If new or updated data sets, has the FIM code, including running fim_pipeline.sh, been updated and tested with the new/adjusted data? You can dev test against subsets if you like.

    • Yes

Notes to DevOps Team or others:

Please add any notes that are helpful for us to make sure it is all done correctly. Do not put actual server names or full true paths, just shortcut paths like 'efs..../inputs/, or 'dev1....inputs', etc.


Issuer Checklist (For developer use)

You may update this checklist before and/or after creating the PR. If you're unsure about any of them, please ask, we're here to help! These items are what we are going to look for before merging your code.

  • Informative and human-readable title, using the format: [_pt] PR: <description>
  • Links are provided if this PR resolves an issue, or depends on another other PR
  • If submitting a PR to the dev branch (the default branch), you have a descriptive Feature Branch name using the format: dev-<description-of-change> (e.g. dev-revise-levee-masking)
  • Changes are limited to a single goal (no scope creep)
  • The feature branch you're submitting as a PR is up to date (merged) with the latest dev branch
  • pre-commit hooks were run locally
  • Any change in functionality is tested
  • New functions are documented (with a description, list of inputs, and expected output)
  • Placeholder code is flagged / future todos are captured in comments
  • CHANGELOG updated with template version number, e.g. 4.x.x.x
  • Add yourself as an assignee in the PR as well as the FIM Technical Lead
  • Where applicable, has fim_pipeline been tested with muliple HUCs, including some other unaffected HUCs?

Reviewer / Approver Checklist

  • Where applicable, has fim_pipeline been tested with muliple HUCs, including some other unaffected HUCs?
  • If there are new inputs, have you confirmed that they have been copied to all enviroments?

Merge Checklist (For Technical Lead use only)

  • Update CHANGELOG with latest version number and merge date
  • Update the Citation.cff file to reflect the latest version number in the CHANGELOG
  • If applicable, update README with major alterations

@AliForghani-NOAA AliForghani-NOAA self-assigned this Mar 17, 2026
@AliForghani-NOAA AliForghani-NOAA added the enhancement New feature or request label Mar 17, 2026
@RobHanna-NOAA RobHanna-NOAA added the Inputs For getting new data, updating data, changing inputs, etc label Apr 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request Inputs For getting new data, updating data, changing inputs, etc

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants