Skip to content

Conversation

@kels271828
Copy link
Member

WHAT
Fixed issues related to custom Pipeline and Config classes when reloading from JSON files.

WHY

  • Custom pipeline objects were being reloaded as the base Pipeline class; would be missing any custom fields or methods and config would not be validated
  • Custom pipeline config objects were being reloaded as base Config class in the stage config; pipeline config fields would not be validated
    • Noticed by Peng when running a stage on Jobmon. In particular, a pipeline config item was not converted from str to Path when loaded from JSON file.

HOW

  • pipeline.py:
    • Added type and module attributes to Pipeline
    • Added __init__() function to Pipeline
  • main.py:
    • Generalized _get_stage() and _get_custom_stage() methods to work with stages and pipelines; renamed to _get_class() and _get_custom_class()
    • Updated load_pipeline to use _get_class() function rather than returning a base Pipeline object
  • config.py: Infer type of _pipeline_config rather than using a base Config object
  • advanced_usage.rst: Updated documentation related to custom Config and StageConfig classes

@kels271828 kels271828 requested review from blsmxiu47 and zhengp0 March 19, 2025 20:03
@kels271828
Copy link
Member Author

kels271828 commented Mar 19, 2025

I wasn't sure where this fix would go (i.e., 1.0.4 vs. 1.1.1), so I haven't updated the version/changelog/etc. Also still working on tests.

@kels271828 kels271828 changed the base branch from main to release/1.1 April 8, 2025 15:11
@kels271828 kels271828 requested a review from saalUW as a code owner April 8, 2025 15:11
@kels271828
Copy link
Member Author

Pulling into release/1.1 instead of as a hotfix to main

@blsmxiu47 blsmxiu47 self-requested a review April 8, 2025 18:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants