Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Move file virtualization in toil-wdl-runner to task boundaries #5028

Open
wants to merge 41 commits into
base: master
Choose a base branch
from
Open

Move file virtualization in toil-wdl-runner to task boundaries #5028

wants to merge 41 commits into from

Commits on Jul 19, 2024

  1. Defer file virtualization to task boundaries and consolidate decl eva… 

    …luation. Also gets rid of monkeypatching in favor of a manual function call
    @stxue1
    stxue1 committed Jul 19, 2024
    Configuration menu
    Copy the full SHA
    89c276a View commit details
    Browse the repository at this point in the history

Commits on Jul 24, 2024

  1. Implement support for importing relative URL paths; import files at s… 

    …tartup and carry through mappings
    @stxue1
    stxue1 committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    c85f173 View commit details
    Browse the repository at this point in the history

  2. Fix possible invalid lookup and don't import raw URLs

    @stxue1
    stxue1 committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    230efa0 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'master' of github.com:DataBiosphere/toil into issues/50… 

    …04-wdl-virtualize-only-at-task-boundaries
    @stxue1
    stxue1 committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    2955bc3 View commit details
    Browse the repository at this point in the history

  4. Get rid of sentinel value implementation + drop files before virtuali… 

    …zation in TaskWrapper for MiniWDL parity
    @stxue1
    stxue1 committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    066f780 View commit details
    Browse the repository at this point in the history

Commits on Jul 25, 2024

  1. Drop missing files during decl eval for outputs + Add a check for inv… 

    …alid coerced-to-null files and raise if exception found
    @stxue1
    stxue1 committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    d5365a2 View commit details
    Browse the repository at this point in the history

  2. Deal with mypy

    @stxue1
    stxue1 committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    bbb098d View commit details
    Browse the repository at this point in the history

  3. Don't drop unnecesssarily

    @stxue1
    stxue1 committed Jul 25, 2024
    Configuration menu
    Copy the full SHA
    1b6bd02 View commit details
    Browse the repository at this point in the history

Commits on Aug 20, 2024

  1. Switch to setattr implementation

    @stxue1
    stxue1 committed Aug 20, 2024
    Configuration menu
    Copy the full SHA
    f65d59c View commit details
    Browse the repository at this point in the history

  2. Fix overwriting files

    @stxue1
    stxue1 committed Aug 20, 2024
    Configuration menu
    Copy the full SHA
    be0203e View commit details
    Browse the repository at this point in the history

Commits on Aug 22, 2024

  1. Fix prototype implementation

    @stxue1
    stxue1 committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    f60d475 View commit details
    Browse the repository at this point in the history

Commits on Aug 23, 2024

  1. Merge branch 'master' of github.com:DataBiosphere/toil into issues/50… 

    …04-wdl-virtualize-only-at-task-boundaries
    @stxue1
    stxue1 committed Aug 23, 2024
    Configuration menu
    Copy the full SHA
    552ac74 View commit details
    Browse the repository at this point in the history

Commits on Aug 27, 2024

  1. Merge master into issues/5004-wdl-virtualize-only-at-task-boundaries

    @github-actions
    github-actions[bot] committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    8163c2c View commit details
    Browse the repository at this point in the history

  2. Fix documentation

    @stxue1
    stxue1 committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    4bbe63d View commit details
    Browse the repository at this point in the history

  3. Merge branch 'issues/5004-wdl-virtualize-only-at-task-boundaries' of … 

    …github.com:DataBiosphere/toil into issues/5004-wdl-virtualize-only-at-task-boundaries
    @stxue1
    stxue1 committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    05dc240 View commit details
    Browse the repository at this point in the history

Commits on Aug 29, 2024

  1. Resolve nested virtualize files issue by converting back to original … 

    …and permanent devirtualization of URLs
    @stxue1
    stxue1 committed Aug 29, 2024
    Configuration menu
    Copy the full SHA
    a6b18c7 View commit details
    Browse the repository at this point in the history

  2. Fix virtualization of URLs that aren't toil URIs

    @stxue1
    stxue1 committed Aug 29, 2024
    Configuration menu
    Copy the full SHA
    94da9af View commit details
    Browse the repository at this point in the history

  3. Mypy

    @stxue1
    stxue1 committed Aug 29, 2024
    Configuration menu
    Copy the full SHA
    644fd51 View commit details
    Browse the repository at this point in the history

  4. Merge branch 'master' of github.com:DataBiosphere/toil into issues/50… 

    …04-wdl-virtualize-only-at-task-boundaries
    @stxue1
    stxue1 committed Aug 29, 2024
    Configuration menu
    Copy the full SHA
    d5bedee View commit details
    Browse the repository at this point in the history

  5. Remove documentation that no longer applies

    @stxue1
    stxue1 committed Aug 29, 2024
    Configuration menu
    Copy the full SHA
    f8c0ef1 View commit details
    Browse the repository at this point in the history

  6. Fix merge conflict issue

    @stxue1
    stxue1 committed Aug 29, 2024
    Configuration menu
    Copy the full SHA
    6365b34 View commit details
    Browse the repository at this point in the history

  7. Enable symlink conformance test

    @stxue1
    stxue1 committed Aug 29, 2024
    Configuration menu
    Copy the full SHA
    0a35c53 View commit details
    Browse the repository at this point in the history

  8. whitespace

    @stxue1
    stxue1 committed Aug 29, 2024
    Configuration menu
    Copy the full SHA
    ce21051 View commit details
    Browse the repository at this point in the history

Commits on Aug 31, 2024

  1. Fix virtualize/devirtualize_filename to be called only during command… 

    … stdlibs and combine_bindings issue with new virtualization representation
    @stxue1
    stxue1 committed Aug 31, 2024
    Configuration menu
    Copy the full SHA
    343d6d8 View commit details
    Browse the repository at this point in the history

Commits on Sep 3, 2024

  1. Merge master into issues/5004-wdl-virtualize-only-at-task-boundaries

    @github-actions
    github-actions[bot] committed Sep 3, 2024
    Configuration menu
    Copy the full SHA
    f5f3eea View commit details
    Browse the repository at this point in the history

Commits on Sep 6, 2024

  1. Add documentation and make convert_files function import greedily

    @stxue1
    stxue1 committed Sep 6, 2024
    Configuration menu
    Copy the full SHA
    37efd55 View commit details
    Browse the repository at this point in the history

Commits on Sep 10, 2024

  1. Get rid of memoization as File is not hashable (I believe inside LRU)

    @stxue1
    stxue1 committed Sep 10, 2024
    Configuration menu
    Copy the full SHA
    491ba1b View commit details
    Browse the repository at this point in the history

  2. Merge master into issues/5004-wdl-virtualize-only-at-task-boundaries

    @github-actions
    github-actions[bot] committed Sep 10, 2024
    Configuration menu
    Copy the full SHA
    8569a81 View commit details
    Browse the repository at this point in the history

  3. Merge master into issues/5004-wdl-virtualize-only-at-task-boundaries

    @github-actions
    github-actions[bot] committed Sep 10, 2024
    Configuration menu
    Copy the full SHA
    731120d View commit details
    Browse the repository at this point in the history

Commits on Sep 12, 2024

  1. Merge master into issues/5004-wdl-virtualize-only-at-task-boundaries

    @github-actions
    github-actions[bot] committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    b25c80d View commit details
    Browse the repository at this point in the history

Commits on Sep 16, 2024

  1. Merge master into issues/5004-wdl-virtualize-only-at-task-boundaries

    @github-actions
    github-actions[bot] committed Sep 16, 2024
    Configuration menu
    Copy the full SHA
    ffb2e08 View commit details
    Browse the repository at this point in the history

  2. Merge master into issues/5004-wdl-virtualize-only-at-task-boundaries

    @github-actions
    github-actions[bot] committed Sep 16, 2024
    Configuration menu
    Copy the full SHA
    7f17413 View commit details
    Browse the repository at this point in the history

  3. Merge master into issues/5004-wdl-virtualize-only-at-task-boundaries

    @github-actions
    github-actions[bot] committed Sep 16, 2024
    Configuration menu
    Copy the full SHA
    82383ab View commit details
    Browse the repository at this point in the history

Commits on Sep 17, 2024

  1. Apply suggestions from code review 

    Co-authored-by: Adam Novak <[email protected]>
    @stxue1 @adamnovak
    stxue1 and adamnovak committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    a7292be View commit details
    Browse the repository at this point in the history

Commits on Sep 18, 2024

  1. Update src/toil/wdl/wdltoil.py 

    Co-authored-by: Adam Novak <[email protected]>
    @stxue1 @adamnovak
    stxue1 and adamnovak committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    e6718cd View commit details
    Browse the repository at this point in the history

  2. Rename, add comments, remove unused code/comments

    @stxue1
    stxue1 committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    2ad130f View commit details
    Browse the repository at this point in the history

  3. Merge branch 'issues/5004-wdl-virtualize-only-at-task-boundaries' of … 

    …github.com:DataBiosphere/toil into issues/5004-wdl-virtualize-only-at-task-boundaries
    @stxue1
    stxue1 committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    cb8b230 View commit details
    Browse the repository at this point in the history

  4. Add comments and adjust wdl context usage

    @stxue1
    stxue1 committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    b0db027 View commit details
    Browse the repository at this point in the history

Commits on Sep 19, 2024

  1. add namespace

    @stxue1
    stxue1 committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    8c24d8f View commit details
    Browse the repository at this point in the history

  2. integrate namespace into wdl_context

    @stxue1
    stxue1 committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    666aef5 View commit details
    Browse the repository at this point in the history

  3. properly name wdl value bases

    @stxue1
    stxue1 committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    0b6fb82 View commit details
    Browse the repository at this point in the history