Roadmap

This page captures the goals of the Toil workflow engine project. These are divided into three categories based on when they will be addressed. Given that the project advances thanks to a team of mostly students and external collaborators, these are only rough estimates.

Feedback is welcome. Every effort will be made to put resources on any critical goals.

Near Term (In Progress)

• Basic WDL support.
• Python 3 support.
• Travis CI.
• Ansible provisioning (Azure and AWS).
• Run Toil within Popper (https://cross.ucsc.edu/tag/popper/).
• Grafana for workflow monitoring

Medium Term (~ 6-month goals)

• Improve the user experience.
  • Change the thread pool model to improve single machine usage.
  • Better feedback (error messages, logging).
• JobStores
  • Update the Azure jobStore.
  • Finish Google (GCP)
  • Add OpenStack
• Batch systems
  • DRMAA
  • Google Pipelines
  • Azure Batch
  • AWS Batch
• Improve the development process.
  • Add a linter
  • Add a code coverage tool.
  • Organize tests.
  • Better access to tests for external developers.
• Better track versions of specifications (e.g. CWL, WDL) and dependencies.
• Add other provisioners: GCP, OpenStack

Longer Term

• Singularity support.
• Uniform configuration (i.e. not just environment variables).
• Add management and monitoring UIs.
• TES support
• WES Support

Finished

• Improve the documentation.
• Improve the release process.
• Bring CWL support up to 100%.
• Better local testing.
• Using Docker API
• Turn off or remove forking to enable easier debugging.