[query] run ServiceBackend in Py4jQueryDriver

[ehigham]

This PR adds EXPERIMENTAL support for query-on-batch-in-batch by running the driver process locally while offloading partition computation to Batch. This differs from the traditional "batch" backend which requires an additional driver job.

By adding a new experimental feature, my hope is to commit this small proof of concept and then improve it incrementally.

Key changes:

• Added pyServiceBackend method to create a ServiceBackend from Python
• Added proper interrupt handling in ServiceBackend to cancel jobs when interrupted
• Implemented retry logic for job group creation to handle situations where multiple driver jobs create batch updates concurrently.
• Added a new experimental Python API for initializing Hail with a local driver and Batch backend

Security Assessment

This change has no impact on the hail batch service in GCP.

[ehigham]

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

• [query] Use Log4j 2 Api #15176
• [query] run ServiceBackend in Py4jQueryDriver #14984 👈 (View in Graphite)
• [query] read hail-generated cloud credentails #15160 : 1 other dependent PR (#15161 )
• [query] simplify DeployContext #15159
• [query] engoodening of the py4j clients #15158
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.