[query] remove global Backend field

[ehigham]

Change Description

This PR refactors the Backend interface to improve separation of concerns between driver and worker contexts. The key changes include:

1. Removes the static Backend.instance singleton pattern in favour of explicit context passing
2. Repurposes BackendContext​ as DriverRuntimeContext to implement mapCollectPartitions​ (formally parallelizeAndComputeWithIndex)
3. Removes canExecuteParallelTasksOnDriver​ from Backend​ - this is now an implementation detail in mapCollectPartitions​.
4. Broadcasts globals in a separate file for ServiceBackend​
5. Disable all semantic-hash code when it's not featured on

Security Assessment

This change cannot impact the Hail Batch instance as deployed by Broad Institute in GCP

[ehigham]

• [query] remove global Backend field #15107 👈 (View in Graphite)
• [query] vanquish HailContext #15104
• [query] HailContext is not required in batch #15101
• [query] remove Backend from HailContext #15100
• [query] move configureLogging and version out of HailContext #15098
• [query] move block matrix read out of HailContext #15096
• [query] remove uses of HailContext.backend  #14964
• [query] implement HailContext.checkRVDKeys via flags #15080
• [query] remove branchingFactor from HailContext #14963
• [query] Configure Optimiser via Flags, not HailContext #14952
• [query] Lowering + Optimisation with implict timing context #14731 : 2 other dependent PRs (#14732 , #14748 )
• [query] Lift backend state into (Batch|Py4J)QueryDriver #14698
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[patrick-schultz]

Can you comment on the reason for 4?

[patrick-schultz]

I like this a lot. Just have a few questions/comments, including the earlier one about the motivation for the change to globals handling.

[patrick-schultz]

I have a wip branch that introduces a pattern to avoid the copy when sorting the result of an ArraySeq builder like this (which we do somewhat often). I made a note to come back and use it here too.

[ehigham]

One reason to upload globals separately is if globals is large, we'd get increased parallelism from writing/reading those while writing/reading the PartitionFn.

Subjectively, if I serialised the globals into the PartitionFn, the worker would invoke it with an empty byte array as the globals. Maybe that's fine but feels a little bit of a gotcha where we have to remember that the driver handles serialisation that way. I'm probably over thinking it.

[patrick-schultz]

For large enough files, we should be getting the optimal amount of parallelism by chunking up the file, so I don't think manually breaking it up should have much effect. But I also can't think of any negatives to writing the globals separately, so I'm fine with leaving it. It might pave the way for someday having each partition only read in the broadcasted values they actually need, as spark does.

[ehigham]

Are we doing any chunking currently?

[patrick-schultz]

I don't know for sure, but if this is the same code we use to localize inputs in batch jobs, then yes, that chunking is what has been causing timeout issues (because we're too eager to timeout and retry chunks).

[ehigham]

That chunking happens in the python fs implementation. From a quick glance at our
scala FSs, i'm not sure that we do.

[chrisvittal]

Hurray for removing another bit of global state. Thanks for doing this.