[query] remove uses of HailContext.backend

[ehigham]

This change removes uses of HailContext.backend​ as part of an effort to remove the HailContext​ singleton. Instead, the current Backend​ is accessed by threading ExecuteContext​.

This change cannot impact the Hail Batch instance as deployed by Broad Institute in GCP

[ehigham]

• [query] remove global Backend field #15107
• [query] vanquish HailContext #15104
• [query] HailContext is not required in batch #15101
• [query] remove Backend from HailContext #15100
• [query] move configureLogging and version out of HailContext #15098
• [query] move block matrix read out of HailContext #15096
• [query] remove uses of HailContext.backend  #14964 👈 (View in Graphite)
• [query] implement HailContext.checkRVDKeys via flags #15080
• [query] remove branchingFactor from HailContext #14963
• [query] Configure Optimiser via Flags, not HailContext #14952
• [query] Lowering + Optimisation with implict timing context #14731 : 2 other dependent PRs (#14732 , #14748 )
• [query] Lift backend state into (Batch|Py4J)QueryDriver #14698
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[ehigham]

Unused/implemented in BackendRpc

[chrisvittal]

I had a thought to avoid the 'hack' you put in. Fantastic work though.

[chrisvittal]

Thought, make this parameter required, and then supply it with a default from python.

This is where we construct this node in python. What if we were to get the parallelism from the branching_factor flag here?

hail/hail/python/hail/linalg/blockmatrix.py

Lines 1773 to 1827 in e899546

	@typecheck_method(n_partitions=nullable(int), maximum_cache_memory_in_bytes=nullable(int))
	def to_table_row_major(self, n_partitions=None, maximum_cache_memory_in_bytes=None):
	"""Returns a table where each row represents a row in the block matrix.
	The resulting table has the following fields:
	- **row_idx** (:py:data.`tint64`, key field) -- Row index
	- **entries** (:py:class:`.tarray` of :py:data:`.tfloat64`) -- Entries for the row
	Examples
	--------
	>>> import numpy as np
	>>> block_matrix = BlockMatrix.from_numpy(np.array([[1, 2], [3, 4], [5, 6]]), 2)
	>>> t = block_matrix.to_table_row_major()
	>>> t.show()
	+---------+---------------------+
	| row_idx | entries |
	+---------+---------------------+
	| int64 | array<float64> |
	+---------+---------------------+
	| 0 | [1.00e+00,2.00e+00] |
	| 1 | [3.00e+00,4.00e+00] |
	| 2 | [5.00e+00,6.00e+00] |
	+---------+---------------------+
	Parameters
	----------
	n_partitions : int or None
	Number of partitions of the table.
	maximum_cache_memory_in_bytes : int or None
	The amount of memory to reserve, per partition, to cache rows of the
	matrix in memory. This value must be at least large enough to hold
	one row of the matrix in memory. If this value is exactly the size of
	one row, then a partition makes a network request for every row of
	every block. Larger values reduce the number of network requests. If
	memory permits, setting this value to the size of one output
	partition permits one network request per block per partition.
	Notes
	-----
	Does not support block-sparse matrices.
	Returns
	-------
	:class:`.Table`
	Table where each row corresponds to a row in the block matrix.
	"""
	path = new_temp_file()
	if maximum_cache_memory_in_bytes and maximum_cache_memory_in_bytes > (1 << 31) - 1:
	raise ValueError(
	f'maximum_cache_memory_in_bytes must be less than 2^31 -1, was: {maximum_cache_memory_in_bytes}'
	)
	self.write(path, overwrite=True, force_row_major=True)
	reader = TableFromBlockMatrixNativeReader(path, n_partitions, maximum_cache_memory_in_bytes)
	return Table(TableRead(reader))

[patrick-schultz]

I see you get rid of this upstack, but I'm curious if there's a simple rule for when we get the backend via an ExecuteContext, vs when we still need to get a HailContext (for now, later using a different mechanism). Is it just a compile-time vs runtime distinction?

[ehigham]

I've been proceeding on the basis of avoiding "global" mutable fields entirely, instead favouring dependency injection in code that we maintain (via ExecuteContext in this case). For generated code, using a constant pool or some such is probably the right thing to do, so long as it doesn't depend on non-generated code.

For Backend specifically, my intention is that that ref in the upcoming change should only be used by BackendUtils.collectDArray . I hope is to remove the mutable ref eventually, either by

• code-generating parallelizeAndComputeWithIndex,
• by initialising a "constant" field in the generated code with either
  • the backend in a similar way to reference genomes etc
  • the BackendContext that's passed to collectDArray

I'm not sure if that answers your question properly...

[ehigham]

I might just do that last one now...

[patrick-schultz]

I'm not sure if that answers your question properly...

It does, thanks!

[ehigham]

https://app.graphite.dev/github/pr/hail-is/hail/15107

[patrick-schultz]

Great change!