[BUG]: The symbol table requires protection for thread-safety

[pentschev]

Is this a duplicate?

• I confirmed there appear to be no duplicate issues for this bug and that I agree to the Code of Conduct

Type of Bug

Runtime Error

Component

cuda.bindings

Describe the bug

Multi-threaded applications may attempt to access the symbol table simultaneously before it's populated and cause concurrency issues, as observed previously.

Sample failure stack

/opt/conda/envs/test/lib/python3.12/site-packages/numba_cuda/numba/cuda/cudadrv/devices.py:125: in ensure_context
    with driver.get_active_context():
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^
/opt/conda/envs/test/lib/python3.12/site-packages/numba_cuda/numba/cuda/cudadrv/driver.py:539: in __enter__
    hctx = driver.cuCtxGetCurrent()
           ^^^^^^^^^^^^^^^^^^^^^^^^
/opt/conda/envs/test/lib/python3.12/site-packages/numba_cuda/numba/cuda/cudadrv/driver.py:396: in safe_cuda_api_call
    return self._check_cuda_python_error(fname, libfn(*args))
                                                ^^^^^^^^^^^^
cuda/bindings/driver.pyx:20135: in cuda.bindings.driver.cuCtxGetCurrent
    ???
cuda/bindings/cydriver.pyx:107: in cuda.bindings.cydriver.cuCtxGetCurrent
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

>   ???
E   RuntimeError: Function "cuCtxGetCurrent" not found

This should have been already addressed by #835.

How to Reproduce

No known trivial reproducer, consistently observed with this test from RapidsMPF and numba-cuda=0.18.1 (required to ensure it's using cuda-bindings, instead of its own ctypes implementation):

Reproducer

@pytest.mark.parametrize("partition_count", [None, 3])
@pytest.mark.parametrize("sort", [True, False])
@pytest.mark.parametrize("cluster_kind", ["auto", "single"])
def test_dask_cudf_integration_single(
    partition_count: int,
    sort: bool,  # noqa: FBT001
    cluster_kind: Literal["distributed", "single", "auto"],
) -> None:
    # Test single-worker cuDF integration with Dask-cuDF
    pytest.importorskip("dask_cudf")

    df = (
        dask.datasets.timeseries(
            freq="3600s",
            partition_freq="2D",
        )
        .reset_index(drop=True)
        .to_backend("cudf")
    )
    partition_count_in = df.npartitions
    expect = df.compute().sort_values(["id", "name", "x", "y"])
    shuffled = dask_cudf_shuffle(
        df,
        ["id", "name"],
        sort=sort,
        partition_count=partition_count,
        cluster_kind=cluster_kind,
        config_options=Options({"single_spill_device": "0.1"}),
    )
    assert shuffled.npartitions == (partition_count or partition_count_in)
    got = shuffled.compute()
    if sort:
        assert got["id"].is_monotonic_increasing
    got = got.sort_values(["id", "name", "x", "y"])

    dd.assert_eq(expect, got, check_index=False)

Expected behavior

Attempting to create a CUDA context from multiple threads should not cause an error.

Operating System

No response

nvidia-smi output

No response

[leofang]

Thanks, Peter! Marked as closed by #835.