Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[enhancement] add dlpack support to to_table #2275

Open
wants to merge 142 commits into
base: main
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
142 commits
Select commit Hold shift + click to select a range
939141e
start from #2195
icfaust Jan 27, 2025
ae038cc
add apache license
icfaust Jan 27, 2025
df87e74
add files from #1568
icfaust Jan 27, 2025
a6d364b
start converting over
icfaust Jan 27, 2025
eabbbed
attempts to fix copyright checker
icfaust Jan 27, 2025
bf459de
remove table_metadata
icfaust Jan 27, 2025
941a8ad
Merge branch 'uxlfoundation:main' into dev/dlpack_integration
icfaust Jan 27, 2025
ce6de53
merge
icfaust Jan 27, 2025
4caedc7
weird merge
icfaust Jan 27, 2025
009a19c
renaming
icfaust Jan 27, 2025
ccc3534
change location
icfaust Jan 27, 2025
3a6c3db
will implement these elsewhere
icfaust Jan 27, 2025
7c36c57
move files to follow naming
icfaust Jan 27, 2025
79496da
change headers further
icfaust Jan 27, 2025
311e4bf
Merge remote-tracking branch 'sklearnex/main' into dev/dlpack_integra…
icfaust Jan 27, 2025
09b6c7c
Merge branch 'uxlfoundation:main' into dev/dlpack_integration
icfaust Jan 27, 2025
5f92352
interim standpoint which will fail
icfaust Jan 27, 2025
1d1076d
interim changes
icfaust Jan 28, 2025
c4461d5
helper -> utils
icfaust Jan 28, 2025
39243b9
move macro to a central spot
icfaust Jan 28, 2025
05a1c15
remove whitespace
icfaust Jan 28, 2025
559a127
commit before merge
icfaust Jan 28, 2025
ba98166
Merge branch 'main' into dev/dlpack_integration
icfaust Jan 28, 2025
ca88f51
current status
icfaust Jan 28, 2025
db1fe9d
interim
icfaust Jan 28, 2025
9b318d0
remove and format
icfaust Jan 28, 2025
5da5f41
more fixes
icfaust Jan 28, 2025
827304d
add fixes
icfaust Jan 28, 2025
d69fd2b
add fixes
icfaust Jan 28, 2025
6f63c15
more fixes
icfaust Jan 28, 2025
e980f84
add fixes
icfaust Jan 28, 2025
d649537
more fixes
icfaust Jan 28, 2025
1893516
more fixes
icfaust Jan 28, 2025
99076a4
updates
icfaust Jan 28, 2025
a6a83d3
updates
icfaust Jan 28, 2025
6076ed2
more fixes
icfaust Jan 28, 2025
7e7aa5b
formatting
icfaust Jan 28, 2025
92c3ae3
formatting
icfaust Jan 28, 2025
b5929f8
formatting
icfaust Jan 28, 2025
7c157df
changes
icfaust Jan 28, 2025
1e9ebc4
move header include
icfaust Jan 28, 2025
a9e83f2
fix tensor issue
icfaust Jan 28, 2025
16ea427
type change
icfaust Jan 28, 2025
29f3565
move literals to see if it helps
icfaust Jan 28, 2025
6940536
status
icfaust Jan 28, 2025
6c1a246
remove inline
icfaust Jan 28, 2025
c522ada
move ordering in table.cpp
icfaust Jan 28, 2025
4fd71a6
missing whitespace
icfaust Jan 28, 2025
2da618b
change macro section
icfaust Jan 28, 2025
dcac239
add some commentary
icfaust Jan 28, 2025
698a7cb
attempt to shorten with a macro
icfaust Jan 29, 2025
dc2c13e
first fixes for array_api_strict
icfaust Jan 29, 2025
9db52f7
formatting
icfaust Jan 29, 2025
7c22a2f
remove unneeded code
icfaust Jan 29, 2025
b9dbaa0
remove unneeded code
icfaust Jan 29, 2025
d0c1c40
oops
icfaust Jan 29, 2025
827fefd
make better logic
icfaust Jan 29, 2025
817dfd1
attempt to include byte_offset
icfaust Jan 29, 2025
0ed6ed4
convert to static_cast entirely
icfaust Jan 29, 2025
4a0aa9d
back to reinterpret_cast
icfaust Jan 29, 2025
8720ab1
add initial tests
icfaust Jan 29, 2025
2fa5929
remove array api from contiguous test
icfaust Jan 29, 2025
1c3aca6
hide behind if statement, definite a TODO
icfaust Jan 29, 2025
12a1f56
working on test case failures
icfaust Jan 29, 2025
23d3222
switch to length_error
icfaust Jan 29, 2025
1ee3c19
Update test_data.py
icfaust Jan 29, 2025
61d8044
Update test_data.py
icfaust Jan 29, 2025
065a0e2
Update test_data.py
icfaust Jan 29, 2025
387c7e3
retest with dpnp and dpctl
icfaust Jan 30, 2025
0355103
fix stride counting
icfaust Jan 30, 2025
20db18d
begin testing strategy change
icfaust Jan 30, 2025
80ac3ad
add tests
icfaust Jan 30, 2025
512d8ee
fix mistakes in test
icfaust Jan 30, 2025
c5bde34
move class out of if statement
icfaust Jan 30, 2025
0a7678c
Update test_validation.py
icfaust Jan 30, 2025
80255dd
remove pandas
icfaust Jan 31, 2025
6b318c9
missing change
icfaust Jan 31, 2025
0691f6e
add initial memory leak checking
icfaust Jan 31, 2025
0399e0c
remove get_namespace
icfaust Jan 31, 2025
d77c922
remove dpc backend skip
icfaust Jan 31, 2025
617b322
fix issue in get_namespace change
icfaust Jan 31, 2025
3f6e8c9
further fixes
icfaust Jan 31, 2025
c0d356b
attempt again
icfaust Jan 31, 2025
81cbc51
Merge branch 'uxlfoundation:main' into dev/dlpack_integration
icfaust Feb 2, 2025
826b2a4
address changes
icfaust Feb 3, 2025
25bc110
add recursion block on suggestion
icfaust Feb 4, 2025
3431c7a
Update doc/third-party-programs-sklearnex.txt
icfaust Feb 5, 2025
52af11c
add testing for emptys and simple types
icfaust Feb 5, 2025
60c4cd4
rewrite test
icfaust Feb 5, 2025
7629706
attempt to solve empty and dlpack scalars
icfaust Feb 6, 2025
77bcb09
deal with odd scenario
icfaust Feb 6, 2025
3db0c5b
fix some tests
icfaust Feb 6, 2025
93f52fe
attempt at making things consistent
icfaust Feb 6, 2025
21514f5
oops
icfaust Feb 6, 2025
8c05300
try again
icfaust Feb 6, 2025
b3667ca
try to swap
icfaust Feb 6, 2025
72b146c
Update test_data.py
icfaust Feb 6, 2025
e90ba16
Update dlpack_utils.cpp
icfaust Feb 7, 2025
a372d0a
Update test_data.py
icfaust Feb 7, 2025
a66f852
Update test_data.py
icfaust Feb 7, 2025
aed0904
Update test_data.py
icfaust Feb 7, 2025
c31450e
Update data_conversion.cpp
icfaust Feb 7, 2025
dabf223
Update data_conversion.cpp
icfaust Feb 7, 2025
aab194b
try again
icfaust Feb 7, 2025
e2d2cd3
bad logic correction
icfaust Feb 7, 2025
bbb1ab1
make consistent
icfaust Feb 7, 2025
d821edc
std -> py
icfaust Feb 7, 2025
cd395c8
missing bracket
icfaust Feb 7, 2025
817dceb
additional fixes
icfaust Feb 7, 2025
2cf1339
further homogenation
icfaust Feb 7, 2025
69a9740
try again
icfaust Feb 7, 2025
f9602e4
fix dlpack again
icfaust Feb 7, 2025
49e3bf7
set a ticket up to solve this edge case
icfaust Feb 7, 2025
129797a
Update test_data.py
icfaust Feb 7, 2025
1f61b02
Update test_data.py
icfaust Feb 7, 2025
05b66fc
Update dtype_conversion.cpp
icfaust Feb 7, 2025
524378f
Update data_conversion.cpp
icfaust Feb 7, 2025
f88be42
Update data_conversion.cpp
icfaust Feb 7, 2025
b661373
Update data_conversion.cpp
icfaust Feb 7, 2025
d1a88b3
Update dtype_conversion.cpp
icfaust Feb 7, 2025
376d125
Update data_conversion.cpp
icfaust Feb 7, 2025
48753d7
Update test_data.py
icfaust Feb 7, 2025
01ae64a
Update test_data.py
icfaust Feb 7, 2025
0655382
Update test_data.py
icfaust Feb 8, 2025
32cef43
Update onedal/datatypes/dlpack/dlpack_utils.cpp
icfaust Feb 18, 2025
6f92764
Update onedal/datatypes/dlpack/data_conversion.cpp
icfaust Feb 18, 2025
7062b0b
Update onedal/datatypes/sycl_usm/dtype_conversion.cpp
icfaust Feb 18, 2025
6d47241
Update test_data.py
icfaust Feb 18, 2025
de8bfb0
Update test_data.py
icfaust Feb 18, 2025
aacb86e
Update test_data.py
icfaust Feb 18, 2025
4eb1817
Update test_data.py
icfaust Feb 18, 2025
52db1fc
Update dlpack_utils.cpp
icfaust Feb 19, 2025
938d545
Update dlpack_utils.cpp
icfaust Feb 19, 2025
d3a94f8
Update data_conversion.cpp
icfaust Feb 19, 2025
a468aad
Update data_conversion.cpp
icfaust Feb 19, 2025
01e419b
Update data_conversion.cpp
icfaust Feb 19, 2025
754728e
Update data_conversion.cpp
icfaust Feb 19, 2025
159fba3
clang formatting
icfaust Feb 19, 2025
173d98b
fight segfault with dpctl
icfaust Feb 20, 2025
7f01aea
Update data_conversion.cpp
icfaust Feb 20, 2025
c102f57
formatting
icfaust Feb 20, 2025
164f90c
Update test_data.py
icfaust Feb 20, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
add initial tests
icfaust committed Jan 29, 2025

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
commit 8720ab12be416b96f40361fcfae179b02bb54a8e
40 changes: 27 additions & 13 deletions onedal/datatypes/tests/test_data.py
Original file line number Diff line number Diff line change
@@ -239,7 +239,7 @@ def test_conversion_to_table(dtype):
)
@pytest.mark.parametrize("order", ["C", "F"])
@pytest.mark.parametrize("dtype", [np.float32, np.float64, np.int32, np.int64])
def test_input_sua_iface_zero_copy(dataframe, queue, order, dtype):
def test_input_zero_copy_sycl_usm(dataframe, queue, order, dtype):
"""Checking that values ​​representing USM allocations `__sycl_usm_array_interface__`
are preserved during conversion to onedal table.
"""
@@ -276,7 +276,7 @@ def test_input_sua_iface_zero_copy(dataframe, queue, order, dtype):
@pytest.mark.parametrize("order", ["F", "C"])
@pytest.mark.parametrize("data_shape", data_shapes)
@pytest.mark.parametrize("dtype", [np.float32, np.float64])
def test_table_conversions(dataframe, queue, order, data_shape, dtype):
def test_table_conversions_sycl_usm(dataframe, queue, order, data_shape, dtype):
"""Checking that values ​​representing USM allocations `__sycl_usm_array_interface__`
are preserved during conversion to onedal table and from onedal table to
sycl usm array dataformat.
@@ -331,7 +331,7 @@ def test_table_conversions(dataframe, queue, order, data_shape, dtype):
"dataframe,queue", get_dataframes_and_queues("dpctl,dpnp", "cpu,gpu")
)
@pytest.mark.parametrize("data_shape", unsupported_data_shapes)
def test_sua_iface_interop_invalid_shape(dataframe, queue, data_shape):
def test_interop_invalid_shape_sycl_usm(dataframe, queue, data_shape):
X = np.zeros(data_shape)
X = _convert_to_dataframe(X, sycl_queue=queue, target_df=dataframe)
sua_iface, _, _ = _get_sycl_namespace(X)
@@ -358,7 +358,7 @@ def test_sua_iface_interop_invalid_shape(dataframe, queue, data_shape):
pytest.param(np.uint64, id=np.dtype(np.uint64).name),
],
)
def test_sua_iface_interop_unsupported_dtypes(dataframe, queue, dtype):
def test_interop_unsupported_dtypes_sycl_usm(dataframe, queue, dtype):
# sua iface interobility supported only for oneDAL supported dtypes
# for input data: int32, int64, float32, float64.
# Checking some common dtypes supported by dpctl, dpnp for exception
@@ -373,7 +373,7 @@ def test_sua_iface_interop_unsupported_dtypes(dataframe, queue, dtype):


@pytest.mark.parametrize(
"dataframe,queue", get_dataframes_and_queues("numpy,dpctl,dpnp", "cpu,gpu")
"dataframe,queue", get_dataframes_and_queues("numpy,dpctl,dpnp,array_api", "cpu,gpu")
)
def test_to_table_non_contiguous_input(dataframe, queue):
if dataframe in "dpnp,dpctl" and not _is_dpc_backend:
@@ -396,7 +396,7 @@ def test_to_table_non_contiguous_input(dataframe, queue):
"dataframe,queue", get_dataframes_and_queues("dpctl,dpnp", "cpu,gpu")
)
@pytest.mark.parametrize("dtype", [np.float32, np.float64])
def test_sua_iface_interop_if_no_dpc_backend(dataframe, queue, dtype):
def test_interop_if_no_dpc_backend_sycl_usm(dataframe, queue, dtype):
X = np.zeros((10, 20), dtype=dtype)
X = _convert_to_dataframe(X, sycl_queue=queue, target_df=dataframe)
sua_iface, _, _ = _get_sycl_namespace(X)
@@ -406,12 +406,7 @@ def test_sua_iface_interop_if_no_dpc_backend(dataframe, queue, dtype):
to_table(X)


@pytest.mark.skipif(
not _is_dpc_backend, reason="Requires DPC backend for dtype conversion"
)
@pytest.mark.parametrize("dtype", [np.float32, np.float64])
@pytest.mark.parametrize("sparse", [True, False])
def test_low_precision_gpu_conversion(dtype, sparse):
def _test_low_precision_gpu_conversion(dtype, sparse, dataframe):
# Use a dummy queue as fp32 hardware is not in public testing

class DummySyclQueue:
@@ -428,7 +423,9 @@ class DummySyclDevice:
if sparse:
X = sp.random(100, 100, format="csr", dtype=dtype)
else:
X = np.random.rand(100, 100).astype(dtype)
X = _convert_to_dataframe(
np.random.rand(100, 100).astype(dtype), target_df=dataframe
)

if dtype == np.float64:
with pytest.warns(
@@ -444,6 +441,23 @@ class DummySyclDevice:
assert_allclose(X, from_table(X_table))


@pytest.mark.skipif(
not _is_dpc_backend, reason="Requires DPC backend for dtype conversion"
)
@pytest.mark.parametrize("dtype", [np.float32, np.float64])
@pytest.mark.parametrize("sparse", [True, False])
def test_low_precision_gpu_conversion_numpy(dtype, sparse):
_test_low_precision_gpu_conversion(dtype, sparse, "numpy")


@pytest.mark.skipif(
not _is_dpc_backend, reason="Requires DPC backend for dtype conversion"
)
@pytest.mark.parametrize("dtype", [np.float32, np.float64])
def test_low_precision_gpu_conversion_array_api(dtype):
_test_low_precision_gpu_conversion(dtype, False, "array_api")


@pytest.mark.parametrize("X", [None, 5, "test", True, [], np.pi, lambda: None])
@pytest.mark.parametrize("queue", get_queues())
def test_non_array(X, queue):

Unchanged files with check annotations Beta

return res_table;
}
dal::table convert_to_table(py::object inp_obj, py::object queue) {
dal::table res;
PyObject *obj = inp_obj.ptr();
if (obj == nullptr || obj == Py_None) {
return res;
}
#ifdef ONEDAL_DATA_PARALLEL
if (!queue.is(py::none()) && !queue.attr("sycl_device").attr("has_aspect_fp64").cast<bool>() &&
hasattr(inp_obj, "dtype")) {
// If the queue exists, doesn't have the fp64 aspect, and the data is float64
// then cast it to float32
int type = reinterpret_cast<PyArray_Descr *>(inp_obj.attr("dtype").ptr())->type_num;
if (type == NPY_DOUBLE || type == NPY_DOUBLELTR) {
PyErr_WarnEx(
PyExc_RuntimeWarning,
"Data will be converted into float32 from float64 because device does not support it",
1);
// use astype instead of PyArray_Cast in order to support scipy sparse inputs
inp_obj = inp_obj.attr("astype")(py::dtype::of<float>());
res = convert_to_table(
inp_obj); // queue will be set to none, as this check is no longer necessary
return res;
}
}
#endif // ONEDAL_DATA_PARALLEL
if (is_array(obj)) {
PyArrayObject *ary = reinterpret_cast<PyArrayObject *>(obj);
if (!PyArray_ISCARRAY_RO(ary) && !PyArray_ISFARRAY_RO(ary)) {
// NOTE: this will make a C-contiguous deep copy of the data
// this is expected to be a special case
obj = reinterpret_cast<PyObject *>(PyArray_GETCONTIGUOUS(ary));
if (obj) {
res = convert_to_table(py::cast<py::object>(obj), queue);
Py_DECREF(obj);
return res;
}
else {
throw std::invalid_argument(
"[convert_to_table] Numpy input could not be converted into onedal table.");
}
}
#define MAKE_HOMOGEN_TABLE(CType) res = convert_to_homogen_impl<CType>(ary);
SET_NPY_FEATURE(array_type(ary),
array_type_sizeof(ary),
MAKE_HOMOGEN_TABLE,
throw std::invalid_argument("Found unsupported array type"));
#undef MAKE_HOMOGEN_TABLE
}
else if (strcmp(Py_TYPE(obj)->tp_name, "csr_matrix") == 0 ||
strcmp(Py_TYPE(obj)->tp_name, "csr_array") == 0) {
PyObject *py_data = PyObject_GetAttrString(obj, "data");
PyObject *py_column_indices = PyObject_GetAttrString(obj, "indices");
PyObject *py_row_indices = PyObject_GetAttrString(obj, "indptr");
PyObject *py_shape = PyObject_GetAttrString(obj, "shape");
if (!(is_array(py_data) && is_array(py_column_indices) && is_array(py_row_indices) &&
array_numdims(py_data) == 1 && array_numdims(py_column_indices) == 1 &&
array_numdims(py_row_indices) == 1)) {
throw std::invalid_argument("[convert_to_table] Got invalid csr_matrix object.");
}
PyObject *np_data = PyArray_FROMANY(py_data, array_type(py_data), 0, 0, NPY_ARRAY_CARRAY);
PyObject *np_column_indices =
PyArray_FROMANY(py_column_indices,
NPY_UINT64,
0,
0,
NPY_ARRAY_CARRAY | NPY_ARRAY_ENSURECOPY | NPY_ARRAY_FORCECAST);
PyObject *np_row_indices =
PyArray_FROMANY(py_row_indices,
NPY_UINT64,
0,
0,
NPY_ARRAY_CARRAY | NPY_ARRAY_ENSURECOPY | NPY_ARRAY_FORCECAST);
PyObject *np_row_count = PyTuple_GetItem(py_shape, 0);
PyObject *np_column_count = PyTuple_GetItem(py_shape, 1);
if (!(np_data && np_column_indices && np_row_indices && np_row_count && np_column_count)) {
throw std::invalid_argument(
"[convert_to_table] Failed accessing csr data when converting csr_matrix.\n");
}
const std::int64_t row_count = static_cast<std::int64_t>(PyLong_AsSsize_t(np_row_count));
const std::int64_t column_count =
static_cast<std::int64_t>(PyLong_AsSsize_t(np_column_count));
#define MAKE_CSR_TABLE(CType) \
res = convert_to_csr_impl<CType>(np_data, \
np_column_indices, \
np_row_indices, \
row_count, \
column_count);
SET_NPY_FEATURE(array_type(np_data),
array_type_sizeof(np_data),
MAKE_CSR_TABLE,
throw std::invalid_argument("Found unsupported data type in csr_matrix"));
#undef MAKE_CSR_TABLE
Py_DECREF(np_column_indices);
Py_DECREF(np_row_indices);
}
else {
throw std::invalid_argument(
"[convert_to_table] Not available input format for convert Python object to onedal table.");
}

Check notice on line 262 in onedal/datatypes/numpy/data_conversion.cpp

codefactor.io / CodeFactor

onedal/datatypes/numpy/data_conversion.cpp#L155-L262

Complex Method
return res;
}