Third-party CI Warning "Falling back to prediction using DMatrix due to mismatched devices"

[jakirkham]

Seeing the following warnings in nightly CI:

test_xgboost.py::test_predict[cuda]
  /opt/conda/envs/test/lib/python3.12/site-packages/xgboost/core.py:158: UserWarning: [07:11:48] WARNING: /workspace/src/common/error_msg.cc:58: Falling back to prediction using DMatrix due to mismatched devices. This might lead to higher memory usage and slower performance. XGBoost is running on: cuda:0, while the input data is on: cpu.
  Potential solutions:
  - Use a data structure that matches the device ordinal in the booster.
  - Set the device for booster before call to inplace_predict.
  
  This warning will only be shown once.
  
    warnings.warn(smsg, UserWarning)

test_xgboost.py::test_predict[cpu]
  /opt/conda/envs/test/lib/python3.12/site-packages/xgboost/core.py:158: UserWarning: [07:11:48] WARNING: /workspace/src/common/error_msg.cc:58: Falling back to prediction using DMatrix due to mismatched devices. This might lead to higher memory usage and slower performance. XGBoost is running on: cpu, while the input data is on: cuda:0.
  Potential solutions:
  - Use a data structure that matches the device ordinal in the booster.
  - Set the device for booster before call to inplace_predict.
  
  This warning will only be shown once.
  
    warnings.warn(smsg, UserWarning)

Note this CI job also fails. AFAICT it is due to these wanings being converted to errors, but please let me know if there is more going on

[jakirkham]

Here is the relevant test from the warning above

cudf/python/cudf/cudf_pandas_tests/third_party_integration_tests/tests/test_xgboost.py

Lines 116 to 140 in 61e47bb

	@pytest.mark.parametrize("device", ["cpu", "cuda"])
	def test_predict(device: str) -> np.ndarray:
	reg = xgb.XGBRegressor(n_estimators=2, device=device)
	X, y = make_regression(n_samples, n_features, random_state=11)
	X_df = pd.DataFrame(X)
	reg.fit(X_df, y)
	booster = reg.get_booster()
	predt0 = reg.predict(X_df)
	predt1 = booster.inplace_predict(X_df)
	# After https://github.com/dmlc/xgboost/pull/11014, .inplace_predict()
	# returns a real cupy array when called on a cudf.pandas proxy dataframe.
	# So we need to ensure we have a valid numpy array.
	if not isinstance(predt1, np.ndarray):
	predt1 = predt1.get()
	np.testing.assert_allclose(predt0, predt1)
	predt2 = booster.predict(xgb.DMatrix(X_df))
	np.testing.assert_allclose(predt0, predt2)
	predt3 = booster.inplace_predict(X)
	np.testing.assert_allclose(predt0, predt3)
	return predt0

[jakirkham]

@hcho3 do you know what we would need to change here?

[hcho3]

"XGBoost is running on: cpu, while the input data is on: cuda:0."

This message might be telling us the truth, given that pd.DataFrame is being redirected to cudf.DataFrame.
One possible cause is that XGBoost could be using isinstance check internally to determine the type of the input, which the proxy module interferes with. We may need to adopt a similar workaround as the one in rapidsai/cuml#6175.