Running rust unit tests

[clbarnes]

Not sure if this is a bug, a feature request, or me just not understanding.

I find there's generally some pure-rust glue code between whatever library I'm wrapping and the pyo3-annotated functions which python actually sees. Often there's a small number of key functions in this glue code. I'd like to unit test them from rust, rather than from python (the functionality exposed to python is sometimes too high-level to test effectively).

However, I can't run cargo test if there's any pyo3 code in the library. The error message is huge, but the top and tail is

error: linking with `cc` failed: exit code: 1
  |
  = note: "cc" "-Wl,--as-needed" "-Wl,-z,noexecstack" "-m64" "-L" "/home/barnesc/.rustup/toolchains/nightly-2019-11-25-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib" "/home/barnesc/work/code/ncollpyde/target/debug/deps/ncollpyde-1e1a0e51329946c7.10mjpmfbaikw05c6.rcgu.o" "/home/barnesc/work/code/ncollpyde/target/debug/deps/ncollpyde-1e1a0e51329946c7.11illn7lkbyjj5od.rcgu.o" "/home/barnesc/work/code/ncollpyde/target/debug/deps/ncollpyde-1e1a0e51329946c7.11nob3mubtrc8rv7.rcgu.o" "/home/barnesc/work/code/ncollpyde/target/debug/deps/ncollpyde-1e1a0e51329946c7.12e809s0aapay8kc.rcgu.o" "/home/barnesc/work/code/ncollpyde/target/debug/deps/ncollpyde-1e1a0e51329946c7.12y9x5pkw1aqmbor.rcgu.o" "/home/barnesc/work/code/ncollpyde/target/debug/deps/ncollpyde-1e1a0e51329946c7.14r17tf13i8fr5m1.rcgu.o" "/home/barnesc/work/code/ncollpyde/target/debug/deps/ncollpyde-1e1a0e51329946c7.14r2ofop261bg7q1.rcgu.o" "/home/barnesc/work/code/ncollpyde/target/debug/deps/ncollpyde-1e1a0e51329946c7.14zkkvo2aiu134l8.rcgu.o" "/home/barnesc/work/code/ncollpyde/target/debug/deps/ncollpyde-1e1a0e51329946c7.15khoy5qd6gsxwca.rcgu.o" "/home/barnesc/work/code/ncollpyde/target/debug/deps/ncollpyde-1e1a0e51329946c7.16tkuyhuv10hmrxn.rcgu.o" "/home/barnesc/work/code/ncollpyde/target/debug/deps/ncollpyde-1e1a0e51329946c7.17p7so2q9naojq01.rcgu.o" "/home/barnesc/work/code/ncollpyde/target/debug/deps/ncollpyde-1e1a0e51329946c7.18u7gpqnnmcw7ibx.rcgu.o" "/home/barnesc/work/code/ncollpyde/target/debug/deps/ncollpyde-1e1a0e51329946c7.1c6xf4atvbyxzrcc.rcgu.o" "/home/barnesc/work/code/ncollpyde/target/debug/deps/ncollpyde-1e1a0e51329946c7.1etoobxnii7g6j5h.rcgu.o" "/home/barnesc/work/code/ncollpyde/target/debug/deps/ncollpyde-1e1a0e51329946c7.1f4wi3f1fycow1t8.rcgu.o" "/home/barnesc/work/code/ncollpyde/target/debug/deps/ncollpyde-1e1a0e51329946c7.1j0rw6tkhmas6v0f.rcgu.o" "/home/barnesc/work/code/ncollpyde/target/debug/deps/ncollpyde-1e1a0e51329946c7.1kec3icgy9gh8ckx.rcgu.o" "/home/barnesc/work/code/ncollpyde/target/debug/deps/ncollpyde-1e1a0e51329946c7.1ld4cor4xs8rg7r9.rcgu.o" "/home/barnesc/work/code/ncollpyde/target/debug/deps/ncollpyde-1e1a0e51329946c7.1ljrommb62qxtv3m.rcgu.o" "/home/barnesc/work/code/ncollpyde/target/debug/deps/ncollpyde-1e1a0e51329946c7.1m5dbar01xzo3nz9.rcgu.o" "/home/barnesc/work/code/ncollpyde/target/debug/deps/ncollpyde-1e1a0e51329946c7.1mw34afyyax1iigt.rcgu.o" "/home/barnesc/work/code/ncollpyde/target/debug/deps/ncollpyde-1e1a0e51329946c7.1o75908a4ylf8r01.rcgu.o" "/home/barnesc/work/code/ncollpyde/target/debug/deps/ncollpyde-1e1a0e51329946c7.1owbxzg86mdl7xlz.rcgu.o" "/home/barnesc/work/code/ncollpyde/target/debug/deps/ncollpyde-1e1a0e51329946c7.1qduy0ykusw1ib5u.rcgu.o" <LOADS MORE SIMILAR-LOOKING LOGS>

...

/home/barnesc/.cargo/registry/src/github.com-1ecc6299db9ec823/pyo3-0.8.3/src/types/string.rs:83: undefined reference to `PyUnicode_AsEncodedString'
/usr/bin/ld: /home/barnesc/.cargo/registry/src/github.com-1ecc6299db9ec823/pyo3-0.8.3/src/types/string.rs:94: undefined reference to `PyBytes_AsString'
/usr/bin/ld: /home/barnesc/.cargo/registry/src/github.com-1ecc6299db9ec823/pyo3-0.8.3/src/types/string.rs:96: undefined reference to `PyBytes_Size'
/usr/bin/ld: /home/barnesc/work/code/ncollpyde/target/debug/deps/libpyo3-cb5b866f38957f83.rlib(pyo3-cb5b866f38957f83.pyo3.b6ihku99-cgu.7.rcgu.o): in function `pyo3::types::tuple::PyTuple::slice':
/home/barnesc/.cargo/registry/src/github.com-1ecc6299db9ec823/pyo3-0.8.3/src/types/tuple.rs:61: undefined reference to `PyTuple_GetSlice'
/usr/bin/ld: /home/barnesc/work/code/ncollpyde/target/debug/deps/libpyo3-cb5b866f38957f83.rlib(pyo3-cb5b866f38957f83.pyo3.b6ihku99-cgu.8.rcgu.o): in function `<T as pyo3::objectprotocol::ObjectProtocol>::str':
/home/barnesc/.cargo/registry/src/github.com-1ecc6299db9ec823/pyo3-0.8.3/src/objectprotocol.rs:318: undefined reference to `PyObject_Str'
/usr/bin/ld: /home/barnesc/work/code/ncollpyde/target/debug/deps/libpyo3-cb5b866f38957f83.rlib(pyo3-cb5b866f38957f83.pyo3.b6ihku99-cgu.8.rcgu.o): in function `pyo3::types::floatob::PyFloat::new':
/home/barnesc/.cargo/registry/src/github.com-1ecc6299db9ec823/pyo3-0.8.3/src/types/floatob.rs:33: undefined reference to `PyFloat_FromDouble'
/usr/bin/ld: /home/barnesc/work/code/ncollpyde/target/debug/deps/libpyo3-cb5b866f38957f83.rlib(pyo3-cb5b866f38957f83.pyo3.b6ihku99-cgu.8.rcgu.o): in function `pyo3::types::floatob::<impl pyo3::conversion::FromPyObject for f64>::extract':
/home/barnesc/.cargo/registry/src/github.com-1ecc6299db9ec823/pyo3-0.8.3/src/types/floatob.rs:58: undefined reference to `PyFloat_AsDouble'
collect2: error: ld returned 1 exit status
          
error: aborting due to previous error

error: could not compile `ncollpyde`.

It seems like it may relate to issue #99: @konstin suggested looking here and indeed I am using pyenv, although recompiling (3.8.3) with that flag enabled didn't seem to help.

[davidhewitt]

Hi @clbarnes, this is a longstanding issue, you can find more information and a couple of workaround in #340.

I hope to have a resolution soon. For now closing this as a duplicate.

[clbarnes]

Thanks, the #[cfg(not(test))] workaround is good enough for me for the time being.

[DavidAntliff]

Hi @davidhewitt, in 2020 you mentioned that you might have a resolution soon, and marked this as Completed soon after. Just wondering if there is now a best-practice solution to this problem?

EDIT: I found the Testing section, and this does resolve the linker errors, but I still have to set LD_LIBRARY_PATH to the parent location of my libpython3.12.so.1.0 file - is there any way to avoid that without doing something like this?

.cargo.config.toml:

[target.'cfg(all())']
rustflags = [
    "-C", "link-arg=-Wl,-rpath,.../pyenv.git/versions/3.12.2/lib",
    "-C", "link-arg=-L.../pyenv.git/versions/3.12.2/lib",
    "-C", "link-arg=-lpython3.12",
]

I am using pyenv with Python 3.12.2, installed with env PYTHON_CONFIGURE_OPTS="--enable-shared" and --keep so the Python source and shared libraries are available.

BTW, I don't get "weird linking errors" even with extension-module feature enabled, if I have that .cargo/config.toml above and the deps below - it actually links and runs correctly. But I do have to hard-code my Python library path.

i.e. this works with .cargo/config.toml above:

[dependencies]
pyo3 = { git = "https://github.com/pyo3/pyo3", features = ["extension-module"] }

[davidhewitt]

BTW, I don't get "weird linking errors" even with extension-module feature enabled, if I have that .cargo/config.toml above and the deps below - it actually links and runs correctly. But I do have to hard-code my Python library path.

I think that's because by having that config.toml you're forcing linking, thus overriding the extension-module feature.

EDIT: I found the Testing section, and this does resolve the linker errors, but I still have to set LD_LIBRARY_PATH to the parent location of my libpython3.12.so.1.0 file - is there any way to avoid that without doing something like this?

I believe that the rpath setting in Cargo.toml should fix this, but I struggled to make it work last time I tried.