zstandard static linkage issue

[brendene]

Solution to issue cannot be found in the documentation.

• I checked the documentation.

Issue

Thank you for the the fix to statically link against libzstd @h-vetinari

zstandard fails to work when the newer libzstd's symbols are loaded into the process table.

Here is a repro:

import ctypes                                                                                                                                                                                                                                                                                                                                                               
import os                                                                                                                                                                                                                                                                                                                                                                   
import sys                                                                                                                                                                                                                                                                                                                                                                  
                                                                                                                                                                                                                                                                                                                                                                            
lib_os = ctypes.CDLL('/miniconda3/envs/dev/lib/libzstd.so', mode=os.RTLD_GLOBAL)  # 1.5.7 libzstd                                                                                                                                                                                                                                                                                
import zstandard # FAILS
# ImportError: zstd C API versions mismatch; Python bindings were not compiled/linked against expected zstd version (10507 returned by the lib, 10506 hardcoded in zstd headers, 10506 hardcoded in the cext)

The call to ZSTD_versionNumber is indirect and points to a relocation entry, which the dynamic linker fixes up with the address of the function in the previously loaded shared object. Presumably the point of statically linking libzstd was to isolate backend_c from the outside world, which seems to be violated here.

$ objdump -Cdr /miniconda3/envs/dev/lib/python3.11/site-packages/zstandard/backend_c.cpython-311-x86_64-linux-gnu.so                                                                                                                                                                                                                                              
...                                                                                                                                                                                                                                                                                                                                                                         
000000000001c690 <zstd_module_init>:                                                                                                                                                                                                                                                                                                                                        
   1c690:       41 54                   push   %r12                                                                                                                                                                                                                                                                                                                         
   1c692:       55                      push   %rbp                                                                                                                                                                                                                                                                                                                         
   1c693:       53                      push   %rbx                                                                                                                                                                                                                                                                                                                         
   1c694:       48 89 fb                mov    %rdi,%rbx                                                                                                                                                                                                                                                                                                                    
   1c697:       ff 15 83 ce 0e 00       call   *0xece83(%rip)        # 109520 <ZSTD_versionNumber@@Base+0xe9e30>                                                                                                                                                                                                                                                            
                                                                                                                                                                                                                                                                                                                                                                            
                                                                                                                                                                                                                                                                                                                                                                            
$ readelf -a /miniconda3/envs/dev/lib/python3.11/site-packages/zstandard/backend_c.cpython-311-x86_64-linux-gnu.so                                                                                                                                                                                                                                                
...                                                                                                                                                                                                                                                                                                                                                                         
Relocation section '.rela.dyn' at offset 0xd258 contains 1150 entries:                                                                                                                                                                                                                                                                                                      
  Offset          Info           Type           Sym. Value    Sym. Name + Addend                                                                                                                                                                                                                                                                                            
000000109520  007a00000006 R_X86_64_GLOB_DAT 000000000001f6f0 ZSTD_versionNumber + 0

Installed packages

zstandard                 0.23.0          py311h9ecbd09_1    conda-forge
zstd                      1.5.7                hb8e6e7a_2    conda-forge

Environment info

conda version : 25.1.1

[h-vetinari]

Hm... The static linkage was really just to work around the issue that we need zstandard 1.5.6 to be able to co-exist with zstd 1.5.7. Of course, the best thing - by a long shot - would be having indygreg/python-zstandard#255 merged and then either waiting for a release or backporting it. The case you point out seems to run into that divergence. I don't have a good suggestion how to fix this, except that if you need to import both zstandard and zstd itself through ctypes, then you'll have to add zstd <1.5.7 to your constraints.

That said, I'm not a maintainer here, and there may be cleverer technical solutions that I'm not aware of.

[brendene]

Agreed, merging in indygreg/python-zstandard#255 would be the best solution. In the event it is merged in, will this feedstock maintain the static linkage or will it revert back to the dynamic linkage?

In case it does not get merged in, would you consider building with -Bsymbolic ? This should cause the dynamic linker to prefer symbols in backend_c itself (static libzstd 1.5.6) rather than from the outside (libzstd 1.5.7).

[h-vetinari]

In the event it is merged in, will this feedstock maintain the static linkage or will it revert back to the dynamic linkage?

I expect it to go back to dynamic linkage, which we strongly prefer in general.

In case it does not get merged in, would you consider building with -Bsymbolic ?

That's a question for the maintainers of this feedstock (I'm not one of them), but from my POV: if we can fix this problem, that would be great. You could try to open a PR and add a test based on the example you've had here; if it can be fixed (and the solution is not super invasive), I think we the chances are high that it'll be merged.

[nehaljwani]

I tried using -Bsymbolic, but it didn't help resolve the issue for me.

I've an alternative approach in #61 . It isn't pretty and is only for Linux, but it kinda does the job.