Making a Conda Package for Pyjulia

[mjohnson541]

I've been trying to conda package a python software I wrote that's dependent on pyjulia and I've been trying to write a conda recipe for pyjulia. This comes with the problem that the python interpreter is statically linked to libpython. However, I'm trying to configure it so that my users don't have to deal with the troubleshooting methods (and that the load times aren't as painful as with the compiled_modules fix). Ideally this would require no non-automatic configuring or installing by the user and would work on scripts, python prompts and in jupyter notebooks.

It's worth noting that I've been unsuccessful installing any non-core julia packages and having them transfer properly within conda (I can't seem to run julia.install() in the recipe and have it not need run again after installing it).

I've tried a number of things:

1. Link python to python-jl
   I've changed python-jl to be run directly by python3 and then link python to python-jl. The conda recipe didn't seem to accommodate this and I had to ask users to do the linking manually. This works for prompts and scripts but seems to prevent users from launching jupyter notebooks at all, which is probably indicative of bigger problems.

2. run a file similar to python-jl as a PYTHONSTARTUP file. This can be done automatically, but it doesn't work for scripts because it only comes into play in interactive mode.

3. run a file similar to python-jl as a part of sitecustomize.py. This seemed like it should work, but I get a strange error doing it. After it starts installing PyCall I get an error that it can't find Pkg in pyjulia's install script:

[ Info: Build log in /opt/anaconda3/envs/pyjtest/share/julia/site/packages/PyCall/zqDXB/src/../deps/build.log
ERROR: LoadError: ArgumentError: Package Pkg not found in current path:
- Run `import Pkg; Pkg.add("Pkg")` to install the Pkg package.

Stacktrace:
 [1] require(::Module, ::Symbol) at ./loading.jl:892
 [2] include(::Module, ::String) at ./Base.jl:377
 [3] exec_options(::Base.JLOptions) at ./client.jl:288
 [4] _start() at ./client.jl:484
in expression starting at /Users/mattjohnson/.local/lib/python3.7/site-packages/julia/install.jl:12
Error in sitecustomize; set PYTHONVERBOSE for traceback:
PyCallInstallError: Installing PyCall failed.

** Important information from Julia may be printed before Python's Traceback **

Is there a better way to do this?

[tkf]

I have high hope for the compile-time preference system JuliaLang/Pkg.jl#1835. I think this would make the hack like python-jl obsolete because we can tweak PyCall.jl to separate precompile cache files used for PyJulia.

Item 3 is a separate issue and should easily be solvable. We need to use something like

pyjulia/src/julia/compile.jl

Lines 7 to 8 in ca04b00

	const Pkg =
	Base.require(Base.PkgId(Base.UUID("44cfe95a-1eb2-52ea-b672-e2afdf69b78f"), "Pkg"))

everywhere instead of using.

[mjohnson541]

Interesting!

Thanks! That solved that issue, although it turns out putting stuff sitecustomize isn't as great of an idea as I hoped. It seems no better than linking python to python-jl.

I've given this some more thought and I think it'd be much more valuable to have a way to automatically create a python-jl jupyter kernel than to find someway to change the python command. It's not too much to ask users to type python-jl to launch scripts, but asking them to make a jupyter kernel is a bit much. Do you think a function that did that would belong in pyjulia? Perhaps even in the install function if it detects that jupyter is installed?

[tkf]

python-jl is a fragile temporary hack especially when multiple environments are involved (as in the use-case with conda). I'm not in favor of increasing its use.

[tkf]

Item 3 should be fixed by #407.

[mjohnson541]

Makes sense! Thanks!

[mjohnson541]

@tfk Does the new preferences system make what you were talking about possible now? I can invest some time into this if I have some advice on where to start.

[MilesCranmer]

We have PyJulia on conda-forge now: https://anaconda.org/conda-forge/pyjulia.

Here is an example Python package which uses it: https://anaconda.org/conda-forge/pysr

[jacobwilliams]

It seems like the conda one also downloads Julia? (which doesn't currently have a Windows package). Last time I installed the PyPi one (which is confusingly named "julia") it didn't install Julia, so I was able to use it on Windows. Is this the expected behavior?

[mkitti]

conda-forge's Window's build system is not currently compatible with Julia's build system of Makefiles and shell scripts. I do not clearly see a path forward on Windows with conda-forge. We would need a cmake build system for Julia that is compatible with Visual Studio.

It might be possible to assemble a conda package that is just a binary repacking of the official Julia. This would not be in the conda-forge channel. Another route is to use the juliaup package in conda-forge. We side step the issue by downloading Julia on the user's time instead of packing the Julia binaries in the conda package.

Another possibility is to go from Julia side. @cjdoris has created another Python-Julia interface ecosystem:

1. https://github.com/cjdoris/PythonCall.jl (PythonCall.jl and JuliaCall)
2. https://github.com/cjdoris/CondaPkg.jl
3. https://github.com/cjdoris/MicroMamba.jl

[MilesCranmer]

Yes this behavior is intended. Any conda-forge install will bring all binary dependencies, but a pip install only installs the Python dependencies. Similarly, when you install, e.g., PyTorch through conda, it installs CUDA for you; when you install PyTorch through pip, it assumes you have already set CUDA it up. PyJulia is similar to this in that Julia is treated as the binary dependency.

Regarding why there is no windows install, see @mkitti’s comment.

[MilesCranmer]

Note that you can always install pip packages in a conda environment, and link them to your host system’s binaries. This will break some assumptions made by conda but it can provide a way around architecture constraints.