Python REPL for Conda environments is broken

[TheDohn]

Description

Python REPL does not use Python installation as determined by Conda environment.

Reproduction guide

• Start Emacs
• Select and activate conda env (conda-env-activate)
• Start Python REPL (run spacemacs/python-start-or-switch-repl)

Observed behaviour:

• Python REPL starts successfully, but points to some sort of default Python installation, implying Conda shim has not been detected in PATH
• This can be seen by running import sys; print(sys.executable), which does not print the Conda installation. More problematically, the libraries in the desired Conda env are not available.

Expected behaviour:

• Python REPL starts successfully, and points to the correct Python installation, as determined by the conda env

System Info 💻

• OS: darwin
• Emacs: 29.4
• Spacemacs: 0.999.0
• Spacemacs branch: develop (rev. 4216cbb)
• Graphic display: t
• Running in daemon: nil
• Distribution: spacemacs
• Editing style: vim
• Completion: helm
• Layers:

(toml ibuffer
      (lua :variables lua-backend 'lsp)
      octave sql html
      (latex :variables latex-backend 'lsp)
      csv vimscript javascript yaml debug
      (dap :variables dap-python-debugger 'debugpy)
      (auto-completion :variables auto-completion-return-key-behavior 'complete auto-completion-private-snippets-directory "/Users/donbunk/dotfiles/yasnippets" auto-completion-enable-snippets-in-popup t auto-completion-enable-sort-by-usage t auto-completion-enable-help-tooltip t auto-completion-use-company-box t auto-completion-tab-key-behavior 'cycle auto-completion-complete-with-key-sequence "jk" auto-completion-minimum-prefix-length 1 auto-completion-idle-delay 0.2)
      emacs-lisp
      (git :variables git-enable-magit-delta-plugin t git-enable-magit-gitflow-plugin t)
      helm
      (lsp :variables lsp-sonarlint t lsp-headerline-breadcrumb-enable nil lsp-lens-enable t)
      markdown multiple-cursors org
      (shell :variables shell-default-height 30 shell-default-position 'bottom)
      outshine spell-checking syntax-checking version-control treemacs
      (python :variables python-backend 'lsp python-spacemacs-indent-guess 'nil python-lsp-server 'pyright)
      (ess :variables ess-indent-level 5 ess-style 'RStudio ess-r-backend 'ess)
      (evil-snipe :variables evil-snipe-enable-alternate-f-and-t-behaviors t)
      themes-megapack theming evil-better-jumper conda eww spacemacs-purpose
      (json :variables json-fmt-tool 'web-beautify)
      (multiple-cursors :variables multiple-cursors-backend 'evil-mc)
      colors pdf command-log tern
      (tree-sitter :variables tree-sitter-syntax-highlight-enable t)
      (spacemacs-layouts :variables spacemacs-layouts-restricted-functions
                         '(spacemacs/window-split-double-columns spacemacs/window-split-triple-columns spacemacs/window-split-grid winner-undo winner-redo)
                         spacemacs-layouts-restrict-spc-tab t))

• System configuration features: ACL GLIB GMP GNUTLS JPEG JSON LIBXML2 MODULES NOTIFY KQUEUE NS PDUMPER PNG RSVG SQLITE3 THREADS TIFF TOOLKIT_SCROLL_BARS TREE_SITTER ZLIB

Backtrace

• I don't think there is a backtrace, b/c there is no formal error being thrown. Please lmk how to add this if it is necessary.

[sunlin7]

Could you give some details for your Conda environment? Like the exec-path after conda-env-activate? I'd like to reproduce the issue but don't have experience on installing/using "Conda".

[sunlin7]

I tried install a miniconda and activate the conda on terminal, then in the terminal run emacs -nw, execute the conda-evn-activate, the exec-path will be changed with the conda path first.
(executable-find "python") will get correct value. But I do find a issue (maybe not same as current issue), that if I have the ipython on system path, it will be used over the python in conda path, this issue always exists, to fix it may lead performance issue.

[sunlin7]

@TheDohn Could you help to verify the PR 16843? It will try searching the python interpreters in exec-path. For example, you have the python3 in conda path, and the ipython3 in the system path, the python-shell-interpreter will:
Before: ipython3 will be deteced and used;
After: python will be deteced and used.

[TheDohn]

@sunlin7 Thank you very much for your responses and time on this. I tried your PR https://github.com/sunlin7/spacemacs/tree/fix-searching-python-intepreter but when I activate my conda env, I still don't get the right Python executable. On your branch, in an Emacs Python REPL, running import sys; sys.executable gives '/Library/Developer/CommandLineTools/usr/bin/python3', which is some other Python installation on my machine. It is worth noting that the value of exec-path contains my conda env path at the beginning, so it does seem like it is being set correctly. ("/opt/homebrew/Caskroom/miniconda/base/envs/p2e_ray_2d6d3/bin")

The full value of exec-path is: ("/opt/homebrew/Caskroom/miniconda/base/envs/p2e_ray_2d6d3/bin" "/usr/bin" "/bin" "/usr/sbin" "/sbin" "/Applications/Emacs.app/Contents/MacOS/bin-arm64-12" "/Applications/Emacs.app/Contents/MacOS/libexec-arm64-12" "/opt/homebrew/bin" "/opt/homebrew/sbin" "/usr/local/bin" "/Library/Tex/texbin" "/opt/homebrew/Caskroom/miniconda/base/condabin" "/System/Cryptexes/App/usr/bin" "/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin" "/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin" "/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin" "/Library/TeX/texbin" "/Applications/Emacs.app/Contents/MacOS/bin-arm64-11" "/Applications/Emacs.app/Contents/MacOS/libexec-arm64-11" "/Applications/Emacs.app/Contents/MacOS/libexec")

[sunlin7]

It seems you have some customer code had set the python-shell-interpreter.
Could you try find you .spacemac and .dir-locals.el which had set the value?
Or put follow two lines in you early-init.el to find who changed the value.

(require 'debug)
(debug-on-variable-change 'python-shell-interpreter)

[TheDohn]

Hmm... thanks for the suggestion. I will have to look into that and get back to you.