Unexpected hatch shell behavior: PATH prepend is shadowed by shell initialization (due to config.toml shell config mistake)

[DylanLukes]

Expected Behavior

When running hatch shell, I expect the hatch virtualenv bin to be prepended to my path, as occurs when sourcing a virtual environment.

Problematic Behavior

When running hatch shell, the hatch virtualenv bin is prepended to my path... before my ~/.zshrc runs again and prepends a bunch of other stuff in front of it.

"Other stuff" includes asdf, which means .hatch/<default env>/bin/python is shadowed by ~/.asdf/shims/python.

This results in a ton of breakage unless I manually source .hatch/<default env>/bin/activate.

Note: I have dirs.env.virtual = ".hatch" set up, so that Hatch creates virtual environments inside project directories. This assists with IDE integration/detection of virtual environments.

Reproduction

I haven't noticed this behavior before, so I'm not sure if this is a regression of some sort.

First, I create a new project:

❯ hatch new wtf
wtf
├── .github
│   └── workflows
│       └── test.yml
├── src
│   └── wtf
│       ├── __about__.py
│       └── __init__.py
├── tests
│   └── __init__.py
├── LICENSE.txt
├── README.md
└── pyproject.toml

Heading inside, I check my path. The contents all make sense to me, I can tell where they're all coming from. There is no mysterious source of shell configuration I'm unaware of contributing to the contents of my PATH.

❯ echo $PATH | tr ":" "\n"
/Users/dylan/Library/pnpm               <-- set with export in my ~/.zshrc
/Users/dylan/.local/bin
/Users/dylan/Application Support/JetBrains/Toolbox/scripts
/Users/dylan/.asdf/shims.               
/opt/homebrew/opt/asdf/libexec/bin      <-- set by asdf oh-my-zsh plugin

/opt/homebrew/bin                       <-- set by ~/.zprofile's `eval "$(/opt/homebrew/bin/brew shellenv)"`
/opt/homebrew/sbin                      

/usr/local/bin                          <-- loaded from /etc/paths.d/*
/System/Cryptexes/App/usr/bin
/usr/bin
/bin
/usr/sbin
/sbin
/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin
/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin
/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin
/opt/X11/bin
/Library/Apple/usr/bin
/Library/TeX/texbin
/Applications/VMware Fusion.app/Contents/Public
/Applications/iTerm.app/Contents/Resources/utilities

/opt/homebrew/opt/fzf/bin               <-- appended by fzf init script from ~/.zshrc
...

Now I try to activate the default environment and compare:

❯ echo $PATH | tr ":" "\n" >original.env
❯ hatch shell
❯ git diff original.env <(echo $PATH | tr ":" "\n")

diff --git a/original.env b/dev/fd/13
index 12ac786..0000000 100644
--- a/original.env
+++ b/dev/fd/13
@@ -1,22 +1,26 @@
 /Users/dylan/Library/pnpm
 /Users/dylan/.local/bin
 /Users/dylan/Application Support/JetBrains/Toolbox/scripts
 /Users/dylan/.asdf/shims
 /opt/homebrew/opt/asdf/libexec/bin
+/Users/dylan/Projects/wtf/.hatch/wtf/bin
+/Users/dylan/Library/pnpm
+/Users/dylan/.local/bin
+/Users/dylan/Application Support/JetBrains/Toolbox/scripts
 /opt/homebrew/bin
 /opt/homebrew/sbin
 /usr/local/bin
 /System/Cryptexes/App/usr/bin
 /usr/bin
 /bin
 /usr/sbin
 /sbin
<etc...>

Discussion

Hatch seems to be first copying my additions to PATH excluding the one set by a plugin (which is asdf). Then, it appears that my ~/.zshrc ran as usual. Adding an echo SOURCING ZSHRC to my .zshrc confirms hatch shell is creating a new login shell and sourcing it.

Running source ~.hatch/wtf/bin/activate places /Users/dylan/Projects/wtf/.hatch/wtf/bin at the top of the PATH as expected. Notably, this file does not appear to be run automatically at any time by hatch shell.

[DylanLukes]

I think I've figured it out. The issue was that my config.toml had shell = "/bin/zsh" rather than just shell = "zsh".

---

I installed hatch from a copy of the current master branch and spliced in the PyCharm visual debugger integration so that I can explore where exactly things go wrong.

# src/hatch/__init__.py
import pydevd_pycharm
pydevd_pycharm.settrace('localhost', port=33333, stdoutToServer=True, stderrToServer=True)

❯ pipx install -e .
⣯ installing hatch from spec '/Users/dylan/PycharmProjects/hatch'
❯ ~/.local/pipx/venvs/hatch/bin/python -m pip install "pydevd-pycharm~=242.20224.347"

Then...

VirtualEnvironment.enter_shell does not find a shell executor, and so defaults to self.safe_activation(), which prepends as one would expect. So, ShellManager.enter_zsh is never actually called.

At this point, os.environ["PATH"] is correct. However, hatch then performs os.execvp(command[0], command) where command = ['/bin/zsh'] which as expected re-runs ~/.zshrc.

But notably, the activate script is not ever run via ShellManager.enter_zsh. This seems odd.

Looking into it further, when enter_shell is called:

    def enter_shell(self, name: str, path: str, args: Iterable[str]):
        shell_executor = getattr(self.shells, f'enter_{name}', None)
        if shell_executor is None:
            # Manually activate in lieu of an activation script
            with self.safe_activation():
                self.platform.exit_with_command([path, *args])
        else:
            with self.expose_uv(), self.get_env_vars():
                shell_executor(path, args, self.virtual_env.executables_directory)

... it appears that name is "/bin/zsh", so the first call resolves to getattr(self.shells, f'enter_/bin/zsh', None). Obviously this isn't found. As a result, the activate script is not called.

Fixing the configuration file resolves this issue.

[DylanLukes]

While this issue is definitely my own fault, it seems likely enough to happen to others and time-consuming enough to troubleshoot that some kind of quality of life countermeasure might be worthwhile.

For example it might be preferable if when shell is set as a bare string in config.toml, and it is set to a path, Hatch either:

1. Emits a warning/error asking if you meant to define shell as a table with name and path keys. (probably better)
2. Detects that the provided string is a path rather than the name of the supported shell and acts accordingly. (too magical?)

Has something in the behavior here changed in 1.12.0? I wasn't experiencing these issues until I updated Hatch.