exp run: doesn't run anything

[mermerico]

Bug Report

Description

dvc exp run returns without running anything even though a dependency has been changed and dvc repro will run run a stage.

Environment information

Output of dvc doctor:

$ dvc doctor
DVC version: 3.61.0 (pip)
-------------------------
Platform: Python 3.11.6 on Linux-6.8.0-63-generic-x86_64-with-glibc2.35
Subprojects:
        dvc_data = 3.16.10
        dvc_objects = 5.1.1
        dvc_render = 1.0.2
        dvc_task = 0.40.2
        scmrepo = 3.3.11
Supports:
        http (aiohttp = 3.12.13, aiohttp-retry = 2.9.1),
        https (aiohttp = 3.12.13, aiohttp-retry = 2.9.1),
        s3 (s3fs = 2025.3.0, boto3 = 1.38.27)
Config:
        Global: /home/mermerico/.config/dvc
        System: /etc/xdg/dvc
Cache types: reflink, symlink
Cache directory: zfs on tank
Caches: local
Remotes: s3
Workspace directory: zfs on tank
Repo: dvc, git
Repo.site_cache_dir: /home/mermerico/dvc_site_cache/repo/6264cc5779bad4edf4deee3c51449b14

Additional Information (if any):

When I turn on verbose mode for dvc exp run I see a bunch of messages talking about failure to protect. I think this is because I am using a shared cache, but from my brief look at the code it doesn't seem like this is supposed to be blocking. Here is a snippet from the end of the log.

2025-07-29 17:37:43,499 DEBUG: Computed stage: '../../<REDACTED>.dvc' md5: 'None'                                                                               
2025-07-29 17:37:43,501 DEBUG: Preparing to transfer data from 'memory://dvc-staging-md5/ec716ca5100f1e34ea43838b968d01c635c2710b09efb8b153754cec36b69c1e' to '/ds/dvc-cache/files/md5'                       
2025-07-29 17:37:43,501 DEBUG: Preparing to collect status from '/ds/dvc-cache/files/md5'
2025-07-29 17:37:43,501 DEBUG: Collecting status from '/ds/dvc-cache/files/md5'
2025-07-29 17:37:43,501 DEBUG: failed to protect '/ds/dvc-cache/files/md5/18/aba611dbc99e8fbf684f82f35c6f2c' - [Errno 1] Operation not permitted: '/ds/dvc-cache/files/md5/18/aba611dbc99e8fbf684f82f35c6f2c' 
Traceback (most recent call last):
  File "/home/mermerico/ds-blackbox.worktrees/dev-2025-q1/.ds/lib/python3.11/site-packages/dvc_data/hashfile/db/local.py", line 117, in protect
    os.chmod(path, self.CACHE_MODE)
PermissionError: [Errno 1] Operation not permitted: '/ds/dvc-cache/files/md5/18/aba611dbc99e8fbf684f82f35c6f2c'

2025-07-29 17:37:43,502 DEBUG: Saving information to '../../production_models/title_to_skills/data/all_lc_skills.pkl.dvc'.                                                                                    
2025-07-29 17:37:45,224 DEBUG: Stashed experiment '97bdd1a' with baseline 'a248568' for future execution.
2025-07-29 17:37:46,078 DEBUG: Analytics is enabled.                                            
2025-07-29 17:37:46,140 DEBUG: Trying to spawn ['daemon', 'analytics', '/tmp/tmp_al9fdwl', '-v']
2025-07-29 17:37:46,151 DEBUG: Spawned ['daemon', 'analytics', '/tmp/tmp_al9fdwl', '-v'] with pid 2637712

I am running the command from a folder with a dvc.yaml file but there is no mention of the relevant dvc.yaml file or its stages in the verbose output. The output only mentions other pipelines that are in this repo that are irrelevant to the pipeline I am interested in running.

[skshetry]

You are not using git worktree, are you? Also, could you please share the full log?

[mermerico]

Oh no... Yes I am using a git worktree.... Running a long-running pipeline right now but I can share the start of the log tomorrow.

EDIT: Ah it looks like worktrees are not supported. I'll switch to my main tree then. Thanks!

[skshetry]

Since we are getting a lot of reports about worktrees, I'll keep this issue open to add a hint that "git worktrees are not supported" in the exp run as a bandaid.

[skshetry]

I investigated this and found some bugs on dulwich side. I have opened a PR to fix them: jelmer/dulwich#1749.

Once that’s merged, we should be able to support worktrees. I’ve also opened a test PR here so that we don't regress: #10834.