Skip to content

Conversation

@oshadura
Copy link

@bryates
Copy link
Contributor

bryates commented Jun 17, 2021

Instead of copying or moving the analysis folder, shortcuts might also work. I used a shortcut earlier to get binder working from a subdirectory. However, if everyone is onboard with moving analysis inside the topcoffea folder, that's fine with me.

@oshadura
Copy link
Author

oshadura commented Jun 17, 2021

@bryates thanks a lot for comment!
My problem is that I need to install topcoffea from my topcoffea fork git+https://github.com/oshadura/topcoffea.git@coffea-casa-analysis on remote Dask workers (not only notebook). For this I am using next class:

from dask.distributed import Client, Worker, WorkerPlugin
import os
from typing import List
class DependencyInstaller(WorkerPlugin):
    def __init__(self, dependencies: List[str]):
        self._depencendies = " ".join(f"'{dep}'" for dep in dependencies)
    def setup(self, worker: Worker):
        os.system(f"pip install {self._depencendies}")
dependency_installer = DependencyInstaller([
    "git+https://github.com/oshadura/topcoffea.git@coffea-casa-analysis","awkward==1.3.0"
])

Since I need to use coffea processor on dask workers as well, I need to ship it there, so it should be packaged :(

oshadura and others added 3 commits June 20, 2021 13:57
@oshadura
Copy link
Author

Most luckily will be replaced by TopEFT/topcoffea#41

@oshadura oshadura closed this Jul 28, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants