Yet another Python Dependencies question

[jgard]

Just starting to explore Windmill CE with local-hosted docker. My team is more on the infrastructure & devops side, with expertise in Python and Powershell. We're in an "enterprise" with an SSL-intercepting proxy (meaning, certificates are changed which causes issues) and an internal Artifactory environment to proxy things like pip packages.

I've had some success customizing the worker image to install some Powershell and Python dependencies with a Dockerfile like:

FROM ghcr.io/windmill-labs/windmill:main

##Debian
COPY cacerts.crt /usr/local/share/ca-certificates
RUN update-ca-certificates

##Python
COPY requirements.txt /tmp
RUN \
    pip config --user set global.index-url https://artifactory.COMPANY.com/artifactory/api/pypi/pypi-virtual/simple && \
    pip config --user set global.trusted-host COMPANY.com && \
    pip install --no-cache-dir -r /tmp/requirements.txt

ENV REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt

##Powershell
RUN \
    pwsh -c "Register-PSRepository -Name Artifactory -SourceLocation https://artifactory.COMPANY.com/artifactory/api/nuget/powershell-virtual -InstallationPolicy Trusted" && \
    pwsh -c "Unregister-PSRepository -Name PSGallery" && \
    pwsh -c "Install-Module -Scope AllUsers -RequiredVersion 4.1.686 -Name AWS.Tools.Common" && \
    pwsh -c "Install-Module -Scope AllUsers -RequiredVersion 4.1.686 -Name AWS.Tools.S3" && \
    pwsh -c "Install-Module -Scope AllUsers -RequiredVersion 13.3.0.24145081 -Name VMware.VimAutomation.Common"

ENV POWERSHELL_TELEMETRY_OPTOUT=1

Three questions from my first experiences:
1. Importing python dependencies at the top of a script invokes the UV package manager that I just can't get to work using our local Artifactory. I've tried adding using UV_DEFAULT_INDEX & WHITELIST_ENVS env vars, and I've tried injecting an /etc/uv/uv.toml configuration file for [[index]]. Nothing I try changes the result of it not using Artifactory, and not trusting the certificate it sees from pypi (due to our proxy):

Error while installing boto3==1.36.3:
Using CPython 3.11.10 interpreter at: /usr/local/bin/python
error: Failed to fetch: `https://pypi.org/simple/boto3/`
  Caused by: Request failed after 3 retries
  Caused by: error sending request for url (https://pypi.org/simple/boto3/)
  Caused by: client error (Connect)
  Caused by: invalid peer certificate: UnknownIssuer

2. Importing python dependencies within the main() function seems to just work, using the pip-installed packages in the image. What downsides will I encounter if I just continue using this method?

3. Importing Powershell dependencies is case-sensitive where the language is not. Import-Module <module name> will properly install and cache the module (from our Artifactory because I registered that PSRepository). import-module <module name> will not try to install and cache the module, and will only work if it is preinstalled in the image.

Any advice on these subjects is appreciated, thanks!

FWIW: I'm aware of the Discord, but it isn't allowed from our enterprise workstations (as I'm sure is the case at many big companies) and having discussions about code from my phone is cumbersome at best.

[rubenfiszel]

Hi @jgard , for using private registries on python, we support it directly in the instance settings but it require EE unfortunately. For Powershell, that looks like an issue we could investigate.

[jgard]

Thanks @rubenfiszel . Are there any downsides to bypassing the cache for Python and just importing what is needed (assuming preinstalled in the image), in each function?

[rubenfiszel]

That seems very tedious but if it works it works. It means you can only have one version shared across all your scripts and incompatible versions will be impossible to track down. Windmill compute a unique lockfile for each script with precise versions and all versions will co-locate on the cache.

[aHVzY2g]

try setting:

      - USE_PIP_COMPILE=true
      - USE_PIP_INSTALL=true

that will use the deprecated pip option but at least it does work with tls interception.