Skip to content

[BUZZOK-27814] Install extra agent dependecies in runtime. #1716

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Oct 20, 2025
Merged

[BUZZOK-27814] Install extra agent dependecies in runtime. #1716

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
c0bccd1
[BUZZOK-27814][BUZZOK-27812] Install extra agent dependecies in runtime.
rmidianyi Oct 17, 2025
f435505
Reconcile dependencies, updated IDs, tags
svc-harness-git2 Oct 17, 2025
7c29e9d
Remove time from uv sync.
rmidianyi Oct 17, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 5 additions & 2 deletions public_dropin_environments/python311_genai_agents/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -90,16 +90,19 @@ EXPOSE 22
FROM base AS builder
# this stage has only bare minimal of dependencies installed to optimize build time for the local development

ENV ANNOY_COMPILER_ARGS="-D_CRT_SECURE_NO_WARNINGS,-DANNOYLIB_MULTITHREADED_BUILD,-march=x86-64"
ENV UV_COMPILE_BYTECODE=1 \
Copy link
Contributor Author

@rmidianyi rmidianyi Oct 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Env variables are not passed between docker stages, so need to explicitly enable bytecode compilation again for uv sync.

ANNOY_COMPILER_ARGS="-D_CRT_SECURE_NO_WARNINGS,-DANNOYLIB_MULTITHREADED_BUILD,-march=x86-64"

ARG WORKDIR
ARG VENV_PATH

COPY ./pyproject.toml ${WORKDIR}/pyproject.toml
COPY ./uv.lock ${WORKDIR}/uv.lock
Copy link
Contributor Author

@rmidianyi rmidianyi Oct 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We didn't use lock file, but it's better to have it for consistency of env.


WORKDIR ${WORKDIR}
RUN . ${VENV_PATH}/bin/activate && \
uv sync
uv sync && \
chmod -R 777 ${VENV_PATH}
Copy link
Contributor Author

@rmidianyi rmidianyi Oct 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Venv should be writable both for notebook user (10101) and custom model user (1000).

WORKDIR /

# Copy agent runtime into work directory
Expand Down
7 changes: 5 additions & 2 deletions public_dropin_environments/python311_genai_agents/Dockerfile.local
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -130,16 +130,19 @@ EXPOSE 22
FROM base AS builder
# this stage has only bare minimal of dependencies installed to optimize build time for the local development

ENV ANNOY_COMPILER_ARGS="-D_CRT_SECURE_NO_WARNINGS,-DANNOYLIB_MULTITHREADED_BUILD,-march=x86-64"
ENV UV_COMPILE_BYTECODE=1 \
ANNOY_COMPILER_ARGS="-D_CRT_SECURE_NO_WARNINGS,-DANNOYLIB_MULTITHREADED_BUILD,-march=x86-64"

ARG WORKDIR
ARG VENV_PATH

COPY ./pyproject.toml ${WORKDIR}/pyproject.toml
COPY ./uv.lock ${WORKDIR}/uv.lock
Copy link
Contributor

@mjnitz02 mjnitz02 Oct 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the user doesn't have a uv.lock file would this be a problem? Say they just to a task start + task deploy, they might not have built a local environment. They "must" have a pyproject.toml, but the uv.lock could be missing

Copy link
Contributor Author

@rmidianyi rmidianyi Oct 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This particular uv.lock is part of execution env (I've added it).

If we are talking about agent itself:

  • uv sync is non-critical - we would still attempt to run the agent.
  • uv.lock is generally supposed to be always present because recommended way to add dependency would be via uv add --no-upgrade --group extras <dependency> or via some Task shortcut. Manual editing of pyproject.toml is not recommended (unless you know what you are doing).

Copy link
Contributor Author

@rmidianyi rmidianyi Oct 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if you mean that uv.lock may be missing when building docker image locally using Dockerfile.local - I guess it's okay to require user to freeze env by generating lock file in this case. Lock file is generally a must in image if we're talking about consistency of dependencies.

Copy link
Contributor

@mjnitz02 mjnitz02 Oct 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yea, thats fair. If you make a ticket or two for updating docks related to this I can help pickup recipe and agent updates next week. I know you'll be out soon.

Copy link
Contributor

@mjnitz02 mjnitz02 Oct 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To your second comment I think that makes sense


WORKDIR ${WORKDIR}
RUN . ${VENV_PATH}/bin/activate && \
uv sync
uv sync && \
chmod -R 777 ${VENV_PATH}
WORKDIR /

# Copy agent runtime into work directory
Expand Down
6 changes: 3 additions & 3 deletions public_dropin_environments/python311_genai_agents/env_info.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
"description": "This template environment can be used to create GenAI-powered agents using CrewAI, LangGraph, or Llama-Index. Similar to other drop-in environments, you can either include a .pth artifact or any other code needed to deserialize your model, and optionally a custom.py file. You can also use this environment in codespaces.",
"programmingLanguage": "python",
"label": "",
"environmentVersionId": "68f0fda6e202621203aaedf7",
"environmentVersionId": "68f23ab0cee3dd12045636c5",
"environmentVersionDescription": "",
"isPublic": true,
"isDownloadable": true,
Expand All @@ -15,8 +15,8 @@
"contextUrl": "https://github.com/datarobot/datarobot-user-models/tree/master/public_dropin_environments/python311_genai_agents",
"imageRepository": "env-python-genai-agents",
"tags": [
"v11.2.0-68f0fda6e202621203aaedf7",
"68f0fda6e202621203aaedf7",
"v11.2.0-68f23ab0cee3dd12045636c5",
"68f23ab0cee3dd12045636c5",
"v11.2.0-latest"
]
}
5 changes: 4 additions & 1 deletion public_dropin_environments/python311_genai_agents/pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -220,4 +220,7 @@ implicit_reexport = true
[tool.uv]
override-dependencies = [
"fastapi==0.118.2",
]
]

[dependency-groups]
extras = []
Comment on lines +225 to +226
Copy link
Contributor Author

@rmidianyi rmidianyi Oct 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Extra user dependencies will go here.

Copy link
Contributor

@mjnitz02 mjnitz02 Oct 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this work well to put it here. We should update the templates pyproject.toml everywhere to have this addition

Copy link
Contributor Author

@rmidianyi rmidianyi Oct 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, working on another PR do do the same changes in af-component-agent.

82 changes: 40 additions & 42 deletions public_dropin_environments/python311_genai_agents/requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,19 +1,17 @@
# This file was autogenerated by uv via the following command:
# uv pip compile --no-annotate --no-emit-index-url --no-emit-trusted-host --output-file=requirements.txt pyproject.toml
Comment on lines -1 to -2
Copy link
Contributor Author

@rmidianyi rmidianyi Oct 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

uv lock 
uv sync
uv pip freeze > requirements.txt

absl-py==2.3.1
ag-ui-protocol==0.1.9
aioboto3==15.2.0
aiobotocore==2.24.2
aiofiles==24.1.0
aiofiles==25.1.0
aiohappyeyeballs==2.6.1
aiohttp==3.13.0
aioitertools==0.12.0
aiosignal==1.4.0
aiosqlite==0.21.0
alembic==1.16.5
alembic==1.17.0
annotated-types==0.7.0
annoy==1.17.3
anthropic==0.69.0
anthropic==0.71.0
anyio==4.11.0
appdirs==1.4.4
appnope==0.1.4
Expand All @@ -24,7 +22,7 @@ arrow==1.3.0
asttokens==3.0.0
attrs==25.4.0
authlib==1.6.5
azure-core==1.35.1
azure-core==1.36.0
azure-identity==1.25.1
azure-storage-blob==12.19.0
backoff==2.2.1
Expand All @@ -37,20 +35,20 @@ boto3==1.40.18
botocore==1.40.18
browserbase==1.4.0
build==1.3.0
cachetools==6.2.0
cachetools==6.2.1
certifi==2025.10.5
cffi==2.0.0
charset-normalizer==3.4.3
charset-normalizer==3.4.4
chromadb==1.1.1
click==8.2.1
cohere==5.18.0
cohere==5.19.0
colorama==0.4.6
coloredlogs==15.0.1
colorlog==6.9.0
colorlog==6.10.1
comm==0.2.3
crewai==0.193.2
crewai-tools==0.76.0
cryptography==46.0.2
cryptography==46.0.3
cuid==0.4
dataclasses-json==0.6.7
datarobot==3.9.1
Expand Down Expand Up @@ -81,7 +79,7 @@ execnet==2.1.1
executing==2.2.1
expandvars==1.1.2
fastapi==0.118.2
fastavro==1.12.0
fastavro==1.12.1
fastembed==0.6.0
fastjsonschema==2.21.2
filechunkio==1.8
Expand All @@ -92,41 +90,41 @@ flatbuffers==25.9.23
fqdn==1.5.1
frozenlist==1.8.0
fsspec==2025.3.0
genai-prices==0.0.31
genai-prices==0.0.34
gevent==25.9.1
google-api-core==2.26.0
google-auth==2.41.1
google-cloud-aiplatform==1.120.0
google-cloud-aiplatform==1.121.0
google-cloud-bigquery==3.38.0
google-cloud-core==2.4.3
google-cloud-resource-manager==1.14.2
google-cloud-storage==2.19.0
google-crc32c==1.7.1
google-genai==1.42.0
google-genai==1.45.0
google-resumable-media==2.7.2
googleapis-common-protos==1.70.0
greenlet==3.2.4
griffe==1.14.0
groq==0.32.0
grpc-google-iam-v1==0.14.2
grpc-google-iam-v1==0.14.3
grpcio==1.75.1
grpcio-status==1.75.1
gunicorn==23.0.0
h11==0.16.0
hf-xet==1.1.10
httpcore==1.0.9
httptools==0.6.4
httptools==0.7.1
httpx==0.28.1
httpx-sse==0.4.0
huggingface-hub==0.35.3
humanfriendly==10.0
idna==3.10
idna==3.11
importlib-metadata==8.7.0
importlib-resources==6.5.2
inflection==0.5.1
iniconfig==2.1.0
instructor==1.11.3
invoke==2.2.0
invoke==2.2.1
ipykernel==6.28.0
ipython==9.6.0
ipython-pygments-lexers==1.1.1
Expand All @@ -149,7 +147,7 @@ jsonschema==4.25.1
jsonschema-specifications==2025.9.1
julia==0.5.7
jupyter-client==8.6.3
jupyter-core==5.8.1
jupyter-core==5.9.1
jupyter-events==0.12.0
jupyter-kernel-gateway==3.0.1
jupyter-server==2.17.0
Expand All @@ -162,7 +160,7 @@ lancedb==0.25.2
langchain==0.3.27
langchain-aws==0.2.35
langchain-community==0.3.31
langchain-core==0.3.78
langchain-core==0.3.79
langchain-litellm==0.2.3
langchain-mcp-adapters==0.1.11
langchain-milvus==0.2.1
Expand All @@ -174,7 +172,7 @@ langgraph==0.6.10
langgraph-checkpoint==2.1.2
langgraph-prebuilt==0.6.4
langgraph-sdk==0.2.9
langsmith==0.4.33
langsmith==0.4.37
lark==1.3.0
legacy-cgi==2.6.3
litellm==1.74.9
Expand All @@ -186,7 +184,7 @@ llama-index-core==0.13.6
llama-index-embeddings-azure-openai==0.4.1
llama-index-embeddings-openai==0.5.1
llama-index-indices-managed-llama-cloud==0.9.4
llama-index-instrumentation==0.4.1
llama-index-instrumentation==0.4.2
llama-index-llms-azure-openai==0.4.1
llama-index-llms-bedrock-converse==0.9.5
llama-index-llms-langchain==0.7.1
Expand All @@ -197,16 +195,16 @@ llama-index-readers-file==0.5.4
llama-index-readers-llama-parse==0.5.1
llama-index-workflows==1.3.0
llama-parse==0.6.54
logfire==4.13.0
logfire-api==4.13.0
logfire==4.13.2
logfire-api==4.13.2
loguru==0.7.3
lxml==6.0.2
mako==1.3.10
markdown-it-py==4.0.0
markupsafe==3.0.3
marshmallow==3.26.1
matplotlib-inline==0.1.7
mcp==1.16.0
mcp==1.18.0
mdurl==0.1.2
memory-profiler==0.61.0
mistralai==1.9.11
Expand Down Expand Up @@ -236,7 +234,7 @@ oauthlib==3.3.1
ollama==0.6.0
onnxruntime==1.22.0
openai==1.109.1
openinference-semantic-conventions==0.1.23
openinference-semantic-conventions==0.1.24
openpyxl==3.1.5
opentelemetry-api==1.37.0
opentelemetry-exporter-otlp==1.37.0
Expand Down Expand Up @@ -312,7 +310,7 @@ prometheus-client==0.23.1
prompt-toolkit==3.0.52
propcache==0.4.1
proto-plus==1.26.1
protobuf==6.32.1
protobuf==6.33.0
psutil==5.9.8
ptyprocess==0.7.0
pure-eval==0.2.3
Expand All @@ -323,12 +321,12 @@ pyasn1==0.6.1
pyasn1-modules==0.4.2
pybase64==1.4.2
pycparser==2.23
pydantic==2.12.0
pydantic-ai==1.0.17
pydantic-ai-slim==1.0.17
pydantic-core==2.41.1
pydantic-evals==1.0.17
pydantic-graph==1.0.17
pydantic==2.12.2
pydantic-ai==1.1.0
pydantic-ai-slim==1.1.0
pydantic-core==2.41.4
pydantic-evals==1.1.0
pydantic-graph==1.1.0
pydantic-settings==2.11.0
pyee==13.0.0
pyfiglet==1.0.4
Expand Down Expand Up @@ -357,7 +355,7 @@ pyvis==0.3.2
pyyaml==6.0.3
pyzmq==27.1.0
ragas @ git+https://github.com/explodinggradients/ragas@5d59549ad5ef511f621502c563bc55ac5aeb9188#subdirectory=ragas
referencing==0.36.2
referencing==0.37.0
regex==2025.9.18
requests==2.32.5
requests-oauthlib==2.0.0
Expand All @@ -373,15 +371,15 @@ ruamel-yaml==0.17.4
s3transfer==0.13.1
scipy==1.16.2
send2trash==1.8.3
sentry-sdk==2.41.0
sentry-sdk==2.42.0
setuptools==80.9.0
shapely==2.1.2
shellingham==1.5.4
simpleeval==1.0.3
six==1.17.0
sniffio==1.3.1
soupsieve==2.8
sqlalchemy==2.0.43
sqlalchemy==2.0.44
sse-starlette==3.0.2
stack-data==0.6.3
stagehand==0.5.4
Expand Down Expand Up @@ -417,12 +415,12 @@ tzdata==2025.2
ujson==5.7.0
uri-template==1.3.0
urllib3==2.5.0
uv==0.9.0
uv==0.9.3
uvicorn==0.35.0
uvloop==0.21.0
uwsgi==2.0.30
uvloop==0.22.1
uwsgi==2.0.31
watchdog==6.0.0
watchfiles==1.1.0
watchfiles==1.1.1
wcwidth==0.2.14
webcolors==24.11.1
webencodings==0.5.1
Expand All @@ -434,7 +432,7 @@ wikipedia==1.4.0
wrapt==1.17.3
xxhash==3.6.0
yarl==1.22.0
youtube-transcript-api==1.2.2
youtube-transcript-api==1.2.3
zipp==3.23.0
zope-event==6.0
zope-interface==8.0.1
Expand Down
22 changes: 22 additions & 0 deletions public_dropin_environments/python311_genai_agents/run_agent.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@
import logging
import os
import socket
import subprocess
import sys
from pathlib import Path
from typing import Any, TextIO, cast
Expand Down Expand Up @@ -257,6 +258,25 @@ def run_agent_procedure(args: Any) -> None:
)


def install_extra_dependencies():
"""
Run uv sync to pick up user's extra dependencies from pyproject.toml.
"""
root.info("Syncing extra dependencies")
env = os.environ.copy()
# Unless directory where pyproject.toml is is not set, default to codespace location.
if env.get("UV_PROJECT") is None:
env["UV_PROJECT"] = "/home/notebooks/storage/"
# uv sync would recompile whole venv, which takes a lot of time. Disable it.
env["UV_COMPILE_BYTECODE"] = "0"

# Sync only extra dependencies to active venv (usually kernel) without upgrading any others.
# --frozen to skip dependency resolution and just install exactly what's in lock file
cmd = "time uv sync --frozen --active --no-progress --no-cache --group extras"
subprocess.run(cmd.split(), env=env, stdout=sys.stdout, stderr=sys.stderr, check=False)
root.info("Sync completed")


def main_stdout_redirect() -> Any:
"""
This is a wrapper around the main function that redirects stdout and stderr to a file.
Expand Down Expand Up @@ -287,6 +307,7 @@ def main_stdout_redirect() -> Any:
sys.stdout = f
sys.stderr = f

install_extra_dependencies()
try:
run_agent_procedure(args)
except Exception as e:
Expand All @@ -301,6 +322,7 @@ def main() -> Any:
setup_logging(logger=root, log_level=logging.INFO)
root.info("Parsing args")
args = argparse_args()
install_extra_dependencies()
run_agent_procedure(args)
# flush stdout and stderr to ensure all output is returned to the caller
sys.stdout.flush()
Expand Down
Loading