To build the agent on Windows, see datadog-agent-buildimages.
The Agent embeds a full-fledged CPython interpreter, so it requires the development files to be available in the dev env. The Agent can embed Python2 and/or Python3, you will need development files for all versions you want to support.
If you're on OSX/macOS, installing Python 2.7 and/or 3.12 with Homebrew brings along all the development files needed:
Please note that not using Python versions explicitly supported, you may have problems running the built Agent's Python checks, especially if using a virtualenv. At this time, only Python 3.12 is confirmed to work as expected in the development environment.
brew install python@2
brew install [email protected]
On Linux, depending on the distribution, you might need to explicitly install the development files, for example on Ubuntu:
sudo apt-get install python2.7-dev
sudo apt-get install python3.12-dev
On Windows, install Python 2.7 and/or 3.12 via the official installer.
Invoke is a task runner written in Python that is extensively used in this project to orchestrate builds and test runs. To run the tasks, you need to have it installed on your machine. We offer two different ways to run our invoke tasks.
The deva CLI tool is a single binary that can be used to install and manage the development environment for the Agent, built by the Datadog team. It will install all the necessary Python dependencies for you. The development environment will be completely independent of your system Python installation. This tool leverages PyApp, a wrapper for Python applications that bootstrap themselves at runtime. In our case, we wrap invoke itself and include the dependencies needed to work on the Agent.
To install deva, you'll need to:
- Download the binary for your platform from the releases page,
- Make it executable (and optionally add it to your PATH),
- Run the invoke command you need, using
devain place ofinvokeorinv.
The Python environment will automatically be created on the first run. and will be reused for subsequent runs. For example:
cd datadog-agent
curl -L -o deva https://github.com/DataDog/datadog-agent-devtools/releases/download/deva-v1.0.0/deva-aarch64-unknown-linux-gnu-1.0.0
chmod +x deva
./deva linter.goBelow a live demo of how the tool works:
Datadog also provides an invoke task to perform some sanity checks and enable some tools. To use it, run:
> deva setup
Fetching git repository...
Checking main branch...
Checking Python version...
Checking Go version...
Updating Python dependencies...
Updating Python dependencies from requirements.txt...
Updating Python dependencies from tasks/requirements.txt...
Downloading go tools...
Installing go tools...
Enabling pre-commit...
Results:
Check git repository OK
Check Python version OK
Check Go version OK
Update Python dependencies from requirements.txt OK
Update Python dependencies from tasks/requirements.txt OK
Download Go tools OK
Install Go tools OK
Enable pre-commit OK
Setup completed successfully.If you want to uninstall deva, you can simply run the ./deva self remove command, which will remove the virtual environment from your system, and remove the binary. That's it.
To protect and isolate your system-wide python installation, a python virtual environment is highly recommended (though optional). It will help keep a self-contained development environment and ensure a clean system Python.
Please note that due to the way
some virtual environments handle executable paths
(e.g. python -m venv), not all virtual environment options will be able to run the built
agent correctly. At this time, the only confirmed virtual environment that is known for
sure to work is virtualenv.
- Install the virtualenv module:
python3 -m pip install virtualenv - Create the virtual environment:
virtualenv $GOPATH/src/github.com/DataDog/datadog-agent/venv - Activate the virtualenv (OS-dependent). This must be done for every new terminal before you start.
If using virtual environments when running the built Agent, you may need to override the built Agent's search path for Python check packages using the PYTHONPATH variable (your target path must have the pre-requisite core integration packages installed though).
PYTHONPATH="./venv/lib/python3.12/site-packages:$PYTHONPATH" ./agent run ...See also some notes in ./checks about running custom python checks.
Our invoke tasks are only compatible with Python 3, thus you will need to use Python 3 to run them.
Though you may install invoke in a variety of way we suggest you use the provided requirements file and pip:
pip install -r tasks/requirements.txtThis procedure ensures you not only get the correct version of invoke, but also any additional python dependencies our development workflow may require, at their expected versions. It will also pull other handy development tools/deps (reno, or docker).
You must install Golang version 1.22.8 or
higher. Make sure that $GOPATH/bin is in your $PATH otherwise invoke
cannot use any additional tool it might need.
Please note that versions of Golang that aren't an exact match to the version specified in our build images (see e.g. here) may not be able to build the agent and/or the rtloader binary properly.
From the root of datadog-agent, run deva install-tools to install go tooling. This uses go to install the necessary dependencies.
When working on the Agent codebase you can choose among two different ways to build the binary, informally named System and Embedded builds. For most contribution scenarios you should rely on the System build (the default) and use the Embedded one only for specific use cases. Let's explore the differences.
System builds use your operating system's standard system libraries to satisfy the Agent's external dependencies. Since, for example, macOS 10.11 may provide a different version of Python than macOS 10.12, system builds on each of these platforms may produce different Agent binaries. If this doesn't matter to you—perhaps you just want to contribute a quick bugfix—do a System build; it's easier and faster than an Embedded build. System build is the default for all build and test tasks, so you don't need to configure anything there. But to make sure you have system copies of all the Agent's dependencies, skip the Embedded build section below and read on to see how to install them via your usual package manager (apt, yum, brew, etc).
Embedded builds download specifically-versioned dependencies and compile them locally from sources. We run Embedded builds to create Datadog's official Agent releases (i.e. RPMs, debs, etc), and while you can run the same builds while developing locally, the process is as slow as it sounds. Hence, you should only use them when you care about reproducible builds. For example:
- you want to build an agent binary that can be used as-is to replace the binary of an existing agent installation
- some dependencies are not available on your system
- you're working or debugging at a very low level: let's say you're adding a function to the Python bindings, you want to make sure you're using the exact same versions of Python as the official Agent packages
Embedded builds rely on Omnibus to download
and build dependencies, so you need a recent ruby environment with bundler
installed. See how to build Agent packages with Omnibus for more
details.
The agent is able to collect systemd journal logs using a wrapper on the systemd utility library.
On Ubuntu/Debian:
sudo apt-get install libsystemd-dev
On Redhat/CentOS:
sudo yum install systemd-devel
If you want to build a Docker image containing the Agent, or if you wan to run system and integration tests you need to run a recent version of Docker in your dev environment.
We use Doxygen to generate the documentation for the rtloader part of the Agent.
To generate it (using the deva rtloader.generate-doc command), you'll need to have Doxygen installed on your system and available in your $PATH. You can compile and install Doxygen from source with the instructions available here.
Alternatively, you can use already-compiled Doxygen binaries from here.
To get the dependency graphs, you may also need to install the dot executable from graphviz and add it to your $PATH.
It is optional but recommended to install pre-commit to run a number of checks done by the CI locally.
To install pre-commit, follow these instructions. The deva setup automatically enables the hooks, but you can do it manually running: pre-commit install.
The shellcheck pre-commit hook requires having the shellcheck binary installed and in your $PATH.
To install it, run:
deva install-shellcheck --destination <path>(by default, the shellcheck binary is installed in /usr/local/bin).
If you want to skip pre-commit for a specific commit you can add --no-verify to the git commit command.
If you want to run one of the checks manually, you can run pre-commit run <check name>.
You can run it on all files with the --all-files flag.
pre-commit run ruff --all-files # run ruff on all filesSee pre-commit run --help for further options.
To configure your IDE to work with datadog-agent repository, specify build tags in .vscode/settings. Run the following command to create the file:
inv setup.vscode-settings
Microsoft Visual Studio Code with the devcontainer plugin allow to use a container as remote development environment in vscode. It simplify and isolate the dependencies needed to develop in this repository.
To configure the vscode editor to use a container as remote development environment you need to:
- Install the devcontainer plugin and the golang language plugin.
- Run the following invoke command
deva vscode.setup-devcontainer --image "<image name>". This command will create the devcontainer configuration file./devcontainer/devcontainer.json. - Start or restart your vscode editor.
- A pop-up should show-up to propose to "reopen in container" your workspace.
- The first start, it might propose you to install the golang plugin dependencies/tooling.
Microsoft Visual Studio Code is recommended as it's lightweight and versatile.
Building on Windows requires multiple 3rd-party software to be installed. To avoid the complexity, Datadog recommends to make the code change in VS Code, and then do the build in Docker image. For complete information, see Build the Agent packages