Skip to content

iterative/dvc-bench

Folders and files

NameName
Last commit message
Last commit date
Aug 21, 2024
Jan 30, 2025
Aug 28, 2024
Aug 27, 2024
May 14, 2020
Mar 29, 2022
Jan 30, 2025
Sep 30, 2021
Sep 23, 2024
Jun 30, 2022
Aug 21, 2024
Aug 21, 2024
Aug 21, 2024

Repository files navigation

dvc-bench

Benchmarking dvc with pytest-benchmark.

Daily benchmark results

Visit bench.dvc.org

Dispatch workflow

Trigger a dispatch workflow with desired dataset and revisions and see results in bench.dvc.org/run_ID_ATTEMPT.html, where ID is github.run_id and ATTEMPT is github.run_attempt. For example, for https://github.com/iterative/dvc-bench/actions/runs/7119039172/attempts/2 it would be http://bench.dvc.org/run_7119039172_2.html

Setting up

$ uv pip install -r requirements.txt
$ dvc pull # optional, otherwise will pull datasets dynamically

Running all benchmarks

$ pytest --pyargs dvc.testing.benchmarks

Running one benchmark

$ pytest --pyargs dvc.testing.benchmarks.cli.commands.test_add

CLI options

$ pytest -h
...
  --dataset=DATASET
                        Dataset name to use in tests (e.g. tiny/small/large/mnist/etc)
  --dvc-bin=DVC_BIN     Path to dvc binary
  --dvc-revs=DVC_REVS   Comma-separated list of DVC revisions to test (overrides `--dvc-bin`)
  --dvc-repo=DVC_GIT_REPO
                        Path or url to dvc git repo
  --dvc-bench-repo=DVC_BENCH_GIT_REPO
                        Path or url to dvc-bench git repo (for loading benchmark dataset)
  --dvc-install-deps=DVC_INSTALL_DEPS
                        Comma-separated list of DVC installation packages
  --project-rev=PROJECT_REV
                        Project revision to test
  --project-repo=PROJECT_GIT_REPO
                        Path or url to dvc project
...

Comparing results

$ pytest-benchmark compare --histogram histograms/ --group-by name --sort name --csv results.csv

and if you want beautiful plots:

$ dvc repro
$ dvc plots show

Contributing

Benchmark test definitions are now part of dvc.testing.