[WIP] Integration with DeepLabCut 3.0 - PyTorch Engine #121

n-poulsen · 2025-02-28T10:58:21Z

This pull requests updates DeepLabCut-Live for models exported with DeepLabCut 3.0. TensorFlow models can still be used, and the code is siloed so that only the engine used to run the code is required as a package (i.e. no need to install TensorFlow if you want to run live pose estimation with PyTorch models).

If you want to give this PR a try, you can install the code in your local conda environment by running:

pip install "git+https://github.com/DeepLabCut/DeepLabCut-live.git@dlclive3"

…in code

…NX inference

…erence

This script will be used for both benchmarking and integration testing, therefore it needs to crash if inference isnt successful

- removing until a new one can be added properly

* Update benchmark.py - changed benchmarking link; needs testing * fix #112 - don't subset cpuinfo (#114) * Introduce CI/CD for TensorFLow version (#122) * Update .gitignore * CI/CD update python version * CI/CD update actions versions * CI/CD update trigger events * CI/CD update MacOS version * dlclibrary set version to >=0.0.6 * Poetry lock * Poetry lock * Pyproject.toml update tensorflow installation * Poetry lock * Install specific tensorflow-io-gcs-filesystem for windows * Poetry lock * Update deprecated section name * Poetry lock * CI/CD test on python 3.11, 3.12, 3.13 as well * Update testing.yml - rename to main - add tables installation --------- Co-authored-by: Mackenzie Mathis <[email protected]> * master --> main * Update testing.yml * Update benchmark.py * Update README & benchmarking script (#126) * Update README.md fix test instructions * Update check_install.py * Update check_install.py * Update check_install.py * Update pyproject.toml (#125) * Update pyproject.toml - adding support for python 3.11, 3.12, 3.13 * Update testing.yml * Update testing.yml * Update pyproject.toml * Update testing.yml * Update pyproject.toml * Update poetry.lock * Update poetry.lock * Update pyproject.toml * Update pyproject.toml * Update pyproject.toml * Update pyproject.toml * Update pyproject.toml * Update pyproject.toml * Update pyproject.toml * Update pyproject.toml * Update pyproject.toml * Update testing.yml * Update pyproject.toml * Update pyproject.toml * Update pyproject.toml * Update testing.yml * Update pyproject.toml * Update pyproject.toml * Update poetry.lock * Update testing.yml * Update testing.yml * Update testing.yml - remove poetry (for now) * Update testing.yml * Update testing.yml * Update testing.yml * Update testing.yml * Update testing.yml * updated poetry.lock * updated * Update testing.yml * Update pyproject.toml * Update pyproject.toml * Update pyproject.toml * Update pyproject.toml * Update testing.yml testing 3.11 * Update testing.yml * Update pyproject.toml * Update pyproject.toml * Update pyproject.toml * Update testing.yml * updated test for win32 * Update testing.yml * Update testing.yml * Update testing.yml * Update testing.yml * Fix download_benchmarking_data() * Ignore __MACOSX directories * Run tests on all PRs * tensorflow-io-gcs-filesystem - python=3.9 * Pin python to 3.10 version * WIP refactor benchmarking: __init__.py * WIP refactor benchmarking: remove dupplicates * WIP refactor benchmarking: warning * WIP refactor benchmarking: close live * WIP refactor benchmarking: resize and pixels * WIP refactor benchmarking: get_sys_info * Update benchmark.py (#123) * WIP refactor benchmarking: Delete dupplicate scripts * WIP refactor benchmarking: Introduce Engine * WIP refactor benchmarking: benchmark() method * WIP refactor benchmarking: benchmark_videos() * CI/CD: add torch installation * WIP refactor benchmarking: extract Engine * Refactor benchmarking: model_type in benchmark_videos() * Delete poetry lock * Fix frame_batch initialization in runner.py when low detection confidence (#138) --------- Co-authored-by: Mackenzie Mathis <[email protected]> Co-authored-by: Jonny Saunders <[email protected]> Co-authored-by: Artur <[email protected]>

MMathisLab · 2026-01-13T11:59:18Z

@copilot resolve conflicts

Copilot · 2026-01-13T11:59:27Z

@MMathisLab I've opened a new pull request, #146, to work on those changes. Once the pull request is ready, I'll request review from you.

Copilot

Pull request overview

This pull request updates DeepLabCut-Live to version 3.0.0a0, adding PyTorch engine support for models exported with DeepLabCut 3.0 while maintaining backward compatibility with TensorFlow models. The code is structured to require only the engine-specific dependencies (PyTorch or TensorFlow) based on the model being used.

Changes:

Updated to version 3.0.0a0 with Python 3.10+ support
Added PyTorch runner and model infrastructure for DLC 3.0 models
Maintained TensorFlow runner for legacy model support
Updated installation documentation for both desktop and Jetson platforms
Added new test and utility scripts for benchmarking and model export

Reviewed changes

Copilot reviewed 79 out of 84 changed files in this pull request and generated 31 comments.

Show a summary per file

File	Description
pyproject.toml	Updated version to 3.0.0a0, Python to 3.10-3.11, made TensorFlow/PyTorch optional dependencies
poetry.lock	Regenerated lock file (removed old dependencies)
dlclive/version.py	Updated version string to 3.0.0a0
dlclive/utils.py	Added type hints, improved exception handling, cleaned up formatting
dlclive/processor/*.py	Modernized class definitions, cleaned up formatting
dlclive/predictor/*.py	New predictor modules for pose estimation
dlclive/pose_estimation_tensorflow/*.py	Refactored TensorFlow inference code into separate module
dlclive/pose_estimation_pytorch/*.py	New PyTorch runner and model infrastructure
tests/test_benchmark_script.py	New functional test for benchmarking
scripts/*.py	New utility scripts for export and import fixing
docs/*.md	Updated installation instructions for PyTorch and new Python versions

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-01-13T12:03:57Z

dlclive/benchmark.py

    benchmark_videos(
-        args.model_path,
-        args.video_path,
-        output=args.output,
-        resize=args.resize,
-        pixels=args.pixels,
-        cropping=args.cropping,
-        dynamic=args.dynamic,
-        n_frames=args.n_frames,
-        print_rate=args.print_rate,
+        video_path=args.video_path,
+        model_path=args.model_path,
+        model_type=args.model_type,
+        device=args.device,
+        precision=args.precision,
        display=args.display,
        pcutoff=args.pcutoff,
        display_radius=args.display_radius,
-        cmap=args.cmap,
+        resize=tuple(args.resize) if args.resize else None,
+        cropping=args.cropping,
+        dynamic=tuple(args.dynamic),
        save_poses=args.save_poses,
+        save_dir=args.save_dir,
+        draw_keypoint_names=args.draw_keypoint_names,
+        cmap=args.cmap,
+        get_sys_info=args.get_sys_info,
        save_video=args.save_video,
    )


Keyword argument 'precision' is not a supported parameter name of function benchmark_videos.
Keyword argument 'save_dir' is not a supported parameter name of function benchmark_videos.
Keyword argument 'device' is not a supported parameter name of function benchmark_videos.
Keyword argument 'draw_keypoint_names' is not a supported parameter name of function benchmark_videos.
Keyword argument 'get_sys_info' is not a supported parameter name of function benchmark_videos.

Copilot · 2026-01-13T12:03:58Z

dlclive/pose_estimation_pytorch/models/predictors/dekr_predictor.py

+            # Assuming you have 'heatmap' tensor
+            max_pooled_heatmap = predictor.max_pool(heatmap)
+        """
+        pool1 = torch.nn.MaxPool2d(3, 1, 1)


Variable pool1 is not used.

Copilot · 2026-01-13T12:03:58Z

dlclive/pose_estimation_pytorch/models/predictors/dekr_predictor.py

+        """
+        pool1 = torch.nn.MaxPool2d(3, 1, 1)
+        pool2 = torch.nn.MaxPool2d(5, 1, 2)
+        pool3 = torch.nn.MaxPool2d(7, 1, 3)


Variable pool3 is not used.

Copilot · 2026-01-13T12:03:58Z

dlclive/pose_estimation_pytorch/models/predictors/dekr_predictor.py

+        pool1 = torch.nn.MaxPool2d(3, 1, 1)
+        pool2 = torch.nn.MaxPool2d(5, 1, 2)
+        pool3 = torch.nn.MaxPool2d(7, 1, 3)
+        map_size = (heatmap.shape[1] + heatmap.shape[2]) / 2.0


Variable map_size is not used.

Copilot · 2026-01-13T12:03:58Z

dlclive/pose_estimation_pytorch/models/predictors/dekr_predictor.py

+            # Assuming you have 'heatmaps' and 'poses' tensors
+            predictor.pose_nms(heatmaps, poses)
+        """
+        pose_scores = poses[:, :, :, 2]


Variable pose_scores is not used.

Copilot · 2026-01-13T12:04:05Z

dlclive/core/inferenceutils.py

+                            assemblies.remove(store[j])
+                            for idx in store[j]._idx:
+                                store[idx] = store[i]
+                    except KeyError:


'except' clause does nothing but pass and there is no explanatory comment.

Suggested change

except KeyError:

except KeyError:

# Some links may reference indices that were never added to `store`;

# in that case we intentionally skip merging for this link.

Copilot · 2026-01-13T12:04:05Z

dlclive/core/inferenceutils.py

+    df = pd.read_hdf(h5_file)
+    try:
+        df.drop("single", axis=1, level="individuals", inplace=True)
+    except KeyError:


'except' clause does nothing but pass and there is no explanatory comment.

Suggested change

except KeyError:

except KeyError:

# The "single" individual column is optional; ignore if it is absent.

Copilot · 2026-01-13T12:04:06Z

dlclive/pose_estimation_tensorflow/graph.py

+if int(vers[0]) == 2 or int(vers[0]) == 1 and int(vers[1]) > 12:
+    tf = tf.compat.v1
+else:
+    tf = tf


This assignment assigns a variable to itself.

Copilot · 2026-01-13T12:04:06Z

dlclive/core/inferenceutils.py

+            for link in assembly._links:
+                i, j = link.idx


Nested for statement uses loop variable 'link' of enclosing for statement.

Suggested change

for link in assembly._links:

i, j = link.idx

for assembly_link in assembly._links:

i, j = assembly_link.idx

Copilot · 2026-01-13T12:04:06Z

dlclive/pose_estimation_pytorch/models/necks/layers.py

+        Returns:
+            Output tensor.
+        """
+        b, n, _, h = *x.shape, self.heads


Left hand side of assignment contains 4 variables, but right hand side is a tuple of length 2.

deruyter92 · 2026-01-13T14:42:09Z

pushed the state of this branch back to it's state before resolving merge conflicts (copilot suggestions broke the benchmark script)

dikraMasrour and others added 24 commits February 24, 2025 15:30

Remove TensorFlow references

b11743c

Add comments on TF questions for changing dlc-live pipeline to Pytorch

c8f5839

Vanilla pytorch inference done; Commenting out TensorFlow references …

030c42c

…in code

change testing directory for Anna and allow to run on CPU

d75339a

Fix display, clean code, adapt frame processing, onnx inference

4de5829

add screenshots

40a9dd4

Fix CPU inference crash + GPU (cuda) & CPU support for Pytorch and ON…

2fc48f1

…NX inference

Video analysis feature

125a072

Improvements on benchmark_pytorch.py

9d2350e

Implement TensorRT optimisation on ONNX models and FP16 precision inf…

13b8fae

…erence

Avs live feed

8b41df6

Tutorial notebook in progress

d1e7df9

bug fixing h5 saving for live video feed

2fb39fc

add code to save numbers in csv and h5 as numbers, not tensor(number)

98b3a13

add timestamp suffix to videos and csv/h5 files

e6a914a

fix live inference and display, black and isort

29c8122

cleaning out unused files

9508a79

update docstrings, clean dlclive script

d61f892

Continued DeepLabCut-Live implementation for DeepLabCut 3.0

f527d38

working on README

4c07e04

improved docs

82daf43

improved docs for PyTorch code

40005ae

improved readme

c6a1f69

fix default top down dynamic cropping parameters

d738a2c

n-poulsen added the DLC 3.0 🔥 label Feb 28, 2025

n-poulsen mentioned this pull request Feb 28, 2025

[WIP] Integration with DeepLabCut 3.0 - PyTorch Engine #120

Closed

n-poulsen requested review from AlexEMG, MMathisLab and maximpavliv February 28, 2025 10:58

n-poulsen mentioned this pull request Feb 28, 2025

MobileNets DeepLabCutAIResidency/DLC_AI2024#24

Open

maximpavliv added 3 commits June 11, 2025 16:20

Benchmark pytorch: add single_animal arg

045de53

Benchmark pytorch: dont setup videowriter if not needed

b493bef

Benchmark pytorch: fix save_poses_to_files()

dfe5817

maximpavliv force-pushed the dlclive3 branch from 5a7a976 to dfe5817 Compare June 13, 2025 08:42

maximpavliv and others added 5 commits June 13, 2025 15:13

Benchmark pytorch: docstring

c6ee1ba

Benchmark pytorch: formatting

9b5d8f1

Benchmark pytorch: introduce n_frames and progress bar

a29415d

Benchmark pytorch: remove try-except block

3660dcf

This script will be used for both benchmarking and integration testing, therefore it needs to crash if inference isnt successful

Delete poetry.lock

f8fb374

- removing until a new one can be added properly

maximpavliv mentioned this pull request Jun 16, 2025

[WIP] Continue work on Integration with DeepLabCut 3.0 #127

Merged

maximpavliv and others added 2 commits August 22, 2025 11:44

SimCCPredictor: add visilibity computation (#131)

a096c94

MMathisLab requested review from C-Achard, Copilot, deruyter92 and maximpavliv and removed request for AlexEMG and maximpavliv January 13, 2026 11:57

Copilot started reviewing on behalf of MMathisLab January 13, 2026 11:59 View session

Copilot AI mentioned this pull request Jan 13, 2026

Merge main branch into dlclive3 to resolve conflicts #146

Closed

Copilot AI reviewed Jan 13, 2026

View reviewed changes

MMathisLab assigned MMathisLab, deruyter92 and C-Achard Jan 13, 2026

deruyter92 force-pushed the dlclive3 branch from 13e9666 to ac75b2a Compare January 13, 2026 14:38

	except KeyError:
	except KeyError:
	# The "single" individual column is optional; ignore if it is absent.

[WIP] Integration with DeepLabCut 3.0 - PyTorch Engine #121

Are you sure you want to change the base?

[WIP] Integration with DeepLabCut 3.0 - PyTorch Engine #121

Uh oh!

Conversation

n-poulsen commented Feb 28, 2025

Uh oh!

MMathisLab commented Jan 13, 2026

Uh oh!

Copilot AI commented Jan 13, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Jan 13, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 13, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 13, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 13, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 13, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 13, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 13, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 13, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 13, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 13, 2026

Choose a reason for hiding this comment

Uh oh!

deruyter92 commented Jan 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants