Jaap/fix no detections case #140
Conversation
Instead using a single_animal parameter for PytorchRunner, which defaults to True, single_animal mode will be inferred from the models metadata configuration. This is useful for cases when you want to safely leave out the single_animal paramter, e.g. when running a multi-animal model in DeepLabCut-live-GUI, just passing a model configuration suffices.
When the detector does not detect any crops (with a supra-threshold confidence), no pose-detection is applied and and a zero-vector is returned for the pose-prediction. This solves DeepLabCut#137 and copies the already existing intended behavior of returning zeros for empty pose-predictions in single animals.
|
@deruyter92 can you fix conflicts? |
There was a problem hiding this comment.
Pull request overview
This PR improves the handling of cases where no detections are found and refactors how multi-animal mode is determined. The changes skip pose estimation entirely when there are no detections and return an all-zeros pose, while also inferring the single vs. multi-animal mode from the model's metadata configuration rather than requiring it as a parameter.
Changes:
- Returns all-zeros pose when no supra-threshold detections are found, skipping the pose estimation step
- Infers single/multi-animal mode from model metadata, deprecating the
single_animalparameter - Adds
n_individualsandn_bodypartsattributes extracted from model configuration
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| np.zeros((self.n_bodyparts, 3)) | ||
| if self.n_individuals < 2 else | ||
| np.zeros((self.n_individuals, self.n_bodyparts, 3)) | ||
| ) |
There was a problem hiding this comment.
When returning early due to no detections, the skip_frames state is not updated. This means the age counter doesn't get incremented and the cached detections don't get cleared. This could lead to stale detections being used in subsequent frames or incorrect frame skipping behavior. Consider updating the skip_frames state before returning, or restructuring the logic to ensure skip_frames.update() is always called when skip_frames is enabled.
| ) | |
| ) | |
| if self.top_down_config.skip_frames is not None: | |
| zero_pose_tensor = torch.from_numpy(zero_pose).to(self.device) | |
| self.top_down_config.skip_frames.update(zero_pose_tensor, w, h) |
| np.zeros((self.n_bodyparts, 3)) | ||
| if self.n_individuals < 2 else | ||
| np.zeros((self.n_individuals, self.n_bodyparts, 3)) |
There was a problem hiding this comment.
The early return uses self.n_individuals < 2 to determine the output shape, while the regular path uses self.single_animal at line 240. These two conditions could differ if a user explicitly sets single_animal to a value that doesn't match the model configuration. For consistency, both code paths should use the same condition (either both use self.single_animal or both use self.n_individuals < 2).
| np.zeros((self.n_bodyparts, 3)) | |
| if self.n_individuals < 2 else | |
| np.zeros((self.n_individuals, self.n_bodyparts, 3)) | |
| np.zeros((self.n_bodyparts, 3)) | |
| if self.single_animal | |
| else np.zeros((self.n_individuals, self.n_bodyparts, 3)) |
| zero_pose = ( | ||
| np.zeros((self.n_bodyparts, 3)) | ||
| if self.n_individuals < 2 else | ||
| np.zeros((self.n_individuals, self.n_bodyparts, 3)) | ||
| ) |
There was a problem hiding this comment.
If get_pose is called before load_model is called, and the model has a detector with no detections, this will fail with a TypeError when comparing self.n_individuals (which is None) with 2. While calling get_pose before load_model is already a misuse of the API, the error message could be improved. Consider adding a check that self.n_individuals and self.n_bodyparts are not None, or document that load_model must be called first.
Co-authored-by: Copilot <[email protected]>
|
@deruyter92 can you check suggestions and fix conflicts, thanks! |
3 participants
This PR introduces two changes:
get_pose(..)returns a pose with all-zeros. Fixes TopDown models fail when detector has a low-confidence prediction #137, and follows earlier intended implementation for single animals. Now it should work for multiple detections as well, and the pose-estimation step is entirely skipped when there are no detections.load_modelstage, rather than the argumentsingle_animalat initialization. This has the advantage that a PytorchPoseRunner can be initialized with only an exported model path, without defaulting to single animal mode.