Add SanitizeKeyPoints transform to remove keypoints outside of the image area
#9235
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/vision/9235
Note: Links to docs will display an error until the docs builds have been completed. ✅ No FailuresAs of commit 1a3ac36 with merge base b851874 ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
Summary: Fixing error `TypeError: unsupported operand type(s) for |: 'type' and 'type'`
|
@Callidior, a huge thanks for the high quality PR. After careful consideration, I did several changes, let let know what you think and if you are aligned with the current version. Here is a list of my modifications:
Illustration of the changesI apply below this logic to the example used for #9236. orig_pts = tv_tensors.KeyPoints(
[
[
[445, 700], # nose
[320, 660],
[370, 660],
[420, 660], # left eye
[300, 620],
[420, 620], # left eyebrow
[475, 665],
[515, 665],
[555, 655], # right eye
[460, 625],
[560, 600], # right eyebrow
[370, 780],
[450, 760],
[540, 780],
[450, 820], # mouth
],
],
canvas_size=(orig_img.size[1], orig_img.size[0]),
# clamping_mode="soft"
)
cropper = v2.RandomCrop(size=(128, 128))
crops = [cropper((orig_img, orig_pts)) for _ in range(4)]
plot([(orig_img, orig_pts)] + crops)
If we keep the points grouped sanitizer = v2.SanitizeKeyPoints()
sanitized_pts = [sanitizer(pts) for _, pts in crops]
for pts in sanitized_pts:
print(pts)
# KeyPoints([], size=(0, 15, 2), dtype=torch.int64, canvas_size=(128, 128))
# KeyPoints([], size=(0, 15, 2), dtype=torch.int64, canvas_size=(128, 128))
# KeyPoints([], size=(0, 15, 2), dtype=torch.int64, canvas_size=(128, 128))
# KeyPoints([], size=(0, 15, 2), dtype=torch.int64, canvas_size=(128, 128))If we ungroup them: sanitizer = v2.SanitizeKeyPoints()
sanitized_pts = [sanitizer(tv_tensors.wrap(pts.squeeze(0), like=pts)) for _, pts in crops]
for pts in sanitized_pts:
print(pts)
# KeyPoints([], size=(0, 2), dtype=torch.int64, canvas_size=(128, 128))
# KeyPoints([], size=(0, 2), dtype=torch.int64, canvas_size=(128, 128))
# KeyPoints([[ 6, 76],
# [ 56, 76],
# [106, 76],
# [106, 36]], canvas_size=(128, 128))
# KeyPoints([[ 16, 106],
# [ 56, 106],
# [ 96, 96],
# [ 1, 66],
# [101, 41]], canvas_size=(128, 128))Testingpytest test/test_transforms_v2.py -vvv -k "SanitizeKeyPoints"
52 passed, 9718 deselected in 1.14s |
|
@AntoineSimoulin Thanks for your help with the PR! I would also be fine with including the reduced version of the transform you proposed. It does simplify the code a lot, indeed. I initially included those additional arguments because we have |
Context
This PR proposes to add a
SanitizeKeyPointstransform, similar to the existingSanitizeBoundingBoxes(#7246). This transform removes keypoints lying outside of the valid image area, which can happen after geometric transformations with the new defaultclamping_modeproposed in #9234, which allows for disabling automatic clamping of keypoints.This implementation follows the proposal in issue #9223 as a solution for the issue that the previous default clamping of keypoints to the image edges modifies their position and creates a misalignment with the actual locations in the transformed image.
This PR hence only makes sense in combination with new clamping modes such as the one proposed in #9234.
Implementation details
To understand the behavior of the proposed
SanitizeKeyPointstransform, we need to distinguish two cases of keypoint formats:tv_tensors.KeyPointscontains a set of keypoints of shape[N_points, 2]or[N_points, 1, 2]. In this case, the transform will remove all keypoints lying outside of the valid image region.tv_tensors.KeyPointscontains groups of keypoints, i.e., several objects, each consisting of a certain number of keypoints (e.g., polygons, polygonal chains, skeletons etc.). It is a tensor of shape[N_objects, N_points, 2]or, in general,[N_objects, ..., 2]. In this case, the transform will remove all objects (first dimension) that have at least a certain number of keypoints lying outside of the valid image region.The behavior of the transform can be controlled with the following arguments:
min_valid_edge_distance(int): The minimum distance that keypoints need to be away from the closest image edge along any axis in order to be considered valid. For example, setting this to 0 will only invalidate/remove keypoints outside of the image area, while a value of 1 will also remove keypoints lying exactly on the edge. Default is 0.min_invalid_points(int or float): Minimum number or fraction of invalid keypoints required for a group of keypoints to be removed. For example, setting this to 1 will remove a group of keypoints if any of its keypoints is invalid, while setting it to 2 will only remove groups with at least 2 invalid keypoints. If a float in(0.0, 1.0]is passed, it represents a fraction of the total number of keypoints in the group. For example, setting this to 0.3 will remove groups of keypoints with at least 30% invalid keypoints. Note that a value of1(integer) is very different from1.0(float). The former will remove groups with any invalid keypoint, while the latter will only remove groups where all keypoints are invalid. Default is 1 (int).In addition, the transform can also remove labels associated with the keypoints (or elements from any other tensors with the same first dimension as the keypoints). This can be achieved by setting the
labels_getterargument, which follows the same logic as the homonymous argument ofSanitizeBoundingBoxes. The only difference is, that the default forSanitizeKeyPointsisNone, in order to avoid accidental conflicts with any additionally present bounding box labels.Illustration of the changes
The following example additionally requires PR #9234.
Unsanitized keypoint coordinates:
Sanitization:
Testing
Please run the following unit tests: