[BUG] Segmentation Mask Uses Instance Index Instead of Class Label and Has Data Type Limitation

[richard-xx]

Bug Report: Segmentation Mask Uses Instance Index Instead of Class Label and Has Data Type Limitation

Describe the bug

The segmentationDecode function in DetectionParserUtils.cpp has two main issues:

1. Semantic Issue: The function currently uses detection instance indices (detIdx) as values in the segmentation mask instead of using the actual class labels from DetectionCandidate.label. This makes the segmentation mask less meaningful because instance indices are arbitrary and change between runs.

2. Data Type Limitation: The segmentation mask uses uint8_t data type which limits the maximum number of distinguishable classes to 254 (since value 255 is reserved for "unassigned" pixels). When class labels exceed 254, they get mapped to the same value, making it impossible to distinguish between different classes.

Current problematic code

In the segmentationDecode function:

const int detIdx = static_cast<int>(i);  // index in outDetections list
// ...
const uint8_t value = static_cast<uint8_t>(std::min(detIdx, 254));

Minimal Reproducible Example

Using any YOLO model with instance segmentation enabled that outputs class labels greater than 254 will demonstrate both issues:

1. The segmentation mask uses instance indices rather than semantic class labels
2. Class labels above 254 all get mapped to value 254 in the mask

Expected behavior

The segmentation mask should:

1. Use class labels from DetectionCandidate.label instead of instance indices:

   const int detLabel = detectionCandidates[i].label;  // Use class label
   const uint8_t value = static_cast<uint8_t>(std::min(detLabel, 254));

2. Either support more classes through a larger data type (e.g., uint16_t) or provide warnings when class labels exceed the supported range

Additional context

These issues affect the usability of the instance segmentation feature in DepthAI. The first issue impacts semantic clarity, while the second imposes a hard limit on the number of classes that can be properly distinguished in segmentation masks.

[aljazkonec1]

Hi @richard-xx ,
I appreciate that you took the time to look into it! Regrading the raised issues:

1. Semantic Issue: We did consider storing classes instead of instances but ultimately decided against it as the notion of detections itself implies separate instances, so search should be by instance. This also solves the situation where two objects of the same class overlap and with using class segmentation mask, we would loose (potentially important) information. For ease of use, a getCvSegmentationMaskByClass() function is available. A standalone SegmentationMask message type is in the works, and will be released in the coming month or so.
2. Data Type Limitation: We were deciding between uint8 and uint16. Now, why did we decide on uint8? Two reasons:
   - We wanted to keep segmentaton data as small as possible to not limit transfer speeds.
   - Typically, NN models have a rather small input / output size ( eg YOLO has 640 x 640) and its very difficult to have a model detect 255 unique objects in a single image. Adding segmentation for 255 objects would, in the YOLO case, result in an average object size of 40 x 40 px. Having more than 255 objects would reduce that size even further and in practice would have very limited usability.

So we decided to go with uint8. Since the data is stored as instance segmentation, class labels above 255 dont have an affect on the storage, as long as there is less than 255 total detections in the image.

Regarding const uint8_t value = static_cast<uint8_t>(std::min(detLabel, 254)); the 254 is a bug and it should have been 255. Thanks for noticing and I will create a PR to adjust it! If you ran into the above two issues in your usage of the new detection parser, I would love to hear more about it so we can further improve performance 😄

Hope this helps! Thanks,

Aljaz

[richard-xx]

Thanks, I understand.

By the way, is it possible to store the instances number in ImgDetection?

In my use case, I need to use ObjectTracker to track objects and get SegmentationMask, but it seems there's a mismatch between Tracklets and ImgDetections (Masks). I think adding an index value to ImgDetection would be helpful.

[aljazkonec1]

Hi,

I see the problem and it most likely has to do with ObjectTracker dropping or changing the order ImgDetections appear. There is not a specific index value attribute currently available in ImgDetection. For a quick and "dirty" fix, you could repurpose the uint32_t label = 0; attribute for indexes instead of classes if you are not using it.

We do plan on updating object tracker to pass along the segmentation mask (by copying it over to dai::Tracklets) at the very least.

Hope this helps,
Aljaz

[richard-xx]

We do plan on updating object tracker to pass along the segmentation mask (by copying it over to dai::Tracklets) at the very least.

Thanks for sharing this — it's great to hear that there's a plan in place. Looking forward to seeing this enhancement when it's ready!