Deciding on features for evalem.cv

[rbavery]

So far we have discussed

• running on N batches, chosen by the user
• selecting the correct batch size depending on the amount of GPU memory.
• defaulting to CPU if no gpu is available
• supporting flexible array types (ndarray, torch tensor, cupy array)
• model specific input processor, possibly using transformers, see integration with huggingface transformers? #18
• selecting the correct set of metrics for semantic segmentation (initially, and then other downstream tasks)
• adding dependencies for torchmetrics, possibly torcheval if needed for semantic segmentation (definitely for object detection, COCO mAP)

[weiji14]

Initial focus will probably be on supporting different dowstream Computer Vision tasks (segmentation, detection, classification), which would require wrapping around torchmetrics, smp, mmeval or some other metric library. We can start with torch.Tensor as the array format to get a proof of concept going, but just keep in mind about flexible array types later.

[NISH1001]

@weiji14 @rbavery I am following up if we can add any metric implementation for evalem.cv.metrics.

[rbavery]

Yes we can, right now I'm focused on a higher priority task to get a reproducible evaluation example in the hls foundation repo. Setting up the aws vm with all the dependencies and files has taken some time but that should be done today and I can start on this after.

[NISH1001]

Eh okay. No worries. Just keeping this thread in the loop.