Discrepancy in reproducing SurroundOcc baseline on TartanGround

[ly3580]

Hi TartanGround Authors,

Thanks for releasing this great dataset and codebase!

We are currently trying to reproduce the SurroundOcc baseline on the ModularNeighborhood / Data_anymal / P2000 trajectory. However, we are seeing a large performance gap and hope to get your expert advice.

Our rerendered inputs yield a binary Occupancy IoU of ~0.07 (compared to the reported ~0.19). Our pipeline is as follows:

Load GT from sem_occ/*.npz.

Generate nuScenes-style 6-camera images via customization_example.py.

Run zero-shot inference using the pre-trained SurroundOcc checkpoint.

Evaluate binary IoU within +/-25m xy range at 0.5m resolution.

We have already verified basic settings like camera ordering, GT voxel alignment, and image resolution, but the gap remains.

To help us align our pipeline with your original experiments, could you kindly share some insights on:

Data Pipeline: Does the paper use the exact customization_example.py script, or are there additional pre/post-processing steps?

Evaluation Split: Were the reported results evaluated on the entire P2000 trajectory or a specific subset?

Scripts: If possible, would you be open to sharing the exact evaluation script or data preparation code? It would be incredibly helpful for our baseline setup.

Thank you so much for your time and guidance!

[frischzenger]

Their labels are inconsistent， surroundocc trained with 16 classes， but tartanground has 1447 classes！and the labels of tartanground are also extremely messy.

[ly3580]

Their labels are inconsistent， surroundocc trained with 16 classes， but tartanground has 1447 classes！and the labels of tartanground are also extremely messy.

Thanks for the comment. But our current evaluation is on binary occupancy IoU rather than semantic IoU. In our setup, we binarize the labels and only evaluate whether each voxel is occupied or free. So in principle, the difference between 16 semantic classes and 1447 semantic classes may not by itself explain the performance gap.

[manthan99]

Hi @ly3580

We use the first 1000 poses of the P0002 of ModularNeighborhood for the numbers reported in the paper. I can share some script over email if you send me an email. It wont be perfect but could be a good reference to use. Feel free to shoot me an email.

Best
Manthan

[ly3580]

Hi @ly3580

We use the first 1000 poses of the P0002 of ModularNeighborhood for the numbers reported in the paper. I can share some script over email if you send me an email. It wont be perfect but could be a good reference to use. Feel free to shoot me an email.

Best Manthan

Thanks a lot! That would be very helpful.

My email is yueliang@tongji.edu.cn.

Best,
Yue