推理结果保存不一致 #12420
Description
在做多数据集推理的时候，第一个数据集的推理结果和最终的写入结果不一致。
12/16 12:06:08 - mmengine - INFO - Evaluating bbox...
Loading and preparing results...
DONE (t=0.78s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
12/16 12:06:10 - mmengine - INFO - start multi processing evaluation ...
0it [00:00, ?it/s]
0it [00:00, ?it/s]
0it [00:00, ?it/s]
0it [00:00, ?it/s]
0it [00:00, ?it/s]
0it [00:00, ?it/s]
0it [00:00, ?it/s]
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████| 105700/105700 [00:17<00:00, 5875.04it/s]
DONE (t=25.80s).
Accumulating evaluation results...
DONE (t=2.46s).
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=1000 ] = 0.232
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=1000 ] = 0.413
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=1000 ] = 0.225
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.014
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.146
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.434
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.328
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=300 ] = 0.328
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=1000 ] = 0.328
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.036
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.242
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.560
12/16 12:06:38 - mmengine - INFO -
+------------+-------+--------+--------+-------+-------+-------+
| category | mAP | mAP_50 | mAP_75 | mAP_s | mAP_m | mAP_l |
+------------+-------+--------+--------+-------+-------+-------+
| person | 0.237 | 0.454 | 0.222 | 0.015 | 0.243 | 0.519 |
| truck | 0.179 | 0.275 | 0.201 | 0.002 | 0.037 | 0.324 |
| rider | 0.239 | 0.457 | 0.219 | 0.003 | 0.203 | 0.477 |
| bus | 0.263 | 0.37 | 0.303 | 0.023 | 0.052 | 0.395 |
| bicycle | 0.152 | 0.331 | 0.123 | 0.012 | 0.11 | 0.331 |
| motorcycle | 0.16 | 0.37 | 0.101 | 0.014 | 0.104 | 0.294 |
| car | 0.395 | 0.634 | 0.407 | 0.028 | 0.272 | 0.695 |
+------------+-------+--------+--------+-------+-------+-------+
12/16 12:06:38 - mmengine - INFO - bbox_mAP_copypaste: 0.232 0.413 0.225 0.014 0.146 0.434
================dw/daytime_foggy================
12/16 12:06:41 - mmengine - INFO - Evaluating bbox...
Loading and preparing results...
DONE (t=0.12s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
12/16 12:06:42 - mmengine - INFO - start multi processing evaluation ...
0it [00:00, ?it/s]
0it [00:00, ?it/s]
0it [00:00, ?it/s]
0it [00:00, ?it/s]
0it [00:00, ?it/s]
0it [00:00, ?it/s]
0it [00:00, ?it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████| 98028/98028 [00:13<00:00, 7256.59it/s]
DONE (t=20.17s).
Accumulating evaluation results...
DONE (t=2.10s).
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=1000 ] = 0.219
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=1000 ] = 0.412
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=1000 ] = 0.207
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.080
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.236
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.320
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.342
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=300 ] = 0.342
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=1000 ] = 0.342
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.175
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.372
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.450
12/16 12:07:04 - mmengine - INFO -
+------------+-------+--------+--------+-------+-------+-------+
| category | mAP | mAP_50 | mAP_75 | mAP_s | mAP_m | mAP_l |
+------------+-------+--------+--------+-------+-------+-------+
| person | 0.188 | 0.439 | 0.13 | 0.104 | 0.243 | 0.261 |
| truck | 0.3 | 0.492 | 0.321 | 0.082 | 0.25 | 0.42 |
| rider | 0.11 | 0.266 | 0.073 | 0.032 | 0.192 | 0.117 |
| bus | 0.292 | 0.454 | 0.317 | 0.066 | 0.221 | 0.439 |
| bicycle | 0.145 | 0.328 | 0.113 | 0.031 | 0.178 | 0.24 |
| motorcycle | 0.089 | 0.169 | 0.09 | 0.049 | 0.115 | 0.108 |
| car | 0.408 | 0.735 | 0.405 | 0.195 | 0.451 | 0.657 |
+------------+-------+--------+--------+-------+-------+-------+
12/16 12:07:04 - mmengine - INFO - bbox_mAP_copypaste: 0.219 0.412 0.207 0.080 0.236 0.320
================dw/dusk_rainy================
12/16 12:07:07 - mmengine - INFO - Evaluating bbox...
Loading and preparing results...
DONE (t=0.14s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
12/16 12:07:08 - mmengine - INFO - start multi processing evaluation ...
0it [00:00, ?it/s]
0it [00:00, ?it/s]
0it [00:00, ?it/s]
0it [00:00, ?it/s]
0it [00:00, ?it/s]
0it [00:00, ?it/s]
0it [00:00, ?it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████| 69832/69832 [00:10<00:00, 6691.41it/s]
DONE (t=15.94s).
Accumulating evaluation results...
DONE (t=1.75s).
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=1000 ] = 0.104
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=1000 ] = 0.210
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=1000 ] = 0.091
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.021
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.087
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.171
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.210
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=300 ] = 0.210
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=1000 ] = 0.210
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.061
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.190
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.300
12/16 12:07:25 - mmengine - INFO -
+------------+-------+--------+--------+-------+-------+-------+
| category | mAP | mAP_50 | mAP_75 | mAP_s | mAP_m | mAP_l |
+------------+-------+--------+--------+-------+-------+-------+
| person | 0.078 | 0.21 | 0.035 | 0.047 | 0.096 | 0.096 |
| truck | 0.158 | 0.279 | 0.159 | 0.008 | 0.122 | 0.212 |
| rider | 0.045 | 0.093 | 0.036 | 0.001 | 0.032 | 0.14 |
| bus | 0.189 | 0.295 | 0.211 | 0.0 | 0.096 | 0.279 |
| bicycle | 0.047 | 0.115 | 0.028 | 0.018 | 0.051 | 0.063 |
| motorcycle | 0.006 | 0.013 | 0.006 | 0.0 | 0.01 | 0.011 |
| car | 0.206 | 0.463 | 0.161 | 0.075 | 0.206 | 0.398 |
+------------+-------+--------+--------+-------+-------+-------+
12/16 12:07:25 - mmengine - INFO - bbox_mAP_copypaste: 0.104 0.210 0.091 0.021 0.087 0.171
================dw/night_rainy================
12/16 12:07:37 - mmengine - INFO - Evaluating bbox...
Loading and preparing results...
DONE (t=2.46s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
12/16 12:07:44 - mmengine - INFO - start multi processing evaluation ...
0it [00:00, ?it/s]
0it [00:00, ?it/s]
0it [00:00, ?it/s]
0it [00:00, ?it/s]
0it [00:00, ?it/s]
0it [00:00, ?it/s]
0it [00:00, ?it/s]
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████| 732424/732424 [01:32<00:00, 7887.07it/s]
DONE (t=132.57s).
Accumulating evaluation results...
DONE (t=20.25s).
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=1000 ] = 0.228
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=1000 ] = 0.448
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=1000 ] = 0.203
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.056
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.208
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.398
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.379
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=300 ] = 0.379
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=1000 ] = 0.379
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=1000 ] = 0.171
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=1000 ] = 0.368
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=1000 ] = 0.545
12/16 12:10:17 - mmengine - INFO -
+------------+-------+--------+--------+-------+-------+-------+
| category | mAP | mAP_50 | mAP_75 | mAP_s | mAP_m | mAP_l |
+------------+-------+--------+--------+-------+-------+-------+
| person | 0.245 | 0.547 | 0.18 | 0.113 | 0.316 | 0.419 |
| truck | 0.296 | 0.465 | 0.329 | 0.045 | 0.223 | 0.42 |
| rider | 0.147 | 0.345 | 0.089 | 0.017 | 0.152 | 0.354 |
| bus | 0.299 | 0.442 | 0.341 | 0.027 | 0.125 | 0.462 |
| bicycle | 0.167 | 0.422 | 0.1 | 0.027 | 0.168 | 0.252 |
| motorcycle | 0.086 | 0.208 | 0.06 | 0.025 | 0.09 | 0.195 |
| car | 0.358 | 0.708 | 0.323 | 0.136 | 0.385 | 0.682 |
+------------+-------+--------+--------+-------+-------+-------+
12/16 12:10:17 - mmengine - INFO - bbox_mAP_copypaste: 0.228 0.448 0.203 0.056 0.208 0.398
================dw/night_sunny================
[rank0]:W1216 12:10:18.888000 4137771 site-packages/torch/distributed/distributed_c10d.py:2945] _object_to_tensor size: 2506 hash value: 2944332299956845755
[rank1]:W1216 12:10:18.894000 4137772 site-packages/torch/distributed/distributed_c10d.py:2960] _tensor_to_object size: 2506 hash value: 2944332299956845755
12/16 12:10:19 - mmengine - INFO - Iter(val) [1123/1123] dw/daytime_foggy/person_precision: 0.2340 dw/daytime_foggy/truck_precision: 0.1694 dw/daytime_foggy/rider_precision: 0.2360 dw/daytime_foggy/bus_precision: 0.2586 dw/daytime_foggy/bicycle_precision: 0.1476 dw/daytime_foggy/motorcycle_precision: 0.1546 dw/daytime_foggy/car_precision: 0.3874 dw/daytime_foggy/bbox_mAP: 0.2268 dw/daytime_foggy/bbox_mAP_50: 0.4060 dw/daytime_foggy/bbox_mAP_75: 0.2186 dw/daytime_foggy/bbox_mAP_s: 0.0132 dw/daytime_foggy/bbox_mAP_m: 0.1404 dw/daytime_foggy/bbox_mAP_l: 0.4276 dw/dusk_rainy/person_precision: 0.1880 dw/dusk_rainy/truck_precision: 0.3000 dw/dusk_rainy/rider_precision: 0.1100 dw/dusk_rainy/bus_precision: 0.2920 dw/dusk_rainy/bicycle_precision: 0.1450 dw/dusk_rainy/motorcycle_precision: 0.0890 dw/dusk_rainy/car_precision: 0.4080 dw/dusk_rainy/bbox_mAP: 0.2190 dw/dusk_rainy/bbox_mAP_50: 0.4120 dw/dusk_rainy/bbox_mAP_75: 0.2070 dw/dusk_rainy/bbox_mAP_s: 0.0800 dw/dusk_rainy/bbox_mAP_m: 0.2360 dw/dusk_rainy/bbox_mAP_l: 0.3200 dw/night_rainy/person_precision: 0.0780 dw/night_rainy/truck_precision: 0.1580 dw/night_rainy/rider_precision: 0.0450 dw/night_rainy/bus_precision: 0.1890 dw/night_rainy/bicycle_precision: 0.0470 dw/night_rainy/motorcycle_precision: 0.0060 dw/night_rainy/car_precision: 0.2060 dw/night_rainy/bbox_mAP: 0.1040 dw/night_rainy/bbox_mAP_50: 0.2100 dw/night_rainy/bbox_mAP_75: 0.0910 dw/night_rainy/bbox_mAP_s: 0.0210 dw/night_rainy/bbox_mAP_m: 0.0870 dw/night_rainy/bbox_mAP_l: 0.1710 dw/night_sunny/person_precision: 0.2450 dw/night_sunny/truck_precision: 0.2960 dw/night_sunny/rider_precision: 0.1470 dw/night_sunny/bus_precision: 0.2990 dw/night_sunny/bicycle_precision: 0.1670 dw/night_sunny/motorcycle_precision: 0.0860 dw/night_sunny/car_precision: 0.3580 dw/night_sunny/bbox_mAP: 0.2280 dw/night_sunny/bbox_mAP_50: 0.4480 dw/night_sunny/bbox_mAP_75: 0.2030 dw/night_sunny/bbox_mAP_s: 0.0560 dw/night_sunny/bbox_mAP_m: 0.2080 dw/night_sunny/bbox_mAP_l: 0.3980 data_time: 0.0197 time: 0.2831
我直接指出异常的地方：
在第一个数据集推理完成后的评估结果是：
12/16 12:06:38 - mmengine - INFO - bbox_mAP_copypaste: 0.232 0.413 0.225 0.014 0.146 0.434
但是最后的写入结果会看到：
dw/daytime_foggy/bbox_mAP: 0.2268 dw/daytime_foggy/bbox_mAP_50: 0.4060 dw/daytime_foggy/bbox_mAP_75: 0.2186 dw/daytime_foggy/bbox_mAP_s: 0.0132 dw/daytime_foggy/bbox_mAP_m: 0.1404 dw/daytime_foggy/bbox_mAP_l: 0.4276
很明显地看到这两个结果是不一致的。
上面的logs记录了四个数据集评估，只有第一个数据集结果不一致，后面三个数据集的保存结果都是正确一致的。
我不清楚这是BUG还是配置文件没有写正确，如果需要更多信息，我也可以提供。
Metadata
Metadata
Assignees
Labels
No labels