evaluation_cards_database.csv

title,subtitle,authors,link,code_link,date,purpose,principles_tested,functional_props,input_modality,output_modality,input_source,output_source,size,splits,design,judge,protocol,model_access,has_heldout,heldout_details,alignment_validation,is_valid,baseline_models,robustness_measures,known_limitations,benchmarks_list
DigiData: Training and Evaluating General-Purpose Mobile Control Agents,"We present DigiData-Bench, a benchmark for evaluating mobile control agents on real-world complex tasks. We demonstrate that the commonly used step-accuracy metric falls short in reliably assessing mobile control agents and, to address this, we propose dynamic evaluation protocols and AI-powered evaluations as rigorous alternatives for agent assessment.",Meta FAIR,https://arxiv.org/abs/2511.07413,https://github.com/facebookresearch/DigiData,2025-11-08,Development; Research,mobile control agents,Core Performance,Text + Vision,Actions,New dataset (released with eval),Human annotations,Small (< 1K samples),,Dynamic data-driven (adaptive/interactive),Model-based: In the wild,"1. Given a goal and a trajectory produced by an agent, an LLM judge classifies whether it successfully achieved 2. the goal. We use LLM judges relying on both the screenshot and UI tree.",Outputs,False,,human operator is asked to judge whether the trajectory successfully achieves the goal or not. Then we assess the alignment between human judge and LLM judge to ensure a high alignment,unknown,,Prompt variations tested; Multiple runs per sample; Temperature sensitivity tested; Repeated evaluations; Ablation studies; Inter-rater reliability; Significance testing,"-live and dynamic environment introduces uncontrollable factors (feature deprecation, version changes etc) that makes certain tasks unavailable after a period -sizable efforts required to manually set up prework in the environment ","AndroidControl, Android in the wild"
IntPhys 2,"IntPhys 2 offers a comprehensive suite of tests, based on the violation of expectation framework, that challenge models to differentiate between possible and impossible events within controlled and diverse virtual environments.",Meta FAIR,https://arxiv.org/abs/2506.09849,https://github.com/facebookresearch/IntPhys2,2025-05-31,Research,Intuitive Physics,Core Performance,Video,Scores/Embeddings,New dataset (released with eval),Simulation-based,Medium (1K - 100K samples),"Debug Set: 60 videos for Model calibration Main Set: 1,012 videos as Main evaluation set Held-Out Set: 344 videos as Test set",Fixed data-driven (static test set),Automatic (Reference-based),Feed a video to the model Ask the model using specific prompts wether the video is physically plausible or not Check if the model's answer match the ground truth label ,Outputs,True,Held-Out Set: 344 videos as Test set,We ran human baselines to ensure that this task is easy for humans. ,unknown,- Human baseline: 96% - Best model baseline: V-JEPA2 56%,Prompt variations tested; Multiple runs per sample; Temperature sensitivity tested,"Model can be very sensitive to the way they are prompted. Also the compression artefact of the video can also impact the results. Those are mostly limitations on the model sides, since human are not sensitive to those. ","IntPhys: A Framework and Benchmark for Visual Intuitive Physics Reasoning, Riochet et al. 2020"
ImageNet,"A large-scale hierarchical image database designed for visual object recognition research, containing over 14 million images across 20,000+ categories.",Princeton University,https://ieeexplore.ieee.org/document/5206848,https://image-net.org/,2009-01-01,Research; Development,"Object recognition, Visual categorization, Multi-class classification",Core Performance,Vision (Image),Structured Data,New dataset (released with eval),Human annotations,Huge (> 10M samples),"Train, Validation, Test",Fixed data-driven (static test set),Automatic (Reference-based),1. Model receives an image as input 2. Model predicts class label from 1000 classes 3. Top-1 and Top-5 accuracy computed against ground truth,Outputs,True,"Private test set maintained by organizers, used for annual ILSVRC competitions","Expert curation of hierarchical categories, manual verification of labels, consistency checks across similar categories",unknown,AlexNet: 63.3% top-1 VGG: 71.5% top-1 ResNet-50: 76.1% top-1 ResNet-152: 78.3% top-1 EfficientNet-B7: 84.3% top-1 Human performance: ~95% top-5,Multiple runs per sample; Significance testing; Inter-rater reliability,Class imbalance in some categories; Label noise in training set; Some ambiguous images with multiple valid labels; Bias towards certain object viewpoints and contexts,"ImageNet-V2, ImageNet-C, ImageNet-R, ImageNet-A, ImageNet-Sketch"
COCO (Common Objects in Context),"A large-scale object detection, segmentation, and captioning dataset containing 330K images with 80 object categories, designed to advance scene understanding.",Microsoft Research,https://arxiv.org/abs/1405.0312,https://github.com/cocodataset/cocoapi,2014-01-01,Research; Development; Selection,"Object detection, Instance segmentation, Keypoint detection, Panoptic segmentation, Image captioning",Core Performance,Vision (Image),Text; Structured Data,New dataset (released with eval),Human annotations,Large (100K - 1M samples),"Train, Validation, Test",Fixed data-driven (static test set),Automatic (Reference-based),"1. Model receives image as input 2. Model outputs bounding boxes, segmentation masks, or captions 3. Metrics computed: mAP for detection, IoU for segmentation, BLEU/CIDEr for captioning",Outputs,True,"Test-dev and test-challenge splits maintained privately, submissions via evaluation server","Multi-annotator consensus for instance annotations, quality control through redundant labeling",unknown,Faster R-CNN: 42.0 mAP Mask R-CNN: 37.1 mask mAP YOLOv8: 53.9 mAP Human performance (detection): ~70 mAP,Inter-rater reliability; Multiple runs per sample; Confidence intervals,Small object detection remains challenging; Occlusion handling difficulties; Dataset bias toward certain object contexts; Annotation inconsistencies in crowded scenes,"LVIS, Objects365, Open Images, Visual Genome"
CIFAR-10 and CIFAR-100,"Small-scale image classification benchmarks with 60K 32x32 color images in 10 (CIFAR-10) or 100 (CIFAR-100) classes, widely used for algorithm development and testing.",University of Toronto,https://www.cs.toronto.edu/~kriz/learning-features-2009-TR.pdf,https://github.com/pytorch/vision,2009-04-08,Research; Development; Selection,"Image classification, Transfer learning, Generalization",Core Performance; Robustness,Vision (Image),Structured Data,New dataset (released with eval),Human annotations,Medium (1K - 100K samples),"Train (50K), Test (10K)",Fixed data-driven (static test set),Automatic (Reference-based),1. Model receives 32x32 RGB image 2. Model predicts class label (10 or 100 classes) 3. Classification accuracy computed,Outputs,False,,"Systematic sampling from larger dataset (80 million tiny images), human verification of labels",unknown,CIFAR-10: ResNet-56: 93.03% Wide ResNet-28-10: 96.11% PyramidNet: 96.54%  CIFAR-100: ResNet-56: 71.35% Wide ResNet-28-10: 81.15% PyramidNet: 83.78%,Multiple runs per sample; Seed variation tested; Ablation studies,Low resolution (32x32) limits fine-grained recognition; Some label noise in CIFAR-100; Dataset size enables memorization in large models; Limited diversity in poses and contexts,"CIFAR-10-C, CIFAR-10.1, CIFAR-100-C, STL-10, Tiny ImageNet"
Pascal VOC,"A pioneering object detection and segmentation benchmark with 20 object classes, providing standardized evaluation for visual recognition tasks.",University of Oxford,https://link.springer.com/article/10.1007/s11263-009-0275-4,https://github.com/pytorch/vision,2007-01-01,Research; Development; Selection,"Object detection, Semantic segmentation, Instance segmentation, Action classification",Core Performance,Vision (Image),Structured Data,New dataset (released with eval),Human annotations,Medium (1K - 100K samples),"Train, Validation, Test",Fixed data-driven (static test set),Automatic (Reference-based),1. Model receives image as input 2. Model predicts bounding boxes and class labels 3. mAP computed at IoU threshold 0.5,Outputs,True,"Test set labels held privately, evaluation via submission to organizers (historically)","Careful manual annotation with quality control, multiple annotators for difficult cases",unknown,R-CNN: 58.5 mAP (VOC 2007) Fast R-CNN: 70.0 mAP Faster R-CNN: 75.9 mAP YOLOv3: 78.6 mAP,Inter-rater reliability; Significance testing,Limited to 20 classes; Small dataset size by modern standards; Some annotation inconsistencies; Relatively simple backgrounds,"COCO, LVIS, Open Images, Cityscapes"
Cityscapes,"A large-scale dataset for urban scene understanding with pixel-level annotations for semantic segmentation, instance segmentation, and depth estimation in autonomous driving contexts.","Daimler AG, MPI for Informatics, TU Darmstadt",https://arxiv.org/abs/1604.01685,https://github.com/mcordts/cityscapesScripts,2016-04-01,Research; Development; Deployment,"Semantic segmentation, Instance segmentation, Scene understanding, Autonomous driving perception",Core Performance; Robustness,Vision (Image); Video,Structured Data,New dataset (released with eval),Expert annotations,Medium (1K - 100K samples),"Train (2975), Val (500), Test (1525)",Fixed data-driven (static test set),Automatic (Reference-based),1. Model receives street scene image (2048×1024) 2. Model predicts per-pixel semantic class (19 or 30 classes) 3. IoU and accuracy metrics computed,Outputs,True,"Test set labels private, evaluation via online server with leaderboard","Expert annotators with domain knowledge, multi-pass quality control, consistency verification across video sequences",unknown,FCN-8s: 65.3 mIoU DeepLab v3+: 82.1 mIoU HRNetV2: 83.0 mIoU SegFormer: 84.0 mIoU,Multiple runs per sample; Ablation studies; Significance testing,Limited to European cities; Weather bias (mostly good conditions); Class imbalance for rare objects; Fine annotation boundaries challenging,"ADE20K, KITTI, Mapillary Vistas, BDD100K, nuScenes"
ADE20K,"A comprehensive scene parsing benchmark with 150 semantic categories, designed for understanding diverse indoor and outdoor scenes with detailed object and part annotations.",MIT CSAIL,https://arxiv.org/abs/1608.05442,https://github.com/CSAILVision/ADE20K,2017-06-01,Research; Development,"Scene parsing, Semantic segmentation, Multi-scale recognition, Part segmentation",Core Performance,Vision (Image),Structured Data,New dataset (released with eval),Human annotations,Medium (1K - 100K samples),"Train (20K), Val (2K), Test (3K)",Fixed data-driven (static test set),Automatic (Reference-based),1. Model receives diverse scene image 2. Model predicts per-pixel semantic labels (150 classes) 3. Mean IoU and pixel accuracy computed,Outputs,False,,"Crowdsourced annotations with expert review, hierarchical consistency checks, multi-round verification",unknown,PSPNet: 43.29 mIoU DeepLab v3: 45.65 mIoU UPerNet: 44.85 mIoU SegFormer-B5: 51.8 mIoU,Inter-rater reliability; Multiple runs per sample,Long-tail distribution of object classes; Annotation granularity varies; Some scenes have ambiguous boundaries; Challenging for rare categories,"Cityscapes, Pascal Context, COCO-Stuff, Mapillary Vistas"
Kinetics,"A large-scale video action recognition dataset with 400/600/700 human action classes, designed to advance video understanding and temporal reasoning.",DeepMind,https://arxiv.org/abs/1705.06950,https://github.com/cvdfoundation/kinetics-dataset,2017-05-01,Research; Development,"Action recognition, Video understanding, Temporal reasoning, Human activity recognition",Core Performance,Video,Structured Data,New dataset (released with eval),Human annotations,Large (100K - 1M samples),"Train, Val, Test",Fixed data-driven (static test set),Automatic (Reference-based),1. Model receives 10-second video clip 2. Model predicts action class (400/600/700 classes) 3. Top-1 and Top-5 accuracy computed,Outputs,False,,"Human verification of video labels, removal of ambiguous clips, consistency checks for similar actions",unknown,I3D: 71.1% top-1 (K400) SlowFast: 79.8% top-1 (K400) X3D: 80.4% top-1 (K400) VideoMAE: 81.5% top-1 (K400),Multiple runs per sample; Temporal ordering tested,YouTube videos may become unavailable over time; Some action classes overlap or are ambiguous; Camera viewpoint bias; Dataset drift as internet content changes,"UCF-101, HMDB-51, ActivityNet, Something-Something, Moments in Time"
KITTI,"An autonomous driving benchmark suite providing stereo vision, optical flow, 3D object detection, and tracking datasets collected from real-world driving scenarios.","Karlsruhe Institute of Technology, Toyota Technological Institute",https://www.cvlibs.net/publications/Geiger2012CVPR.pdf,https://github.com/bostondiditeam/kitti,2012-01-01,Research; Development; Deployment,"3D object detection, Stereo vision, Optical flow, Visual odometry, Tracking",Core Performance; Robustness,Vision (Image); Video,Structured Data,New dataset (released with eval),Human annotations; Programmatically generated,Medium (1K - 100K samples),"Train, Val, Test",Fixed data-driven (static test set),Automatic (Reference-based),"1. Model receives stereo image pair or point cloud 2. Model predicts 3D bounding boxes, orientation, class 3. 3D AP computed at different difficulty levels (easy/moderate/hard)",Outputs,True,"Test set labels held privately, evaluation via online server with public leaderboard","LiDAR ground truth for 3D positions, manual verification of annotations, multi-sensor fusion for accuracy",unknown,"PointPillars: 79.05 AP (Car, Moderate) PV-RCNN: 83.90 AP (Car, Moderate) CenterPoint: 85.15 AP (Car, Moderate)",Multiple difficulty levels; Ablation studies; Significance testing,Limited to specific geographic region; Weather bias (mostly clear); Limited nighttime data; Class imbalance (cars dominate),"nuScenes, Waymo Open Dataset, Argoverse, A2D2, Lyft Level 5"
Places365,"A scene recognition benchmark with 365 scene categories and over 10 million images, designed to understand high-level visual concepts and environmental context.",MIT CSAIL,https://arxiv.org/abs/1610.02055,https://github.com/CSAILVision/places365,2017-07-01,Research; Development,"Scene recognition, Scene classification, Environmental understanding, Context recognition",Core Performance,Vision (Image),Structured Data,New dataset (released with eval),Human annotations,Very Huge (> 100M samples),"Train, Val, Test",Fixed data-driven (static test set),Automatic (Reference-based),1. Model receives scene image 2. Model predicts scene category (365 classes) 3. Top-1 and Top-5 accuracy computed,Outputs,False,,"Human verification, consistency checks for scene categories, hierarchical taxonomy validation",unknown,ResNet-152: 55.24% top-1 DenseNet-161: 56.12% top-1 ResNeXt-101: 56.05% top-1,Inter-rater reliability; Multiple runs per sample,Some scene categories overlap semantically; Cultural bias in scene definitions; Indoor scenes better represented than outdoor; Ambiguous boundary cases,"SUN397, MIT Indoor 67, Scene-15, ADE20K"
UCF-101,"An action recognition benchmark with 101 action categories and 13,320 videos collected from YouTube, widely used for video understanding research.",University of Central Florida,https://arxiv.org/abs/1212.0402,https://github.com/pytorch/vision,2012-01-01,Research; Development,"Action recognition, Video classification, Temporal understanding",Core Performance,Video,Structured Data,New dataset (released with eval),Human annotations,Medium (1K - 100K samples),Train/Test (3 splits provided),Fixed data-driven (static test set),Automatic (Reference-based),1. Model receives video clip 2. Model predicts action class (101 classes) 3. Average accuracy across 3 splits reported,Outputs,False,,"Manual verification of action labels, removal of ambiguous videos",unknown,Two-Stream CNN: 88.0% I3D: 95.6% SlowFast: 96.8% VideoMAE: 97.2%,Multiple evaluation splits; Seed variation tested,YouTube videos may become unavailable; Camera motion and quality vary; Some action classes are very similar; Dataset saturation with modern methods,"HMDB-51, Kinetics, ActivityNet, Something-Something-V2"
NYU Depth V2,"An RGB-D dataset for indoor scene understanding with 1449 densely labeled pairs of aligned RGB and depth images, designed for depth estimation and semantic segmentation.",New York University,https://cs.nyu.edu/~fergus/datasets/indoor_seg_support.pdf,https://github.com/ankurhanda/nyuv2-meta-data,2012-06-01,Research; Development,"Depth estimation, RGB-D understanding, Indoor scene parsing, 3D reconstruction",Core Performance,Vision (Image); Structured Data,Structured Data,New dataset (released with eval),Expert annotations,Small (< 1K samples),"Train (795), Test (654)",Fixed data-driven (static test set),Automatic (Reference-based),"1. Model receives RGB image 2. Model predicts depth map or semantic segmentation 3. RMSE, absolute relative error, and accuracy metrics computed",Outputs,False,,"Kinect sensor ground truth with manual alignment corrections, multi-view consistency",unknown,Depth Estimation: Eigen et al.: 0.641 RMSE AdaBins: 0.364 RMSE BTS: 0.392 RMSE,Multiple runs per sample; Ablation studies,"Small dataset size; Limited to indoor scenes; Kinect depth sensor limitations (range, IR interference); Mostly residential environments","ScanNet, Matterport3D, SUNRGB-D, KITTI Depth"
CelebA,"A large-scale face attributes dataset with 202,599 face images annotated with 40 binary attributes, 5 landmark locations, and identity information for face recognition and attribute prediction.",The Chinese University of Hong Kong,https://arxiv.org/abs/1411.7766,https://github.com/tkarras/progressive_growing_of_gans,2015-09-01,Research; Development,"Face attribute recognition, Face detection, Facial landmark detection, Identity recognition",Core Performance; Fairness,Vision (Image),Structured Data,New dataset (released with eval),Human annotations,Large (100K - 1M samples),"Train (162,770), Val (19,867), Test (19,962)",Fixed data-driven (static test set),Automatic (Reference-based),1. Model receives face image 2. Model predicts binary attributes (40 dimensions) 3. Per-attribute accuracy and mean accuracy computed,Outputs,False,,"Manual annotation with quality control, consistency verification across multiple attributes",unknown,LNets+ANet: 87.30% mean accuracy Walk and Learn: 88.06% FaceNet: 89.35%,Inter-rater reliability; Multiple runs per sample,Demographic bias (celebrity images); Some attributes are subjective; Label noise in some attributes; Privacy concerns with celebrity images; Lighting and pose variation,"LFW, VGGFace2, MS-Celeb-1M, FFHQ"
Visual Genome,"A comprehensive visual understanding dataset with dense annotations of objects, attributes, relationships, and scene graphs across 108K images for structured scene understanding.",Stanford University,https://arxiv.org/abs/1602.07332,https://github.com/ranjaykrishna/visual_genome_python_driver,2016-05-01,Research; Development,"Scene graph generation, Visual relationship detection, Visual question answering, Dense captioning",Core Performance,Vision (Image),Structured Data; Text,MS COCO,Crowdsourced annotations,Large (100K - 1M samples),"Train, Val, Test",Fixed data-driven (static test set),Automatic (Reference-based),"1. Model receives image 2. Model predicts objects, attributes, and relationships 3. Recall@K metrics for scene graph components",Outputs,False,,"Multi-round crowdsourced annotations with validation, consistency checks for relationships",unknown,IMP: 14.6 R@50 (scene graph detection) Motifs: 21.4 R@50 VCTree: 22.0 R@50 GPS-Net: 24.0 R@50,Inter-rater reliability; Multiple runs per sample,Long-tail distribution of relationships; Annotation inconsistencies; Subjective relationship definitions; Incomplete annotations (not all relationships captured),"GQA, Scene Graph Benchmark, VRD, OpenImages V6"
LVIS (Large Vocabulary Instance Segmentation),A large-scale instance segmentation dataset with 1203 object categories designed to address the long-tail distribution challenge in object recognition.,Facebook AI Research,https://arxiv.org/abs/1908.03195,https://github.com/lvis-dataset/lvis-api,2019-08-01,Research; Development,"Instance segmentation, Long-tail recognition, Object detection, Fine-grained categorization",Core Performance; Robustness,Vision (Image),Structured Data,MS COCO,Expert annotations,Large (100K - 1M samples),"Train, Val, Test",Fixed data-driven (static test set),Automatic (Reference-based),"1. Model receives image 2. Model predicts instance masks and categories (1203 classes) 3. AP computed separately for rare, common, and frequent categories",Outputs,True,"Test set labels private, evaluation via online server","Expert annotators with WordNet taxonomy, quality control for long-tail categories",unknown,Mask R-CNN: 21.2 AP (v1.0) Cascade R-CNN: 26.2 AP Swin Transformer: 50.9 AP,Multiple runs per sample; Ablation studies; Category frequency stratification,Rare categories have very few examples; Annotation cost for 1203 categories; Some category definitions overlap; Challenging for zero-shot generalization,"COCO, Objects365, OpenImages, iNaturalist"
Mapillary Vistas,"A diverse street-level imagery dataset with pixel-level annotations for 66 object categories, designed for robust semantic segmentation across varied geographic locations and conditions.",Mapillary AB,https://openaccess.thecvf.com/content_ICCV_2017/papers/Neuhold_The_Mapillary_Vistas_ICCV_2017_paper.pdf,https://github.com/mapillary/mapillary_vistas,2017-08-01,Research; Development; Deployment,"Semantic segmentation, Panoptic segmentation, Scene understanding, Robust perception",Core Performance; Robustness,Vision (Image),Structured Data,User-generated content,Expert annotations,Medium (1K - 100K samples),"Train (18K), Val (2K), Test (5K)",Fixed data-driven (static test set),Automatic (Reference-based),1. Model receives street-level image 2. Model predicts per-pixel semantic labels (66 classes) 3. mIoU computed across all classes,Outputs,True,"Test set labels private, evaluation via online platform","Expert annotators, multi-stage quality control, geographic diversity validation",unknown,PSPNet: 42.7 mIoU DeepLab v3+: 45.8 mIoU HRNetV2: 50.3 mIoU,Geographic diversity tested; Weather variations included; Multiple runs per sample,Varying image quality from crowdsourced data; Camera parameter diversity; Some regions overrepresented; Annotation inconsistencies across diverse scenes,"Cityscapes, BDD100K, IDD, WildDash"
MPII Human Pose,"A benchmark for human pose estimation with 25K images containing over 40K annotated people with 16 body joints, covering diverse activities and viewpoints.",Max Planck Institute for Informatics,https://openaccess.thecvf.com/content_cvpr_2014/papers/Andriluka_2D_Human_Pose_2014_CVPR_paper.pdf,https://www.mpi-inf.mpg.de/departments/computer-vision-and-machine-learning/software-and-datasets/mpii-human-pose-dataset,2014-06-01,Research; Development,"Human pose estimation, Keypoint detection, Articulated pose estimation, Activity recognition",Core Performance,Vision (Image),Structured Data,New dataset (released with eval),Human annotations,Medium (1K - 100K samples),"Train (~29K people), Test (~12K people)",Fixed data-driven (static test set),Automatic (Reference-based),1. Model receives image with person 2. Model predicts 2D joint locations (16 joints) 3. PCKh (Percentage of Correct Keypoints) at various thresholds,Outputs,False,,"Manual annotation with consistency checks, multi-annotator agreement for difficult poses",unknown,Hourglass Network: 90.9 PCKh@0.5 HRNet: 92.3 PCKh@0.5 SimpleBaseline: 91.5 PCKh@0.5,Multiple difficulty levels; Inter-rater reliability; Occlusion analysis,2D annotations only (no 3D); Occlusion and truncation challenges; Some joint definitions ambiguous; Dataset bias toward certain activities,"COCO Keypoints, Human3.6M, PoseTrack, CrowdPose"
Open Images,"A large-scale multi-task dataset with ~9M images annotated for image classification (20K classes), object detection (600 classes), visual relationships, and segmentation masks.",Google Research,https://arxiv.org/abs/1811.00982,https://github.com/openimages/dataset,2018-01-01,Research; Development; Selection,"Object detection, Image classification, Visual relationship detection, Instance segmentation",Core Performance; Robustness,Vision (Image),Structured Data,New dataset (released with eval),Human annotations; Crowdsourced annotations,Very Huge (> 100M samples),"Train, Val, Test",Fixed data-driven (static test set),Automatic (Reference-based),"1. Model receives image 2. Model performs classification, detection, or segmentation 3. mAP computed for detection, top-k accuracy for classification",Outputs,True,"Test set labels private for challenges, public validation set available","Multi-stage crowdsourced annotation with verification, automated quality filters",unknown,Faster R-CNN: 54.3 mAP (detection) YOLOv4: 55.8 mAP EfficientDet: 56.1 mAP,Multiple runs per sample; Confidence intervals; Large-scale diversity,Long-tail class distribution; Annotation inconsistencies at scale; Some classes poorly defined; Label noise in crowdsourced annotations,"COCO, LVIS, Objects365, ImageNet"
ScanNet,"A richly-annotated 3D dataset of indoor scenes with RGB-D scans, semantic segmentation, instance segmentation, and 3D object bounding boxes for 1513 scenes.","Stanford University, Princeton University, Technical University of Munich",https://arxiv.org/abs/1702.04405,https://github.com/ScanNet/ScanNet,2017-04-01,Research; Development,"3D scene understanding, Semantic segmentation, Instance segmentation, 3D reconstruction",Core Performance,Vision (Image); Video; Structured Data,Structured Data,New dataset (released with eval),Human annotations; Programmatically generated,Medium (1K - 100K samples),"Train (1201), Val (312), Test (100)",Fixed data-driven (static test set),Automatic (Reference-based),"1. Model receives 3D scene (RGB-D scans) 2. Model predicts semantic labels or instance masks 3. mIoU for semantic segmentation, mAP for instance segmentation",Outputs,True,"Test set labels private, evaluation via online benchmark server","Manual verification of 3D reconstructions, multi-annotator consistency for semantic labels",unknown,PointNet++: 53.5 mIoU SparseConvNet: 72.5 mIoU MinkowskiNet: 73.6 mIoU,Multiple runs per sample; Ablation studies,Limited to indoor scenes; Reconstruction artifacts; Scanning noise and occlusions; Limited scene diversity (mostly offices and apartments),"Matterport3D, S3DIS, 2D-3D-S, ARKitScenes"
nuScenes,"A large-scale autonomous driving dataset with 1000 scenes (40K frames) featuring 3D object annotations, tracking IDs, and multimodal sensor data including cameras, LiDAR, and radar.",Motional (formerly nuTonomy),https://arxiv.org/abs/1903.11027,https://github.com/nutonomy/nuscenes-devkit,2019-03-26,Research; Development; Deployment,"3D object detection, Multi-object tracking, Prediction, Sensor fusion, Scene understanding",Core Performance; Robustness,Vision (Image); Video; Structured Data,Structured Data,New dataset (released with eval),Human annotations,Medium (1K - 100K samples),"Train (700), Val (150), Test (150)",Fixed data-driven (static test set),Automatic (Reference-based),"1. Model receives multimodal sensor data (cameras, LiDAR, radar) 2. Model predicts 3D bounding boxes and tracking IDs 3. NDS (nuScenes Detection Score) and mAP computed",Outputs,True,"Test set labels private, evaluation via online leaderboard","Expert annotators, multi-sensor consistency checks, temporal coherence validation",unknown,PointPillars: 45.3 NDS CenterPoint: 65.5 NDS BEVFusion: 72.9 NDS,Geographic diversity; Weather conditions; Day/night variations; Multiple runs per sample,"Limited geographic coverage (Boston, Singapore); Sensor calibration challenges; Annotation latency for fast-moving objects; Class imbalance","Waymo Open Dataset, KITTI, Argoverse, Lyft Level 5, A2D2"
ActivityNet,"A large-scale video dataset for human activity understanding with 200 activity classes and temporal annotations, designed for action recognition and temporal action localization.","Universidad del Norte, KAUST",https://www.cv-foundation.org/openaccess/content_cvpr_2015/papers/Heilbron_ActivityNet_A_Large-Scale_2015_CVPR_paper.pdf,https://github.com/activitynet/ActivityNet,2015-06-01,Research; Development,"Temporal action detection, Action recognition, Dense video captioning, Video understanding",Core Performance,Video,Structured Data; Text,New dataset (released with eval),Human annotations,Medium (1K - 100K samples),"Train (50%), Val (25%), Test (25%)",Fixed data-driven (static test set),Automatic (Reference-based),"1. Model receives untrimmed video 2. Model predicts action segments with class labels 3. mAP at different IoU thresholds (0.5, 0.75, 0.95)",Outputs,True,"Test set used for annual challenges, labels withheld","Multi-annotator temporal boundary agreement, consistency checks for activity definitions",unknown,SSN: 41.3 mAP@0.5 BMN: 50.1 mAP@0.5 TALLFormer: 59.8 mAP@0.5,Inter-rater reliability; Multiple IoU thresholds; Temporal boundary sensitivity,YouTube video availability issues; Temporal boundary ambiguity; Action class overlap; Video quality variance,"THUMOS14, Kinetics, Charades, MultiTHUMOS, AVA"
DAVIS (Densely Annotated VIdeo Segmentation),A video object segmentation benchmark with high-quality pixel-level annotations for densely segmenting objects in video sequences.,"ETH Zurich",https://arxiv.org/abs/1704.00675,https://github.com/davisvideochallenge/davis,2016-10-01,Research; Development,"Video object segmentation, Temporal consistency, Object tracking, Dense prediction",Core Performance; Robustness,Video,Structured Data,New dataset (released with eval),Human annotations,Small (< 1K samples),"Train/Val (60 sequences), Test-dev (30 sequences)",Fixed data-driven (static test set),Automatic (Reference-based),1. Model receives video sequence with first frame annotation 2. Model propagates segmentation to subsequent frames 3. J&F metric (region similarity and contour accuracy),Outputs,True,"Test-challenge set for competitions, labels withheld",High-quality manual annotations with temporal consistency verification,unknown,OSVOS: 79.8 J&F STM: 84.3 J&F XMem: 86.2 J&F,Temporal consistency tested; Occlusion robustness; Multiple runs per sample,Small dataset size; Limited object categories; Short video sequences; Primarily objects with clear boundaries,"YouTube-VOS, FBMS, SegTrack, OVIS"
VQA (Visual Question Answering),"A dataset with open-ended questions about images requiring visual understanding and reasoning, containing 265K images and over 1M questions.","Virginia Tech, Facebook AI Research",https://arxiv.org/abs/1505.00468,https://github.com/GT-Vision-Lab/VQA,2015-10-01,Research; Development,"Visual question answering, Visual reasoning, Multimodal understanding, Common sense reasoning",Core Performance; Robustness,Text + Vision,Text,MS COCO,Crowdsourced annotations,Huge (> 10M samples),"Train, Val, Test",Fixed data-driven (static test set),Automatic (Reference-based); Human: Representative sample,1. Model receives image and question 2. Model generates answer 3. Accuracy computed with consensus matching (multiple human answers),Outputs,True,"Test-dev and test-std splits, evaluation via server","Multiple human answers per question (10 answers), consensus-based evaluation",unknown,Bottom-Up Top-Down: 70.3% VILBERT: 72.4% OSCAR: 73.8% BLIP: 78.3%,Multiple human references; Prompt variations tested; Inter-rater reliability,Language bias (can answer many questions without image); Dataset bias toward common objects; Answer distribution imbalance; Ambiguous questions,"GQA, VQA v2, OK-VQA, TextVQA, VizWiz"
CLEVR,"A diagnostic dataset for compositional visual reasoning with 100K synthetic images and 1M questions testing spatial relations, counting, and logic.","Stanford University, Facebook AI Research",https://arxiv.org/abs/1612.06890,https://github.com/facebookresearch/clevr-dataset-gen,2017-04-01,Research; Development,"Compositional reasoning, Spatial reasoning, Counting, Logical reasoning, Visual reasoning",Core Performance; Robustness,Text + Vision,Text,Synthetic/Generated,Programmatically generated,Huge (> 10M samples),"Train (70K), Val (15K), Test (15K)",Fixed data-driven (static test set),Automatic (Reference-based),"1. Model receives synthetic image and question 2. Model predicts answer from predefined set 3. Accuracy computed, can analyze by question type",Outputs,False,,"Programmatically generated with known ground truth, exhaustive question type coverage",unknown,CNN+LSTM: 52.3% Film: 97.7% NS-VQA: 99.8% MAC: 98.9%,Ablation studies; Question type analysis; Compositional generalization tested,Synthetic domain (limited real-world applicability); Simple shapes and colors; No natural language variation; Programmatic biases,"CLEVR-CoGenT, GQA, NLVR2, CLOSURE"
Waymo Open Dataset,"A large-scale autonomous driving dataset with 1000 diverse driving scenes, 12M LiDAR points per frame, high-resolution camera images, and rich 3D annotations.",Waymo LLC,https://arxiv.org/abs/1912.04838,https://github.com/waymo-research/waymo-open-dataset,2019-08-21,Research; Development; Deployment,"3D object detection, 2D object detection, Tracking, Domain adaptation, Sensor fusion",Core Performance; Robustness,Vision (Image); Video; Structured Data,Structured Data,New dataset (released with eval),Human annotations; Programmatically generated,Huge (> 10M samples),"Train (798), Val (202), Test (150)",Fixed data-driven (static test set),Automatic (Reference-based),1. Model receives multimodal sensor data 2. Model predicts 3D bounding boxes with tracking IDs 3. AP/APH metrics at different IoU thresholds and difficulty levels,Outputs,True,"Test set labels private, evaluation via online leaderboard","Multi-stage annotation pipeline with quality checks, temporal consistency validation",unknown,PointPillars: 63.8 L2 APH (Vehicle) CenterPoint: 73.9 L2 APH PV-RCNN++: 77.8 L2 APH,Geographic diversity; Time of day variations; Weather conditions; Multiple difficulty levels,Geographic concentration in specific cities; Sensor-specific challenges; Annotation latency for distant objects; Class imbalance,"nuScenes, KITTI, Argoverse 2, Once, Lyft Level 5"
Fashion-MNIST,"A drop-in replacement for MNIST with 70K grayscale images of fashion products across 10 categories, designed to be a more challenging benchmark for image classification.",Zalando Research,https://arxiv.org/abs/1708.07747,https://github.com/zalandoresearch/fashion-mnist,2017-08-25,Research; Development; Selection,"Image classification, Transfer learning, Benchmark comparison",Core Performance,Vision (Image),Structured Data,New dataset (released with eval),Human annotations,Medium (1K - 100K samples),"Train (60K), Test (10K)",Fixed data-driven (static test set),Automatic (Reference-based),1. Model receives 28x28 grayscale image 2. Model predicts fashion category (10 classes) 3. Classification accuracy computed,Outputs,False,,Product catalog labels verified by domain experts,unknown,Linear Classifier: 83.7% CNN: 93.5% ResNet: 94.9% Vision Transformer: 95.1%,Multiple runs per sample; Seed variation tested,Low resolution (28x28); Grayscale only; Limited intra-class variation; Some categories overlap visually,"MNIST, EMNIST, Kuzushiji-MNIST, DeepFashion"
MMLU (Massive Multitask Language Understanding),"A comprehensive benchmark with 57 tasks spanning STEM, humanities, social sciences, and more, designed to measure multitask accuracy and knowledge breadth in language models.","UC Berkeley, Columbia University",https://arxiv.org/abs/2009.03300,https://github.com/hendrycks/test,2020-09-07,Research; Development; Selection,"World knowledge, Reasoning, Domain expertise across 57 subjects",Core Performance,Text,Text,New dataset (released with eval),Expert annotations,Medium (1K - 100K samples),"Dev (5 shot examples per task), Test (285 questions per task average)",Fixed data-driven (static test set),Automatic (Reference-based),1. Model receives multiple choice question with 4 options 2. Model predicts answer (A/B/C/D) 3. Accuracy computed per subject and overall,Outputs,False,,"Questions sourced from practice exams and tests, expert review for correctness",unknown,GPT-3 (175B): 43.9% GPT-3.5: 70.0% GPT-4: 86.4% Claude 3.5 Sonnet: 88.7% Random baseline: 25%,Multiple evaluation runs; Few-shot prompting variations; Subject-wise analysis,Multiple choice format may not reflect real-world usage; Some questions have ambiguous answers; Dataset contamination concerns with web-trained models; Cultural and knowledge cutoff biases,"MMLU-Pro, AGIEval, C-Eval, CMMLU"
HumanEval,A code generation benchmark with 164 hand-written programming problems to evaluate functional correctness of synthesized Python code.,OpenAI,https://arxiv.org/abs/2107.03374,https://github.com/openai/human-eval,2021-07-07,Research; Development; Selection,"Code generation, Programming ability, Functional correctness, Code understanding",Core Performance,Text; Code,Code,New dataset (released with eval),Author-provided,Small (< 1K samples),Test (164 problems),Fixed data-driven (static test set),Automatic (Execution-based),1. Model receives function signature and docstring 2. Model generates function implementation 3. Generated code executed against unit tests 4. pass@k metric computed (% passing all tests in k samples),Outputs,False,,"Hand-written problems with comprehensive unit tests, manual verification of test correctness",unknown,Codex (12B): 28.8% pass@1 GPT-3.5-turbo: 48.1% pass@1 GPT-4: 67.0% pass@1 Claude 3.5 Sonnet: 92.0% pass@1,Multiple samples per problem (pass@k); Temperature sensitivity tested; Execution-based verification,Limited to Python; Small dataset size (164 problems); Relatively simple problems; May be contaminated in training data; No testing of code efficiency or style,"MBPP, APPS, CodeContests, HumanEval+, MultiPL-E"
HellaSwag,"A benchmark for commonsense natural language inference about physical situations, requiring models to complete scenarios with the most plausible continuation.","University of Washington, Allen Institute for AI",https://arxiv.org/abs/1905.07830,https://github.com/rowanz/hellaswag,2019-05-19,Research; Development,"Commonsense reasoning, Physical understanding, Situation modeling, Plausibility judgment",Core Performance; Robustness,Text,Text,New dataset (released with eval),Crowdsourced annotations,Medium (1K - 100K samples),"Train (39,905), Val (10,042), Test (10,003)",Fixed data-driven (static test set),Automatic (Reference-based),1. Model receives scenario context 2. Model selects most plausible continuation from 4 options 3. Accuracy computed,Outputs,False,,"Adversarial filtering using BERT to ensure difficulty, human validation of plausibility",unknown,BERT-Large: 47.3% GPT-2: 50.9% GPT-3: 78.9% GPT-4: 95.3% Human performance: 95.6%,Adversarial filtering; Multiple runs per sample; Human baseline comparison,Dataset may be easier than originally intended for modern LLMs; Multiple choice format; Adversarial examples may have artifacts; Potential data contamination,"PIQA, WinoGrande, CommonsenseQA, ARC"
GSM8K (Grade School Math 8K),"A dataset of 8,500 grade school math word problems requiring multi-step arithmetic reasoning to solve.",OpenAI,https://arxiv.org/abs/2110.14168,https://github.com/openai/grade-school-math,2021-10-27,Research; Development,"Mathematical reasoning, Multi-step problem solving, Arithmetic reasoning, Chain-of-thought reasoning",Core Performance,Text,Text,New dataset (released with eval),Author-provided,Medium (1K - 100K samples),"Train (7,473), Test (1,319)",Fixed data-driven (static test set),Automatic (Reference-based),1. Model receives math word problem 2. Model generates solution with reasoning steps 3. Final numerical answer extracted and compared to ground truth,Outputs,False,,"Human-written problems with verified solutions, consistency checks for answer correctness",unknown,GPT-3 (175B): 34.2% GPT-3.5: 57.1% GPT-4: 92.0% GPT-4o: 96.1% Claude 3.5 Sonnet: 96.4%,Multiple runs per sample; Chain-of-thought prompting tested; Answer extraction robustness,Limited to grade-school level math; Answer extraction can be brittle; Some problems have ambiguous wording; Potential contamination in training data,"MATH, GSM-Hard, SVAMP, ASDiv, MathQA"
TruthfulQA,A benchmark measuring whether language models generate truthful answers to questions that humans might answer falsely due to misconceptions or false beliefs.,"Oxford University, OpenAI",https://arxiv.org/abs/2109.07958,https://github.com/sylinrl/TruthfulQA,2021-09-16,Research; Development; Safety,"Truthfulness, Factual accuracy, Resistance to misconceptions, Calibration",Safety; Core Performance; Calibration,Text,Text,New dataset (released with eval),Expert annotations,Small (< 1K samples),Test (817 questions across 38 categories),Fixed data-driven (static test set),Model-based: Expert; Human: Experts,1. Model receives question designed to elicit false beliefs 2. Model generates answer 3. Answers judged for truthfulness and informativeness using GPT-judge or human evaluation,Outputs,False,,"Expert-curated questions targeting known misconceptions, multi-rater validation of truth labels",unknown,GPT-3 (175B): 58.0% GPT-3.5: 47.0% GPT-4: 59.0% Claude 2: 62.0% Human baseline: 94.0%,"Multiple evaluation methods (MC1, MC2, generative); Human validation; Model-based evaluation correlation",Subjective truthfulness judgments in some cases; Cultural bias in what constitutes 'truth'; Model-based evaluation may not align with human judgment; Limited coverage of misconceptions,"FactScore, HaluEval, SelfCheckGPT, FELM"
BIG-bench (Beyond the Imitation Game),"A collaborative benchmark with 204 tasks spanning linguistics, child development, math, commonsense reasoning, biology, physics, social bias, and software development.","Google Research, 450+ authors",https://arxiv.org/abs/2206.04615,https://github.com/google/BIG-bench,2022-06-09,Research; Development,"Diverse capabilities across 204 tasks including reasoning, knowledge, language understanding, bias detection",Core Performance; Fairness; Robustness,Text,Text,New dataset (released with eval),Multiple/Mixed sources,Large (100K - 1M samples),Varies by task,Composite,Automatic (Reference-based); Model-based: In the wild,1. Model evaluated on 204 diverse tasks 2. Each task has its own evaluation protocol 3. Performance aggregated across tasks,Outputs,False,,"Crowdsourced task creation with quality review, diverse authorship for broad coverage",unknown,Average human rater: 89.0% Few-shot PaLM (540B): 65.7% GPT-4: ~83% (estimated on BIG-Bench Hard),Multiple tasks provide robustness; Human baseline comparison; Cross-task analysis,Task quality varies; Some tasks too easy or too hard; Computational cost of running all 204 tasks; Aggregation methodology debatable,"BIG-Bench Hard, MMLU, HELM, SuperGLUE"
MATH,"A dataset of 12,500 challenging competition mathematics problems from high school math competitions, requiring advanced mathematical reasoning and problem-solving.","UC Berkeley, OpenAI",https://arxiv.org/abs/2103.03874,https://github.com/hendrycks/math,2021-03-05,Research; Development,"Advanced mathematical reasoning, Problem solving, Multi-step reasoning, Mathematical knowledge",Core Performance,Text,Text,New dataset (released with eval),Published references,Medium (1K - 100K samples),"Train (7,500), Test (5,000)",Fixed data-driven (static test set),Automatic (Reference-based),1. Model receives competition math problem 2. Model generates solution with steps 3. Final answer extracted and checked against ground truth 4. Problems span 7 subjects with 5 difficulty levels,Outputs,False,,"Problems from real math competitions with verified solutions, difficulty levels validated",unknown,GPT-3: 6.9% GPT-4: 42.5% Minerva (540B): 33.6% GPT-4 Turbo: 52.9% Claude 3.5 Sonnet: 71.1%,Multiple difficulty levels; Subject-wise analysis; Chain-of-thought evaluation,Answer extraction challenges; LaTeX formatting issues; Symbolic vs numeric answers; High difficulty may not reflect practical math usage,"GSM8K, MathQA, SVAMP, ASDiv, Hendrycks MATH"
ARC (AI2 Reasoning Challenge),"A dataset of 7,787 science exam questions from grade 3-9, designed to require reasoning beyond simple retrieval or pattern matching.",Allen Institute for AI,https://arxiv.org/abs/1803.05457,https://github.com/allenai/arc,2018-03-14,Research; Development,"Scientific reasoning, Commonsense reasoning, Knowledge retrieval, Multi-hop reasoning",Core Performance; Robustness,Text,Text,New dataset (released with eval),Existing dataset labels,Medium (1K - 100K samples),"Easy (2,376 train, 570 test), Challenge (1,119 train, 1,172 test)",Fixed data-driven (static test set),Automatic (Reference-based),1. Model receives science exam question with multiple choice options 2. Model selects answer 3. Accuracy computed separately for Easy and Challenge sets,Outputs,False,,"Real exam questions filtered to require reasoning, partitioned by difficulty using retrieval-based baseline",unknown,ARC-Challenge: BERT: 59.1% GPT-3: 51.4% GPT-3.5: 85.2% GPT-4: 96.3%,Two difficulty levels; Multiple choice format variations; Retrieval baseline comparison,Multiple choice format; Some questions solvable without reasoning; Challenge set becoming saturated; Grade-school level may not test advanced reasoning,"OpenBookQA, CommonsenseQA, QASC, SciQ"
DROP (Discrete Reasoning Over Paragraphs),"A reading comprehension benchmark requiring discrete reasoning operations over paragraph content, including sorting, counting, and arithmetic.","Allen Institute for AI, University of Washington",https://arxiv.org/abs/1903.00161,https://github.com/allenai/drop,2019-03-01,Research; Development,"Reading comprehension, Numerical reasoning, Discrete operations, Multi-hop reasoning",Core Performance,Text,Text,New dataset (released with eval),Crowdsourced annotations,Large (100K - 1M samples),"Train (77,409), Dev (9,536), Test (9,622)",Fixed data-driven (static test set),Automatic (Reference-based),"1. Model receives paragraph and question 2. Model generates answer (number, span, or date) 3. F1 and Exact Match metrics computed",Outputs,False,,"Crowdsourced questions with verification, requires discrete reasoning operations confirmed through analysis",unknown,BERT: 47.0 F1 RoBERTa: 80.9 F1 GPT-3: 29.0 F1 GPT-4: 80.9 F1 Human performance: 96.4 F1,Multiple answer types; Question type analysis; Human baseline comparison,Requires careful answer extraction; Some questions ambiguous; Numerical reasoning can be brittle; Limited diversity in reasoning types,"SQuAD, NewsQA, NaturalQuestions, NumGLUE, TAT-QA"
WinoGrande,"A large-scale commonsense reasoning benchmark with 44,000 problems requiring resolving pronoun ambiguity through world knowledge and commonsense.","Allen Institute for AI, University of Washington",https://arxiv.org/abs/1907.10641,https://github.com/allenai/winogrande,2019-07-24,Research; Development,"Commonsense reasoning, Coreference resolution, World knowledge, Causal reasoning",Core Performance; Robustness,Text,Text,New dataset (released with eval),Crowdsourced annotations,Medium (1K - 100K samples),"Train (40,398), Dev (1,267), Test (1,767)",Fixed data-driven (static test set),Automatic (Reference-based),1. Model receives sentence with pronoun ambiguity 2. Model selects correct referent from 2 options 3. Accuracy computed,Outputs,False,,"Adversarial filtering using language models, crowdsourced generation with validation",unknown,BERT-Large: 59.4% RoBERTa-Large: 79.1% GPT-3: 70.2% GPT-4: 87.5% Human performance: 94.0%,Adversarial filtering; Large-scale dataset; Multiple difficulty levels,Binary choice may be limiting; Adversarial filtering may introduce artifacts; Saturation with modern models; Limited reasoning depth,"Winograd Schema Challenge, COPA, CommonsenseQA, PIQA"
BBH (BIG-Bench Hard),"A curated subset of 23 challenging tasks from BIG-Bench where language models perform below human raters, focusing on tasks requiring multi-step reasoning.",Google Research,https://arxiv.org/abs/2210.09261,https://github.com/suzgunmirac/BIG-Bench-Hard,2022-10-17,Research; Development,"Complex reasoning, Multi-step thinking, Challenging cognitive tasks, Chain-of-thought reasoning",Core Performance,Text,Text,One or Multiple existing datasets,Multiple/Mixed sources,Medium (1K - 100K samples),"Test (6,511 examples across 23 tasks)",Fixed data-driven (static test set),Automatic (Reference-based),1. Model receives task from one of 23 challenging categories 2. Model generates answer 3. Performance aggregated across all tasks 4. Chain-of-thought prompting typically used,Outputs,False,,"Tasks selected where models underperform humans, difficulty validated through empirical testing",unknown,PaLM (540B): 56.5% average PaLM (540B) + CoT: 78.1% GPT-4: ~91% Human raters: ~92% average,Multiple tasks; Chain-of-thought evaluation; Human baseline comparison; Cross-model validation,Only 23 tasks (limited coverage); Chain-of-thought prompting required for good performance; Task aggregation methodology; Rapidly saturating with newer models,"BIG-Bench, MMLU, AGIEval, HELM"
MT-Bench,"A multi-turn conversational benchmark with 80 high-quality multi-turn questions spanning 8 categories, evaluated using GPT-4 as a judge.","UC Berkeley, UCSD, CMU, MBZUAI",https://arxiv.org/abs/2306.05685,https://github.com/lm-sys/FastChat,2023-06-09,Development; Selection,"Multi-turn conversation, Instruction following, Reasoning, Writing, Role-playing, Knowledge",Core Performance; Core Quality Dimensions,Text,Text,New dataset (released with eval),Author-provided,Small (< 1K samples),Test (80 questions with 2 turns each),Fixed data-driven (static test set),Model-based: In the wild,1. Model engages in 2-turn conversation 2. GPT-4 evaluates responses on 10-point scale 3. Scores averaged across turns and categories,Outputs,False,,"Strong correlation with human preferences validated on Chatbot Arena data, GPT-4 judge agreement measured",unknown,Vicuna-13B: 6.39 GPT-3.5-turbo: 7.94 Claude 2: 8.06 GPT-4: 8.99 GPT-4-turbo: 9.32,Multiple categories; Position bias mitigation; Agreement with human ratings validated; Pairwise comparison,Small dataset (80 questions); GPT-4 judge may have biases; Evaluation cost; Model-based judge limitations; Prompt sensitivity,"Chatbot Arena, AlpacaEval, Arena-Hard, LiveBench"
GPQA (Google-Proof Q&A),"A challenging multiple-choice benchmark of 448 expert-level questions in biology, physics, and chemistry designed to be difficult even for skilled non-experts with internet access.",New York University,https://arxiv.org/abs/2311.12022,https://github.com/idavidrein/gpqa,2023-11-20,Research; Development,"Expert-level knowledge, Scientific reasoning, Domain expertise, Graduate-level understanding",Core Performance; Robustness,Text,Text,New dataset (released with eval),Expert annotations,Small (< 1K samples),"Main (198), Extended (246), Diamond (198 highest quality)",Fixed data-driven (static test set),Automatic (Reference-based),1. Model receives graduate-level science question 2. Model selects from 4 multiple choice options 3. Accuracy computed 4. Questions validated to be difficult for non-experts with Google access,Outputs,False,,"PhD-level experts write and validate questions, non-expert validators with Google access achieve <35% accuracy",unknown,Random: 25% Non-expert w/ Google: 34% Expert: 81% GPT-4: 39% Claude 3 Opus: 59.4% GPT-4o: 53.6%,Expert validation; Non-expert baseline; Multiple subject areas; Quality tiers (Diamond subset),Small dataset size; Limited to 3 scientific domains; Multiple choice format; High cost of expert question creation; Cultural bias toward Western scientific education,"MMLU-Pro, JEE-Advanced, SciBench, LiveBench"
IFEval (Instruction-Following Eval),"A benchmark of 541 prompts with verifiable instructions testing models' ability to follow precise formatting, length, and structural constraints.",Google DeepMind,https://arxiv.org/abs/2311.07911,https://github.com/google-research/google-research/tree/master/instruction_following_eval,2023-11-13,Research; Development; Selection,"Instruction following, Constraint satisfaction, Format compliance, Precise control",Core Performance; Robustness,Text,Text,New dataset (released with eval),Programmatically generated,Small (< 1K samples),Test (541 prompts with ~25 verifiable instructions),Fixed data-driven (static test set),Automatic (Reference-free),"1. Model receives prompt with verifiable instructions (e.g., 'respond in exactly 3 paragraphs', 'include word X at least 5 times') 2. Model generates response 3. Programmatic checks verify instruction compliance 4. Strict and loose accuracy metrics computed",Outputs,False,,"Verifiable instructions with programmatic checking, no ambiguity in correctness",unknown,GPT-3.5: 57.4% strict GPT-4: 76.9% strict Claude 2: 66.7% strict Gemini Ultra: 79.4% strict,Strict and loose metrics; Multiple instruction types; Programmatic verification; No human evaluation needed,Limited to verifiable instructions only; May not reflect realistic usage; Some instructions may be conflicting; Excludes semantic quality assessment,"MT-Bench, AlpacaEval, InstructGPT evals, FollowBench"
RULER (Long Context Evaluation),A synthetic benchmark for evaluating long-context language models with tasks requiring retrieval and reasoning over contexts up to 128K tokens.,NVIDIA,https://arxiv.org/abs/2404.06654,https://github.com/hsiehjackson/RULER,2024-04-10,Research; Development,"Long-context understanding, Needle-in-haystack retrieval, Multi-hop reasoning, Variable tracking",Core Performance; Robustness,Text,Text,Synthetic/Generated,Programmatically generated,Medium (1K - 100K samples),Test (configurable context lengths: 4K to 128K+),Fixed data-driven (static test set),Automatic (Reference-based),"1. Model receives long context (4K-128K tokens) with embedded tasks 2. Tasks include retrieval, multi-hop tracing, aggregation, QA 3. Accuracy computed at different context lengths and depths",Outputs,False,,"Synthetic generation with known ground truth, controlled difficulty levels",unknown,"GPT-4 Turbo (128K): ~90% at 64K, degrades beyond Claude 2.1 (200K): ~85% at 100K Gemini 1.5 Pro: >90% up to 128K",Multiple context lengths tested; Position-dependent analysis; Task type variations; Controlled synthetic generation,Synthetic tasks may not reflect real-world usage; Limited task diversity; Computational cost for long contexts; Does not test semantic understanding deeply,"Needle-in-Haystack, ZeroSCROLLS, L-Eval, LongBench, InfiniteBench"
SQuAD (Stanford Question Answering Dataset),"A reading comprehension dataset with 100K+ questions posed by crowdworkers on Wikipedia articles, where answers are text spans from the passage.",Stanford University,https://arxiv.org/abs/1606.05250,https://github.com/rajpurkar/SQuAD-explorer,2016-06-16,Research; Development,"Reading comprehension, Span extraction, Question answering, Natural language understanding",Core Performance,Text,Text,Wikipedia,Crowdsourced annotations,Large (100K - 1M samples),"Train (87,599), Dev (10,570)",Fixed data-driven (static test set),Automatic (Reference-based),"Model receives passage and question, predicts answer span from passage, F1 and Exact Match scores computed against human answers",Outputs,False,,"Crowdsourced questions with multiple answer annotations, consistency validation",unknown,"BERT-base: 88.5 F1, BERT-large: 90.9 F1, RoBERTa: 94.6 F1, Human performance: 91.2 F1",Multiple human answers; F1 and EM metrics; Unanswerable questions in v2.0,Saturated with modern models; Adversarial examples exist; Answer extraction biases; Limited reasoning depth required,"SQuAD 2.0, DROP, NaturalQuestions, NewsQA, TriviaQA"
GLUE (General Language Understanding Evaluation),"A collection of 9 natural language understanding tasks including sentiment analysis, textual entailment, and question answering for evaluating general-purpose language models.","New York University, University of Washington, DeepMind",https://arxiv.org/abs/1804.07461,https://github.com/nyu-mll/GLUE-baselines,2018-04-19,Research; Development; Selection,"Natural language understanding, Textual entailment, Sentiment analysis, Semantic similarity, Question answering",Core Performance,Text,Text,One or Multiple existing datasets,Multiple/Mixed sources,Large (100K - 1M samples),Varies by task,Composite,Automatic (Reference-based),"Model evaluated on 9 diverse NLU tasks, performance aggregated into single GLUE score, tasks include CoLA, SST-2, MRPC, QQP, STS-B, MNLI, QNLI, RTE, WNLI",Outputs,True,"Test set labels private, evaluation via online leaderboard","Diverse task selection covering multiple NLU phenomena, human baseline validation",unknown,"BERT-base: 78.3, BERT-large: 80.5, RoBERTa-large: 88.5, Human baseline: 87.1",Multiple tasks; Diagnostic dataset; Human baseline comparison,Saturated by modern models; Some tasks correlate highly; Diagnostic set limited; Replaced by SuperGLUE,"SuperGLUE, XTREME, CLUE, IndoNLU"
SuperGLUE,A more challenging benchmark than GLUE with 8 language understanding tasks designed to be difficult for models that exceed human baseline on GLUE.,"New York University, Facebook AI, DeepMind, University of Washington",https://arxiv.org/abs/1905.00537,https://github.com/nyu-mll/jiant,2019-05-02,Research; Development; Selection,"Advanced NLU, Reasoning, Coreference resolution, Natural language inference, Question answering",Core Performance; Robustness,Text,Text,One or Multiple existing datasets,Multiple/Mixed sources,Large (100K - 1M samples),Varies by task,Composite,Automatic (Reference-based),"Model evaluated on 8 challenging NLU tasks including BoolQ, CB, COPA, MultiRC, ReCoRD, RTE, WiC, WSC, performance aggregated into SuperGLUE score",Outputs,True,"Test set labels private, evaluation via leaderboard","Tasks selected to challenge state-of-art models, human baseline exceeds initial model performance",unknown,"BERT-large: 69.0, RoBERTa-large: 84.6, T5-11B: 89.3, Human baseline: 89.8",Multiple challenging tasks; Human baseline comparison; Diagnostic dataset,Becoming saturated; Some tasks highly correlated; Limited multilingual coverage; Aggregate score masks per-task performance,"GLUE, BIG-bench, XTREME, MMLU"
CommonsenseQA,"A multiple-choice question answering dataset requiring commonsense reasoning, with 12,247 questions generated from ConceptNet.","Tel Aviv University, Allen Institute for AI, University of Washington",https://arxiv.org/abs/1811.00937,https://github.com/jonathanherzig/commonsenseqa,2019-01-01,Research; Development,"Commonsense reasoning, Conceptual understanding, World knowledge, Reasoning over concepts",Core Performance,Text,Text,New dataset (released with eval),Crowdsourced annotations,Medium (1K - 100K samples),"Train (9,741), Dev (1,221), Test (1,140)",Fixed data-driven (static test set),Automatic (Reference-based),"Model receives question and 5 answer choices, selects most appropriate answer requiring commonsense reasoning, accuracy computed",Outputs,True,"Test set labels withheld, evaluation via leaderboard","Questions derived from ConceptNet knowledge graph, human validation of question quality",unknown,"BERT-large: 56.7%, RoBERTa-large: 72.1%, GPT-3: 65.4%, Human performance: 88.9%",Multiple answer distractors; Human baseline comparison; ConceptNet grounding,Multiple choice format; ConceptNet bias; Some questions solvable without commonsense; Cultural biases,"PIQA, WinoGrande, HellaSwag, ARC, COPA"
PIQA (Physical Interaction QA),"A benchmark for physical commonsense reasoning about everyday situations, with 20,000 multiple choice questions about physical interactions.","Allen Institute for AI, University of Washington",https://arxiv.org/abs/1911.11641,https://github.com/ybisk/ybisk.github.io/tree/master/piqa,2019-11-01,Research; Development,"Physical commonsense, Intuitive physics, Everyday reasoning, Goal-oriented reasoning",Core Performance,Text,Text,New dataset (released with eval),Crowdsourced annotations,Medium (1K - 100K samples),"Train (16,113), Dev (1,838), Test (3,084)",Fixed data-driven (static test set),Automatic (Reference-based),"Model receives goal description and two possible solutions, selects physically more plausible solution, accuracy computed",Outputs,False,,"Crowdsourced goals and solutions with filtering, human validation of physical plausibility",unknown,"BERT-large: 76.5%, RoBERTa-large: 79.2%, GPT-3: 81.0%, Human performance: 94.9%",Adversarial filtering; Human baseline comparison; Binary choice validation,Binary choice may be limiting; Cultural bias in physical scenarios; Dataset saturation with large models,"HellaSwag, CommonsenseQA, WinoGrande, COPA"
NaturalQuestions,"A question answering dataset with real Google search queries and answers extracted from Wikipedia, containing 307K training examples with long and short answers.",Google Research,https://aclanthology.org/Q19-1026/,https://github.com/google-research-datasets/natural-questions,2019-01-01,Research; Development,"Open-domain question answering, Reading comprehension, Information retrieval, Answer extraction",Core Performance,Text,Text,Wikipedia,Crowdsourced annotations,Large (100K - 1M samples),"Train (307,373), Dev (7,830), Test (7,842)",Fixed data-driven (static test set),Automatic (Reference-based),"Model receives question from real user query and Wikipedia page, predicts short answer (entity) and/or long answer (paragraph), F1 score computed",Outputs,False,,"Real user queries from Google search, annotators find answers in Wikipedia with validation",unknown,"BERT-joint: 64.1 F1 (short), T5: 76.4 F1, Human performance: 87.1 F1",Long and short answers; Real user queries; Multiple annotator validation,Wikipedia-only answers; Some questions unanswerable; Annotation subjectivity; Requires large-scale retrieval,"SQuAD, TriviaQA, MS MARCO, ELI5"
LAMBADA,"A word prediction dataset requiring broad context understanding, with 10K passages where the last word requires understanding the entire passage to predict correctly.","University of Trento, University of Amsterdam",https://arxiv.org/abs/1606.06031,https://huggingface.co/datasets/cimec/lambada,2016-06-01,Research; Development,"Long-range context understanding, Word prediction, Discourse comprehension, Semantic coherence",Core Performance,Text,Text,New dataset (released with eval),Author-provided,Medium (1K - 100K samples),"Test (5,153)",Fixed data-driven (static test set),Automatic (Reference-based),"Model receives passage with last word removed, must predict final word requiring full passage context, accuracy computed",Outputs,False,,"Passages selected where humans can predict final word only with full context, validation through human testing",unknown,"GPT-2: 45.99%, GPT-3: 76.2%, GPT-4: ~90% (estimated), Human performance: 100%",Human baseline validation; Long-range dependency testing,Single word prediction limited; Dataset saturated with modern LLMs; Limited diversity in passages,"HellaSwag, StoryCloze, RACE"
MBPP (Mostly Basic Python Problems),"A benchmark of 974 short Python programming problems designed for entry-level programmers, testing basic programming knowledge and code synthesis.",Google Research,https://arxiv.org/abs/2108.07732,https://github.com/google-research/google-research/tree/master/mbpp,2021-08-17,Research; Development,"Code synthesis, Basic programming, Algorithm implementation, Python fundamentals",Core Performance,Text; Code,Code,New dataset (released with eval),Author-provided,Small (< 1K samples),"Train (374), Test (500), Validation (90)",Fixed data-driven (static test set),Automatic (Execution-based),"Model receives natural language description of programming task, generates Python function, solution tested against 3 test cases, pass@k metric computed",Outputs,False,,"Crowdsourced problems with manual review, test case validation",unknown,"Codex (12B): 59.6% pass@80, GPT-3.5: 52.2% pass@1, GPT-4: 75.8% pass@1, Code Llama 34B: 62.6% pass@1",Multiple test cases; Entry-level focus; Pass@k evaluation,Only 3 test cases per problem; Python-only; Entry-level problems may not test advanced capabilities; Small dataset size,"HumanEval, APPS, CodeContests, MultiPL-E"
AlpacaEval,An LLM-based automatic evaluation benchmark with 805 questions for assessing instruction-following models using GPT-4 as a judge against reference outputs.,Stanford University,https://arxiv.org/abs/2305.14387,https://github.com/tatsu-lab/alpaca_eval,2023-05-22,Development; Selection,"Instruction following, General capabilities, Open-ended generation, Helpfulness",Core Performance; Core Quality Dimensions,Text,Text,One or Multiple existing datasets,Model-generated references,Small (< 1K samples),Test (805 instructions),Fixed data-driven (static test set),Model-based: In the wild,"Model generates responses to diverse instructions, GPT-4 compares model output to reference (text-davinci-003), win rate computed",Outputs,False,,"Strong correlation with human preferences demonstrated, multiple judge models available",unknown,"Alpaca 7B: 26.5% win rate, Vicuna 13B: 76.8%, GPT-3.5-turbo: 89.4%, GPT-4: 95.3%, Claude 2: 91.4%",Multiple judge models; Correlation with human preferences; Position bias controls; Length bias analysis,"GPT-4 judge biases (length, style); Reference model may not be optimal; Small evaluation set; Model-based evaluation limitations","MT-Bench, Chatbot Arena, Arena-Hard, IFEval"
GQA (Visual Reasoning),"A visual reasoning dataset with 22M questions on 113K images, featuring compositional questions requiring multi-step reasoning over scene graphs.",Stanford University,https://arxiv.org/abs/1902.09506,https://github.com/stanfordnlp/gqa-dataset,2019-02-01,Research; Development,"Visual reasoning, Compositional question answering, Scene understanding, Spatial reasoning",Core Performance; Robustness,Text + Vision,Text,MS COCO,Programmatically generated,Huge (> 10M samples),"Train (943K), Val (132K), Test (12.6M)",Fixed data-driven (static test set),Automatic (Reference-based),"Model receives image and compositional question, generates answer, accuracy computed with focus on reasoning consistency and grounding",Outputs,True,"Test set available but unannotated, challenge submission for evaluation","Programmatically generated from scene graphs with human validation, consistency checks for reasoning chains",unknown,"Bottom-Up: 49.7%, LXMERT: 60.0%, OSCAR: 61.6%, BLIP-2: 65.0%, Human performance: 89.3%",Compositional consistency; Grounding verification; Multiple question types; Validity and plausibility checks,Programmatic generation may have artifacts; Limited to COCO images; Some questions require implicit reasoning; Scene graph annotation noise,"VQA, CLEVR, Visual Genome, NLVR2"
COCO Keypoints,"Human pose estimation benchmark with keypoint annotations for 250K people across 200K images, part of the COCO dataset focusing on articulated pose detection.",Microsoft Research,https://arxiv.org/abs/1405.0312,https://github.com/cocodataset/cocoapi,2016-05-01,Research; Development,"Human pose estimation, Keypoint detection, Multi-person pose, Articulated object detection",Core Performance,Vision (Image),Structured Data,MS COCO,Human annotations,Large (100K - 1M samples),"Train, Val, Test",Fixed data-driven (static test set),Automatic (Reference-based),"Model receives image, predicts 17 keypoints per person, Object Keypoint Similarity (OKS) based mAP computed",Outputs,True,"Test set labels private, evaluation via online server","Manual annotation with quality control, consistency verification across annotators",unknown,"OpenPose: 61.8 AP, HRNet: 75.5 AP, ViTPose: 80.9 AP, Human performance: ~85 AP (estimated)",Multi-person scenarios; Occlusion handling; Scale variation; Multiple runs per sample,17 keypoints may be insufficient for fine-grained pose; Occlusion challenges; Crowded scene difficulties; Annotation noise,"MPII Human Pose, Human3.6M, PoseTrack, CrowdPose"
Chatbot Arena,"A crowdsourced platform for evaluating LLMs through pairwise comparisons, where users chat with two anonymous models and vote for the better response.","UC Berkeley, Stanford, UCSD",https://arxiv.org/abs/2403.04132,https://github.com/lm-sys/FastChat,2023-05-03,Development; Selection,"Conversational ability, Helpfulness, Instruction following, Overall quality",Core Performance; Core Quality Dimensions,Text,Text,User-generated content,No reference (reference-free),Huge (> 10M samples),Continuous (live platform),Dynamic data-driven (adaptive/interactive),Human: Representative sample,"Users submit prompts, receive responses from two anonymous models, vote for better response, Elo ratings computed from pairwise preferences",Outputs,False,,"Large-scale human preferences, Elo rating system with statistical validation, correlates with other benchmarks",unknown,"GPT-4-turbo: ~1250 Elo, Claude 3.5 Sonnet: ~1270 Elo, GPT-4o: ~1280 Elo, Llama 3.1 405B: ~1230 Elo",Large-scale human evaluation; Pairwise comparison; Continuous updates; Multiple model comparisons; Statistical significance testing,User population bias; Prompt distribution may not be representative; Style preferences over correctness; Elo inflation over time; No ground truth,"MT-Bench, AlpacaEval, Arena-Hard, LiveBench"
APPS (Automated Programming Progress Standard),"A benchmark of 10,000 programming problems at varying difficulty levels from introductory to competition-level, requiring code generation and algorithmic reasoning.","UC Berkeley",https://arxiv.org/abs/2105.09938,https://github.com/hendrycks/apps,2021-05-20,Research; Development,"Code generation, Algorithmic reasoning, Problem solving, Programming proficiency",Core Performance,Text; Code,Code,New dataset (released with eval),Existing dataset labels,Medium (1K - 100K samples),"Train (5,000), Test (5,000) across 3 difficulty levels",Fixed data-driven (static test set),Automatic (Execution-based),"Model receives problem description, generates Python solution, solution tested against hidden test cases, pass@k and strict accuracy computed",Outputs,False,,"Problems from coding practice platforms, test cases validate correctness",unknown,"GPT-3: 3.5% (competition), GPT-Neo 2.7B: 3.2% (competition), Codex: 25% (interview level), GPT-4: ~40% (estimated)",Three difficulty levels; Multiple test cases; Pass@k evaluation; Execution-based verification,Very challenging even for modern models; Test case coverage may be incomplete; Python-only; Time and memory constraints not enforced,"HumanEval, MBPP, CodeContests, Codeforces"
AGIEval,"A human-centric benchmark using official admission and qualification exams (SAT, LSAT, GRE, etc.) to evaluate general intelligence and reasoning in foundation models.",Microsoft Research,https://arxiv.org/abs/2304.06364,https://github.com/microsoft/AGIEval,2023-04-13,Research; Development,"General intelligence, Human-level reasoning, Exam performance, Domain knowledge",Core Performance,Text,Text,New dataset (released with eval),Existing dataset labels,Medium (1K - 100K samples),"20 official exams including SAT, LSAT, GRE, GMAT, Chinese Gaokao",Fixed data-driven (static test set),Automatic (Reference-based),"Model takes official human exams, answers multiple choice and generation questions, accuracy compared to human test-taker performance",Outputs,False,,"Official standardized exams with known difficulty and validity, human baseline available",unknown,"GPT-3.5: 42.9%, GPT-4: 56.4%, Claude 2: 51.2%, Human average: varies by exam (50-70%)",Multiple exam types; Cross-cultural exams; Human baseline comparison; Standardized test validation,Exam-specific knowledge may not reflect general intelligence; Cultural bias in exams; Translation issues; Potential training data contamination,"MMLU, BIG-bench, C-Eval, GAOKAO-Bench"
Human3.6M,"A large-scale dataset for 3D human pose estimation with 3.6 million video frames of 11 subjects performing 15 activities, captured with motion capture system.","University of Bonn",https://ieeexplore.ieee.org/document/6682899,http://vision.imar.ro/human3.6m/description.php,2014-06-01,Research; Development,"3D pose estimation, Human motion analysis, Activity recognition, Temporal modeling",Core Performance,Vision (Image); Video,Structured Data,New dataset (released with eval),Programmatically generated,Huge (> 10M samples),"Train (5 subjects), Test (2 subjects), Validation (1 subject)",Fixed data-driven (static test set),Automatic (Reference-based),"Model receives RGB video or images, predicts 3D joint positions, Mean Per Joint Position Error (MPJPE) computed",Outputs,False,,"Motion capture ground truth with high precision, multi-camera validation",unknown,"Martinez et al.: 62.9mm MPJPE, VideoPose3D: 46.8mm MPJPE, PoseFormer: 44.3mm MPJPE",Multiple subjects; Multiple activities; Multi-camera validation; Temporal consistency,Limited to indoor laboratory setting; Only 11 subjects; Controlled environment; Mocap suit artifacts,"MPII Human Pose, COCO Keypoints, PoseTrack, 3DPW"
BDD100K (Berkeley DeepDrive),"A diverse driving video dataset with 100K videos and annotations for multiple tasks including object detection, lane marking, drivable area, and instance segmentation.",UC Berkeley,https://arxiv.org/abs/1805.04687,https://github.com/bdd100k/bdd100k,2018-05-30,Research; Development; Deployment,"Autonomous driving perception, Multi-task learning, Object detection, Segmentation, Tracking",Core Performance; Robustness,Vision (Image); Video,Structured Data,New dataset (released with eval),Human annotations,Large (100K - 1M samples),"Train (70K), Val (10K), Test (20K)",Fixed data-driven (static test set),Automatic (Reference-based),"Model performs multiple tasks on driving videos, mAP for detection, IoU for segmentation, MOTA for tracking",Outputs,True,"Test set labels private, evaluation via online server","Crowdsourced annotations with quality control, diverse driving conditions validated",unknown,"Faster R-CNN: 30.1 mAP (detection), Mask R-CNN: 32.4 mAP, Multi-task: varies by task",Geographic diversity; Time of day variations; Weather conditions; Multi-task evaluation,Annotation quality varies; Limited to US driving; Class imbalance; Some tasks have sparse annotations,"nuScenes, Waymo Open Dataset, Argoverse, KITTI, Cityscapes"
ScanNet v2,"An indoor RGB-D video dataset of 1513 scanned scenes with semantic segmentation, instance segmentation, and CAD model alignments for 3D scene understanding.","Princeton University, TU Munich",https://arxiv.org/abs/1702.04405,https://github.com/ScanNet/ScanNet,2018-04-01,Research; Development,"3D scene understanding, Semantic segmentation, Instance segmentation, 3D object detection",Core Performance,Vision (Image); Video; Structured Data,Structured Data,New dataset (released with eval),Human annotations,Medium (1K - 100K samples),"Train (1201), Val (312), Test (100)",Fixed data-driven (static test set),Automatic (Reference-based),"Model receives 3D scene data, predicts semantic or instance segmentation, mIoU and mAP computed",Outputs,True,"Test set labels private, evaluation via online benchmark","Manual annotation of 3D reconstructions, multi-view consistency validation",unknown,"PointNet++: 53.5 mIoU, SparseConvNet: 72.5 mIoU, MinkowskiNet: 73.6 mIoU, Mask3D: 76.8 mIoU",Multiple indoor scene types; 3D consistency validation; Multi-task evaluation,Indoor scenes only; Reconstruction quality varies; Annotation effort intensive; Limited scene diversity,"Matterport3D, S3DIS, 2D-3D-S, ARKitScenes, ScanNet++"
HMDB-51,"A video action recognition dataset with 6,766 clips across 51 action categories extracted from movies and YouTube videos.","MIT, Brown University",https://ieeexplore.ieee.org/document/6126543,https://github.com/artix41/hmdb51-pytorch,2011-11-01,Research; Development,"Action recognition, Human motion understanding, Video classification, Temporal reasoning",Core Performance,Video,Structured Data,New dataset (released with eval),Human annotations,Medium (1K - 100K samples),3 train/test splits provided,Fixed data-driven (static test set),Automatic (Reference-based),"Model receives video clip, predicts action category, accuracy averaged across 3 splits",Outputs,False,,"Manual annotation and verification, category definitions validated",unknown,"Two-Stream CNN: 59.4%, I3D: 74.5%, SlowFast: 80.3%, VideoMAE: 83.1%",Three evaluation splits; Multiple runs per sample; Camera variation,Small dataset size; Video quality varies; Some categories overlap; Dataset saturated with modern models,"UCF-101, Kinetics, ActivityNet, Something-Something"
LiveBench,"A continuously updating benchmark with new questions released monthly to prevent test set contamination, covering math, coding, reasoning, and instruction following.","Abacus.AI, Various institutions",https://arxiv.org/abs/2406.19314,https://github.com/LiveBench/LiveBench,2024-06-01,Research; Development; Selection,"Math, Coding, Reasoning, Data analysis, Language, Instruction following",Core Performance; Leakage/Contamination,Text,Text; Code,New dataset (released with eval),Multiple/Mixed sources,Medium (1K - 100K samples),Monthly releases with new questions,Dynamic data-driven (adaptive/interactive),Automatic (Reference-based); Model-based: In the wild,"Model evaluated on fresh questions released monthly, prevents memorization, aggregate scores across categories computed",Outputs,False,,"Monthly question generation with quality review, contamination prevention through continuous updates",unknown,"GPT-4o: 55.4%, Claude 3.5 Sonnet: 57.2%, Gemini 1.5 Pro: 52.1% (scores vary monthly)",Monthly updates prevent contamination; Multiple task categories; Model-based and automatic judging,Limited historical data; Monthly score fluctuations; Judge model biases; Computational cost of continuous evaluation,"MT-Bench, Arena-Hard, MMLU, HumanEval, GPQA"
COPA (Choice of Plausible Alternatives),"A causal reasoning benchmark with 1,000 questions requiring selection of the most plausible cause or effect for a given premise."," Indiana University, University of Southern California",https://aaai.org/papers/02418-2418-choice-of-plausible-alternatives-an-evaluation-of-commonsense-causal-reasoning/,https://github.com/drwiner/COPA,2011-01-01,Research; Development,"Causal reasoning, Commonsense inference, Plausibility judgment, World knowledge",Core Performance,Text,Text,New dataset (released with eval),Expert annotations,Small (< 1K samples),"Dev (500), Test (500)",Fixed data-driven (static test set),Automatic (Reference-based),"Model receives premise and question (cause or effect), selects more plausible alternative from 2 choices, accuracy computed",Outputs,False,,Expert annotation with inter-annotator agreement validation,unknown,"BERT-large: 70.6%, RoBERTa-large: 90.6%, GPT-3: 92.0%, Human performance: 100%",Binary choice validation; Human baseline comparison; Causal reasoning focus,Small dataset size; Binary choice limiting; Saturated with modern models; Limited causal reasoning depth,"WinoGrande, CommonsenseQA, PIQA, HellaSwag"
RACE (Reading Comprehension from Examinations),"A reading comprehension dataset from Chinese English exams with 28,000 passages and 100,000 questions designed for middle and high school students.","Carnegie Mellon University",https://arxiv.org/abs/1704.04683,https://github.com/qizhex/RACE_AR_baselines,2017-04-01,Research; Development,"Reading comprehension, Reasoning, Educational assessment, Question answering",Core Performance,Text,Text,New dataset (released with eval),Existing dataset labels,Large (100K - 1M samples),"Train (87,866), Dev (4,887), Test (4,934)",Fixed data-driven (static test set),Automatic (Reference-based),"Model receives passage and multiple choice question, selects correct answer from 4 options, accuracy computed for middle and high school levels",Outputs,False,,"Real exam questions with verified answers, difficulty validated by educational use",unknown,"BERT-large: 72.0%, RoBERTa-large: 83.2%, GPT-3: 75.2%, Human (middle school): 85.1%",Two difficulty levels; Multiple choice validation; Educational standard alignment,Multiple choice format; Cultural context (Chinese exams); Some questions require cultural knowledge; Becoming saturated,"SQuAD, DREAM, ReClor, LogiQA"
Argoverse,"A dataset for 3D tracking and motion forecasting with HD maps, containing sensor data from autonomous vehicles in Pittsburgh and Miami.","Argo AI, Carnegie Mellon University",https://arxiv.org/abs/1911.02620,https://github.com/argoverse/argoverse-api,2019-11-01,Research; Development; Deployment,"3D object tracking, Motion forecasting, HD mapping, Sensor fusion, Scene understanding",Core Performance,Vision (Image); Video; Structured Data,Structured Data,New dataset (released with eval),Human annotations; Programmatically generated,Large (100K - 1M samples),"Train, Val, Test for tracking and forecasting tasks",Fixed data-driven (static test set),Automatic (Reference-based),"Model performs 3D tracking or trajectory forecasting with HD map context, metrics include ADE, FDE for forecasting and MOTA for tracking",Outputs,True,"Test set labels private, evaluation via leaderboard","Multi-sensor fusion validation, HD map ground truth verification",unknown,"Tracking: AB3DMOT 41.5 MOTA, Forecasting: various baselines for ADE/FDE metrics",Geographic diversity (2 cities); HD map integration; Multi-task evaluation,Limited to 2 cities; HD map dependency; Sensor-specific challenges; Annotation complexity,"nuScenes, Waymo Open Dataset, KITTI, Lyft Level 5, Argoverse 2"
Matterport3D,"A large-scale RGB-D dataset of 90 building-scale scenes with 10,800 panoramic views and dense semantic annotations for indoor scene understanding.","Princeton University, Stanford University, TU Munich",https://arxiv.org/abs/1709.06158,https://github.com/niessner/Matterport,2017-12-01,Research; Development,"3D scene understanding, Semantic segmentation, Visual navigation, Embodied AI, Room layout estimation",Core Performance,Vision (Image); Video; Structured Data,Structured Data,New dataset (released with eval),Human annotations,Medium (1K - 100K samples),"Train (61 scenes), Val (11 scenes), Test (18 scenes)",Fixed data-driven (static test set),Automatic (Reference-based),"Model performs semantic segmentation or navigation tasks on building-scale 3D scenes, mIoU for segmentation, SPL for navigation",Outputs,False,,"Professional 3D scanning with manual annotation verification, multi-view consistency",unknown,"Segmentation: PointNet++ 47.4 mIoU, Navigation: various agents with SPL metrics",Building-scale scenes; Multi-room navigation; Dense annotations,Limited to residential buildings; Expensive data collection; Annotation effort intensive; Texture variation limited,"ScanNet, Gibson, Habitat, AI2-THOR"
VTAB (Visual Task Adaptation Benchmark),"A diverse benchmark of 19 visual classification tasks across natural, specialized, and structured domains for evaluating few-shot transfer learning.",Google Research,https://arxiv.org/abs/1910.04867,https://github.com/google-research/task_adaptation,2019-10-01,Research; Development,"Transfer learning, Few-shot learning, Domain adaptation, Visual representation learning",Core Performance; Robustness; Meta-Learning,Vision (Image),Structured Data,One or Multiple existing datasets,Existing dataset labels,Medium (1K - 100K samples),"1000 examples per task for training, standard test sets",Composite,Automatic (Reference-based),"Pre-trained model fine-tuned on 1000 examples per task, evaluated on 19 diverse tasks, mean accuracy across tasks computed",Full,False,,"Diverse task selection validated for transfer learning evaluation, task difficulty balanced",unknown,"BiT-L (ResNet152x4): 76.3% mean, ViT-L: 77.6%, CLIP: 73.2%",19 diverse tasks; Natural/Specialized/Structured domain split; Few-shot setting; Transfer learning focus,Limited to 1000 training examples; Task selection bias; Aggregate metric may mask task-specific performance; Requires full model access,"Meta-Dataset, FGVC, CrossTransformers"
OpenBookQA,"A question answering dataset requiring multi-hop reasoning over open book facts and common knowledge, with 6,000 elementary science questions.",Allen Institute for AI,https://arxiv.org/abs/1809.02789,https://github.com/allenai/OpenBookQA,2018-09-01,Research; Development,"Multi-hop reasoning, Science reasoning, Open-book QA, Knowledge integration, Commonsense reasoning",Core Performance,Text,Text,New dataset (released with eval),Crowdsourced annotations,Medium (1K - 100K samples),"Train (4,957), Dev (500), Test (500)",Fixed data-driven (static test set),Automatic (Reference-based),"Model receives science question and core facts, selects answer from 4 choices requiring reasoning beyond provided facts, accuracy computed",Outputs,False,,"Crowdsourced with expert review, multi-hop reasoning validated through question design",unknown,"BERT-large: 62.0%, RoBERTa-large: 76.0%, UnifiedQA: 87.2%, Human performance: 91.7%",Multi-hop reasoning required; Core facts provided; Multiple distractors,Small dataset size; Multiple choice format; Core facts may not cover all needed knowledge; Becoming saturated,"ARC, CommonsenseQA, QASC, SciQ"
C-Eval,"A comprehensive Chinese evaluation suite with 13,948 questions across 52 subjects for assessing advanced knowledge and reasoning in Chinese language models.","Shanghai Jiao Tong University, Tsinghua University",https://arxiv.org/abs/2305.08322,https://github.com/SJTU-LIT/ceval,2023-05-01,Research; Development; Selection,"Chinese language understanding, Domain knowledge (Chinese context), Reasoning, Multitask learning",Core Performance,Text,Text,New dataset (released with eval),Expert annotations,Medium (1K - 100K samples),"Dev (1,346), Val (1,346), Test (12,342)",Fixed data-driven (static test set),Automatic (Reference-based),"Model answers multiple choice questions across 52 subjects in Chinese, accuracy computed per subject and overall",Outputs,True,Test set labels private for some splits,"Expert-curated questions from Chinese educational materials, subject matter expert validation",unknown,"GPT-4: 68.7%, ChatGPT: 52.5%, Claude: 54.2%, Best Chinese models: ~70%",52 diverse subjects; Multiple difficulty levels; Chinese-specific knowledge,Chinese language only; Cultural and educational system bias; Translation challenges for evaluation; Multiple choice format,"MMLU, AGIEval, CMMLU, GAOKAO-Bench"
SimpleBench,"A benchmark testing if language models can perform basic tasks that humans find trivial, focusing on instruction following and simple reasoning without tricks.",Independent Researchers,https://drive.google.com/file/d/1mddNFK5UbBFVr3oDftd2Kyc6D8TFctfe/view,https://github.com/simple-bench/SimpleBench,2024-01-01,Research; Development,"Basic instruction following, Simple reasoning, Practical task completion, Reliability on easy tasks",Core Performance; Robustness,Text,Text,New dataset (released with eval),Author-provided,Small (< 1K samples),Test (varies),Fixed data-driven (static test set),Automatic (Reference-based); Model-based: In the wild,"Model performs simple tasks humans find trivial, correctness validated automatically or by judge model, failure rate computed",Outputs,False,,"Tasks validated to be trivial for humans, baseline human performance near 100%",unknown,"GPT-4: ~90%, GPT-3.5: ~75%, Claude 2: ~85% (estimates vary by task type)",Human baseline comparison; Simple task focus; Multiple task types,Small benchmark; Subjective task selection; May not reflect real-world complexity; Limited coverage,"IFEval, MT-Bench, BigBench"
LongBench,A bilingual benchmark for long-context understanding with 21 tasks testing models on contexts from 4K to 32K+ tokens in English and Chinese.,Tsinghua University,https://arxiv.org/abs/2308.14508,https://github.com/THUDM/LongBench,2023-08-28,Research; Development,"Long-context understanding, Multi-document QA, Summarization, Code completion, Synthetic tasks",Core Performance,Text; Code,Text; Code,One or Multiple existing datasets,Multiple/Mixed sources,Medium (1K - 100K samples),Test sets for 21 tasks,Composite,Automatic (Reference-based),"Model evaluated on 21 tasks with long contexts (4K-32K+ tokens), tasks span QA, summarization, code, synthetic challenges, scores aggregated",Outputs,False,,"Diverse task selection covering multiple long-context scenarios, bilingual validation",unknown,"GPT-3.5-Turbo-16K: 44.0 avg, Claude 2: 45.7 avg, GPT-4-32K: 54.0 avg",21 diverse tasks; Bilingual (English/Chinese); Multiple context lengths; Multiple domains,Aggregate metrics may hide task-specific performance; Context length limitations vary by model; Computational cost,"RULER, ZeroSCROLLS, L-Eval, InfiniteBench, Needle-in-Haystack"
VQA v2.0,"An improved version of VQA with balanced pairs to reduce language bias, containing questions that cannot be answered without understanding the image.","Virginia Tech, Georgia Institute of Technology",https://arxiv.org/abs/1612.00837,https://github.com/GT-Vision-Lab/VQA,2017-01-01,Research; Development,"Visual question answering, Visual reasoning, Language grounding, Bias reduction",Core Performance; Robustness,Text + Vision,Text,MS COCO,Crowdsourced annotations,Huge (> 10M samples),"Train, Val, Test",Fixed data-driven (static test set),Automatic (Reference-based),"Model receives image and question, generates answer, accuracy computed with balanced pairs to prevent language-only solutions",Outputs,True,Test-dev and test-std splits with private labels,"Balanced complementary pairs reduce language bias, multiple human answers validate consistency",unknown,"Bottom-Up Attention: 65.3%, LXMERT: 72.4%, OSCAR: 73.6%, BLIP-2: 82.2%",Balanced pairs; Language bias reduction; Multiple human answers; Test-dev validation,Still has residual language bias; Some questions remain ambiguous; Limited reasoning depth; Answer distribution imbalance,"VQA, GQA, OK-VQA, TextVQA, VizWiz"
SQuAD 2.0,"An extension of SQuAD adding 50,000 unanswerable questions requiring models to determine when no answer exists in the passage.",Stanford University,https://arxiv.org/abs/1806.03822,https://github.com/rajpurkar/SQuAD-explorer,2018-06-11,Research; Development,"Reading comprehension, Span extraction, Answerability detection, Robustness",Core Performance; Robustness,Text,Text,Wikipedia,Crowdsourced annotations,Large (100K - 1M samples),"Train (130,319), Dev (11,873)",Fixed data-driven (static test set),Automatic (Reference-based),"Model receives passage and question, predicts answer span or indicates no answer exists, F1 and EM computed including no-answer examples",Outputs,False,,"Crowdsourced unanswerable questions with validation, plausible distractors verified",unknown,"BERT-large: 81.9 F1, RoBERTa: 89.8 F1, XLNet: 90.1 F1, Human performance: 89.5 F1",Unanswerable questions; Plausibility checks; Multiple human answers,Saturated with modern models; Answerability detection can be gamed; Limited reasoning complexity,"SQuAD, NaturalQuestions, DROP, NewsQA"
TextVQA,"A VQA benchmark requiring reading and reasoning about text in images, with 28,408 images containing text and 45,336 questions.",Facebook AI Research,https://arxiv.org/abs/1904.08920,https://github.com/facebookresearch/mmf,2019-04-01,Research; Development,"Text reading in images, Visual question answering, OCR reasoning, Scene text understanding",Core Performance,Text + Vision,Text,New dataset (released with eval),Crowdsourced annotations,Medium (1K - 100K samples),"Train (34,602), Val (5,000), Test (5,734)",Fixed data-driven (static test set),Automatic (Reference-based),"Model receives image containing text and question, generates answer requiring reading scene text, accuracy computed with multiple reference answers",Outputs,True,"Test set labels private, evaluation via leaderboard",Manual verification of text presence and question answering requiring text reading,unknown,"LoRRA: 26.6%, M4C: 40.5%, TAP: 54.7%, BLIP-2: 62.3%, Human performance: 80.8%",Multiple answer annotations; OCR token accuracy; Human baseline comparison,OCR quality dependency; Text reading vs reasoning ambiguity; Limited to horizontal text; Scene complexity varies,"VQA v2, ST-VQA, OCR-VQA, DocVQA"
OK-VQA (Outside Knowledge VQA),"A VQA dataset requiring external commonsense and world knowledge beyond the image content, with 14,055 questions.","Carnegie Mellon University, University of Washington",https://arxiv.org/abs/1906.00067,https://github.com/allenai/ok-vqa,2019-06-01,Research; Development,"Knowledge-based VQA, Commonsense reasoning, External knowledge integration, Visual grounding",Core Performance,Text + Vision,Text,MS COCO,Crowdsourced annotations,Medium (1K - 100K samples),"Train (9,009), Val (5,046)",Fixed data-driven (static test set),Automatic (Reference-based),"Model receives image and question requiring external knowledge, generates answer, accuracy computed against multiple references",Outputs,False,,"Questions filtered to require outside knowledge, human validation of knowledge requirements",unknown,"KRISP: 38.4%, REVIVE: 47.5%, Prophet: 61.1%, Flamingo: 50.6%",Knowledge requirement validation; Multiple answer annotations; External knowledge integration testing,Knowledge source ambiguity; Difficult to validate knowledge requirements; Answer diversity; Outdated knowledge,"VQA v2, GQA, A-OKVQA, FVQA"
NLVR2 (Natural Language for Visual Reasoning),A visual reasoning benchmark requiring understanding relationships between two images based on natural language descriptions.,Cornell University,https://arxiv.org/abs/1811.00491,https://github.com/lil-lab/nlvr,2019-01-01,Research; Development,"Visual reasoning, Multi-image understanding, Language grounding, Compositional reasoning",Core Performance,Text + Vision,Text,New dataset (released with eval),Crowdsourced annotations,Large (100K - 1M samples),"Train (86,373), Dev (6,982), Test (6,967)",Fixed data-driven (static test set),Automatic (Reference-based),"Model receives natural language statement and two images, determines if statement is true about the pair, accuracy computed",Outputs,True,Test set labels private for public test,"Crowdsourced descriptions with validation, compositional reasoning verified",unknown,"MaxEnt: 56.6%, LXMERT: 76.2%, UNITER: 79.1%, ALBEF: 82.6%, Human performance: 95.4%",Compositional structure; Two-image reasoning; Human baseline comparison,Binary classification limiting; Synthetic image pairs; Description complexity varies; Approaching saturation,"CLEVR, GQA, VQA v2, NLVR"
MS MARCO,A large-scale information retrieval and question answering dataset with real Bing queries and passages from web documents.,Microsoft,https://arxiv.org/abs/1611.09268,https://github.com/microsoft/MSMARCO-Passage-Ranking,2016-11-01,Research; Development; Deployment,"Information retrieval, Passage ranking, Question answering, Reading comprehension",Core Performance,Text,Text,New dataset (released with eval),Human annotations; Crowdsourced annotations,Huge (> 10M samples),"Train (8.8M passages), Dev, Eval",Fixed data-driven (static test set),Automatic (Reference-based),"Model ranks passages for queries or generates answers from passages, MRR@10 for ranking, ROUGE-L and BLEU for QA",Outputs,True,Evaluation set for annual competitions,"Real search queries with human-labeled relevant passages, answer generation validated",unknown,"BM25: 16.7 MRR@10, BERT: 34.7, ANCE: 38.1, ColBERT: 39.7",Large-scale real queries; Multiple evaluation metrics; Passage ranking and QA tasks,Sparse annotations; Single relevant passage assumption; Query ambiguity; Web document quality varies,"NaturalQuestions, TriviaQA, SQuAD, TREC-DL"
HELM (Holistic Evaluation of Language Models),"A comprehensive framework evaluating LLMs across 42 scenarios and 7 metrics categories including accuracy, calibration, robustness, fairness, bias, toxicity, and efficiency.",Stanford University,https://arxiv.org/abs/2211.09110,https://github.com/stanford-crfm/helm,2022-11-16,Research; Development; Selection,"Holistic evaluation, Multi-scenario performance, Fairness, Robustness, Calibration, Efficiency, Bias detection",Core Performance; Core Quality Dimensions; Fairness; Robustness; Calibration; Efficiency; Safety,Text,Text,One or Multiple existing datasets,Multiple/Mixed sources,Large (100K - 1M samples),Varies by scenario,Composite,Automatic (Reference-based); Model-based: In the wild,"Models evaluated on 42 scenarios across 7 metric categories, standardized prompting, comprehensive reporting of accuracy, calibration, robustness, fairness, bias, toxicity, efficiency",Outputs,False,,"Comprehensive scenario coverage validated, multiple metric categories ensure holistic assessment",unknown,"Performance varies by scenario and model, comparative analysis across models provided",42 scenarios; 7 metric categories; Perturbation testing; Demographic analysis; Calibration testing,Computational cost; Scenario selection bias; Aggregate metrics complexity; Continuous updates needed,"BIG-bench, MMLU, SuperGLUE, Chatbot Arena"
MMLU-Pro,"An enhanced version of MMLU with more challenging questions, 10 answer choices instead of 4, and focus on reasoning over memorization.","University of Waterloo, University of Toronto, Carnegie Mellon University",https://arxiv.org/abs/2406.01574,https://github.com/TIGER-AI-Lab/MMLU-Pro,2024-06-03,Research; Development; Selection,"Advanced knowledge, Reasoning, Domain expertise, Distractor resistance",Core Performance; Robustness,Text,Text,One or Multiple existing datasets,Expert annotations,Medium (1K - 100K samples),"Test (12,000 questions)",Fixed data-driven (static test set),Automatic (Reference-based),"Model receives multiple choice question with 10 options, selects answer, accuracy computed with chain-of-thought reasoning encouraged",Outputs,False,,"Questions filtered for difficulty and reasoning requirements, expert review of distractors",unknown,"GPT-4o: 72.6%, Claude 3.5 Sonnet: 78.0%, Random baseline: 10%",10-way multiple choice; Reasoning-focused questions; Distractor quality validation,Still multiple choice format; Limited to 14 subjects; Potential for future saturation; Question source overlap with MMLU,"MMLU, GPQA, AGIEval, JEE-Advanced"
Arena-Hard,"A benchmark of 500 challenging user queries designed to differentiate top-tier LLMs, evaluated using GPT-4-Turbo as judge with baseline comparisons.","UC Berkeley",https://arxiv.org/abs/2406.11939,https://github.com/lm-sys/arena-hard-auto,2024-04-01,Development; Selection,"Challenging queries, Instruction following, Reasoning, Multi-step tasks, Real-world complexity",Core Performance; Core Quality Dimensions,Text,Text,User-generated content,No reference (reference-free),Small (< 1K samples),Test (500 queries),Fixed data-driven (static test set),Model-based: In the wild,"Model generates response to challenging query, GPT-4-Turbo compares against GPT-4-0314 baseline, win rate computed",Outputs,False,,"Queries selected from Chatbot Arena for discriminative power, correlation with human preferences validated",unknown,"GPT-4-0314: 50.0% (baseline), Claude 3.5 Sonnet: 79.2%, GPT-4o: 78.0%, Gemini 1.5 Pro: 72.0%",Challenging query selection; Agreement with Chatbot Arena; GPT-4-Turbo judge calibration,Small dataset (500 queries); GPT-4 judge bias; Static benchmark (will saturate); Model-based evaluation limits,"MT-Bench, Chatbot Arena, AlpacaEval, LiveBench"
VizWiz,"A VQA dataset from blind users with 31,000 visual questions taken by people who are blind, testing real-world accessibility scenarios.","University of Texas at Austin, Microsoft, University of Colorado Boulder",https://arxiv.org/abs/1802.08218,https://github.com/VizWiz-dev/vizwiz,2018-04-01,Research; Development; Deployment,"Accessibility, Visual question answering, Real-world robustness, Image quality handling, Unanswerable questions",Core Performance; Robustness; Fairness,Text + Vision,Text,User-generated content,Crowdsourced annotations,Medium (1K - 100K samples),"Train (20,523), Val (4,319), Test (8,000)",Fixed data-driven (static test set),Automatic (Reference-based),"Model receives real-world image from blind user and question, generates answer or indicates unanswerable, accuracy computed with human annotations",Outputs,True,"Test set labels private, evaluation via leaderboard","Real user questions from blind community, multiple annotators per question for quality",unknown,"Pythia: 46.0%, LXMERT: 51.5%, BLIP-2: 60.5%, Human performance: 91.2%",Real-world images; Poor quality handling; Unanswerable detection; Multiple annotators,Image quality highly variable; Some questions ambiguous; Annotation challenges with unclear images; Domain-specific (accessibility),"VQA v2, TextVQA, GQA, OK-VQA"
Something-Something V2,"A video dataset of 220,847 clips showing humans performing pre-defined actions with objects, requiring temporal reasoning to distinguish similar actions.",TwentyBN,https://arxiv.org/abs/1706.04261,https://github.com/TwentyBN/smth-smth-v2-baseline,2018-05-01,Research; Development,"Temporal reasoning, Action recognition, Fine-grained motion understanding, Object-action relationships",Core Performance,Video,Structured Data,New dataset (released with eval),Crowdsourced annotations,Large (100K - 1M samples),"Train (168,913), Val (24,777), Test (27,157)",Fixed data-driven (static test set),Automatic (Reference-based),"Model receives video clip of object interaction, predicts action category (174 classes), top-1 and top-5 accuracy computed",Outputs,True,Test set labels withheld for challenge,"Crowdsourced with template-based actions ensuring consistency, temporal reasoning validated",unknown,"TSN: 19.7%, TSM: 63.4%, SlowFast: 67.2%, VideoMAE: 74.8%",Template-based collection; Temporal reasoning required; Fine-grained actions; Object variation,Template constraints may limit diversity; Short clips (2-6 sec); Controlled recording environment; Label noise,"Kinetics, ActivityNet, UCF-101, Moments in Time"
TriviaQA,"A large-scale reading comprehension dataset with 95K question-answer pairs authored by trivia enthusiasts, with supporting documents from Wikipedia and web search.",University of Washington,https://arxiv.org/abs/1705.03551,https://github.com/mandarjoshi90/triviaqa,2017-05-01,Research; Development,"Reading comprehension, Open-domain QA, Evidence-based reasoning, Distant supervision",Core Performance,Text,Text,New dataset (released with eval),Crowdsourced annotations,Large (100K - 1M samples),"Train (87,622), Dev (11,313), Test (10,790)",Fixed data-driven (static test set),Automatic (Reference-based),"Model receives question and evidence documents, extracts answer span, Exact Match and F1 scores computed for Wikipedia and web domains",Outputs,False,,"Trivia questions with multiple evidence documents, distant supervision validated with answer aliases",unknown,"BiDAF: 40.3 F1, BERT: 72.5 F1, RoBERTa: 83.2 F1, T5: 85.1 F1",Multiple evidence documents; Answer aliases; Two domains (Wikipedia and Web),Distant supervision noise; Answer extraction challenges; Multiple valid answer forms; Evidence document quality varies,"SQuAD, NaturalQuestions, MS MARCO, HotpotQA"
Winograd Schema Challenge,"A commonsense reasoning test with 273 carefully crafted pronoun resolution problems requiring world knowledge, designed as an alternative to the Turing test.",University of Toronto,https://cdn.aaai.org/ocs/4492/4492-21843-1-PB.pdf,https://github.com/vnmsklnk/Winograd-Schema-Challenge,2012-01-01,Research; Development,"Commonsense reasoning, Coreference resolution, World knowledge, Causal understanding",Core Performance,Text,Text,New dataset (released with eval),Expert annotations,Small (< 1K samples),Test (273 problems),Fixed data-driven (static test set),Automatic (Reference-based),"Model resolves pronoun ambiguity in sentence pairs differing by one word that flips the answer, binary choice, accuracy computed",Outputs,False,,"Carefully constructed to require commonsense, cannot be solved by simple statistics or associations",unknown,"Random: 50%, GPT-2: 70.7%, RoBERTa: 90.1%, GPT-3: 88.3%, Human performance: ~92%",Paired examples; Commonsense requirement validation; Expert construction,Very small dataset (273 problems); Binary choice; Some problems ambiguous; Saturated with modern LLMs,"WinoGrande, COPA, CommonsenseQA, PIQA"
VGGFace2,"A large-scale face recognition dataset with 3.31 million images of 9,131 subjects with large variations in pose, age, illumination, ethnicity, and profession.",University of Oxford,https://arxiv.org/abs/1710.08092,https://github.com/ox-vgg/vgg_face2,2018-05-01,Research; Development,"Face recognition, Identity verification, Face attribute recognition, Pose variation handling",Core Performance; Robustness,Vision (Image),Structured Data,New dataset (released with eval),Programmatically generated,Huge (> 10M samples),"Train (3.31M images, 9,131 identities)",Fixed data-driven (static test set),Automatic (Reference-based),"Model performs face recognition or verification, accuracy and verification metrics computed on evaluation sets",Outputs,False,,"Automated collection with manual verification, pose and age variation validated",unknown,"ResNet-50: 96.0% verification, SENet-50: 97.3%, ArcFace: 99.8%",Large pose variation; Age variation; Ethnicity diversity; Profession diversity,Celebrity bias; Ethical concerns with face recognition; Some identity label noise; Imbalanced distribution,"LFW, CelebA, MS-Celeb-1M, IJB-C"
iNaturalist,"A large-scale fine-grained species classification dataset with millions of images across thousands of species, collected from citizen scientists.","Caltech, Google, Cornell",https://arxiv.org/abs/1707.06642,https://github.com/visipedia/inat_comp,2017-01-01,Research; Development,"Fine-grained recognition, Species classification, Long-tail learning, Real-world classification",Core Performance; Robustness,Vision (Image),Structured Data,User-generated content,Expert annotations,Huge (> 10M samples),"Train, Val, Test (varies by year, typically 500K+ training images)",Fixed data-driven (static test set),Automatic (Reference-based),"Model classifies species from image (10,000+ categories in recent versions), top-1 and top-3 accuracy computed",Outputs,False,,"Expert taxonomist validation, citizen scientist observations verified",unknown,"ResNet-50: 55-65% top-1 (varies by year), EfficientNet: 75-80%, ViT-L: 80-85%",Long-tail distribution; Real-world images; Expert validation; Large category count,Extreme class imbalance; Location bias; Seasonal bias; Image quality variation; Taxonomic changes over time,"FGVC Aircraft, CUB-200, Stanford Cars, NABirds"
StoryCloze,A commonsense reasoning framework for evaluating story understanding with four-sentence stories and two alternative endings.,University of Rochester,https://arxiv.org/abs/1604.01696,https://github.com/snigdhac/StoryComprehension_EMNLP,2016-04-01,Research; Development,"Story understanding, Commonsense reasoning, Narrative coherence, Causal reasoning",Core Performance,Text,Text,New dataset (released with eval),Crowdsourced annotations,Medium (1K - 100K samples),"Val (1,871), Test (1,871)",Fixed data-driven (static test set),Automatic (Reference-based),"Model receives four-sentence story context, selects correct ending from two choices, accuracy computed",Outputs,False,,"Crowdsourced stories with right and wrong endings, coherence validated",unknown,"DSSM: 58.5%, GPT: 86.5%, BERT: 90.4%, RoBERTa: 91.8%, Human performance: 100%",Binary choice validation; Coherence requirement; Human baseline,Binary choice limiting; Saturated with modern models; Some endings debatable; Small test set,"HellaSwag, COPA, WinoGrande, LAMBADA"
ImageNet-C,A robustness benchmark with 15 corruption types applied to ImageNet validation images to test model resilience to common image perturbations.,UC Berkeley,https://arxiv.org/abs/1903.12261,https://github.com/hendrycks/robustness,2019-03-01,Research; Development,"Robustness to corruptions, Out-of-distribution generalization, Perturbation resilience",Robustness,Vision (Image),Structured Data,ImageNet,Programmatically generated,Medium (1K - 100K samples),"50,000 validation images with 15 corruption types at 5 severity levels",Fixed data-driven (static test set),Automatic (Reference-based),"Model classifies corrupted images, mean Corruption Error (mCE) computed across corruption types and severities",Outputs,False,,"Systematic corruption application with human-verified realism, severity levels calibrated",unknown,"ResNet-50: 76.7 mCE, ResNet-152: 68.2 mCE, EfficientNet-B7: 42.5 mCE, ViT-L: 38.7 mCE",15 corruption types; 5 severity levels; Systematic perturbation testing,Limited corruption types; Severity calibration subjective; May not reflect real-world distribution shifts; Clean accuracy correlation,"ImageNet-A, ImageNet-R, ImageNet-Sketch, CIFAR-10-C"
DocVQA,"A VQA dataset for document understanding with 50,000 questions on 12,000 document images requiring reading and reasoning about text, tables, and layouts.","CVIT, IIIT Hyderabad, India",https://arxiv.org/abs/2007.00398,https://github.com/doc-analysis/DocVQA,2020-07-01,Research; Development; Deployment,"Document understanding, Visual question answering, OCR reasoning, Layout understanding, Table reasoning",Core Performance,Text + Vision,Text,New dataset (released with eval),Crowdsourced annotations,Medium (1K - 100K samples),"Train (39,463), Val (5,349), Test (5,188)",Fixed data-driven (static test set),Automatic (Reference-based),"Model receives document image and question, generates answer requiring document understanding, ANLS metric computed",Outputs,True,Test set labels private for challenge,"Crowdsourced questions on real documents, answer verification through multiple annotations",unknown,"T5: 33.6 ANLS, LayoutLM: 36.4 ANLS, Donut: 67.5 ANLS, GPT-4V: 88.4 ANLS",Real document diversity; Complex layouts; Multiple question types; ANLS metric for fuzzy matching,OCR dependency; Document type bias; Complex layout challenges; Answer extraction ambiguity,"TextVQA, ST-VQA, InfographicVQA, ChartQA"
YouTube-VOS,"A large-scale video object segmentation dataset with 4,453 YouTube videos and 94 object categories with pixel-level annotations.","Adobe Research",https://arxiv.org/abs/1809.03327,https://youtube-vos.org/,2018-09-01,Research; Development,"Video object segmentation, Temporal consistency, Object tracking, Multi-object segmentation",Core Performance,Video,Structured Data,New dataset (released with eval),Human annotations,Medium (1K - 100K samples),"Train (3,471 videos), Val (474 videos), Test (508 videos)",Fixed data-driven (static test set),Automatic (Reference-based),"Model segments objects in video given first-frame annotations, J (region similarity) and F (contour accuracy) metrics computed",Outputs,True,Test set labels private for challenge,"Pixel-level manual annotations with quality control, temporal consistency verified",unknown,"OSMN: 51.2 J&F, STM: 79.4 J&F, AOT: 84.1 J&F, XMem: 87.7 J&F",Large object vocabulary; Temporal consistency testing; Multi-object scenarios; Occlusion handling,Annotation cost limits coverage; YouTube video bias; Some categories underrepresented; First-frame dependency,"DAVIS, OVIS, MOSE, VIPSeg"
Objects365,"A large-scale object detection dataset with 2 million images and 365 object categories, designed to improve detection across diverse objects.",Megvii,https://openaccess.thecvf.com/content_ICCV_2019/papers/Shao_Objects365_A_Large-Scale_High-Quality_Dataset_for_Object_Detection_ICCV_2019_paper.pdf,https://www.objects365.org/download.html,2019-09-01,Research; Development,"Object detection, Large-scale recognition, Category diversity, Multi-scale detection",Core Performance,Vision (Image),Structured Data,New dataset (released with eval),Human annotations,Huge (> 10M samples),"Train (1.7M images), Val (80K images)",Fixed data-driven (static test set),Automatic (Reference-based),"Model detects objects and assigns categories (365 classes), mAP at IoU 0.5 computed",Outputs,False,,"Professional annotation with multi-stage quality control, category taxonomy validated",unknown,"Faster R-CNN: 20.8 mAP, Cascade R-CNN: 25.6 mAP, DINO: 63.3 mAP",365 diverse categories; Large-scale dataset; Multiple object scales,Annotation consistency challenges at scale; Category imbalance; Some categories overlap; Quality variation,"COCO, LVIS, Open Images, V3Det"
PoseTrack,"A large-scale video-based pose estimation and tracking dataset with 550 videos containing 23,000 frames annotated with human pose and tracking IDs.","University of Bonn",https://arxiv.org/abs/1710.10000,https://github.com/anDoer/PoseTrack21,2017-01-01,Research; Development,"Multi-person pose estimation, Pose tracking, Temporal consistency, Video understanding",Core Performance,Video,Structured Data,New dataset (released with eval),Human annotations,Medium (1K - 100K samples),"Train (292 videos), Val (50 videos), Test (208 videos)",Fixed data-driven (static test set),Automatic (Reference-based),"Model estimates poses and tracks identities across video frames, mAP for detection and MOTA for tracking computed",Outputs,True,Test set labels private for challenge,"Manual pose annotation with tracking IDs, temporal consistency validated across frames",unknown,"OpenPose: 55.2 mAP, HRNet: 74.3 mAP, DCPose: 78.1 mAP",Temporal consistency; Multi-person scenarios; Tracking evaluation; Video-based challenges,Limited to 15 keypoints; Annotation effort intensive; Some frames have occlusion; Video quality varies,"COCO Keypoints, MPII Human Pose, Human3.6M, CrowdPose"
Moments in Time,A large-scale dataset with 1 million labeled 3-second videos covering 339 action and activity classes for event recognition.,MIT-IBM Watson AI Lab,https://arxiv.org/abs/1801.03150,https://github.com/metalbubble/moments_models,2017-01-01,Research; Development,"Event recognition, Action classification, Temporal understanding, Diverse activity recognition",Core Performance,Video,Structured Data,New dataset (released with eval),Human annotations,Huge (> 10M samples),"Train (800K), Val (33K), Test (67K)",Fixed data-driven (static test set),Automatic (Reference-based),"Model classifies 3-second video clip into one of 339 event categories, top-1 and top-5 accuracy computed",Outputs,False,,"Crowdsourced with quality control, event categories validated for diversity and clarity",unknown,"ResNet-50 (2D): 26.9%, I3D: 33.4%, Multi-Moments: 40.0%",339 diverse categories; Large-scale dataset; Short clips focus on events,Short 3-second clips may miss context; Some categories overlap; Dataset bias toward common events,"Kinetics, ActivityNet, Something-Something, UCF-101"
EMNIST (Extended MNIST),"An extended MNIST dataset with 814,255 characters including digits and letters (uppercase and lowercase) for handwritten character recognition.","The MARCS Institute for Brain, Behaviour and Development",https://arxiv.org/abs/1702.05373,https://github.com/pytorch/vision,2017-02-01,Research; Development,"Handwritten character recognition, Digit recognition, Letter recognition, Balanced classification",Core Performance,Vision (Image),Structured Data,New dataset (released with eval),Human annotations,Large (100K - 1M samples),"Multiple splits (ByClass, ByMerge, Balanced, Letters, Digits, MNIST)",Fixed data-driven (static test set),Automatic (Reference-based),"Model classifies 28x28 grayscale character images, accuracy computed for various splits (10 digits, 26 letters, 47 balanced, 62 classes)",Outputs,False,,Based on NIST Special Database with verified character labels,unknown,"CNN: 85-99% (varies by split), ResNet: 90-99%, ViT: 92-99%",Multiple dataset splits; Balanced and unbalanced versions; Character diversity,Low resolution (28x28); Grayscale only; Limited to English characters; Handwriting style variation limited,"MNIST, Fashion-MNIST, Kuzushiji-MNIST, QMNIST"
SUN397 (Scene Understanding),"A scene categorization benchmark with 108,754 images across 397 scene categories covering diverse indoor and outdoor environments.",MIT,https://faculty.cc.gatech.edu/~hays/papers/sun.pdf,https://3dvision.princeton.edu/projects/2010/SUN/,2010-01-01,Research; Development,"Scene recognition, Scene categorization, Environmental understanding, Context recognition",Core Performance,Vision (Image),Structured Data,New dataset (released with eval),Human annotations,Large (100K - 1M samples),10 pre-defined train/test splits,Fixed data-driven (static test set),Automatic (Reference-based),"Model classifies scene into one of 397 categories, accuracy computed across multiple splits",Outputs,False,,"Manual scene categorization with taxonomy validation, multi-annotator agreement",unknown,"AlexNet: 51.3%, VGG-16: 59.4%, ResNet-152: 68.5%, Places-CNN: 73.5%",397 diverse categories; Multiple evaluation splits; Indoor and outdoor scenes,Some category overlap; Unbalanced distribution; Scene definition ambiguity; Approaching saturation,"Places365, MIT Indoor 67, Scene-15, ADE20K"
CrowdPose,"A challenging multi-person pose estimation dataset with 20,000 images containing crowded scenes with occlusions and close interactions.","Shanghai Jiao Tong University",https://arxiv.org/abs/1812.00324,https://github.com/Jeff-sjtu/CrowdPose,2019-01-01,Research; Development,"Multi-person pose estimation, Crowded scene understanding, Occlusion handling, Dense pose estimation",Core Performance; Robustness,Vision (Image),Structured Data,New dataset (released with eval),Human annotations,Medium (1K - 100K samples),"Train (10,000), Val (2,000), Test (8,000)",Fixed data-driven (static test set),Automatic (Reference-based),"Model estimates 14 keypoints per person in crowded scenes, AP computed at multiple IoU thresholds",Outputs,False,,"Manual annotation with focus on crowded scenarios, occlusion levels validated",unknown,"Mask R-CNN: 57.3 AP, HRNet: 67.6 AP, HigherHRNet: 69.9 AP",Crowded scenarios; Occlusion levels measured; Close interactions; Multiple difficulty levels,Limited to 14 keypoints; Annotation challenges in extreme crowding; Some keypoints not visible,"COCO Keypoints, MPII Human Pose, OCHuman, PoseTrack"
Stanford Cars,"A fine-grained car classification dataset with 16,185 images of 196 car classes at the level of make, model, and year.",Stanford University,https://ai.stanford.edu/~jkrause/papers/fgvc13.pdf,https://huggingface.co/datasets/tanganke/stanford_cars,2013-01-01,Research; Development,"Fine-grained classification, Vehicle recognition, Subordinate categorization, Visual discrimination",Core Performance,Vision (Image),Structured Data,New dataset (released with eval),Expert annotations,Medium (1K - 100K samples),"Train (8,144), Test (8,041)",Fixed data-driven (static test set),Automatic (Reference-based),"Model classifies car into one of 196 classes (make, model, year), accuracy computed",Outputs,False,,Expert annotation of car makes and models with year verification,unknown,"ResNet-50: 92.2%, Bilinear CNN: 91.3%, TransFG: 94.8%, ViT-L: 95.3%",Fine-grained distinctions; Expert validation; Viewpoint variation,Dataset no longer maintained; Limited viewpoint diversity; Bias toward certain makes; Approaching saturation,"FGVC Aircraft, CUB-200-2011, Stanford Dogs, CompCars"
Conceptual Captions,A large-scale image captioning dataset with 3.3 million image-caption pairs automatically harvested from the web with alt-text.,Google Research,https://aclanthology.org/P18-1238.pdf,https://github.com/google-research-datasets/conceptual-captions,2018-05-01,Research; Development,"Image captioning, Visual-language learning, Web-scale learning, Zero-shot transfer",Core Performance,Vision (Image),Text,New dataset (released with eval),Programmatically generated,Huge (> 10M samples),"Train (3.3M), Val (15K)",Fixed data-driven (static test set),Automatic (Reference-based),"Model generates caption for image, BLEU, METEOR, CIDEr, SPICE metrics computed",Outputs,False,,"Automatic extraction from alt-text with filtering pipeline, quality sampling validated",unknown,Baseline models trained on CC show strong transfer to COCO and other datasets,Large-scale diversity; Web-based collection; Filtering pipeline for quality,Alt-text noise; Caption quality varies; Some captions not descriptive; Automated collection biases,"COCO Captions, Flickr30k, SBU Captions, Conceptual 12M"
NewsQA,"A reading comprehension dataset with 120K questions on 12K CNN news articles, requiring reasoning over long documents.","Microsoft Research, Maluuba",https://arxiv.org/abs/1611.09830,https://github.com/Maluuba/newsqa,2017-04-01,Research; Development,"Reading comprehension, Long document understanding, Question answering, News article reasoning",Core Performance,Text,Text,New dataset (released with eval),Crowdsourced annotations,Large (100K - 1M samples),"Train (92,549), Dev (5,166), Test (5,126)",Fixed data-driven (static test set),Automatic (Reference-based),"Model receives news article and question, extracts answer span, F1 and EM computed",Outputs,False,,"Crowdsourced questions with multiple answer validations, answer span verification",unknown,"BERT-large: 69.1 F1, RoBERTa: 73.8 F1, Human performance: 69.4 F1",Long documents; Multiple annotators; News domain diversity,Answer ambiguity in some cases; News article bias; Span extraction challenges; Approaching saturation,"SQuAD, NaturalQuestions, TriviaQA, RACE"
Caltech-101,"An object recognition dataset with 9,146 images across 101 object categories plus background, widely used for few-shot learning evaluation.",California Institute of Technology,https://authors.library.caltech.edu/7694/,https://github.com/pytorch/vision,2003-09-01,Research; Development,"Object recognition, Few-shot learning, Transfer learning, Zero-shot learning",Core Performance; Meta-Learning,Vision (Image),Structured Data,New dataset (released with eval),Human annotations,Medium (1K - 100K samples),"Typically 30 train per class, rest for test",Fixed data-driven (static test set),Automatic (Reference-based),"Model classifies objects into 101 categories, accuracy computed with few-shot setting (typically 30 shots)",Outputs,False,,"Manual annotation with category validation, few-shot protocol standardized",unknown,"ResNet-50: 92%, ViT: 95%, CLIP: 93%",Few-shot evaluation; Multiple evaluation protocols,Small dataset size; Category imbalance; Some categories have few examples; Saturated with modern methods,"Caltech-256, CIFAR-10, ImageNet, FGVC"
CUB-200-2011 (Caltech-UCSD Birds),"A fine-grained bird species classification dataset with 11,788 images of 200 bird species, including part annotations and attributes.","Caltech, UC San Diego",https://authors.library.caltech.edu/27452/,https://github.com/visipedia/cub,2011-06-01,Research; Development,"Fine-grained recognition, Bird species classification, Part-based recognition, Attribute learning",Core Performance,Vision (Image),Structured Data,New dataset (released with eval),Expert annotations,Medium (1K - 100K samples),"Train (5,994), Test (5,794)",Fixed data-driven (static test set),Automatic (Reference-based),"Model classifies bird species from image, accuracy computed with optional part/attribute annotations",Outputs,False,,"Expert ornithologist annotations, part locations and attributes verified",unknown,"ResNet-50: 84%, Bilinear CNN: 85%, TransFG: 91.7%, ViT-L: 91%",Fine-grained evaluation; Part annotations for analysis,Limited to North American birds; Pose variation challenges; Background clutter; Approaching saturation,"FGVC Aircraft, Stanford Cars, Stanford Dogs, NABirds, iNaturalist"
Oxford-IIIT Pets,"A pet breed classification dataset with 37 categories of cats and dogs, containing 7,349 images with pixel-level segmentation masks.",University of Oxford,https://ieeexplore.ieee.org/document/6248092,https://github.com/pytorch/vision,2012-06-01,Research; Development,"Fine-grained recognition, Pet breed classification, Segmentation, Species classification",Core Performance,Vision (Image),Structured Data,New dataset (released with eval),Human annotations,Medium (1K - 100K samples),"Train (3,680), Test (3,669)",Fixed data-driven (static test set),Automatic (Reference-based),"Model classifies pet breed (37 classes), accuracy computed for classification and segmentation tasks",Outputs,False,,Expert breed identification with segmentation masks validated,unknown,"ResNet-50: 93%, ViT: 96%, Fine-tuned models: 97%+",Fine-grained breed distinction; Segmentation quality validation,Limited to 37 breeds; Pose variation; Background complexity; Approaching saturation,"Stanford Dogs, CUB-200, FGVC, iNaturalist"
Oxford Flowers-102,"A flower species classification dataset with 102 flower categories, containing 8,189 images with large variations in scale, pose, and light.",University of Oxford,https://www.robots.ox.ac.uk/~vgg/publications/2008/Nilsback08/,https://github.com/pytorch/vision,2008-01-01,Research; Development,"Fine-grained classification, Flower recognition, Species classification, Visual discrimination",Core Performance,Vision (Image),Structured Data,New dataset (released with eval),Expert annotations,Medium (1K - 100K samples),"Train (1,020), Val (1,020), Test (6,149)",Fixed data-driven (static test set),Automatic (Reference-based),"Model classifies flower species (102 classes), accuracy computed on test set",Outputs,False,,Expert botanist annotations with species verification,unknown,"ResNet-50: 89%, ViT: 99.1%, Fine-tuned models: 99.5%+",Scale variation; Pose variation; Lighting variation,Small training set; High test/train ratio; Limited to UK flowers; Saturated with modern methods,"CUB-200, FGVC, iNaturalist, PlantCLEF"
Food-101,"A food recognition dataset with 101,000 images across 101 food categories from Foodspotting.com, featuring realistic user-captured images.",ETH Zurich,https://data.vision.ee.ethz.ch/cvl/datasets_extra/food-101/,https://github.com/pytorch/vision,2014-01-01,Research; Development; Deployment,"Food recognition, Fine-grained classification, Real-world image understanding",Core Performance,Vision (Image),Structured Data,User-generated content,Human annotations,Large (100K - 1M samples),"Train (75,750), Test (25,250)",Fixed data-driven (static test set),Automatic (Reference-based),"Model classifies food dish (101 classes), top-1 accuracy computed",Outputs,False,,"Crowdsourced from food images with verification, noisy labels intentionally included",unknown,"ResNet-50: 83%, EfficientNet: 90%, ViT-L: 92%",Real-world noise; User-generated images; Label noise tolerance,Label noise (up to 20%); Cultural bias (mostly Western food); Intra-class variation high,"Recipe1M, Food-101N, ISIA Food-500, UEC Food-256"
DTD (Describable Textures Dataset),"A texture classification dataset with 5,640 images across 47 texture categories, each described by human-understandable attributes.",University of Oxford,https://ieeexplore.ieee.org/document/6909856,https://github.com/pytorch/vision,2014-04-01,Research; Development,"Texture recognition, Material classification, Perceptual attributes, Fine-grained texture",Core Performance,Vision (Image),Structured Data,New dataset (released with eval),Human annotations,Medium (1K - 100K samples),10 pre-defined train/val/test splits,Fixed data-driven (static test set),Automatic (Reference-based),"Model classifies texture into 47 categories, accuracy computed across splits",Outputs,False,,"Human-describable attributes validated, perceptual consistency verified",unknown,"ResNet-50: 73%, ViT: 79%, CLIP: 77%",10 evaluation splits; Perceptual attribute grounding,Small dataset; Some categories overlap; Attribute subjectivity; Context dependency,"FMD, MINC, KTH-TIPS, CUReT"
EuroSAT,"A satellite image classification dataset with 27,000 labeled images across 10 land use and land cover classes from Sentinel-2 satellite.",German Research Center for Artificial Intelligence,https://arxiv.org/abs/1709.00029,https://github.com/phelber/eurosat,2019-01-01,Research; Development; Deployment,"Land use classification, Satellite image analysis, Remote sensing, Earth observation",Core Performance,Vision (Image),Structured Data,New dataset (released with eval),Expert annotations,Medium (1K - 100K samples),"10 land use classes, 2,000-3,000 images each",Fixed data-driven (static test set),Automatic (Reference-based),"Model classifies land use type (10 classes), accuracy computed",Outputs,False,,"Expert remote sensing annotations, ground truth validated with maps",unknown,"ResNet-50: 98%, ViT: 99%, Domain-adapted models: 99%+",Multi-spectral data; Geographic diversity; Seasonal variation,Limited to Europe; 10 broad categories; Seasonal bias; Approaching saturation,"UCMerced, AID, RESISC45, PatternNet"
RESISC45,"A remote sensing image scene classification dataset with 31,500 images across 45 scene classes from Google Earth.",Northwestern Polytechnical University,https://arxiv.org/abs/1703.00121,https://github.com/pytorch/vision,2017-03-01,Research; Development,"Scene classification, Remote sensing, Aerial image understanding, Land use recognition",Core Performance,Vision (Image),Structured Data,New dataset (released with eval),Expert annotations,Medium (1K - 100K samples),45 classes with 700 images each,Fixed data-driven (static test set),Automatic (Reference-based),"Model classifies aerial scene (45 classes), accuracy computed with various train/test splits",Outputs,False,,"Expert annotation with remote sensing knowledge, geographic diversity validated",unknown,"ResNet-50: 92%, ViT: 95%, Attention models: 96%",Geographic diversity; Scale variation; Multiple viewpoints,Fixed spatial resolution; Google Earth bias; Some class overlap; Intra-class variation,"UCMerced, AID, EuroSAT, PatternNet"
SVHN (Street View House Numbers),"A digit recognition dataset with 600,000+ labeled digits from Google Street View images, designed as a more challenging MNIST alternative.",Stanford University,http://ufldl.stanford.edu/housenumbers/nips2011_housenumbers.pdf,https://github.com/pytorch/vision,2011-01-01,Research; Development,"Digit recognition, Character recognition, Real-world OCR, Multi-digit recognition",Core Performance; Robustness,Vision (Image),Structured Data,New dataset (released with eval),Human annotations,Large (100K - 1M samples),"Train (73,257), Test (26,032), Extra (531,131)",Fixed data-driven (static test set),Automatic (Reference-based),"Model classifies digits (10 classes) from cropped images, accuracy computed",Outputs,False,,"Automated detection with human verification, bounding boxes validated",unknown,"CNN: 95%, ResNet: 98%, ViT: 98.5%",Real-world variation; Multiple fonts; Lighting variation; Perspective distortion,Limited to digits; Format-specific (house numbers); Cropping artifacts; Approaching saturation,"MNIST, EMNIST, USPS, MNIST-M"
STL-10,An image recognition dataset inspired by CIFAR-10 with higher resolution (96x96) images and only 500 labeled training images per class for unsupervised learning.,Stanford University,http://ai.stanford.edu/~acoates/papers/coatesleeng_aistats_2011.pdf,https://github.com/pytorch/vision,2011-04-01,Research; Development,"Unsupervised learning, Semi-supervised learning, Self-supervised learning, Transfer learning",Core Performance; Meta-Learning,Vision (Image),Structured Data,New dataset (released with eval),Human annotations,Large (100K - 1M samples),"Train (5,000 labeled, 100,000 unlabeled), Test (8,000)",Fixed data-driven (static test set),Automatic (Reference-based),"Model classifies images (10 classes) with limited labels, leveraging unlabeled data, accuracy computed",Full,False,,"Subset of ImageNet with manual verification, unsupervised protocol standardized",unknown,"Supervised ResNet: 78%, Self-supervised SimCLR: 91%, MoCo v2: 92%",Limited labeled data; Large unlabeled set; Self-supervised evaluation,Small labeled set; Limited to 10 classes; Derived from ImageNet; Saturation with self-supervised methods,"CIFAR-10, ImageNet, CIFAR-100, Tiny ImageNet"
Country211,"A country recognition dataset with 211 countries, designed to test geographic diversity recognition in vision models using OpenAI CLIP.",OpenAI,https://github.com/openai/CLIP/blob/main/data/country211.md,https://github.com/openai/CLIP,2021-02-26,Research; Development,"Geographic recognition, Country classification, Geographic diversity, Zero-shot transfer",Core Performance; Fairness,Vision (Image),Structured Data,New dataset (released with eval),Programmatically generated,Large (100K - 1M samples),Test set for evaluation,Fixed data-driven (static test set),Automatic (Reference-based),"Model classifies images by country (211 classes), accuracy computed for geographic recognition",Outputs,False,,"Geotag-based collection with verification, geographic distribution validated",unknown,"CLIP ViT-B/32: 26%, CLIP ViT-L/14: 34%, Fine-tuned models: 50%+",211 countries; Geographic diversity; Zero-shot evaluation,Geographic bias in image availability; Urban/landmark bias; Some countries underrepresented; Ambiguous images,"Dollar Street, GeoDE, ImageNet geographic subsets"
ObjectNet,"A dataset of 50,000 images designed to test object recognition robustness with novel backgrounds, rotations, and viewpoints not in training sets.","MIT, IBM",https://objectnet.dev/objectnet-a-large-scale-bias-controlled-dataset-for-pushing-the-limits-of-object-recognition-models.pdf,https://objectnet.dev/download.html,2019-12-01,Research; Development,"Robustness, Out-of-distribution generalization, Object recognition, Viewpoint invariance",Robustness,Vision (Image),Structured Data,New dataset (released with eval),Human annotations,Medium (1K - 100K samples),"Test only (50,000 images, 313 object classes)",Fixed data-driven (static test set),Automatic (Reference-based),"Model classifies objects with novel viewpoints and backgrounds, accuracy drop from ImageNet measured",Outputs,False,,"Controlled collection to maximize distribution shift, viewpoint diversity validated",unknown,"ImageNet models see 40-45% accuracy drop, ResNet-50: 25%, EfficientNet: 30%",Novel viewpoints; Unusual backgrounds; Rotation variation; Control over distribution shift,Test-only dataset; Limited to objects in ImageNet; Controlled collection may not reflect real distribution,"ImageNet-A, ImageNet-R, ImageNet-C, ImageNet-Sketch"
Winoground,A vision-language compositional reasoning task with 400 image pairs testing whether models can match images to captions requiring compositional understanding.,Facebook AI Research,https://arxiv.org/abs/2204.03162,https://github.com/facebookresearch/Winoground,2022-04-01,Research; Development,"Compositional reasoning, Vision-language understanding, Word order sensitivity, Fine-grained understanding",Core Performance; Robustness,Text + Vision,Structured Data,New dataset (released with eval),Expert annotations,Small (< 1K samples),Test (400 examples),Fixed data-driven (static test set),Automatic (Reference-based),"Model matches two captions to two images where word order matters, group score computed (all 4 matches correct)",Outputs,False,,"Expert-designed examples requiring compositional understanding, human validation of difficulty",unknown,"CLIP ViT-B/32: 11% group, CLIP ViT-L/14: 16.5% group, Human performance: 89.5% group",Compositional requirement; Word order sensitivity; Expert validation,Small dataset (400 examples); Binary choice per pair; Limited compositional patterns; Challenging for all models,"CLEVR, GQA, VQA v2, NLVR2"
RenderedSST2 (Rendered Sentiment Treebank),"A vision-language benchmark with 10,000 rendered images of text snippets from SST-2 for sentiment classification via vision.",OpenAI,https://arxiv.org/abs/2103.00020,https://github.com/openai/CLIP/blob/main/data/rendered-sst2.md,2021-01-01,Research; Development,"OCR reasoning, Visual text understanding, Sentiment analysis, Vision-language transfer",Core Performance,Vision (Image),Text,One or Multiple existing datasets,Existing dataset labels,Medium (1K - 100K samples),"Train, Val, Test from SST-2",Fixed data-driven (static test set),Automatic (Reference-based),"Model reads rendered text in images and classifies sentiment (positive/negative), accuracy computed",Outputs,False,,"Rendered from SST-2 with verified sentiment labels, OCR readability validated",unknown,"CLIP: 68%, ViT with OCR: 85%, Multimodal models: 90%+",OCR capability testing; Vision-language integration,Rendered text not natural; Limited to SST-2 data; OCR dependency; Binary sentiment only,"TextVQA, DocVQA, ST-VQA, SST-2"
Sugar Crepe,"A compositional vision-language benchmark with 3,000 hard negative captions testing fine-grained compositional understanding in vision-language models.","University of Washington",https://arxiv.org/abs/2306.14610,https://github.com/RAIVNLab/sugar-crepe,2023-06-01,Research; Development,"Compositional understanding, Hard negative discrimination, Fine-grained vision-language alignment, Object-relation understanding",Core Performance; Robustness,Text + Vision,Structured Data,One or Multiple existing datasets,Expert annotations,Small (< 1K samples),"Test (3,000 examples across multiple subsets)",Fixed data-driven (static test set),Automatic (Reference-based),"Model distinguishes correct caption from hard negatives differing in object, relation, or attribute, accuracy computed",Outputs,False,,"Expert-created hard negatives requiring fine-grained understanding, human validation of difficulty",unknown,"CLIP ViT-L/14: 67%, BLIP: 71%, Flamingo: 74%, Human performance: ~95%",Hard negative design; Compositional variations; Fine-grained distinctions,Small dataset; Limited compositional patterns; Binary choice per example; Focused on specific phenomena,"Winoground, ARO, CREPE, VL-CheckList"
Dollar Street,"A dataset of 38,479 images from homes across 54 countries at different income levels, designed to test model bias and geographic fairness.","Coactive AI",https://proceedings.neurips.cc/paper_files/paper/2022/file/5474d9d43c0519aa176276ff2c1ca528-Paper-Datasets_and_Benchmarks.pdf,https://www.kaggle.com/datasets/mlcommons/the-dollar-street-dataset,2021-11-9,Research; Development,"Geographic fairness, Income-level representation, Object recognition across cultures, Bias detection",Fairness; Robustness; Core Performance,Vision (Image),Structured Data,New dataset (released with eval),Human annotations,Medium (1K - 100K samples),Images organized by income level and geography,Fixed data-driven (static test set),Automatic (Reference-based),"Model classifies everyday objects with evaluation stratified by income level and geography, fairness metrics computed",Outputs,False,,"Systematic collection across income levels, geographic and economic diversity validated",unknown,"Performance varies significantly by income level and geography, revealing systematic biases",Income stratification; Geographic diversity; Fairness analysis; Bias detection,Limited object categories; Incomplete income coverage; Sampling bias; Cultural context variation,"ImageNet, GeoDE, Country211, Cultural datasets"
CountBench,A counting benchmark testing models' ability to count objects in real-world images across various difficulty levels.,Google,https://arxiv.org/abs/2302.12066,https://huggingface.co/datasets/nielsr/countbench,2020-01-01,Research; Development,"Object counting, Numeracy, Attention mechanisms, Subitizing vs counting",Core Performance,Vision (Image),Text,New dataset (released with eval),Human annotations,Medium (1K - 100K samples),Various difficulty levels,Fixed data-driven (static test set),Automatic (Reference-based),"Model counts objects in images, exact match and mean absolute error computed",Outputs,False,,"Human counting with verification, difficulty levels calibrated",unknown,"CNNs struggle with large counts, attention models perform better",Multiple difficulty levels; Object density variation; Real-world complexity,Counting ground truth can be ambiguous; Occlusion challenges; Object definition issues,"TallyQA, HowMany-QA, CLEVR Count, Visual Genome counting"
DMLab (DeepMind Lab),"A 3D learning environment with tasks testing navigation, memory, and spatial reasoning in first-person perspective.",DeepMind,https://arxiv.org/abs/1612.03801,https://github.com/deepmind/lab,2016-12-01,Research; Development,"Navigation, Spatial memory, Visual reasoning, Agent learning, First-person understanding",Core Performance; Robustness,Vision (Image); Video,Actions,Synthetic/Generated,Programmatically generated,Large (100K - 1M samples),Multiple task environments,Dynamic data-driven (adaptive/interactive),Automatic (Execution-based),"Agent navigates 3D environment completing tasks, success rate and reward computed",Full,False,,"Programmatic task generation with clear success criteria, human playability verified",unknown,"Various RL agents, performance varies significantly by task complexity",Multiple task types; Difficulty levels; Generalization testing,3D synthetic environment; Sim-to-real gap; Task-specific training needed; Computational requirements,"VizDoom, Habitat, AI2-THOR, Minecraft"
PUG ImageNet,Photorealistic Unreal Graphics ImageNet - a synthetic reimplementation of ImageNet using Unreal Engine for controlled robustness testing.,Meta,https://arxiv.org/abs/2308.03977,https://github.com/facebookresearch/PUG,2023-01-01,Research; Development,"Robustness to rendering, Photorealism understanding, Controlled distribution shift, 3D understanding",Robustness; Core Performance,Vision (Image),Structured Data,Synthetic/Generated,Programmatically generated,Large (100K - 1M samples),Aligned with ImageNet classes,Fixed data-driven (static test set),Automatic (Reference-based),"Model classifies synthetic photorealistic objects, accuracy and robustness to rendering parameters measured",Outputs,False,,"3D models verified against ImageNet categories, photorealism validated",unknown,ImageNet models show performance drops on synthetic renderings despite photorealism,Controlled rendering parameters; Systematic perturbations; Photorealism testing,Synthetic domain; Limited to available 3D models; Rendering artifacts; Distribution shift from real images,"ImageNet, ImageNet-R, ObjectNet, Stylized ImageNet"
Humanity's Last Exam,"An expert-level reasoning benchmark with questions designed to be extremely difficult, testing the limits of AI academic reasoning capabilities.","Scale AI, Center for AI Safety",https://arxiv.org/abs/2501.14249,https://huggingface.co/datasets/cais/hle,2024-12-01,Research; Development,"Academic reasoning, Expert-level knowledge, Complex problem solving, Multi-domain expertise",Core Performance,Text,Text,New dataset (released with eval),Expert annotations,Small (< 1K samples),Test set with various difficulty levels,Fixed data-driven (static test set),Automatic (Reference-based),"Model answers extremely challenging academic questions across multiple domains, accuracy computed with and without tools",Outputs,False,,"Expert-created questions validated for difficulty and correctness, human expert baselines established",unknown,"Gemini 3 Pro: 37.5% (no tools), 45.8% (with tools), Claude Sonnet 4.5: 13.7%, GPT-5.1: 26.5%",Multiple evaluation settings; Tool use vs no tools; Expert validation,Very small dataset; Extremely high difficulty may limit practical insights; Tool-assisted performance varies,"GPQA, MMLU-Pro, AGIEval, Graduate-level exams"
AIME 2025,"Problems from the 2025 American Invitational Mathematics Examination, testing advanced high school mathematics problem-solving.",Mathematical Association of America,https://artificialanalysis.ai/evaluations/aime-2025,https://huggingface.co/datasets/MathArena/aime_2025,2025-01-01,Research; Development,"Advanced mathematics, Competition math, Problem solving, Mathematical reasoning",Core Performance,Text,Text,New dataset (released with eval),Published references,Small (< 1K samples),15 problems per exam,Fixed data-driven (static test set),Automatic (Reference-based),"Model solves AIME-level math problems, answers are integers 0-999, exact match computed with and without code execution",Outputs,False,,"Official competition problems with verified solutions, difficulty calibrated for top students",unknown,"Gemini 3 Pro: 95.0% (no tools), 100% (with code), Claude Sonnet 4.5: 87.0% (no tools), 100% (with code), GPT-5.1: 94.0%",Competition-level difficulty; Code execution option; Answer verification,Very small dataset (15 questions); Annual release only; Integer answers may allow guessing strategies,"MATH, AMC, IMO, GPQA"
MathArena Apex,A challenging mathematics competition benchmark with extremely difficult contest problems requiring advanced problem-solving skills.,"ETHZ",https://matharena.ai/,https://github.com/matharena/apex,2024-01-01,Research; Development,"Competition mathematics, Advanced problem solving, Mathematical creativity, Multi-step reasoning",Core Performance,Text,Text,New dataset (released with eval),Expert annotations,Small (< 1K samples),Test set of challenging problems,Fixed data-driven (static test set),Automatic (Reference-based),"Model solves extremely challenging math competition problems, accuracy computed",Outputs,False,,Competition-level problems validated by expert mathematicians,unknown,"Gemini 3 Pro: 23.4%, Gemini 2.5 Pro: 0.5%, Claude Sonnet 4.5: 1.6%, GPT-5.1: 1.0%",Extreme difficulty level; Multiple solution approaches; Expert validation,Extremely difficult even for top models; Small dataset; May require mathematical creativity beyond current AI,"MATH, AIME, IMO, Putnam"
MMMU-Pro,An enhanced version of MMMU with more challenging multimodal questions requiring deeper understanding and reasoning across multiple modalities.,"MMMU Team",https://mmmu-benchmark.github.io/,https://github.com/MMMU-Benchmark/MMMU,2024-01-01,Research; Development,"Multimodal understanding, Visual reasoning, Domain expertise, Cross-modal reasoning",Core Performance,Text + Vision,Text,New dataset (released with eval),Expert annotations,Medium (1K - 100K samples),Test set with increased difficulty,Fixed data-driven (static test set),Automatic (Reference-based),"Model answers multimodal questions requiring expert knowledge, accuracy computed across subjects",Outputs,False,,"Expert-level questions validated across multiple domains, multimodal alignment verified",unknown,"Gemini 3 Pro: 81.0%, Gemini 2.5 Pro: 68.0%, Claude Sonnet 4.5: 68.0%, GPT-5.1: 76.0%",Multi-domain evaluation; Expert-level difficulty; Multimodal integration,Requires domain expertise; Multimodal alignment challenges; May have annotation inconsistencies,"MMMU, M3Exam, ScienceQA, AI2D"
ScreenSpot-Pro,An advanced screen understanding benchmark testing models' ability to locate and interact with UI elements across diverse applications.,"National University of Singapore",https://arxiv.org/abs/2504.07981,https://huggingface.co/blog/Ziyang/screenspot-pro,2024-01-01,Research; Development; Deployment,"Screen understanding, UI element localization, Visual grounding, GUI automation, Spatial reasoning",Core Performance,Text + Vision,Structured Data,New dataset (released with eval),Human annotations,Medium (1K - 100K samples),Test set across multiple applications,Fixed data-driven (static test set),Automatic (Reference-based),"Model localizes UI elements given natural language instructions, accuracy computed based on localization precision",Outputs,False,,"Human validation of UI element locations, diverse application coverage verified",unknown,"Gemini 3 Pro: 72.7%, Gemini 2.5 Pro: 11.4%, Claude Sonnet 4.5: 36.2%, GPT-5.1: 3.5%",Multiple application types; Diverse UI patterns; Real-world screenshots,UI changes over time; Application version dependencies; Resolution sensitivity; Localization precision requirements,"Mind2Web, WebShop, MiniWoB++, AndroidInTheWild"
CharXiv Reasoning,A benchmark testing information synthesis and reasoning from complex charts and figures in scientific papers from arXiv.,"Princeton University",https://charxiv.github.io/,https://github.com/charxiv/charxiv,2024-01-01,Research; Development,"Chart understanding, Information synthesis, Scientific reasoning, Visual data interpretation, Multi-hop reasoning",Core Performance,Text + Vision,Text,New dataset (released with eval),Expert annotations,Medium (1K - 100K samples),Test set from arXiv papers,Fixed data-driven (static test set),Automatic (Reference-based),"Model answers questions requiring synthesis of information from complex scientific charts, accuracy computed",Outputs,False,,"Expert validation of chart interpretations, scientific accuracy verified",unknown,"Gemini 3 Pro: 81.4%, Gemini 2.5 Pro: 69.6%, Claude Sonnet 4.5: 68.5%, GPT-5.1: 69.5%",Complex chart types; Multi-hop reasoning; Scientific domain diversity,Chart quality varies; OCR challenges; Domain-specific knowledge required; Annotation complexity,"ChartQA, PlotQA, FigureQA, DVQA"
OmniDocBench 1.5,"A comprehensive document understanding benchmark testing OCR, layout analysis, and document reasoning across diverse document types.","Shanghai AI Laboratory",https://arxiv.org/abs/2412.07626,https://github.com/opendatalab/OmniDocBench,2024-01-01,Research; Development; Deployment,"Document understanding, OCR, Layout analysis, Table reasoning, Multi-page understanding",Core Performance,Text + Vision,Text,New dataset (released with eval),Human annotations,Medium (1K - 100K samples),Test set across document types,Fixed data-driven (static test set),Automatic (Reference-based),"Model performs document understanding tasks, Overall Edit Distance metric computed (lower is better)",Outputs,False,,"Human annotations across diverse document types, quality control for OCR accuracy",unknown,"Gemini 3 Pro: 0.115, Gemini 2.5 Pro: 0.145, Claude Sonnet 4.5: 0.145, GPT-5.1: 0.147 (lower is better)",Diverse document types; Multiple evaluation metrics; OCR quality assessment,Document format diversity; OCR dependency; Layout complexity; Multi-page challenges,"DocVQA, InfographicVQA, DeepForm, FUNSD"
Video-MMMU,A video-based extension of MMMU testing multimodal understanding and knowledge acquisition from video content across expert domains.,"Carnegie Mellon University",https://arxiv.org/abs/2501.13826,https://huggingface.co/datasets/lmms-lab/VideoMMMU,2024-01-01,Research; Development,"Video understanding, Temporal reasoning, Knowledge acquisition from videos, Multimodal integration",Core Performance,Text + Vision; Video,Text,New dataset (released with eval),Expert annotations,Medium (1K - 100K samples),Test set across domains,Fixed data-driven (static test set),Automatic (Reference-based),"Model answers questions about video content requiring domain expertise, accuracy computed",Outputs,False,,"Expert validation of video content and questions, temporal understanding verified",unknown,"Gemini 3 Pro: 87.6%, Gemini 2.5 Pro: 83.6%, Claude Sonnet 4.5: 77.8%, GPT-5.1: 80.4%",Temporal reasoning; Multi-domain videos; Expert-level questions,Video processing requirements; Temporal annotation challenges; Domain expertise needed; Computational cost,"MMMU, VideoQA, ActivityNet-QA, TVQA"
LiveCodeBench Pro,"An advanced version of LiveCodeBench with competitive programming problems from recent Codeforces, ICPC, and IOI competitions.","UC Berkeley, MIT, Cornell University",https://livecodebench.github.io/,https://github.com/LiveCodeBench/LiveCodeBench,2024-01-01,Research; Development,"Competitive programming, Algorithm design, Complex problem solving, Code generation",Core Performance; Leakage/Contamination,Text,Code,New dataset (released with eval),Existing dataset labels,Medium (1K - 100K samples),Continuously updated with new problems,Dynamic data-driven (adaptive/interactive),Automatic (Execution-based),"Model solves competitive programming problems, Elo rating computed based on problem difficulty and success rate (higher is better)",Outputs,False,,"Competition problems with verified test cases, difficulty ratings from platforms",unknown,"Gemini 3 Pro: 2,439 Elo, Gemini 2.5 Pro: 1,775 Elo, Claude Sonnet 4.5: 1,418 Elo, GPT-5.1: 2,243 Elo (higher is better)",Continuous updates prevent contamination; Elo rating system; Multiple problem sources,Platform-specific formatting; Test case coverage varies; Elo rating calibration; Computational requirements,"CodeContests, APPS, Codeforces, USACO"
Terminal-Bench 2.0,A benchmark for agentic terminal/command-line coding tasks using the Terminus-2 agent framework for bash automation and scripting.,Research community,https://www.tbench.ai/news/announcement-2-0,https://www.tbench.ai/,2025-11-07,Research; Development; Deployment,"Agentic coding, Terminal automation, Bash scripting, System administration, Tool use",Core Performance,Text,Code,New dataset (released with eval),Programmatically generated,Small (< 1K samples),Test tasks for terminal operations,Dynamic data-driven (adaptive/interactive),Automatic (Execution-based),"Agent performs terminal-based coding tasks using Terminus-2 framework, success rate computed",Outputs,False,,"Execution-based validation with clear success criteria, real terminal environment",unknown,"Gemini 3 Pro: 54.2%, Gemini 2.5 Pro: 32.6%, Claude Sonnet 4.5: 42.8%, GPT-5.1: 47.6%",Real terminal execution; Multiple task types; Success verification,Terminal environment dependencies; Platform-specific commands; Execution safety concerns; Limited task diversity,"SWE-Bench, WebArena, BashEval, CommandLine-Tasks"
SWE-Bench Verified,A verified subset of SWE-Bench with human-validated problem statements and solutions for software engineering tasks from GitHub issues.,Princeton University,https://www.swebench.com/,https://github.com/princeton-nlp/SWE-bench,2024-01-01,Research; Development,"Software engineering, Bug fixing, Code understanding, Repository-level coding, Real-world debugging",Core Performance,Text; Code,Code,New dataset (released with eval),Human annotations,Small (< 1K samples),Verified subset with single attempt evaluation,Fixed data-driven (static test set),Automatic (Execution-based),"Model fixes real GitHub issues with single attempt, success rate computed based on test passage",Outputs,False,,"Human verification of problem statements and solutions, test case validation",unknown,"Gemini 3 Pro: 76.2%, Gemini 2.5 Pro: 59.6%, Claude Sonnet 4.5: 77.2%, GPT-5.1: 76.3%",Human verification; Real-world issues; Single attempt constraint; Test-based validation,Limited to verified subset; Repository-specific knowledge needed; Test coverage varies; Single attempt is strict,"SWE-Bench, SWE-Bench Lite, RepoEval, CodeR"
t2-bench,A benchmark for agentic tool use testing models' ability to use tools effectively to solve complex tasks.,"Sierra",https://arxiv.org/abs/2506.07982,https://github.com/sierra-research/tau2-bench,2024-01-01,Research; Development,"Tool use, Agentic behavior, Multi-step planning, API usage, Task decomposition",Core Performance,Text,Text; Code,New dataset (released with eval),Programmatically generated,Small (< 1K samples),Test tasks requiring tool use,Dynamic data-driven (adaptive/interactive),Automatic (Execution-based),"Model uses provided tools to complete tasks, success rate computed based on task completion",Outputs,False,,"Clear task success criteria with tool availability, execution-based validation",unknown,"Gemini 3 Pro: 85.4%, Gemini 2.5 Pro: 54.9%, Claude Sonnet 4.5: 84.7%, GPT-5.1: 80.2%",Multiple tool types; Task complexity variation; Success verification,Tool availability assumptions; Task definition clarity; Execution environment dependencies; Limited tool diversity,"ToolBench, API-Bank, RestBench, ToolLLM"
Vending-Bench 2,A benchmark for long-horizon agentic tasks measuring net worth in complex economic simulation environments.,"Andon Labs",https://arxiv.org/abs/2502.15840,https://andonlabs.com/evals/vending-bench,2024-01-01,Research; Development,"Long-horizon planning, Economic reasoning, Strategic decision making, Multi-step optimization",Core Performance,Text,Actions,New dataset (released with eval),Simulation-based,Small (< 1K samples),Simulation scenarios,Dynamic data-driven (adaptive/interactive),Automatic (Execution-based),"Agent makes decisions in economic simulation, net worth (mean) computed as success metric (higher is better)",Outputs,False,,"Simulation-based with clear economic metrics, reproducible outcomes",unknown,"Gemini 3 Pro: $5,478.16, Gemini 2.5 Pro: $573.64, Claude Sonnet 4.5: $3,838.74, GPT-5.1: $1,473.43 (higher is better)",Long-horizon evaluation; Economic metric validation; Multiple scenarios,Simulation realism; Economic model assumptions; Variance in outcomes; Limited scenario diversity,"WebShop, ALFWorld, BabyAI, MiniGrid"
SimpleQA Verified,A verified version of SimpleQA with human-validated short fact-seeking questions testing factual accuracy and calibration.,OpenAI,https://openai.com/index/introducing-simpleqa/,https://github.com/openai/simple-evals,2024-10-01,Research; Development,"Factual accuracy, Parametric knowledge, Calibration, Short-form QA, Correctness",Core Performance; Calibration; Safety,Text,Text,New dataset (released with eval),Human annotations,Small (< 1K samples),Test set with verified answers,Fixed data-driven (static test set),Automatic (Reference-based); Human: Experts,"Model answers short factual questions, accuracy and calibration computed with human verification",Outputs,False,,"Human verification of questions and answers, factual correctness validated",unknown,"Gemini 3 Pro: 72.1%, Gemini 2.5 Pro: 54.5%, Claude Sonnet 4.5: 29.3%, GPT-5.1: 34.9%",Human verification; Factual validation; Calibration measurement,Small dataset; Short-form only; Knowledge cutoff sensitivity; Verification cost,"SimpleQA, TruthfulQA, NaturalQuestions, TriviaQA"
MMMLU (Massive Multitask Multilingual Language Understanding),A multilingual extension of MMLU testing language understanding across 100+ languages and 57 subjects for global AI evaluation.,"UC Berkeley",https://arxiv.org/abs/2009.03300/,https://huggingface.co/datasets/openai/MMMLU,2024-01-01,Research; Development; Selection,"Multilingual understanding, Cross-lingual reasoning, Global knowledge, Language diversity",Core Performance; Fairness,Text,Text,One or Multiple existing datasets,Expert annotations,Large (100K - 1M samples),Test sets across 100+ languages,Fixed data-driven (static test set),Automatic (Reference-based),"Model answers questions in multiple languages across subjects, accuracy computed per language and overall",Outputs,False,,"Expert translations and validation across languages, cultural appropriateness verified",unknown,"Gemini 3 Pro: 91.8%, Gemini 2.5 Pro: 89.5%, Claude Sonnet 4.5: 89.1%, GPT-5.1: 91.0%",100+ languages; Cross-lingual evaluation; Translation quality validation,Translation quality varies; Cultural context challenges; Language resource imbalance; Some languages underrepresented,"MMLU, XGLUE, XTREME, XCOPA, Belebele"
Global PIQA,A globally diverse version of PIQA testing physical commonsense reasoning across 100 languages and cultures.,"UC San Diego",https://arxiv.org/abs/2510.24081,https://mrlbenchmarks.github.io/,2024-01-01,Research; Development,"Physical commonsense, Cross-cultural reasoning, Multilingual understanding, Global knowledge",Core Performance; Fairness,Text,Text,One or Multiple existing datasets,Expert annotations,Medium (1K - 100K samples),Test sets across 100 languages,Fixed data-driven (static test set),Automatic (Reference-based),"Model selects physically plausible solution in multiple languages, accuracy computed per language",Outputs,False,,"Cultural appropriateness validation, translation quality verification, physical plausibility across cultures",unknown,"Gemini 3 Pro: 93.4%, Gemini 2.5 Pro: 91.5%, Claude Sonnet 4.5: 90.1%, GPT-5.1: 90.9%",100 languages; Cultural diversity; Translation validation; Cross-lingual consistency,Translation challenges; Cultural assumptions in physical scenarios; Language resource imbalance; Some scenarios culturally specific,"PIQA, XCOPA, XCSQA, XWinograd, Belebele"