Dangers of Benchmarking

[BradKML]

Something that I want to test is using Spearman's correlation to see if benchmarks that claim to test for the same goals correlate with one another. Some of the older benchmarks have been known to be overfitted (MMLU + HumanEval + BBH, + math benchmarks like GSM8K, MATH, and AIME), so if we pick a benchmark with a similar age but less popularity, it could show patterns. Alternatively, maybe some skills are correlated with one another?

Some mediocre reference points https://archive.fo/YneaH