Skip to content

Latest commit

 

History

History
87 lines (60 loc) · 3.11 KB

01.overview.md

File metadata and controls

87 lines (60 loc) · 3.11 KB

Sotabench Documentation

## Overview

You have reached the docs for Sotabench! These docs will explain how the website and library work, how you can benchmark your own machine learning models, and how you can create your own benchmarks.

What is Sotabench?

Sotabench is a new resource for benchmarking machine learning models. Pre-trained models are a growing dependency for machine learning projects, but it is hard to verify their quality:

  • How do I know if a model reproduces the results of the original paper?
  • How does this model compare to other models on the same task? Which to choose?

Sotabench solves this problem through a build system that benchmarks pretrained models on public benchmarks. With minimal setup, people can submit their models for evaluation on popular benchmarks like ImageNet. Benchmarks enable comparison: using Sotabench the community can easily compare models to see whether they reproduced the results of the original paper and whether they are state-of-the-art for a task. This helps the community decide which models to use as a starting point for a project.

How Do I Benchmark My Model?

The full documentation for this use case can be accessed here: Benchmarking Your Model.

TLDR Summary: You add a benchmark.py file to your GitHub repo and connect your repo to Sotabench. Sotabench will automatically evaluate your models on benchmarks for free with GPUs, and feature your repository on the site's public benchmark pages.

How Do I Create A Benchmark?

The full documentation for this use case can be accessed here: Create a Benchmark.

TLDR Summary: You write a function that takes in a model and writes results to an evaluation.json, and publish it as a Python library. When other users import your library for their model's benchmark.py file, their model results will be published to your benchmark page.

How Do I Contribute to the Resource?

The full documentation for this user case can be accessed here: Getting Started.

TLDR Summary: Sotabench is a great place to publish paper implementations! You can use sotabench to discover what implementations the community is crying out for, publish them, and get awards and recognition for your work.

## Library Installation

If you just want to submit models, you do not need the sotabench library, but will probably want to install the libraries that power the benchmarks you are interesting - for example torchbench.

If you want to create custom benchmarks, then you can use the sotabench library as a starting point:

There are two ways to install sotabench:

Install Sotabench from PyPi

pip install sotabenchapi

Install Sotabench from GitHub source

git clone https://github.com/sotabench/sotabenchapi.git
cd sotabench
python setup.py install

Support

If you get stuck you can head to our Discourse forum where you ask questions on how to use the project. You can also find ideas for contributions, and work with others on exciting projects.