Schnapsen — ML Ablation Study

This repository contains:

• A Schnapsen game engine (api).
• Baseline bots (rand, rdeep).
• Four ML bot variants (ml_none, ml_deck, ml_trump, ml_both)
• Scripts to train ML models and to reproduce the ablation experiments used in the report.
• A script to perform the statistical test on the results obtained.

The main experiment compares win/loss performance of the ML variants against rand and rdeep, using seeds and seat-swapping for fairness.

---

1) Repository Structure Overview

Core Engine

• api/
  • State and supporting game logic (phases, legal moves, transitions)
  • engine.play(...) runs a full game by repeatedly calling get_move() on bots and applying State.next(...)
  • util.load_player(name) dynamically loads bots from bots/<name>/<name>.py (expects a Bot class)

Bots

• bots/rand/
  • Random baseline: chooses a random legal move
• bots/rdeep/
  • Stronger baseline: uses deeper evaluation (slower than rand)
• bots/ml/
  • Shared ML infrastructure:
    • featuresets.py converts a State into a feature vector
    • common.py implements MLBot:
      • enumerates legal moves
      • (phase 1) samples a plausible full-information state via make_assumption()
      • simulates next_state = state.next(move)
      • scores with model.predict_proba(...)
      • picks max (Player 1) or min (Player 2)
    • SMT.py quick smoke test runner for ML variants (for debugging).
• bots/ml_none/, bots/ml_deck/, bots/ml_trump/, bots/ml_both/
  • Each contains a minimal wrapper Bot class configuring:
    • whether to use deck knowledge features
    • whether to use trump one-hot features
  • Each directory must contain a model.pkl used at runtime.

Experiment & Training Scripts

• run_ablation.py
  • Runs the experiment used in the report:
    • all 4 ML variants vs both opponents
    • deterministic seeds
    • seat swapping enabled for fairness
  • Writes tables and figures to results/
• train-ml-bot.py
  • Generates a dataset by playing games and recording feature vectors per decision state
  • Trains an MLPClassifier and saves a model.pkl
• stat_tests.py
  • Runs statistical tests on results/ablation_results.csv
    • Direct comparison: exact binomial test of win-rate vs p0.
    • Indirect comparison: two-proportion z-test vs a baseline.
  • Exports results/stat_tests.csv

---

2) Environment Setup

Create & Activate a Virtual Environment (.venv)

From the project root:

python3 -m venv .venv
source .venv/bin/activate

Install Dependencies

pip install -U pip
pip install pandas matplotlib scikit-learn joblib scipy

3) Model files requirement (model.pkl)

Each ML variant loads its model from its own folder:

• bots/ml_none/model.pkl
• bots/ml_deck/model.pkl
• bots/ml_trump/model.pkl
• bots/ml_both/model.pkl

If any of these files is missing, run_ablation.py will fail with FileNotFoundError.

Note: You may encounter an InconsistentVersionWarning from scikit-learn if the model.pkl was saved using a different scikit-learnversion than your current environment.

The experiment can still run, but for strict reproducibility, either:

• install the same scikit-learn version used to train the model
• retrain the models in your current environment

4) Quick Sanity Check

Before running long experiments, confirm that the ML bots load correctly and can play:

PYTHONPATH=$(pwd) python -m bots.ml.SMT

The expected output of that is:

• four lines printed (one per ML bot) with winner=... and score=....

5) Reproducing the Ablation Experiment

Run the experiment script from the project root:

PYTHONPATH=$(pwd) python run_ablation.py

What The Script Does:

• Evaluates each ML bot variant against:

  • rand
  • rdeep

• Uses a deterministic seed range to generate initial states

• Uses swap_seats=True so each seed is played twice (A first, then A second) to reduce first-player bias

• Aggregates:

  • games
  • wins
  • losses
  • draws
  • win rate
  • points for/against

• Produces plots and exports tables

• Outputs (in results):

  • ablation_results.csv
  • ablation_results.xlsx
  • wins_losses_vs_rand.png
  • wins_losses_vs_rdeep.png
  • wins_losses_combined.png

Runtime note

• rdeep is substantially slower than rand
• Large seed ranges with seat swapping can take a long time (because total games doubles).

During our personal attempt, maybe due to the machine we used, the script took between 4 & 5 hours to finish running.

6) Statistical Testing

After running run_ablation.py, you will have the obtained results in results/ablation_results.csv

You can obtain the statistical testing used in the report with:

python stats_testing.py --csv results/ablation_results.csv

7) Training Models From Scratch

Use this only if you need to regenerate model.pkl files (or align scikit-learn versions).

7.1 Train a Model:

PYTHONPATH=$(pwd) python train-ml-bot.py -d dataset.pkl -m model.pkl -o

• -d sets the dataset file path
• -m sets the output model name under ./bots/ml/
• -o overwrites dataset if it already exists

7.2 Match Training Configuration to Each Variant:

Inside train-ml-bot.py, set:

USE_DECK = ...
USE_TRUMP = ...

Variant Mapping:

• ml_none → USE_DECK=False, USE_TRUMP=False
• ml_deck → USE_DECK=True, USE_TRUMP=False
• ml_trump → USE_DECK=False, USE_TRUMP=True
• ml_both → USE_DECK=True, USE_TRUMP=True

7.3 Place the Trained Model Where the Bot Expects It

train-ml-bot.py saves to ./bots/ml/<model_name>.

To use it in the experiment, copy it to the corresponding bot directory as model.pkl.

Example:

cp bots/ml/model.pkl bots/ml_trump/model.pkl

Repeat for each variant as needed.

8) Troubleshooting

FileNotFoundError: ... bots/ml_none/model.pkl

You are missing a model file. Ensure all four exist:

ls -la bots/ml_none/model.pkl bots/ml_deck/model.pkl bots/ml_trump/model.pkl bots/ml_both/model.pkl

ModuleNotFoundError for pandas / matplotlib / sklearn / joblib:

Install dependencies into the active environment:

pip install pandas matplotlib scikit-learn joblib

For a fast validation, temporarily reduce the seed range in run_ablation.py (e.g., 10–100 seeds) and run:

PYTHONPATH=$(pwd) python run_ablation.py

Then restore the full seed range used in the report to reproduce final results.

Thank You for Reading This File

Our group hopes things run flawlessly

Special thanks to Vrije University Amsterdam & all contributors of the original Schnapsen repository