EXAMINE

EXAMINE (Expedient Analysis of Management Information to Notice Errors) is a system that helps CCS identify potential errors in supplier-reported Management Information (MI), particularly under-reported or missing spend.

Installation

Python Virtual Environment

python3 -m venv venv

Activate:

• Linux/macOS: source venv/bin/activate
• Windows: .\venv\Scripts\activate

Install dependencies:

python -m pip install -r requirements.txt

Install the repo itself to find the util functions:

python -m pip install -e .

Environment Variables

Create a local .env file (not committed) based on env.example:

cp env.example .env

SQL Drivers

bash install_drivers.sh

Developer Tooling (Pre-commit, Ruff, pytest)

This project uses:

• pre-commit for running checks automatically before each commit.
• Ruff for fast linting.
• pytest for unit testing.

Install tooling

If you already installed dependencies from requirements.txt, install the remaining developer tools:

python -m pip install pre-commit ruff

Or install all at once:

python -m pip install -r requirements.txt pre-commit ruff

Set up pre-commit hooks

Install hooks locally:

pre-commit install

Run all hooks manually across the repository:

pre-commit run --all-files

Run Ruff and pytest manually

Run Ruff:

ruff check .

Run tests:

pytest -q

CI also runs Ruff and pytest on every push.

EXAMINE Data Analysis Pipeline

The EXAMINE data analysis pipeline is fully reproducible using DVC. The pipeline supports dummy and live modes via the data_mode parameter in params.yaml. Live mode requires database credentials, dummy mode runs without external access.

Running the Pipeline

To run the pipeline, set the data_mode in params.yaml to either dummy:

data_mode: dummy

Or live:

data_mode: live

Then run DVC:

python -m dvc repro

How it works

DVC templating passes the mode into scripts:

--mode ${data_mode}
--outdir data/${data_mode}
--indir data/${data_mode}

No Python files contain hard-coded paths.

Pipeline Stages

Stage	Script	Outputs
get_data	scripts/get_data.py	contracts.csv, mi.csv, reg_number_supplier_key.csv
combine	scripts/combine_data.py	combined.csv, unmatched.csv
summarise	scripts/summarise_data.py	summary_stats.csv, line_level.csv

DVC Troubleshooting

If the DVC pipeline is not running:

1. Ensure DVC is initialized: python -m dvc init
2. Check params.yaml has correct data_mode (dummy or live)
3. Verify all dependencies installed: pip install -r requirements.txt