NervCTF Stress Testers

Two complementary load generators for a NervCTF + CTFd deployment. Both create load against per-team instance challenges and exercise flag submission, including flag-sharing detection.

Script	Path it drives	Measures	Use when
monitor_stress.py	Remote-monitor directly (/api/v1/plugin/*)	Monitor latency, throughput, concurrency — logs + metrics	Heavy-duty, repeatable benchmark of the monitor in isolation
stress_test.py	Full stack through CTFd (login → CSRF → API)	End-to-end correctness under load	Validating the whole CTFd + plugin + monitor pipeline

The two answer different questions. stress_test.py exercises the realistic end-to-end path but is bounded by the cost of creating and authenticating CTFd sessions, so it cannot scale to many teams. monitor_stress.py targets the monitor's authenticated plugin endpoints with synthesised team/user IDs (the monitor trusts the team_id the plugin supplies), so it drives far more teams and isolates the monitor as the component under measurement.

---

Install

python3 -m venv ~/.venv
~/.venv/bin/pip install -r requirements.txt

(The command*.sh wrappers invoke ~/.venv/bin/python.)

---

monitor_stress.py — monitor benchmark (recommended for metrics)

Runs two phases:

1. Provision — for every (synthetic team × challenge) pair, POST /plugin/request at bounded concurrency, polling /plugin/info until the instance is running and reading the per-team flag back from the monitor (no SSH needed).
2. Submit — sustain N concurrent workers for a fixed duration, each submitting, by configurable probability, the team's own correct flag, another team's flag (exercising flag-sharing detection), or random noise.

Run

Edit command-monitor.sh to set MONITOR_TOKEN (the monitor's admin token — not the CTFd token), then:

MONITOR_TOKEN=<token> ./command-monitor.sh

Or directly:

~/.venv/bin/python monitor_stress.py \
    --monitor-url http://ctf-landing.lan.xstf.pt:33133 \
    --monitor-token <monitor token> \
    --challenge-source ctfd \
    --teams 100 \
    --provision-concurrency 16 \
    --submit-concurrency 64 --submit-duration 60 \
    --correct-chance 0.2 --sharing-chance 0.2 \
    --cleanup

The CTFd URL/token default to the testbed (http://ctf-landing.lan.xstf.pt) and are only consulted when --challenge-source ctfd (to read the instance-challenge list from CTFd instead of the monitor).

Run modes

Mode	Provisions containers?	Needs deployed challenges?	Measures
full (default)	Yes	Yes — nervctf deploy first	provision request + time-to-running + submission
--no-submit	Yes	Yes	provisioning only
--provision-no-wait	Requests only (no waiting)	Yes	provision request-acceptance throughput at scale
--submit-only	No	No	the flag-attempt hot path alone (command-submit-only.sh)

Two provisioning metrics (important)

Provisioning is measured as two separate numbers, because they scale differently:

• provision_request — how fast the monitor accepts a /plugin/request (inserts the stub, returns). A pure monitor metric, independent of container build time. Push this hard with --provision-no-wait (e.g. --teams 500) to find the monitor's request ceiling.
• time_to_running — how long until the instance is actually up. Bounded by the runner and the monitor's MAX_CONCURRENT_PROVISIONS semaphore (default ~4), so it saturates far earlier. Measure this by ramping --teams small (5, 10, 20, …) and watching when not_running_in_time becomes non-zero — that is the runner's capacity knee.

Asking for teams × challenges containers all at once (e.g. 1000 teams × 11 challenges = 11,000) will not finish: the monitor accepts the requests but the runner cannot build that many, so instances time out at provisioning. That is the saturation result, not a bug. Use --provision-no-wait for request throughput, and small --teams for time-to-running.

Request-throughput run (high scale, cleans up the background backlog afterwards):

./command-monitor.sh --teams 500 --provision-no-wait --no-submit

Prerequisite for provisioning: the monitor only knows a challenge after it has been registered via nervctf deploy (which calls /api/v1/instance/register). If /instance/list is empty, no instance challenges have been deployed to that monitor yet — deploy them first, or use --submit-only, which needs nothing deployed.

--submit-only benchmarks /api/v1/plugin/attempt — the monitor's hottest path, hit on every flag submission — without provisioning any containers. It submits garbage and cross-team flags against synthetic (team, challenge) pairs (challenge names from --challenges, or a synthetic default), measuring attempt throughput, latency, and the flag-ownership lookup. It writes flag_attempts rows to the monitor's SQLite store, so run it against a testbed.

Key options

Flag	Default	Meaning
--monitor-url	testbed :33133	Remote-monitor base URL
--monitor-token	(required)	Monitor admin token
--challenge-source	monitor	monitor (/instance/list) or ctfd (/api/v1/challenges)
--challenges	(all)	Comma-separated names; overrides --challenge-source
--teams	50	Number of synthetic teams
--team-base	900000	First synthetic team_id (kept high to avoid clashing with real teams)
--provision-concurrency	16	Max simultaneous provisioning requests
--submit-concurrency	64	Max simultaneous flag submissions
--submit-duration	60	Seconds to sustain the submission phase
--correct-chance / --sharing-chance	0.1 / 0.1	Per-submission probabilities
--no-provision / --no-submit	off	Isolate a single phase
--cleanup	off	stop_all the targeted challenges before and after

Artifacts (logs + metrics)

All artifacts are written into a per-run directory: logs/stress-test-<timestamp>/ (anchored to the stress-tester directory, regardless of where you launch from).

File	Contents
monitor_stress.log	Timestamped event log; per-second progress; errors
monitor_stress_provision_request.csv	One row per provision request: iso_ts, latency_ms, ok
monitor_stress_time_to_running.csv	One row per wait-to-running: iso_ts, latency_ms, ok
monitor_stress_attempt.csv	One row per submission: iso_ts, latency_ms, ok, sharing
monitor_stress_timeseries.csv	Per second: t_s, phase, prov_inflight, att_inflight, prov_done, att_done, att_per_s
results.json	Summary: per-phase latency p50/p95/p99/max, throughput, peak concurrency, error counts, reached-running / not-running-in-time, sharing detected/attempted

⚠️ Provisioning starts real containers on the provisioning node — --teams N times the challenge count. Start small (--teams 10) and scale up. Use --cleanup to tear everything down, and --no-submit to benchmark provisioning alone.

The results.json from a run feeds directly into the thesis Benchmark chapter (thesis/Chapters/v2/Benchmark.tex, Table~\ref{tab:stress_results}).

---

stress_test.py — full-stack CTFd path

Creates real CTFd users/teams, logs each in, provisions an instance of every visible instance challenge per user, then hammers /api/v1/challenges/attempt. Flags are read from the monitor API where available, falling back to SSH + docker exec into the per-team container. Requires --ssh-target for the fallback. See command.sh for a configured invocation.

---

Safety

• Synthetic team_base IDs are high (≥ 900000) so monitor rows never collide with real CTFd teams.
• Always pair large runs with --cleanup.
• Run against a testbed, never a live competition.