Backtesting Framework for Historical Validation

[ironcladgeek]

Issue #102: Backtesting Framework for Historical Validation

Context & Clarification

FalconSignals generates investment recommendations (buy/hold/sell), not algorithmic trading strategies. Backtesting here means validating recommendation quality through historical analysis, not simulating portfolio returns or order execution.

Key Distinction

What We ARE Building	What We Are NOT Building
✅ Recommendation quality validation	❌ Portfolio return simulation
✅ Signal accuracy tracking over time	❌ Order execution modeling
✅ Confidence calibration analysis	❌ Capital allocation strategies
✅ Mode comparison (LLM vs rule-based)	❌ Multi-asset portfolio optimization

Why Traditional Backtesting Libraries Don't Apply

Libraries like backtrader, zipline, bt, and vectorbt are designed for:

• Systematic trading strategies with deterministic entry/exit rules
• Automatic position sizing and rebalancing
• Order execution simulation (slippage, commissions)

FalconSignals is different:

• Generates human-readable recommendations requiring judgment
• Users decide when/how to act on signals
• Focus is on signal quality, not portfolio mechanics

What We Already Have

✅ Historical Analysis (analyze --date 2024-06-01)

• Generates recommendations using only historical data
• Prevents look-ahead bias
• Works in both LLM and rule-based modes

✅ Performance Tracking (track-performance)

• Fetches current prices for active recommendations
• Stores daily price snapshots
• Calculates price changes and benchmark comparison

✅ Performance Metrics (performance-report)

• Win rate, average return, median return
• Alpha vs benchmark (SPY)
• Sharpe ratio, max drawdown
• Confidence calibration error

✅ Database Infrastructure

• recommendations table stores all signals
• price_tracking table stores daily prices
• performance_summary table caches metrics

What's Missing

1. Backtesting Orchestration

Problem: Manual iteration through dates is tedious

# Currently required
uv run python -m src.main analyze --ticker AAPL --date 2024-01-01
uv run python -m src.main analyze --ticker AAPL --date 2024-02-01
# ...repeat 50+ times

Solution: Automated backtest command

# Proposed
uv run python -m src.main backtest \
  --start-date 2024-01-01 \
  --end-date 2024-12-31 \
  --frequency weekly \
  --ticker AAPL,MSFT \
  --mode rule_based

Tasks:

• Create BacktestEngine class (src/backtesting/engine.py)
• Implement date range iteration (daily/weekly/monthly)
• Reuse existing analyze logic for each date
• Handle failures gracefully (continue on error)
• Store results in existing database tables
• Create backtest CLI command (src/cli/commands/backtest.py)

2. Cost Estimation & Controls

Problem: LLM backtests can be expensive (€0.50-0.80 per ticker)

Solution: Pre-calculate costs and require confirmation

uv run python -m src.main backtest \
  --start-date 2024-01-01 \
  --end-date 2024-12-31 \
  --frequency weekly \
  --ticker AAPL,MSFT,GOOGL \
  --mode llm \
  --dry-run

# Output:
# 📊 Backtest Plan:
#   Date range: 2024-01-01 to 2024-12-31
#   Frequency: weekly (52 dates)
#   Tickers: 3
#   Total analyses: 156
#   Estimated cost (LLM): €78-125
#   Estimated duration: ~2 hours
#
# Continue? [y/N]

Tasks:

• Add cost estimation logic to BacktestEngine
• Support --dry-run flag (show plan without executing)
• Implement confirmation prompt for expensive backtests
• Add cost limits to configuration
• Track actual costs and compare to estimates

3. Backtest-Specific Reports

Problem: Current performance-report shows aggregated metrics, not backtest insights

Solution: Enhanced reports with:

• Performance by signal type (strong_buy vs buy vs hold)
• Confidence calibration tables
• Mode comparison (LLM vs rule-based)
• Time-series visualizations
• CSV export for custom analysis

Example output:

# Backtest Report: 2024-01-01 to 2024-12-31

## Performance by Signal Type
| Signal      | Count | Win Rate | Avg Return (30d) | Alpha vs SPY |
|-------------|-------|----------|------------------|--------------|
| strong_buy  | 12    | 75.0%    | +8.2%            | +3.1%        |
| buy         | 60    | 63.3%    | +4.5%            | +1.2%        |
| hold        | 54    | 51.9%    | +1.8%            | -0.5%        |

## Confidence Calibration
| Confidence Range | Count | Actual Win Rate | Calibration Error |
|------------------|-------|-----------------|-------------------|
| 80-100%          | 23    | 78.3%           | -3.7%             |
| 60-80%           | 89    | 61.8%           | -8.2%             |

## Mode Comparison
| Metric       | Rule-Based | LLM Mode | Winner        |
|--------------|------------|----------|---------------|
| Win Rate     | 58.2%      | 64.7%    | LLM (+6.5%)   |
| Avg Return   | +3.1%      | +4.8%    | LLM (+1.7%)   |
| Sharpe Ratio | 0.82       | 1.05     | LLM           |
| Cost         | €0         | €95      | Rule-Based    |

Tasks:

• Create BacktestReport class (src/backtesting/report.py)
• Add backtest-specific queries to PerformanceRepository
• Implement signal type comparison analysis
• Implement confidence calibration analysis
• Implement mode comparison (if both modes tested)
• Support multiple export formats (Markdown, JSON, CSV)
• Add backtest-report CLI command
• Optional: Add visualizations (charts)

4. Configuration Management

Problem: No centralized backtest configuration

Solution: Add to config/default.yaml:

backtesting:
  # Default parameters
  default_frequency: "weekly"  # daily, weekly, monthly
  default_period_days: 90  # Track outcomes for X days
  max_concurrent_analyses: 5  # Parallel execution limit

  # Cost controls
  llm_cost_limit_per_backtest: 100.0  # EUR
  require_confirmation: true

  # Reporting
  report_formats:
    - markdown
    - json
    - csv
  include_visualizations: true

Tasks:

• Add BacktestConfig schema to src/config/schemas.py
• Add backtesting section to config/default.yaml
• Validate configuration on load
• Document all options in config comments

5. Parallel Execution (Optional)

Problem: Sequential backtests are slow (50+ dates × multiple tickers)

Solution: Run analyses in parallel (careful with API rate limits)

Tasks:

• Implement parallel execution in BacktestEngine
• Add --max-workers CLI option
• Handle API rate limits gracefully
• Ensure thread-safe database writes
• Add progress bar for long-running backtests

6. Resumable Backtests (Optional)

Problem: Long LLM backtests may fail mid-run

Solution: Save progress and allow resuming

Tasks:

• Store backtest session metadata
• Track which dates have been analyzed
• Add --resume flag to continue failed backtests
• Implement idempotent analysis (skip existing data)

Implementation Priority

Phase 1 (Must Have)

1. Core BacktestEngine with date iteration
2. Basic backtest CLI command
3. Cost estimation for LLM mode
4. Reuse existing analyze/track/report logic

Phase 2 (Should Have)

1. Enhanced backtest-specific reports
2. Signal type comparison
3. Confidence calibration analysis
4. Mode comparison (LLM vs rule-based)
5. CSV export

Phase 3 (Nice to Have)

1. Visualizations (charts, plots)
2. Parallel execution
3. Resumable backtests
4. Interactive HTML reports

Success Criteria

• Run full-year backtest in < 30 minutes (rule-based, single ticker)
• Accurate cost estimation (within 10% of actual for LLM mode)
• Clear comparison of signal types and modes
• Confidence calibration error clearly visible
• Export capabilities for custom analysis

Example Workflows

Validate System Before Using

# 1. Run 3-month backtest (rule-based)
uv run python -m src.main backtest \
  --start-date 2024-09-01 \
  --end-date 2024-12-01 \
  --frequency weekly \
  --ticker AAPL,MSFT,NVDA \
  --mode rule_based

# 2. Review results
uv run python -m src.main backtest-report --session-id 1

# 3. If good, test LLM on subset
uv run python -m src.main backtest \
  --start-date 2024-11-01 \
  --end-date 2024-12-01 \
  --frequency weekly \
  --ticker NVDA \
  --mode llm

# 4. Compare modes
uv run python -m src.main backtest-report --session-id 1,2 --compare

Evaluate LLM ROI

# Run both modes on same data
uv run python -m src.main backtest \
  --start-date 2024-06-01 \
  --end-date 2024-12-01 \
  --frequency weekly \
  --ticker AAPL,MSFT,GOOGL,AMZN,META \
  --mode both  # Run both modes

# Compare cost vs performance
uv run python -m src.main backtest-report \
  --session-id 3 \
  --compare-modes \
  --format markdown

# Decision: If LLM only adds 2% return but costs €80,
# use rule-based for daily scans, LLM for final validation

Libraries to Consider (Optional)

Potentially Useful

• quantstats - For tearsheet-style reports and visualizations
• plotly - For interactive charts
• seaborn - For statistical visualizations

NOT Applicable

• ❌ backtrader - For algorithmic trading strategies
• ❌ zipline - For quantitative finance strategies
• ❌ bt - For portfolio optimization
• ❌ vectorbt - For vectorized backtesting

These are designed for systematic trading, not recommendation validation.

Open Questions

1. Default frequency: Weekly (good balance) or monthly (faster but less data)?
2. Parallel execution: Enable by default or opt-in?
3. Visualizations: Which charts are most valuable?
   • Cumulative returns by signal type?
   • Confidence calibration curves?
   • Mode comparison over time?
4. Incremental analysis: Should we skip dates that already have data?

Benefits

• ✅ Validate system effectiveness before relying on recommendations
• ✅ Identify which analysis mode works best (LLM vs rule-based)
• ✅ Optimize confidence calibration
• ✅ Data-driven system improvements
• ✅ Build confidence in recommendations

Related Documentation

• See docs/BACKTESTING_PROPOSAL.md for detailed design
• See docs/CLI_GUIDE.md for existing commands
• See src/data/repository.py for performance tracking logic