Shenannigans

This repository contains a cryptocurrency trading system with components for data fetching, backtesting, and live trading.

Components

Data Fetcher

The data fetcher component is responsible for fetching cryptocurrency price data from exchanges, storing it in a SQLite database, and publishing updates to Redis.

Configuration

The data fetcher is configured using the data.yaml file. Example configuration:

exchange: binance
symbols:
  - symbol: BTC/USDT
    frequency: 5  # minutes
    history_steps: 1000
  - symbol: ETH/USDT
    frequency: 5
    history_steps: 1000

Running Manually

To run the data fetcher manually:

python -m src.genetic_rl.data.data_fetcher

Running with Supervisor

Supervisor is used to ensure the data fetcher runs continuously and restarts automatically if it crashes.

Here are the steps to run DataFetcher with Supervisor:

1. Install Supervisor (if not already installed):

   pip install supervisor
   # or
   apt-get install supervisor  # on Debian/Ubuntu

2. Make sure your logs directory exists:

   mkdir -p logs

3. Create a main supervisord.conf file (if you don't have one already):

   echo_supervisord_conf > supervisord.conf

4. Include your data_fetcher config in the main supervisord.conf: Add this line at the end of supervisord.conf:

   [include]
   files = configs/supervisor/*.conf
   

5. Start Supervisor:

   supervisord -c supervisord.conf

6. Check the status of your data_fetcher process:

   supervisorctl status data_fetcher

7. Control the process as needed:

   supervisorctl start data_fetcher
   supervisorctl stop data_fetcher
   supervisorctl restart data_fetcher

8. View logs:

   tail -f logs/data_fetcher_out.log
   tail -f logs/data_fetcher_err.log

Managing with Supervisor

Basic Operations:

• Check status: supervisorctl -c supervisord.conf status
• Stop: supervisorctl -c supervisord.conf stop data_fetcher
• Start: supervisorctl -c supervisord.conf start data_fetcher
• Restart: supervisorctl -c supervisord.conf restart data_fetcher
• View logs: Check the log files in the logs directory

Supervisor Daemon Management:

When you need to fully restart supervisord (not just services), use these commands:

1. Proper shutdown then restart:

   supervisorctl shutdown  # Stops supervisord completely
   supervisord -c supervisord.conf  # Starts fresh

2. Reload configuration without restart:

   supervisorctl reread  # Re-read config files
   supervisorctl update  # Apply configuration changes

3. Restart specific services:

   supervisorctl restart data_fetcher  # Restart just one service

Note: Using supervisorctl stop data_fetcher only stops the service but leaves the supervisord daemon running. To avoid port conflicts when restarting supervisord, always use supervisorctl shutdown first, then start supervisord again.

Adding Symbols Dynamically

You can add new symbols to track without restarting the data_fetcher process:

1. Using the provided scripts:

   # Add a new symbol
   python src/genetic_rl/data/add_symbol.py SOL/USDT --frequency 5 --history-steps 500
   
   # Remove a symbol
   python src/genetic_rl/data/remove_symbol.py SOL/USDT

2. Using Redis CLI:

   redis-cli PUBLISH config_updates '{"add_symbol": {"symbol": "LTC/USDT", "frequency": 5, "history_steps": 500}}'
   redis-cli PUBLISH config_updates '{"remove_symbol": "LTC/USDT"}'

3. Required Fields for Adding Symbols:

   • symbol: The trading pair (e.g., "LTC/USDT")
   • frequency: How often to fetch data in minutes
   • history_steps: How many historical data points to fetch initially

How It Works

The data fetcher:

1. Reads configuration from data.yaml
2. Connects to the specified exchange
3. Fetches historical data for each configured symbol
4. Starts continuous price fetching tasks for each symbol
5. Stores data in SQLite database (data/prices.db)
6. Publishes price updates to Redis on the "price_updates" channel
7. Listens for configuration updates on the "config_updates" channel

When adding a new symbol dynamically:

1. The symbol is added to the in-memory configuration
2. The configuration is saved to the YAML file
3. Historical data is fetched for the new symbol
4. A notification is published that historical data is complete
5. A new task is started to fetch real-time data for the symbol

Handling Interruptions

If the data fetcher is stopped and restarted:

1. It checks for existing data in the database for each symbol
2. For symbols with existing data, it only fetches data since the last recorded timestamp
3. This ensures no data gaps while avoiding duplicate entries

Data Flow

1. src/genetic_rl/data/data_fetcher.py → SQLite database src/genetic_rl/data/prices.db
2. src/genetic_rl/backtesting/features/features.py FeatureEngine reads from prices.db and creates FeatureEngineResult according to src/genetic_rl/config/yaml/features.yaml

for backtesting:

3. src/genetic_rl/backtesting/orchestrator.py is the main entrypoint for the backtest.

for live trading:

4. Redis messaging for real-time coordination between components

Development Notes

• Models use genetic optimization instead of gradient descent
• No labeled data required - fitness determined by trading performance
• Real-time components use Redis for inter-process communication
• Supervisor manages long-running processes in production

Backtesting

Testing

Run the automated tests with:

uv run pytest

The suite uses a bundled SQLite fixture, exercising data loading, walk-forward execution, CLI wiring, and plotting without extra setup.

Live Trading

The live trading component executes trading strategies in real-time.

Utils

The utils directory contains utility scripts for various tasks:

Plot DB

The plot_db.py script allows you to visualize price data from the SQLite database:

python src/genetic_rl/utils/plot_db.py --symbol "BTC/USDT" --frequency 60 --days 30

Options:

• --db-path: Path to the SQLite database file (default: 'data/prices.db')
• --symbol: Symbol to plot (default: 'BTC/USDT')
• --frequency: Data frequency in minutes (default: 60)
• --days: Number of days to plot (default: 30)
• --output: Output file path (if not provided, will display plot)