Merge #13: Use MYSQL instead of SQLite by default

fa296f1 docs: [#12] complete Phase 2 documentation updates for MySQL migration (Jose Celano)
46ac92a feat: [#12] complete VM integration testing with MySQL x86-64-v2 CPU fix (Jose Celano)
69090dc feat: [#12] complete local functionality and persistence tests for mysql (Jose Celano)
4b5dc4e feat: [#12] add healthcheck to mysql service (Jose Celano)
343aa27 feat: [#12] migrate to mysql and update documentation (Jose Celano)
13f03d8 docs: [#12] add ADR-003 for MySQL vs MariaDB decision (Jose Celano)
b0b779b docs: [#12] add MySQL migration implementation plan (Jose Celano)

Pull request description:

  Relates to: torrust#66

  Use MYSQL instead of SQLite by default.

  ### Subtasks

  - [x] Create a plan to implement the issue
  - [x] Add MYSQL service to Docker Compose config, and make necessary changes to tracker config or env variables to make MySQL the default DB driver.

ACKs for top commit:
  josecelano:
    ACK fa296f1

Tree-SHA512: c4181d589e0ca83d8e8208bc866f3a2a752600582f73d17125e66062d2d7445a32b1c5920febc18280af9fd11c664fd6a33c5fbdd8c443ac2d11f471694237c7

Author: josecelano

.github/copilot-instructions.md

@@ -232,6 +232,13 @@ The project includes a comprehensive linting script that validates all file type
 - **No secrets**: Never commit SSH keys, passwords, or tokens
 - **Documentation**: Update docs for any infrastructure changes
 
+#### End-to-End Smoke Testing
+
+For verifying the functionality of the tracker from an end-user's perspective (e.g., simulating announce/scrape requests), refer to the **Smoke Testing Guide**. This guide explains how to use the official `torrust-tracker-client` tools to perform black-box testing against a running tracker instance without needing a full BitTorrent client.
+
+- **Guide**: [Smoke Testing Guide](../docs/guides/smoke-testing-guide.md)
+- **When to use**: After a deployment (`make apply`) or to validate that all services are working together correctly.
+
 ### Security Guidelines
 
 #### Secrets Management

README.md

@@ -96,7 +96,7 @@ make destroy                                 # Clean up
 - [Application Overview](application/README.md) - Application components and
   deployment
 - [Production Setup](application/docs/production-setup.md) - Production
-  deployment
+  deployment with MySQL
 - [Deployment Guide](application/docs/deployment.md) - Deployment procedures
 - [Backup Procedures](application/docs/backups.md) - Data backup and recovery
 - [Rollback Guide](application/docs/rollbacks.md) - Application rollbacks
@@ -112,6 +112,8 @@ make destroy                                 # Clean up
     main Makefile is at repository root
   - [ADR-002: Docker for All Services](docs/adr/002-docker-for-all-services.md) -
     Why we use Docker for all services including UDP tracker
+  - [ADR-003: Use MySQL Over MariaDB](docs/adr/003-use-mysql-over-mariadb.md) -
+    Why we chose MySQL instead of MariaDB for the database backend
 
 ## 🛠️ Development
 

application/.env.production

@@ -1,5 +1,20 @@
+# Torrust Tracker Demo - Production Environment Configuration
+# 
+# This configuration uses MySQL as the default database backend.
+# Make sure to change the default passwords before deployment!
+
 USER_ID=1000
 
+# Database Configuration (MySQL)
+MYSQL_ROOT_PASSWORD=secure_root_password_change_me
+MYSQL_DATABASE=torrust_tracker
+MYSQL_USER=torrust
+MYSQL_PASSWORD=secure_password_change_me
+
+# Tracker Database Configuration
+TORRUST_TRACKER_CONFIG_OVERRIDE_CORE__DATABASE__DRIVER=mysql
+TORRUST_TRACKER_CONFIG_OVERRIDE_CORE__DATABASE__PATH=mysql://torrust:${MYSQL_PASSWORD}@mysql:3306/torrust_tracker
+
 # Tracker
 TORRUST_TRACKER_CONFIG_TOML=
 TORRUST_TRACKER_CONFIG_OVERRIDE_HTTP_API__ACCESS_TOKENS__ADMIN='MyAccessToken'

application/README.md

@@ -59,9 +59,30 @@ application/
 - **Nginx**: Reverse proxy and SSL termination
 - **Prometheus**: Metrics collection and storage
 - **Grafana**: Metrics visualization and dashboards
-- **MySQL**: Database (migrating from SQLite)
+- **MySQL**: Default database backend for production
 - **Certbot**: SSL certificate management
 
+### Database Configuration
+
+The Torrust Tracker Demo uses **MySQL as the default database backend** for
+production deployments. This provides:
+
+- **Reliability**: Production-grade database with ACID compliance
+- **Scalability**: Support for high-throughput tracking operations
+- **Data Integrity**: Consistent data storage and retrieval
+- **Performance**: Optimized for concurrent tracker operations
+
+**Database Service**: The MySQL service is automatically configured with:
+
+- Database initialization scripts
+- Proper networking and security
+- Data persistence across container restarts
+- Health checks and monitoring
+
+For development and testing environments, you can optionally configure SQLite
+by modifying the tracker configuration, though MySQL is recommended for all
+production use cases.
+
 ## 🚀 Quick Start
 
 ### Application Deployment

application/compose.yaml

@@ -63,7 +63,7 @@ services:
     networks:
       - backend_network
     ports:
-      - "9090:9090" # This port should not be exposed to the internet
+      - "9090:9090"  # This port should not be exposed to the internet
     volumes:
       - ./storage/prometheus/etc:/etc/prometheus:Z
     logging:
@@ -73,15 +73,45 @@ services:
     depends_on:
       - tracker
 
+  mysql:
+    image: mysql:8.0
+    container_name: mysql
+    restart: unless-stopped
+    environment:
+      - MYSQL_ROOT_PASSWORD=${MYSQL_ROOT_PASSWORD}
+      - MYSQL_DATABASE=${MYSQL_DATABASE}
+      - MYSQL_USER=${MYSQL_USER}
+      - MYSQL_PASSWORD=${MYSQL_PASSWORD}
+    networks:
+      - backend_network
+    ports:
+      - "3306:3306"  # Only for debugging, remove in production
+    volumes:
+      - mysql_data:/var/lib/mysql
+      - ./storage/mysql/init:/docker-entrypoint-initdb.d:ro
+    command: >
+      --character-set-server=utf8mb4
+      --collation-server=utf8mb4_unicode_ci
+    healthcheck:
+      test: ["CMD", "mysqladmin", "ping", "-h", "localhost", "-p${MYSQL_PASSWORD}"]
+      interval: 10s
+      timeout: 5s
+      retries: 5
+    logging:
+      options:
+        max-size: "10m"
+        max-file: "10"
+
   tracker:
     image: torrust/tracker:develop
     container_name: tracker
     tty: true
     restart: unless-stopped
     environment:
       - USER_ID=${USER_ID}
-      - TORRUST_TRACKER_DATABASE=${TORRUST_TRACKER_DATABASE:-sqlite3}
-      - TORRUST_TRACKER_CONFIG_OVERRIDE_CORE__DATABASE__DRIVER=${TORRUST_TRACKER_CONFIG_OVERRIDE_CORE__DATABASE__DRIVER:-sqlite3}
+      - TORRUST_TRACKER_DATABASE=${TORRUST_TRACKER_DATABASE:-mysql}
+      - TORRUST_TRACKER_CONFIG_OVERRIDE_CORE__DATABASE__DRIVER=${TORRUST_TRACKER_CONFIG_OVERRIDE_CORE__DATABASE__DRIVER:-mysql}
+      - TORRUST_TRACKER_CONFIG_OVERRIDE_CORE__DATABASE__PATH=${TORRUST_TRACKER_CONFIG_OVERRIDE_CORE__DATABASE__PATH:-mysql://torrust:password@mysql:3306/torrust_tracker}
       - TORRUST_TRACKER_CONFIG_OVERRIDE_HTTP_API__ACCESS_TOKENS__ADMIN=${TORRUST_TRACKER_CONFIG_OVERRIDE_HTTP_API__ACCESS_TOKENS__ADMIN:-MyAccessToken}
     networks:
       - backend_network
@@ -98,6 +128,8 @@ services:
       options:
         max-size: "10m"
         max-file: "10"
+    depends_on:
+      - mysql
 
 networks:
   frontend_network: {}

application/docs/deployment.md

@@ -1,46 +1,93 @@
 # Deployment
 
-1. SSH into the server.
-2. Execute the deployment script: `./bin/deploy-torrust-tracker-demo.com.sh`.
-3. Execute the smoke tests:
-
-   ```console
-   # Clone Torrust Tracker
-   [email protected]:torrust/torrust-tracker.git
-   cd torrust-tracker
+This document outlines the deployment process for the Torrust Tracker demo application.
+
+## 1. Prerequisites
+
+- Ensure you have SSH access to the production server.
+- The server should be provisioned and configured according to the
+  [Production Setup Guide](./production-setup.md).
+
+## 2. Deployment Steps
+
+1. **SSH into the server**.
+
+2. **Navigate to the application directory**:
+
+   ```bash
+   cd /home/torrust/github/torrust/torrust-tracker-demo
    ```
 
-   Execute the following commands to run the tracker client and checker.
+3. **Pull the latest changes** from the repository:
+
+   ```bash
+   git pull
+   ```
+
+4. **Run the deployment script**:
+
+   ```bash
+   ./share/bin/deploy-torrust-tracker-demo.com.sh
+   ```
+
+   This script handles:
+
+   - Stopping services
+   - Rebuilding containers
+   - Starting services
+
+## 3. Verification and Smoke Testing
+
+After deployment, verify that all services are running correctly.
 
-   Simulate a torrent announce to the tracker using UDP:
+### Service Status
 
-   ```console
+Check the status of all Docker containers:
+
+```bash
+docker compose ps
+```
+
+### Application Logs
+
+Check the logs for the tracker container to ensure it started without errors:
+
+```bash
+./share/bin/tracker-filtered-logs.sh
+```
+
+### Smoke Tests
+
+Execute the following smoke tests from a machine with the `torrust-tracker` repository cloned.
+
+1. **UDP Announce**:
+
+   ```bash
    cargo run -p torrust-tracker-client --bin udp_tracker_client announce \
      udp://tracker.torrust-demo.com:6969/announce \
      9c38422213e30bff212b30c360d26f9a02136422 | jq
    ```
 
-   Simulate a torrent scrape to the tracker using HTTP:
+2. **HTTP Announce**:
 
-   ```console
+   ```bash
    cargo run -p torrust-tracker-client --bin http_tracker_client announce \
-     https://tracker.torrust-demo.com \
+     https://tracker.torrust-demo.com/announce \
      9c38422213e30bff212b30c360d26f9a02136422 | jq
    ```
 
-   Make a request to the health check endpoint:
+3. **Health Check Endpoint**:
 
-   ```console
+   ```bash
+   curl https://tracker.torrust-demo.com/api/health_check
+   ```
+
+4. **Run the comprehensive tracker checker**:
+
+   ```bash
    TORRUST_CHECKER_CONFIG='{
        "udp_trackers": ["udp://tracker.torrust-demo.com:6969/announce"],
-       "http_trackers": ["https://tracker.torrust-demo.com"],
+       "http_trackers": ["https://tracker.torrust-demo.com/announce"],
        "health_checks": ["https://tracker.torrust-demo.com/api/health_check"]
    }' cargo run -p torrust-tracker-client --bin tracker_checker
-
-   ```
-
-4. Check the logs of the tracker container to see if everything is working:
-
-   ```console
-   ./share/bin/tracker-filtered-logs.sh
    ```

application/docs/firewall-requirements.md

@@ -109,8 +109,6 @@ The tracker uses four main ports, each serving a specific purpose:
     - `/api/v1/metrics` - Prometheus metrics
   - Used by Prometheus for monitoring and Grafana dashboards
 
-### Application Service Mapping
-
 ### Torrust Tracker
 
 - **UDP ports 6868, 6969**: BitTorrent announce endpoints

application/docs/production-setup.md

@@ -1,13 +1,54 @@
-# Setup
+# Production Environment Setup
 
-Follow instructions on [Deploying Torrust To Production](https://torrust.com/blog/deploying-torrust-to-production).
+This guide details the steps required to configure the production environment
+for the Torrust Tracker demo.
 
-You need to also enable a [firewall](./firewall-requirements.md).
+## 1. Initial Setup
+
+For the initial server setup, follow the instructions on [Deploying Torrust To Production](https://torrust.com/blog/deploying-torrust-to-production).
+
+You also need to enable a [firewall](./firewall-requirements.md).
 
 The application is located in the directory: `/home/torrust/github/torrust/torrust-tracker-demo`.
 
-To run docker compose commands you need to cd to the app dir:
+To run Docker Compose commands, you need to be in the application directory:
 
 ```console
 cd /home/torrust/github/torrust/torrust-tracker-demo
 ```
+
+## 2. Database Configuration
+
+The production environment uses MySQL as the database backend.
+
+### Environment Variables
+
+Create a `.env` file by copying the production template:
+
+```bash
+cp .env.production .env
+```
+
+**Crucially, you must edit the `.env` file and set secure passwords** for the following variables:
+
+- `MYSQL_ROOT_PASSWORD`: The root password for the MySQL server.
+- `MYSQL_PASSWORD`: The password for the `torrust` user.
+- `TRACKER_ADMIN_TOKEN`: The admin token for the tracker API.
+
+### Database Initialization
+
+The MySQL service automatically initializes the database and creates the necessary tables on first startup.
+
+## 3. Running the Application
+
+Once the `.env` file is configured, you can start all services:
+
+```bash
+docker compose up -d
+```
+
+You can check the status of the services with:
+
+```bash
+docker compose ps
+```

application/share/container/default/config/prometheus.yml

@@ -1,3 +1,4 @@
+---
 global:
   scrape_interval: 15s # How often to scrape metrics
 

docs/README.md

@@ -18,6 +18,10 @@ This directory currently contains cross-cutting documentation:
 
 - [ADR-001: Makefile Location](adr/001-makefile-location.md) - Decision to keep
   Makefile at repository root level
+- [ADR-002: Docker for All Services](adr/002-docker-for-all-services.md) - Decision
+  to use Docker for all services including UDP tracker
+- [ADR-003: Use MySQL Over MariaDB](adr/003-use-mysql-over-mariadb.md) - Decision
+  to use MySQL instead of MariaDB for database backend
 
 ### 📅 [`plans/`](plans/) (Ongoing Plans and Roadmaps)
 
@@ -26,6 +30,13 @@ This directory currently contains cross-cutting documentation:
 - [Hetzner Migration Plan](plans/hetzner-migration-plan.md) - Comprehensive plan
   for migrating from Digital Ocean to Hetzner infrastructure
 
+### 🎯 [`issues/`](issues/) (Implementation Plans)
+
+**Issue Implementation Plans:**
+
+- [Phase 1: MySQL Migration](issues/12-use-mysql-instead-of-sqlite-by-default.md) -
+  Detailed implementation plan for database migration from SQLite to MySQL
+
 ### Future Categories
 
 The following directories can be created as needed: