feat: [#14] Phase 1 - 12-Factor App configuration management system

[josecelano]

Phase 1: Configuration Management System

This PR implements the first phase of the 12-Factor App refactoring for the Torrust Tracker Demo, focusing on configuration management and environment-based deployment.

🎯 Objectives Completed

• ✅ Environment-based configuration using .env files and templates
• ✅ Configuration templates with envsubst for dynamic generation
• ✅ Automated configuration scripts with validation and colored output
• ✅ Docker Compose integration with generated environment files
• ✅ Makefile automation for configuration and deployment workflows
• ✅ MySQL database migration from SQLite for production readiness

🏗️ Architecture Changes

Configuration Structure

infrastructure/config/
├── environments/
│   ├── local.env          # Local development configuration
│   └── production.env     # Production deployment configuration
└── templates/
    ├── tracker.toml.tpl         # Torrust Tracker configuration template
    ├── prometheus.yml.tpl       # Prometheus monitoring template
    └── docker-compose.env.tpl   # Docker Compose environment template

New Automation Scripts

• configure-env.sh - Processes templates and generates configuration files
• validate-config.sh - Validates generated configurations for syntax and completeness

Enhanced Makefile Targets

• configure-local / configure-production - Generate environment-specific configs
• validate-config / validate-config-production - Validate configurations
• deploy-local / deploy-production - Full deployment workflows

🔧 Key Features

1. Template-Based Configuration

All configuration files are now generated from templates using environment variables:

• No more hardcoded values in configuration files
• Environment-specific settings (logging levels, database URLs, etc.)
• Secrets management through environment variables

2. Environment Separation

• Local: Debug logging, direct access, SQLite/MySQL support
• Production: Info logging, reverse proxy mode, MySQL database, SSL ready

3. Validation & Quality Assurance

• TOML and YAML syntax validation
• Environment-specific configuration checks
• Template variable substitution verification
• Linting integration with project standards

4. Docker Compose Integration

• Generated .env files for Docker Compose
• Consistent variable naming across all services
• Environment-specific service configurations

🚀 Usage

Quick Start - Local Development

make configure-local    # Generate local configurations
make validate-config    # Validate configurations
make deploy-local       # Deploy with local settings

Production Deployment

# Set required secrets as environment variables
export TORRUST_PROD_DATABASE_URL="mysql://user:pass@host:3306/db"
export TORRUST_PROD_API_TOKEN="your-api-token"
# ... other production secrets

make configure-production  # Generate production configurations
make validate-config-production  # Validate configurations
make deploy-production    # Deploy to production

🛡️ Security Improvements

• No hardcoded secrets - All sensitive values use environment variables
• Environment isolation - Clear separation between local and production configs
• Validation safeguards - Prevent deployment with misconfigured settings

📋 Breaking Changes

• Configuration files are now generated and should not be edited directly
• .env files must be generated using the new scripts
• Database configuration now defaults to MySQL for both environments

🧪 Testing

All changes have been validated with:

• ✅ Linting - yamllint, shellcheck, markdownlint
• ✅ Configuration validation - Both local and production environments
• ✅ Template processing - All templates generate correctly
• ✅ Docker Compose integration - Services start with generated configs

📚 Documentation

Updated documentation includes:

• Phase 1 implementation guide with detailed migration steps
• Configuration management workflow documentation
• Makefile target reference and usage examples

🔄 Next Steps (Future PRs)

• Phase 2: Infrastructure as Code enhancements
• Phase 3: Service isolation and containerization improvements
• Phase 4: Monitoring and observability integrations

---

Ready for Review: This PR represents a complete Phase 1 implementation of the 12-Factor App methodology. All existing functionality is preserved while adding robust configuration management capabilities.

[da2ce7]

@josecelano I did open a Pull Request on this branch:

• Proposal: Migrate Automation from Makefile/sh to a Test-Driven Meson/Perl Toolchain #17
• docs(adr): [#17] propose migration to Meson/Perl automation toolchain #16

Good Work so far! :)

[josecelano]

@josecelano I did open a Pull Request on this branch:

• Proposal: Migrate Automation from Makefile/sh to a Test-Driven Meson/Perl Toolchain #17
• docs(adr): [#17] propose migration to Meson/Perl automation toolchain #16

Good Work so far! :)

Hi @da2ce7, thanks!. I will review and reply to your issue and PR properly. This is my "fast" reply.

All your proposals look excellent, and I agree that they would move the repo to the next level. There have been some decisions from my side that have not been shared publicly in the repo yet, but there are some reasons why the project is in its current state.

The first objective for this project was:

• Migrate the tracker demo to a new provider (cheaper and hopefully more performant)
• Split the demo into: tracker demo and index demo. This repository is only for the tracker.

Then I decided to take the opportunity to also:

• Create a more declarative installation:
  • Increasing the percentage of the automated task in the tracker install from 5% to 90%.
  • Improving the documentation about the tracker infrastructure requirements.

At the same time, I decided to:

• Learn more about how to use AI models to increase productivity.

I have not written a single line directly, but I have learned a lot about providing context and guidance to the model.

That was the initial state.

On the other hand, I don't have a good level of system administration skills, so I decided to let the model propose the initial architecture. I assumed the model would be better suited to building a fast solution using the most common technologies for this type of work. I didn't want to expend that much time on this repo since it's not one of the highest priorities in the Torrust roadmap.

There were several things I didn't like, and I wouldn't do it that way if I were to start doing it on my own. For example:

• I don't like makefiles, or at least the current code. I would have a hierarchy (subcommand), cleaner code and tests.
• I would do it in small increments, starting from the infrastructure requirements.
• I would try to add unit tests for all like: creating the base VM, installing dependencies, etc. Right now, there are only high-level tests. But the best one has to be run manually (docs/guides/integration-testing-guide.md). I mean, with the AI.

However, I decided to continue in this way because:

• It gives us a working product without expending too much time. And it's valuable for sys admins who want to install the tracker. It's a "live" documentation.
• I thought we could use it as a base implementation. I thought it would be easy to instruct the AI model to modify this example to use a different tool rather than starting with another solution that is not commonly used. I guess that would require much more context and guidance. But I'm learning in the process.

Later on, since I was moving quickly, I decided to refactor a little to at least apply the 12-factor methodology to the current solution. I'm in that process now. I think it will take me a couple more days to finish this refactor. In fact, I asked you if I should continue with this refactor, as I thought it might be totally out of the initial scope.

Regarding your proposals, if we want to continue improving the repo, I suggest reviewing and estimating them. One of the benefits of using something more popular is that it would be easier for a sysadmin with experience if we use well-known tools. However, given that we can now use AI, a more reliable and robust solution is a better alternative.

In your proposal, I missed two tools:

• https://www.nushell.sh/
• https://kdl.dev/

A friend of mine (sysadmin) is a big fan of them, and I tried to use them in another project (without AI) with no success. It was hard to write nutshell scripts.

[josecelano]

ACK 77c3fa3