Skip to content

Add Docker Compose scanner with multi-container support #4442

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 30 commits into
base: master
Choose a base branch
from

Conversation

rubys
Copy link
Contributor

@rubys rubys commented Jun 25, 2025

Summary

This PR adds comprehensive Docker Compose support to flyctl, enabling deployment of existing Docker Compose applications to Fly.io with minimal changes by translating compose configurations to multi-container machine specs.

Key Features

🐳 Docker Compose Scanner

  • Detects and parses docker-compose.yml files automatically during fly launch
  • Maps Docker Compose services to Fly.io multi-container machine configuration
  • Supports compose versions 3.x with comprehensive service configuration

🔧 Service Configuration Support

  • Ports: Maps exposed ports and port mappings
  • Environment Variables: Handles both map and array formats
  • Volumes: Converts bind mounts to container file configurations
  • Health Checks: Translates Docker Compose health checks to Fly.io format
  • Dependencies: Preserves service dependencies with condition mapping
  • Restart Policies: Maps Docker restart policies to Fly.io equivalents

🗄️ Database Integration

  • Intelligent Detection: Automatically detects PostgreSQL, MySQL, and Redis services
  • Managed Service Recommendations: Proposes Fly managed databases during launch
  • Credential Extraction: Extracts database credentials as secrets when using external containers
  • Flexible Deployment: Choose between managed services or container-based databases
  • Single-Container Optimization: When database services are replaced with managed services and only one container remains, automatically switches to single-container deployment mode

🏗️ Build Support

  • Single Build Service: Supports one service with build section for traditional workflows
  • Multiple Identical Builds: Supports multiple services with identical build definitions
  • Build Validation: Ensures all build configurations are truly identical (context, dockerfile, args, target)
  • Image Distribution: All containers with identical builds receive the same built image
  • Mixed Deployments: Combines built and pre-built images in the same application

🔐 Security & Secrets

  • Docker Compose Secrets: Processes secrets section and file-based secrets
  • Database Credentials: Automatically extracts and secures database connection details
  • Environment Isolation: Separates sensitive configuration from regular environment variables

🌐 Service Discovery

  • Internal Networking: Containers can communicate using service names
  • Automatic /etc/hosts: Generates service discovery for multi-container deployments
  • Localhost Communication: Services accessible via localhost and respective ports

Intelligent Deployment Optimization

  • Processes Mode: Automatically converts services with identical builds and environments to processes-based deployment
  • Multi-Container Mode: Used when services have different configurations or volumes
  • Single-Container Mode: Selected when only one service remains after database filtering
  • Volume Awareness: Preserves multi-container mode when services use volumes

Implementation Highlights

Multiple Build Services Support

Docker Compose files with multiple services sharing identical build definitions are fully supported:

version: '3.8'
services:
  web:
    build: .
    ports: ["3000:3000"]
  
  worker:
    build: .  # Same build definition as web
    environment: [WORKER=true]
  
  api:
    build: .  # Same build definition as web  
    ports: ["4000:4000"]

All three services receive the same built image, with web as the primary container in fly.toml.

Processes-Based Deployment

When services have identical builds and environments (and no volumes), they're automatically converted to processes:

version: '3.8'
services:
  web:
    build: .
    environment: [NODE_ENV=production]
    # Uses CMD from Dockerfile
  
  worker:
    build: .
    environment: [NODE_ENV=production]
    command: ["npm", "run", "worker"]
  
  scheduler:
    build: .
    environment: [NODE_ENV=production]
    command: ["npm", "run", "scheduler"]

Results in a processes-based fly.toml with one build and multiple processes sharing the same image.

Deployment Mode Selection

  • Processes Mode: Services with identical builds, environments, and no volumes
  • Multi-Container Mode: Services with different configurations, images, or volumes
  • Single-Container Mode: Only one service remains after database filtering
  • Automatic Detection: No manual configuration needed, optimal mode chosen automatically

Architecture

  • Scanner Integration: Plugs into existing flyctl scanner framework
  • Machine Configuration: Generates proper fly.machine.json for multi-container deployments
  • Launch Callback: Post-processing logic filters services based on user database choices
  • Backward Compatibility: Maintains compatibility with existing single-container workflows

File Changes

Core Implementation

  • scanner/docker_compose.go - Main Docker Compose parsing and configuration logic
  • scanner/scanner.go - Enhanced SourceInfo struct with container support
  • internal/command/launch/launch_frameworks.go - Multi-container launch integration

Configuration & Deploy

  • internal/appconfig/config.go - Extended Config struct for multi-container builds
  • internal/command/deploy/machines_launchinput.go - Deploy logic for multiple build containers

Testing

  • scanner/docker_compose_test.go - Comprehensive test coverage including deployment mode selection

Test Coverage

Basic Docker Compose Detection
Service Configuration Mapping
Database Service Detection
Build Context Handling
Health Check Translation
Volume and Bind Mount Processing
Environment Variable Handling
Multiple Identical Build Sections
Different Build Section Validation
Secrets and Credential Extraction
Single-Container Mode Optimization
Processes-Based Deployment
Volume-Aware Mode Selection

Usage

Simply run fly launch in a directory containing a docker-compose.yml file:

fly launch

The scanner will:

  1. Detect the Docker Compose configuration
  2. Analyze services and dependencies
  3. Recommend managed databases for detected database services
  4. Generate appropriate fly.toml and fly.machine.json configurations
  5. Support deployment with multiple containers receiving the same built image
  6. Automatically optimize deployment mode based on service characteristics

Backward Compatibility

  • Existing single-container Docker applications continue to work unchanged
  • All previous flyctl functionality remains intact
  • Multi-container support is additive, not breaking

🤖 Generated with Claude Code

This commit adds comprehensive Docker Compose support to flyctl launch:

- New Docker Compose scanner that detects and parses compose files
- Supports services, ports, volumes, dependencies, and health checks
- Maps to Fly.io multi-container machine configuration with Pilot init
- Generates fly.machine.json for multi-container deployments
- Intelligent database service detection with managed service recommendations
- Handles build contexts, pre-built images, and environment variables
- Comprehensive test coverage and documentation

The scanner enables deploying existing Docker Compose applications
to Fly.io with minimal changes by translating compose configurations
to multi-container machine specs.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>
@rubys rubys marked this pull request as draft June 25, 2025 14:16
rubys and others added 16 commits June 25, 2025 17:23
Remove all trailing whitespace from documentation file to
satisfy linting requirements.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>
Implements automatic service discovery to facilitate inter-container communication:

• Generates entrypoint script that configures /etc/hosts with service names
• Maps all container names to 127.0.0.1 for localhost communication
• Chains to original entrypoint/command to preserve container behavior
• Allows containers to access each other using Docker Compose service names
• Updates documentation to reflect new service discovery capabilities
• Maintains backward compatibility with existing Docker Compose configurations

This enables seamless migration of Docker Compose applications where
services reference each other by name (e.g., database connections)
without requiring connection string modifications.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>
Significantly expanded the documentation to include:

• Comprehensive service discovery explanation and examples
• Detailed machine configuration with entrypoint script examples
• Enhanced troubleshooting section with debugging commands
• Advanced configuration topics for complex scenarios
• Best practices for migration and deployment
• Clear explanation of how service names are preserved
• More detailed examples showing the files configuration

Also cleaned up formatting and removed any trailing whitespace issues.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>
Remove the explicit 'init' field from the machine configuration as it's
not needed - Pilot init is used automatically for multi-container machines.
This fixes the error: 'cannot unmarshal string into Go struct field
MachineConfig.init of type fly.MachineInit'

The correct structure only needs the 'containers' array at the top level.

Also updated documentation to reflect that Pilot init is automatic.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>
…tc/hosts handling

This commit makes two important improvements:

1. Machine configuration now uses 'local_path' instead of 'raw_value' for files:
   - Writes fly-entrypoint.sh to disk during launch
   - References it via local_path in machine config
   - Much more readable than base64-encoded content
   - Easier to debug and inspect

2. Fixed /etc/hosts handling to preserve Fly.io networking:
   - Appends to existing /etc/hosts instead of overwriting
   - Preserves critical Fly.io entries (6PN, fly-local-6pn, etc.)
   - Uses grep to avoid duplicate entries
   - Uses tab spacing to match Fly.io format

The entrypoint script now safely adds service discovery entries while
maintaining all existing Fly.io networking functionality.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>
When database services (PostgreSQL, MySQL, Redis) are detected but not
included as containers, we now automatically remove any depends_on
references to them. This prevents invalid dependencies in the generated
machine configuration.

Changes:
- Two-pass processing: first identify excluded services, then create containers
- extractDependencies now filters out dependencies on excluded services
- Added comprehensive test for mixed dependency scenarios
- Updated documentation to clarify this behavior

This ensures the generated configuration is valid and doesn't reference
non-existent containers.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>
Run gofmt to properly format:
- internal/command/launch/launch_frameworks.go
- scanner/docker_compose.go
- scanner/docker_compose_test.go

All files now pass make lint checks.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>
- Error if multiple services have build sections
- Set container field in fly.toml for single build service
- Omit build/image sections in fly.machine.json for build service
- Add comprehensive tests for build section validation

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>
- Automatically detect database-related environment variables
- Move database credentials to Fly.io secrets for better security
- Remove credentials from plain text container environment
- Support common database URL patterns and individual credential fields
- Add comprehensive test coverage for credential extraction
- Update documentation to explain credential management

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>
- Add Secrets field to Container struct to track which secrets each container needs
- Populate container secrets list when extracting database credentials
- Include secrets array in fly.machine.json for each container
- Update tests to verify containers have proper access to secrets
- Update documentation to explain container secret access
- Skip DATABASE_URL/REDIS_URL when Fly managed databases detected

This ensures containers only have access to the secrets they need, following the principle of least privilege.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>
…cess

- Parse Docker Compose secrets: section and convert to Fly.io secrets
- Read file-based secrets and store their contents
- Track which secrets each service references
- Ensure containers have DATABASE_URL/REDIS_URL in secrets list when managed DBs detected
- Add comprehensive tests for Docker Compose secrets handling
- Update documentation to explain secrets support

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>
…et_name

- Change secrets from array of strings to array of objects
- Each secret now has env_var and secret_name fields
- Update documentation to show correct format
- Fixes "cannot unmarshal string into Go struct field" error

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>
…tring

- Change "permissions" field to "mode" in file configuration
- Use octal integer 0755 instead of string "0755"
- Update documentation to show correct format (mode: 493)
- Ensures fly-entrypoint.sh is executable by non-root users

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>
- Remove secret_name field from machine config secrets
- Use only env_var field when secret name matches environment variable name
- Update documentation to show simplified format
- Keeps configuration clean by omitting redundant information

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>
- Re-parse machine config in launchInputForUpdate to get fresh container file content
- This ensures files like fly-entrypoint.sh are updated when changed locally
- Matches the behavior of fly launch where container files are properly copied
…upport

This implements comprehensive Docker Compose scanner support for flyctl launch,
enabling automatic conversion of docker-compose.yml files to Fly.io deployments.

Key features:
- Multi-container machine support using Fly.io Pilot init system
- Automatic database service detection and exclusion
- Docker Compose secrets conversion to Fly.io environment variables
- Service discovery via localhost networking for container communication
- Smart entrypoint handling that preserves image defaults when needed
- Support for build and image-based services with single build constraint
- Health check translation from Docker Compose to Fly.io format
- Volume mapping and dependency management

Technical improvements:
- Fixed registry authentication for Fly.io container registry
- Resolved CMD extraction complexity by using image defaults approach
- Implemented proper secret processing for file-based Docker Compose secrets
- Added comprehensive error handling and validation
- Created detailed documentation and troubleshooting guide

The scanner now successfully handles complex Docker Compose configurations
including Rails applications with database dependencies and file-based secrets.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>
@rubys rubys changed the title Add Docker Compose scanner for multi-container machine support Complete Docker Compose scanner implementation with multi-container support Jun 28, 2025
rubys and others added 8 commits June 28, 2025 07:25
- Skip creating DATABASE_URL, POSTGRES_USER, POSTGRES_PASSWORD secrets when managed Postgres is proposed
- Skip creating REDIS_URL and Redis secrets when managed Redis is proposed
- Fix extractDatabaseSecrets to filter based on proposed database types during scanning
- Update test files to fix context parameter issues
- Remove unused CMD extraction function and imports
- Update documentation to reflect smart secret management

This prevents duplicate secret creation when users choose managed databases,
resolving the issue where manual secrets were created despite selecting
Fly.io managed database services.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>
- Track DATABASE_URL and REDIS_URL presence before extractDatabaseSecrets deletes them
- Ensure managed database URLs are added to container's secrets list
- Container can now access DATABASE_URL provided by Fly.io managed Postgres

This fixes the issue where DATABASE_URL was missing from the container's
secrets array in fly.machine.json when using managed databases.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>
When all Docker Compose services use external images (no build sections),
the generated fly.toml should not contain an empty [build] section.

This change fixes the issue by modifying determineSourceInfo() to only
create a Build struct when there are actual build settings present
(Builder, DockerfilePath, or Buildpacks). When all fields are empty,
build is left as nil, which properly omits the [build] section from
the TOML output.

Also includes additional improvements:
- Added bind mount volume support for Docker Compose
- Fixed restart policy format to use object with "policy" field
- Updated documentation to reflect new capabilities
- Added proper file handling for entrypoint script and bind mounts

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>
Similar to how we skip creating a second machine when volumes are present,
we should also skip when containers are defined in the machine configuration.
Multi-container deployments run all containers on a single machine, so
creating a second machine would duplicate all containers unnecessarily.

Changes:
- Added container detection in deployCreateMachinesForGroups()
- Skip second machine creation when hasContainers is true
- Updated warning message logic to account for containers
- Updated comment to reflect containers alongside mounts

This ensures multi-container Docker Compose deployments only create
one machine per process group, maintaining proper resource isolation
and avoiding container duplication.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>
Updated all tests to match the current scanner behavior where:
- All containers are returned during scanning (filtering happens in callback)
- Entrypoint scripts are only set when containers have explicit commands
- Database secret extraction logic refined to preserve passwords/keys
  while skipping connection URLs that Fly.io will provide

Changes:
- Updated container count expectations (3 containers instead of 1 for multi-service)
- Fixed entrypoint expectations (nil for image-only containers)
- Refined secret extraction logic to preserve DB_PASSWORD while skipping DATABASE_URL
- Added proper container finding logic in all relevant tests
- Fixed formatting issues

All Docker Compose scanner tests now pass with the current implementation.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>
The TestFlyLaunchWithBuildSecrets test was failing due to Dockerfile path
duplication. The issue occurred because srcInfo.DockerfilePath contains an
absolute path, but resolveDockerfilePath() in deploy_build.go expects a
relative path to join with the config directory.

When we set build.Dockerfile = srcInfo.DockerfilePath directly, the absolute
path gets joined again with the working directory, resulting in paths like:
/tmp/test/tmp/test/Dockerfile

Fixed by making DockerfilePath relative to the working directory before
setting it in the build configuration, so resolveDockerfilePath() can
properly resolve it.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>
@rubys rubys marked this pull request as ready for review June 29, 2025 04:23
…Compose

- Add BuildContainers field to SourceInfo and Config structs to track all containers requiring the built image
- Modify Docker Compose scanner to allow multiple services with identical build configurations
- Add validation to ensure build definitions are truly identical (context, dockerfile, args, target)
- Update deploy logic to apply built image to all containers in BuildContainers
- Maintain backward compatibility: first service with build becomes primary container in fly.toml
- Add comprehensive tests for multiple identical and different build scenarios

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>
@rubys rubys changed the title Complete Docker Compose scanner implementation with multi-container support Support multiple services with identical build definitions in Docker Compose Jun 29, 2025
@rubys rubys changed the title Support multiple services with identical build definitions in Docker Compose Add Docker Compose scanner with multi-container support Jun 29, 2025
rubys and others added 4 commits June 29, 2025 12:26
- Format struct field alignment in config.go
- Remove trailing whitespace in docker_compose.go

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>
…base filtering

When database services are replaced with managed Fly services and only one
container remains, clear the multi-container configuration to use standard
single-container deployment. This avoids creating unnecessary machine_config
files and container fields in fly.toml.

- Modified composeCallback to detect single remaining container after filtering
- Clear Containers, Container, and BuildContainers fields for single-container mode
- Added comprehensive test to verify behavior with managed Postgres
- Maintains backward compatibility and existing functionality

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>
When all remaining services after database filtering have identical build
configurations and environments, convert to processes-based deployment
instead of multi-container. This provides a cleaner configuration for
services that share the same built image but run different commands.

Key features:
- Detects services with identical builds and environments
- Extracts commands from explicit compose commands or Dockerfile CMD
- Parses both JSON array and shell format CMD instructions
- Prevents conversion when services have volumes (preserves multi-container)
- Clears multi-container config fields to enable processes mode
- Comprehensive test coverage for both conversion and prevention scenarios

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>
Modified Docker Compose scanner to maintain the original order of services
as they appear in the YAML file, ensuring deterministic and predictable
behavior for build service selection and container ordering.

Key changes:
- Added getServiceOrder() function to parse YAML and extract service names in order
- Modified service iteration to use ordered service list instead of map iteration
- Ensures first build service in YAML becomes primary Container field
- BuildContainers list maintains original YAML order
- Fixes test flakiness caused by non-deterministic map iteration

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>
@rubys rubys marked this pull request as draft June 30, 2025 16:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant