-
Notifications
You must be signed in to change notification settings - Fork 274
Add Docker Compose scanner with multi-container support #4442
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Draft
rubys
wants to merge
30
commits into
master
Choose a base branch
from
docker-compose-scanner
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This commit adds comprehensive Docker Compose support to flyctl launch: - New Docker Compose scanner that detects and parses compose files - Supports services, ports, volumes, dependencies, and health checks - Maps to Fly.io multi-container machine configuration with Pilot init - Generates fly.machine.json for multi-container deployments - Intelligent database service detection with managed service recommendations - Handles build contexts, pre-built images, and environment variables - Comprehensive test coverage and documentation The scanner enables deploying existing Docker Compose applications to Fly.io with minimal changes by translating compose configurations to multi-container machine specs. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
Remove all trailing whitespace from documentation file to satisfy linting requirements. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
Implements automatic service discovery to facilitate inter-container communication: • Generates entrypoint script that configures /etc/hosts with service names • Maps all container names to 127.0.0.1 for localhost communication • Chains to original entrypoint/command to preserve container behavior • Allows containers to access each other using Docker Compose service names • Updates documentation to reflect new service discovery capabilities • Maintains backward compatibility with existing Docker Compose configurations This enables seamless migration of Docker Compose applications where services reference each other by name (e.g., database connections) without requiring connection string modifications. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
Significantly expanded the documentation to include: • Comprehensive service discovery explanation and examples • Detailed machine configuration with entrypoint script examples • Enhanced troubleshooting section with debugging commands • Advanced configuration topics for complex scenarios • Best practices for migration and deployment • Clear explanation of how service names are preserved • More detailed examples showing the files configuration Also cleaned up formatting and removed any trailing whitespace issues. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
Remove the explicit 'init' field from the machine configuration as it's not needed - Pilot init is used automatically for multi-container machines. This fixes the error: 'cannot unmarshal string into Go struct field MachineConfig.init of type fly.MachineInit' The correct structure only needs the 'containers' array at the top level. Also updated documentation to reflect that Pilot init is automatic. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
…tc/hosts handling This commit makes two important improvements: 1. Machine configuration now uses 'local_path' instead of 'raw_value' for files: - Writes fly-entrypoint.sh to disk during launch - References it via local_path in machine config - Much more readable than base64-encoded content - Easier to debug and inspect 2. Fixed /etc/hosts handling to preserve Fly.io networking: - Appends to existing /etc/hosts instead of overwriting - Preserves critical Fly.io entries (6PN, fly-local-6pn, etc.) - Uses grep to avoid duplicate entries - Uses tab spacing to match Fly.io format The entrypoint script now safely adds service discovery entries while maintaining all existing Fly.io networking functionality. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
When database services (PostgreSQL, MySQL, Redis) are detected but not included as containers, we now automatically remove any depends_on references to them. This prevents invalid dependencies in the generated machine configuration. Changes: - Two-pass processing: first identify excluded services, then create containers - extractDependencies now filters out dependencies on excluded services - Added comprehensive test for mixed dependency scenarios - Updated documentation to clarify this behavior This ensures the generated configuration is valid and doesn't reference non-existent containers. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
Run gofmt to properly format: - internal/command/launch/launch_frameworks.go - scanner/docker_compose.go - scanner/docker_compose_test.go All files now pass make lint checks. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
- Error if multiple services have build sections - Set container field in fly.toml for single build service - Omit build/image sections in fly.machine.json for build service - Add comprehensive tests for build section validation 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
- Automatically detect database-related environment variables - Move database credentials to Fly.io secrets for better security - Remove credentials from plain text container environment - Support common database URL patterns and individual credential fields - Add comprehensive test coverage for credential extraction - Update documentation to explain credential management 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
- Add Secrets field to Container struct to track which secrets each container needs - Populate container secrets list when extracting database credentials - Include secrets array in fly.machine.json for each container - Update tests to verify containers have proper access to secrets - Update documentation to explain container secret access - Skip DATABASE_URL/REDIS_URL when Fly managed databases detected This ensures containers only have access to the secrets they need, following the principle of least privilege. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
…cess - Parse Docker Compose secrets: section and convert to Fly.io secrets - Read file-based secrets and store their contents - Track which secrets each service references - Ensure containers have DATABASE_URL/REDIS_URL in secrets list when managed DBs detected - Add comprehensive tests for Docker Compose secrets handling - Update documentation to explain secrets support 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
…et_name - Change secrets from array of strings to array of objects - Each secret now has env_var and secret_name fields - Update documentation to show correct format - Fixes "cannot unmarshal string into Go struct field" error 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
…tring - Change "permissions" field to "mode" in file configuration - Use octal integer 0755 instead of string "0755" - Update documentation to show correct format (mode: 493) - Ensures fly-entrypoint.sh is executable by non-root users 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
- Remove secret_name field from machine config secrets - Use only env_var field when secret name matches environment variable name - Update documentation to show simplified format - Keeps configuration clean by omitting redundant information 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
- Re-parse machine config in launchInputForUpdate to get fresh container file content - This ensures files like fly-entrypoint.sh are updated when changed locally - Matches the behavior of fly launch where container files are properly copied
…upport This implements comprehensive Docker Compose scanner support for flyctl launch, enabling automatic conversion of docker-compose.yml files to Fly.io deployments. Key features: - Multi-container machine support using Fly.io Pilot init system - Automatic database service detection and exclusion - Docker Compose secrets conversion to Fly.io environment variables - Service discovery via localhost networking for container communication - Smart entrypoint handling that preserves image defaults when needed - Support for build and image-based services with single build constraint - Health check translation from Docker Compose to Fly.io format - Volume mapping and dependency management Technical improvements: - Fixed registry authentication for Fly.io container registry - Resolved CMD extraction complexity by using image defaults approach - Implemented proper secret processing for file-based Docker Compose secrets - Added comprehensive error handling and validation - Created detailed documentation and troubleshooting guide The scanner now successfully handles complex Docker Compose configurations including Rails applications with database dependencies and file-based secrets. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
- Skip creating DATABASE_URL, POSTGRES_USER, POSTGRES_PASSWORD secrets when managed Postgres is proposed - Skip creating REDIS_URL and Redis secrets when managed Redis is proposed - Fix extractDatabaseSecrets to filter based on proposed database types during scanning - Update test files to fix context parameter issues - Remove unused CMD extraction function and imports - Update documentation to reflect smart secret management This prevents duplicate secret creation when users choose managed databases, resolving the issue where manual secrets were created despite selecting Fly.io managed database services. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
- Track DATABASE_URL and REDIS_URL presence before extractDatabaseSecrets deletes them - Ensure managed database URLs are added to container's secrets list - Container can now access DATABASE_URL provided by Fly.io managed Postgres This fixes the issue where DATABASE_URL was missing from the container's secrets array in fly.machine.json when using managed databases. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
When all Docker Compose services use external images (no build sections), the generated fly.toml should not contain an empty [build] section. This change fixes the issue by modifying determineSourceInfo() to only create a Build struct when there are actual build settings present (Builder, DockerfilePath, or Buildpacks). When all fields are empty, build is left as nil, which properly omits the [build] section from the TOML output. Also includes additional improvements: - Added bind mount volume support for Docker Compose - Fixed restart policy format to use object with "policy" field - Updated documentation to reflect new capabilities - Added proper file handling for entrypoint script and bind mounts 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
Similar to how we skip creating a second machine when volumes are present, we should also skip when containers are defined in the machine configuration. Multi-container deployments run all containers on a single machine, so creating a second machine would duplicate all containers unnecessarily. Changes: - Added container detection in deployCreateMachinesForGroups() - Skip second machine creation when hasContainers is true - Updated warning message logic to account for containers - Updated comment to reflect containers alongside mounts This ensures multi-container Docker Compose deployments only create one machine per process group, maintaining proper resource isolation and avoiding container duplication. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
Updated all tests to match the current scanner behavior where: - All containers are returned during scanning (filtering happens in callback) - Entrypoint scripts are only set when containers have explicit commands - Database secret extraction logic refined to preserve passwords/keys while skipping connection URLs that Fly.io will provide Changes: - Updated container count expectations (3 containers instead of 1 for multi-service) - Fixed entrypoint expectations (nil for image-only containers) - Refined secret extraction logic to preserve DB_PASSWORD while skipping DATABASE_URL - Added proper container finding logic in all relevant tests - Fixed formatting issues All Docker Compose scanner tests now pass with the current implementation. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
The TestFlyLaunchWithBuildSecrets test was failing due to Dockerfile path duplication. The issue occurred because srcInfo.DockerfilePath contains an absolute path, but resolveDockerfilePath() in deploy_build.go expects a relative path to join with the config directory. When we set build.Dockerfile = srcInfo.DockerfilePath directly, the absolute path gets joined again with the working directory, resulting in paths like: /tmp/test/tmp/test/Dockerfile Fixed by making DockerfilePath relative to the working directory before setting it in the build configuration, so resolveDockerfilePath() can properly resolve it. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
…Compose - Add BuildContainers field to SourceInfo and Config structs to track all containers requiring the built image - Modify Docker Compose scanner to allow multiple services with identical build configurations - Add validation to ensure build definitions are truly identical (context, dockerfile, args, target) - Update deploy logic to apply built image to all containers in BuildContainers - Maintain backward compatibility: first service with build becomes primary container in fly.toml - Add comprehensive tests for multiple identical and different build scenarios 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
- Format struct field alignment in config.go - Remove trailing whitespace in docker_compose.go 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
…base filtering When database services are replaced with managed Fly services and only one container remains, clear the multi-container configuration to use standard single-container deployment. This avoids creating unnecessary machine_config files and container fields in fly.toml. - Modified composeCallback to detect single remaining container after filtering - Clear Containers, Container, and BuildContainers fields for single-container mode - Added comprehensive test to verify behavior with managed Postgres - Maintains backward compatibility and existing functionality 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
When all remaining services after database filtering have identical build configurations and environments, convert to processes-based deployment instead of multi-container. This provides a cleaner configuration for services that share the same built image but run different commands. Key features: - Detects services with identical builds and environments - Extracts commands from explicit compose commands or Dockerfile CMD - Parses both JSON array and shell format CMD instructions - Prevents conversion when services have volumes (preserves multi-container) - Clears multi-container config fields to enable processes mode - Comprehensive test coverage for both conversion and prevention scenarios 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
Modified Docker Compose scanner to maintain the original order of services as they appear in the YAML file, ensuring deterministic and predictable behavior for build service selection and container ordering. Key changes: - Added getServiceOrder() function to parse YAML and extract service names in order - Modified service iteration to use ordered service list instead of map iteration - Ensures first build service in YAML becomes primary Container field - BuildContainers list maintains original YAML order - Fixes test flakiness caused by non-deterministic map iteration 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds comprehensive Docker Compose support to flyctl, enabling deployment of existing Docker Compose applications to Fly.io with minimal changes by translating compose configurations to multi-container machine specs.
Key Features
🐳 Docker Compose Scanner
fly launch
🔧 Service Configuration Support
🗄️ Database Integration
🏗️ Build Support
🔐 Security & Secrets
🌐 Service Discovery
⚡ Intelligent Deployment Optimization
Implementation Highlights
Multiple Build Services Support
Docker Compose files with multiple services sharing identical build definitions are fully supported:
All three services receive the same built image, with
web
as the primary container in fly.toml.Processes-Based Deployment
When services have identical builds and environments (and no volumes), they're automatically converted to processes:
Results in a processes-based fly.toml with one build and multiple processes sharing the same image.
Deployment Mode Selection
Architecture
File Changes
Core Implementation
scanner/docker_compose.go
- Main Docker Compose parsing and configuration logicscanner/scanner.go
- Enhanced SourceInfo struct with container supportinternal/command/launch/launch_frameworks.go
- Multi-container launch integrationConfiguration & Deploy
internal/appconfig/config.go
- Extended Config struct for multi-container buildsinternal/command/deploy/machines_launchinput.go
- Deploy logic for multiple build containersTesting
scanner/docker_compose_test.go
- Comprehensive test coverage including deployment mode selectionTest Coverage
✅ Basic Docker Compose Detection
✅ Service Configuration Mapping
✅ Database Service Detection
✅ Build Context Handling
✅ Health Check Translation
✅ Volume and Bind Mount Processing
✅ Environment Variable Handling
✅ Multiple Identical Build Sections
✅ Different Build Section Validation
✅ Secrets and Credential Extraction
✅ Single-Container Mode Optimization
✅ Processes-Based Deployment
✅ Volume-Aware Mode Selection
Usage
Simply run
fly launch
in a directory containing a docker-compose.yml file:The scanner will:
Backward Compatibility
🤖 Generated with Claude Code