Fix docker-compose.yml for development on Linux #5832
Conversation
…load-customer-test-data from reaching MinIO - Update minio client commands to newer mc versions
Summary of ChangesHello @pgarciaq, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request addresses a critical issue preventing the Docker development environment from functioning correctly on Linux, specifically when nise attempts to interact with MinIO. The changes ensure that MinIO is properly accessible and configured within the Docker setup, resolving 404 errors encountered during test data loading and enabling a smooth development workflow. Highlights
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request fixes the Docker development environment on Linux by updating the S3 endpoint configuration and modernizing the MinIO client commands. The changes are correct and improve the robustness of the local setup. I've added one suggestion to refactor the bucket creation script in docker-compose.yml to make it more concise and maintainable by using a loop.
| /usr/bin/mc rb --force local/koku-bucket/ || true; | ||
| /usr/bin/mc mb --quiet local/koku-bucket/; | ||
| /usr/bin/mc anonymous set public local/koku-bucket; | ||
| /usr/bin/mc rb --force local/ocp-ingress/; | ||
| /usr/bin/mc rb --force local/ocp-ingress/ || true; | ||
| /usr/bin/mc mb --quiet local/ocp-ingress/; | ||
| /usr/bin/mc anonymous set public local/ocp-ingress; | ||
| /usr/bin/mc rb --force local/metastore/; | ||
| /usr/bin/mc rb --force local/metastore/ || true; | ||
| /usr/bin/mc mb --quiet local/metastore/; | ||
| /usr/bin/mc anonymous set public local/metastore; |
There was a problem hiding this comment.
The commands for creating buckets are repeated three times. You can use a for loop to make this script more concise and easier to maintain. This avoids repetition and makes it clearer which buckets are being created.
for bucket in koku-bucket ocp-ingress metastore; do
/usr/bin/mc rb --force "local/$bucket/" || true;
/usr/bin/mc mb --quiet "local/$bucket/";
/usr/bin/mc anonymous set public "local/$bucket";
done;
|
As a curiosity, it was Cursor the one that found the root cause and provided the fix. Here's the report it wrote. MinIO Bucket Creation Debugging GuideProblemWhen running or similar errors indicating that the bucket doesn't exist. Where Buckets Are CreatedThe buckets are created by the Location: Container: Buckets Created:
Root CauseThe
Error LogsWhen the container fails, you'll see errors like: SolutionOption 1: Quick Fix - Manually Create BucketsIf you need the buckets immediately, you can create them manually: # Set up MinIO alias
docker exec koku-minio mc alias set local http://localhost:9000 kokuminioaccess kokuminiosecret --api s3v4
# Create buckets
docker exec koku-minio mc mb local/koku-bucket
docker exec koku-minio mc mb local/ocp-ingress
docker exec koku-minio mc mb local/metastore
# Set public access permissions
docker exec koku-minio mc anonymous set public local/koku-bucket
docker exec koku-minio mc anonymous set public local/ocp-ingress
docker exec koku-minio mc anonymous set public local/metastoreOption 2: Permanent Fix - Update docker-compose.ymlThe create-parquet-bucket:
image: minio/mc:latest
depends_on:
- minio
environment:
- MINIO_ENDPOINT=${MINIO_ENDPOINT-http://koku-minio:9000}
- S3_ACCESS_KEY=${S3_ACCESS_KEY-kokuminioaccess}
- S3_SECRET=${S3_SECRET-kokuminiosecret}
entrypoint: >
/bin/sh -c "
/usr/bin/mc alias rm local || true;
/usr/bin/mc alias set local $MINIO_ENDPOINT $S3_ACCESS_KEY $S3_SECRET --api s3v4;
/usr/bin/mc rb --force local/koku-bucket/ || true;
/usr/bin/mc mb --quiet local/koku-bucket/;
/usr/bin/mc anonymous set public local/koku-bucket;
/usr/bin/mc rb --force local/ocp-ingress/ || true;
/usr/bin/mc mb --quiet local/ocp-ingress/;
/usr/bin/mc anonymous set public local/ocp-ingress;
/usr/bin/mc rb --force local/metastore/ || true;
/usr/bin/mc mb --quiet local/metastore/;
/usr/bin/mc anonymous set public local/metastore;
"Key Changes:
DebuggingCheck Container Statusdocker ps -a --filter "name=create-parquet" --format "table {{.Names}}\t{{.Status}}"View Container Logsdocker logs koku-create-parquet-bucket-1Check if Buckets Existdocker exec koku-minio mc alias set local http://localhost:9000 kokuminioaccess kokuminiosecret --api s3v4
docker exec koku-minio mc ls local/Use Debug ScriptA debug script is available at ./dev/scripts/debug_minio_buckets.shThis script will:
Environment VariablesThe following environment variables are used:
VerificationAfter creating buckets, verify they exist: docker exec koku-minio mc ls local/You should see: Related Files
Additional Notes
Troubleshooting Checklist
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #5832 +/- ##
=====================================
Coverage 94.3% 94.3%
=====================================
Files 345 345
Lines 29768 29768
Branches 3254 3254
=====================================
+ Hits 28073 28074 +1
+ Misses 1101 1100 -1
Partials 594 594 🚀 New features to boost your workflow:
|
1 participant
Fix development environment when using Docker on Linux. Probably applicable to any from-scratch development environment, even Mac.
Jira Ticket
No Jira ticket, just a very stubborn product manager on a weekend.
Description
When following the "Developing with Docker" instructions on Fedora Linux 43, "make load-customer-test-data" fails with 404's because nise cannot reach MinIO.
This PR includes 3 fixes:
Testing
Release Notes