A way to work with docker-compose ?

[aesteve]

Hello, thanks for your work, and sorry to bother your issues with a question :\

Is there a way to use a docker-compose.yml file in order to setup our integration tests stack ?
Similar to: https://www.testcontainers.org/modules/docker_compose/

Or is it limited to only images defined in the images folder (redis etc.).

If not possible atm, is it a feature you'd like to have in the future ?

Thank you.

[thomaseizinger]

No need to apologize :)

At the moment, there is no docker-compose support and there are currently also no plans to add it.
We would probably need to put in some thoughts on how this can be useful as part of running tests from within Rust. We haven't had the need for this yet. If you do, it would be very helpful if you could write it down!

Regarding the availability of images: As soon as #58 lands, you should be able to very easily create containers for images that are not yet in images.
However, the long-term goal is to provide a rich set of images that just work out-of-the box, so if you are missing some, feel free to send a PR :)

[aesteve]

Thanks for your answer!

One standard use-case of docker-compose in "tests within rust" would be Kafka I guess.
In fact, when opening the issue I was reading : https://github.com/spicavigo/kafka-rust/blob/master/tests/run-all-tests
Because a working Kafka cluster relies on two images : kafka & zookeeper.

So for instance, in order to test this library, one has to setup Kafka through docker-compose (obviously).
I guess this may be a common pattern among library developers.
And the most well-tested libraries, the better :)

[thomaseizinger]

Good point.

Reading up on how testcontainers-java does it, they actually run docker-compose using a docker container 😮

We could also start with a similar approach as we did for docker and simply call the native docker-compose binary :)

Unfortunately, we don't have the resources and the need to tackle this at the moment. Feel free to give it a go :)

[aesteve]

I can give this a try although I'm a complete beginner.
Could you point me at what you mean by "as we did for docker" ? Please ?
It's not totally clear for me.
Thank you

[thomaseizinger]

To talk to the docker host, we currently simply shell out to the docker binary here: https://github.com/testcontainers/testcontainers-rs/blob/master/cli_client/src/cli.rs

The whole thing is abstracted over through the Docker trait so that we can replace it with an HTTP client at some point and talk to the docker daemon directly.

For docker-compose, we can do something similar although we don't need the a trait here.
I think having a separate crate tc_docker_compose would be sufficient.
You'd probably have a struct/module that represents the docker-compose binary and provides an interface for all the commands that you want to execute.

[aesteve]

Perfect pointer! Thank you.
I think up, down would be sufficient.

If it's just calling the docker-compose command under the hood, it may be easy and I can give it a try.
Just curious about the "wait for container started" thing though. Maybe, it's not needed with the "health checks" provided by docker-compose ? Not sure, I need to try.

[thomaseizinger]

That one is a bit tricky.
What you could try to do is redirect all the traffic into individual files and read them again as things are starting up.

I am not too familiar with the healthchecks though. Will have to look it up :)

Looking at what testcontainers-java does might not hurt in terms of possible features ;)

[IgorLaborieWefox]

Hello,

I have a prototype that works with shell out and docker compose v2.x. The code is planned to be open-sourced later, but at the time it's still in a private crate.
It's async with Tokio, but if necessary I could create a sync version.

Are you interested in a PR?
Or maybe it's a better idea to create a separate crate?

[thomaseizinger]

Feel free to open a PR! I cant promise that it will be merged in though, that depends a lot on the design and how it fits into this crate :)

[DDtKey]

Just informing everyone interested, we are doing a major revision of testcontainers and adding docker-compose to our plans. Please note the changed API (see readme)

So anyone who wants to help with this is welcome 🙂

[feelingsonice]

Would love this as well. Not sure if I can try it out right now?

[DDtKey]

Not at the moment unfortunately as I've had limited bandwidth for the last couple of months

I'll continue working on this once have time

[listepo]

any news?

[DDtKey]

Unfortunately I haven't had enough bandwidth.

So we are happy to welcome anyone who wants to take on this.

P.S. if no one wants to take it on, I'll finish it when I have the chance (but can't commit any deadline).

[horacimacias]

I'd like to give it a try. I'm probably underestimating the effort but for sure it will be a good learning experience.
@DDtKey, do you have any partial work or ideas to continue from?
Any specific direction to make sure I don't go completely offtrack?

At least for me, I need to start various containers (e.g. couchdb, yugabyte) with some specific network configuration and using docker-compose from testcontainers seems the most straightforward.

[DDtKey]

Hi @horacimacias 👋
That's awesome!

Some preliminary work has been done and it even partially works, I will collect all the code, thoughts and design vision to provide more details a bit later (gimme couple of days)

Also, do not hesitate to contact me directly via Testcontainers Slack

[DDtKey]

@horacimacias sorry for the delay 🙏

The goal is to follow a similar feature set (impl can be totally different though) to Testcontainers for Java (to get familiar see their docs, containerized implementation, and general wrapper) or Go (however Go has the advantage of reusing Compose’s source code since it’s written in Go).

As a first step, I wanted to implement both containerized and local setups via a unified trait. The containerized version simply runs docker-compose CLI commands, so users don’t need the CLI installed.

I also think we can avoid the socat "ambassador" container — practically speaking, people already define exposed ports in their compose YAML (often reused between tests and dev to a certain extent), and ports can always be overridden with an additional Compose file. However we can extend socat part later.

The initial PoC has been sitting unfinished for the past N months in my local branch with some outdated code.
I’ve now updated and published it so we can focus on finalizing the missing parts: #774.

It already works in general, but we still need to:

• Provide an API to access containers by service name
• Ability to pass ready-conditions to compose
• Handle errors properly
• Verify that the interface is aligned and covers the main use cases

It doesn’t seem like there’s much left to finish IMO.

@horacimacias please, take a look and let me know if you're still interested and planning to complete the work.

[horacimacias]

thanks! yes, still interested. Let me review all this and see where I can contribute.