[Testing] Use real backend services instead of mocks

[mdelapenya]

Hi 👋 from Testcontainers for Go

Giving we are collaborating with the otel-collector folks, we wonder if it would be possible to evaluate if replacing the existing test mocks for the backends (zipkin, jaeger, prometheus) with their containerised version would provide value to the project.

We do see it, as it will be testing against real services, not pre-defined mocks, but wanted to discuss it with you first here before opening any issue/PR.

Do you think it's something on your radar? Thanks in advance

[MrAlias]

What kind of runtime overhead is expected?

What does maintenance look like? Will things be updated by dependabot?

[mdelapenya]

Regarding the automated updates, indeed, dependabot will make that work for you. Please take a look at how the collector project receives the updates: https://github.com/open-telemetry/opentelemetry-collector-contrib/pulls?q=testcontainers+is%3Apr+author%3Aapp%2Fdependabot+

In terms of maintenance, you are already "maintaining" the mocks, as there are certain behaviours that are hardcoded into the mock, although they could not receive updates. The same for the containers, once they are working you keep them next to the code.

And I think that the possible runtime overhead of running a docker container for each backend, compared to the benefits of testing against real instances -not mocks- of them, it really pays off. In any case, I'm open to discuss and collaborate in the process of deciding what to do :)

[MrAlias]

Much of my life is spent waiting on our CI system to complete. I'm not motivated to increase that time given what we have is already working.

Based on what @pellared mentioned below, this would be similar to dockertest. That had significant overhead to CI times when added to https://github.com/signalfx/splunk-otel-go. If https://github.com/testcontainers/testcontainers-go has similar overhead I would not want to change to that setup.

[mdelapenya]

I understand your concerns and that such decisions always come with tradeoffs. Feel free to let us know if you are interested in using testcontainers-go in the future, we are always happy to help.

[pellared]

In https://github.com/signalfx/splunk-otel-go we are using https://github.com/ory/dockertest which is imported by more public repositories, has fewer issues, more stars, and it looks that it has more stable API.

Are there any technical reasons to use https://github.com/testcontainers/testcontainers-go over https://github.com/ory/dockertest?

[mdelapenya]

Hi @pellared thanks for your feedback. There are no technical reasons that come to my mind, and I'd not like to say anything against dockertest, mainly because it's a fabulous OSS project that we, as testcontainers-go maintainers, want to compete against as another OSS competitor, in really really friendly manner :)

Said that, I can only talk about of the good things of using testcontainers-go, which is very actively maintained, with a community that is growing day by day, not only from the active contributors standpoint but in the projects it's used, such as ClickHouse, Otel-collector, Telegraf, Elastic Agent, influxdb, to name a few.

On OSS metrics, I found this dashboard useful for both projects: https://cauldron.io/compare?projects=2296&projects=7062&tab=overview You can take a look and evaluate it for yourself and in the end we can consider both projects as active OSS projects.

Regarding the projects importing the library, I'm currently seeing testcontainers-go being imported by a big number of projects https://pkg.go.dev/github.com/testcontainers/testcontainers-go?tab=importedby, which makes me think that, even though the library is +3 years younger than ory, it's indeed very popular (in adoption and GH stars). At the same time, the project is backed by the authors of testcontainers-java, who come with all their knowledge on how to run a successful OSS project.

At the same time, there other Testcontainers implementations (Java, .NET) that are widely used across other open-telemetry projects, such as the opentelemetry-collector and the Java agent. That's why we considered the Go agent a good fit for using Testcontainers for Go.

And about the stability of the API, we are actively working on building a great OSS project, so I'm pretty sure the stability you mention will come sooner than later. On the other hand I'm interested in your feedback on what instabilities you see, in order to fix them asap.

With that in mind, we understand that the maintainers have their vision for where to go, therefore we are here to support and assist in the case we, as an OSS community, are needed.

[pellared]

I also want to say, that I do not have any concrete arguments why we should not use https://github.com/testcontainers/testcontainers-go

I can also add that https://github.com/testcontainers/testcontainers-dotnet is used in https://github.com/open-telemetry/opentelemetry-dotnet-instrumentation where I am a maintainer 😉

At first glance, I see more reasons to use https://github.com/testcontainers/testcontainers-go:

1. Used in Collector = more unified tooling.
2. TestContainers are already used in OpenTelemetry Java and .NET AutoInstrumentation
3. I assume you can help us with proper adoption (e.g. by creating initial PRs) 😉

Anyway, I think it is up to the @open-telemetry/go-maintainers to make a final decision.

[mdelapenya]

Thanks again for your feedback! Indeed, we do not want to fire&forget a request of improving the integration tests, but instead we want to build trust across OSS projects/communities using the integration tests as the "handshake". Therefore yes, we would help/assist/support during the process if the maintainers see the transition with good eyes.