KAFKA-17587 Refactor test infrastructure

[mumrah]

This patch reorganizes our test infrastructure into three Gradle modules:

":test-common:test-common-internal-api" is now a minimal dependency which exposes interfaces and annotations only. It has one project dependency on server-common to expose commonly used data classes (MetadataVersion, Feature, etc). Since this pulls in server-common, this module is Java 17+. It cannot be used by ":clients" or other Java 11 modules.

":test-common:test-common-util" includes the auto-quarantined JUnit extension. The @Flaky annotation has been moved here. Since this module has no project dependencies, we can add it to the Java 11 list so that ":clients" and others can utilize the @Flaky annotation

":test-common:test-common-runtime" now includes all of the test infrastructure code (TestKitNodes, etc). This module carries heavy dependencies (core, etc) and so it should not normally be included as a compile-time dependency.

In addition to this reorganization, this patch leverages JUnit SPI service discovery so that modules can utilize the integration test framework without depending on ":core". This will allow us to start moving integration tests out of core and into the appropriate sub-module. This is done by adding ":test-common:test-common-runtime" as a testRuntimeOnly dependency rather than as a testImplementation dependency. A trivial example was added to QuorumControllerTest to illustrate this.

This patch does not expose ClusterInstance through the SPI since it has many direct dependencies on core. A stripped-down version of ClusterInstance that can be used by non-core modules remains as future work.

[chia7712]

@mumrah thanks for this refactor. I run the command ./gradlew cleanTest test-common:test-common-runtime:test and it results in some error below:

Gradle Test Run :test-common:test-common-runtime:test > Gradle Test Executor 109 > TestKitNodeTest > testSecurityProtocol(SecurityProtocol) > initializationError FAILED
    java.lang.IllegalStateException: Please annotate test methods with @ClusterTemplate, @ClusterTest, or @ClusterTests when using the ClusterTestExtensions provider
        at org.apache.kafka.common.test.junit.ClusterTestExtensions.provideTestTemplateInvocationContexts(ClusterTestExtensions.java:147)
        at java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:273)
        at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1708)
        at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
        at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
        at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
        at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
        at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
        at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596)
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)

[chia7712]

why only metadata module declares test-common:test-common-runtime as testRuntimeOnly?

[mumrah]

This was just done to illustrate the approach (and test it in the build). I'll revert this bit before merging

[chia7712]

we don't need it, right?

[mumrah]

Correct. I'll update these docs.

[chia7712]

Should we remove @ExtendWith(value = Array(classOf[ClusterTestExtensions])) from this class?

[mumrah]

Wow, good catch 😄 I left this one here intentionally to make sure it still worked if people explicitly add the annotation in the future

[mumrah]

Well this was a bad choice for a test since this one is disabled. I tried it locally on another test and verified keeping the annotation is harmless

[mumrah]

@chia7712 I think this commit should fix the initialization issue. Since SPI is being used, it loads the extension for all tests. We need some additional filtering to not apply the ClusterTest logic to regular tests.

[ijuma]

Thanks for the PR. A couple of questions:

1. Do we intend to have a common test module for clients and streams (aside from junit specific classes)?
2. If we were to provide a test module for external usage, what would we call it?

These questions will help validate the names we have chosen.

[mumrah]

@ijuma,

Do we intend to have a common test module for clients and streams (aside from junit specific classes)?

I think that would make sense. There's quite a bit of utility test code that could be kept at Java 11. In this case we might consider test-common-util instead of test-common-junit

If we were to provide a test module for external usage, what would we call it?

I was thinking testkit or test-kit (so kafka-testkit.jar or kafka-test-kit.jar). Are you thinking we go ahead and establish a public-friendly name for the test-common-api module?

[ijuma]

Makes sense to use the testkit name for the external dependency and going with util for a more general module name that can include shared junit stuff but also other useful things needed by all test modules.

The only open question for me is the test-common-api name. Even though it makes sense, we typically use api for modules that expose apis externally (group-coordinator-api, storage-api, tools-api, etc.). In this case, it's a purely internal module, so I wonder if we can find a different name to describe it.

[mumrah]

Hm, yea.. I see what you mean. Here are some ideas for alternatives to test-common-api

• test-common-private-api
• test-common-defs
• test-common-lib
• test-common-spi (sort of a misnomer)
• test-common

[ijuma]

Since you're calling one test-common-runtime, would test-common-compile be appropriate? Or perhaps test-common-internal-api (we already use internals in our package names). The latter could be used for any module that exposes internal apis, so perhaps it a reasonable pattern to establish.

[mumrah]

I like "internal-api" 👍