Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Devservices Keycloak breaks integration tests #45793

Closed
constantin-ungureanu-github opened this issue Jan 22, 2025 · 32 comments · Fixed by #46710
Closed

Devservices Keycloak breaks integration tests #45793

constantin-ungureanu-github opened this issue Jan 22, 2025 · 32 comments · Fixed by #46710
Labels
Milestone

Comments

@constantin-ungureanu-github

Describe the bug

With the quarkus-oidc in place, the restriction to the REST API are added.
The @QuarkusTest run okay. all of them.
The @QuarkusIntegrationTest do no run okay.

So adding quarkus-oidc to a quarkus project, adding few REST API restrictions (as in security-keycloak-authorization-quickstart demo), and @QuarkusTest runs fine, both with/withouth oidc restrictions, also healthchecks.

Same tests run as @quarkusIntegrationTest fail. (build native but is not related to the build as I understand)
The difference observed is that --hostname-port is sent when running the integration tests. Please remove this because was deprecated (25) and with keycloak 26+ was removed and now the keycloak is failing to start.

Workaround found for this was to:
quarkus.keycloak.devservices.service-name=keycloak
quarkus.keycloak.devservices.start-command=start --http-enabled=true --hostname=keycloak --hostname-strict=false --spi-user-profile-declarative-user-profile-config-file=/opt/keycloak/upconfig.json --features=hostname:v1

However, this works only with quarkus less than 3.18 (CR1), and it fails still in case of healthchecks (integration tests only). Sure, the keyloack can be downgraded but that still doesn't completely fix the issues.
For keycloak the v1 is replaced with v2 (--features=hostname:v2) . However, that does not work anymore, container fails to start.
The 3.17.7 with the workaround fails for the API that don't include oidc restrictions and also for the healthcheck IT tests.

This is what I get from the keycloak container (when running IT tests that fail):

Changes detected in configuration. Updating the server image.
Updating the configuration and installing your custom providers, if any. Please wait.
2025-01-22 12:56:41,951 INFO [io.qua.dep.QuarkusAugmentor] (main) Quarkus augmentation completed in 6640ms
Server configuration updated and persisted. Run the following command to review the configuration:

    kc.sh show-config

Next time you run the server, just run:

� kc.sh start --http-enabled=true --hostname-debug=true --hostname=localhost --spi-user-profile-declarative-user-profile-config-file=/opt/keycloak/upconfig.json --hostname-port=53483 --optimized

Unknown option: '--hostname-port'
Possible solutions: --hostname, --hostname-admin, --hostname-backchannel-dynamic, --hostname-strict, --hostname-debug
Try 'kc.sh start --help' for more information on the available options.

Expected behavior

All test run nicely.

Actual behavior

Integration tests fails

  • in 3.18 CR1, no integration test runs and because you updated the Keycloak to verison 26.0.0.7 there's no workaround but to downgrade this version
  • in version less than 3.18 (ex. 3.17.7) integration tests are failing as well, but for some there are ways to mitigate, see in the description.

I see there is a difference that the --hostname-port option has been removed

How to Reproduce?

Run integration tests involving devservices keycloak and this pops up immediately.

Also test this when endpoints are healtchecks and there are no restrictions and when the management port is used (with healthchecks on that port).

Output of uname -a or ver

No response

Output of java -version

java 21

Quarkus version or git rev

3.18 CR1 and also lower (3.17)

Build tool (ie. output of mvnw --version or gradlew --version)

maven 3.9.9

Additional information

I see for the version 3.18 there were numerous improvements related to OIDC, so would be nice to have the fix for quarkus-oidc running IT tests in 3.18. If needed, more details can be provided, but if you just run IT tests with OIDC this should pop-up.

Copy link

quarkus-bot bot commented Jan 22, 2025

/cc @geoand (devservices), @pedroigor (keycloak), @sberyozkin (keycloak), @stuartwdouglas (devservices)

@sberyozkin
Copy link
Member

@constantin-ungureanu-github Can you please create a basic reproducer ? There are a few integration tests running in Quarkus with the Keycloak devservices

@constantin-ungureanu-github
Copy link
Author

https://github.com/constantin-ungureanu-github/quarkus-oidc-example

Let me know if you need anything more.

@sberyozkin
Copy link
Member

sberyozkin commented Jan 22, 2025

Thanks @constantin-ungureanu-github, that is helpful, it is a shared container case. #42509 related to it but as it happens, #42509 is currently blocked, hopefully not for too long.

@constantin-ungureanu-github
Copy link
Author

constantin-ungureanu-github commented Jan 25, 2025

I wonder if the shared container is the root cause. It fails regardless if it is shared or not (quarkus.keycloak.devservices.shared=true/false).

The cause seems to be that when running IT tests, it sets the '--hostname-port' that with Keycloak version 26+ is no longer available (has been removed completely), and that Keycloak container fails to start.

However, a more serious issue, that it seems it is a different problem, is that the management port and quarkus-oidc don't work together. This is not regarding only the tests, but the resulting container fails to start in a Kubernetes environment where the health probes are used. Also tested to directly access the health checks when the management port is used, they just don't work anymore. The sample code provided is built in this way and can be verified.

Please have these issues fixed. Let me know if you need more details.

@constantin-ungureanu-github
Copy link
Author

I made some updates to the repository, to add running capabilities. Just run docker compose up or podman compose up and the container should pop-up, alongside a preconfigured Keycloak.

I found the root cause for the management port not working, that is only when "quarkus.http.non-application-root-path=/". The default is /q and I wanted to remove that altogether. Commenting out this change and I have back the metrics and the health checks.
One notice is that if I put the "quarkus.http.non-application-root-path=/", the image size is much smaller than when I take this out.
So the workaround to bypass this issue is to just not touch the quarkus.http.non-application-root-path altogether.

Now the "quarkus.http.non-application-root-path=/" works fine without using quarkus-oidc, so the issue resides in the combination of using all those in this configuration.

Please fix all these issues.

@jedla97
Copy link
Contributor

jedla97 commented Jan 25, 2025

Hi @constantin-ungureanu-github I look at your reproducer

For Keycloak issue I come up with some solution. Will discribe them later in the text.

For liveness check. Why test are not passing is because the quarkus test framework not exposing the management port. As workaround you can use -Dquarkus.test.container.additional-exposed-ports."9000"=9000 as addition to your mvn command. I'll try to look at this weekend or next week and raise PR for exposing the management port when running docker integration tests.

When running your docker compose I'm able to see http://localhost:9000/q/health/live.
If you want it to be pointed to http://localhost:9000/health/live you need to set the quarkus.management.root-path=/.
From quarkus.http.non-application-root-path description

If the management interface is enabled, the root path for the endpoints exposed on the management interface is configured using the quarkus.management.root-path property instead of this property.



Now for Keycloak. We need to definitely remove " --hostname-port=" + fixedExposedPort.getAsInt() from Keycloak start command. Looking at migration guide I don't see any example how the --hostname-port be migrated

There are two option and I'm not sure which is correct so here is needed input of @sberyozkin as he have much more knowledge about how thing work with Quarkus. Also I didn't encounter the blocking issue as mentioned in #42509 as blocking issue seems only affecting windows not linux.

1st option is to replace " --hostname-port=" + fixedExposedPort.getAsInt() with " --hostname-backchannel-dynamic true" and change KEYCLOAK_QUARKUS_HOSTNAME from "localhost" to "http://localhost"

The keycloak command would be start --http-enabled=true --spi-user-profile-declarative-user-profile-config-file=/opt/keycloak/upconfig.json --hostname=http://localhost --hostname-backchannel-dynamic true



2nd option it replace KEYCLOAK_QUARKUS_HOSTNAME value from "localhost" to "http://" + hostName + ":" + KEYCLOAK_PORT this will expose the generated Keycloak hostname (http://keycloak-vu5vi:8080)

Maybe there is some other option. I come with these option when playing with it. Here is a command which is executed when the integration test are executed

Executing "docker run --name quarkus-integration-test-KXPdv -i --rm -p 8081:8081 -p 8444:8444 -p 9000:9000
--net=e227ae9ccbe1166628c9d9f786458e8c654d568492ea5721027b18caaed17c31 
--env QUARKUS_LOG_CATEGORY__IO_QUARKUS__LEVEL=INFO 
--env QUARKUS_HTTP_PORT=8081 
--env QUARKUS_HTTP_SSL_PORT=8444 
--env TEST_URL=http://localhost:8081 
--env QUARKUS_PROFILE=prod 
--env QUARKUS_OIDC_CLIENT_ID=quarkus-app 
--env CLIENT_QUARKUS_OIDC_AUTH_SERVER_URL=http://localhost:34667/realms/quarkus 
--env KEYCLOAK_URL=http://keycloak-vu5vi:8080 
--env QUARKUS_OIDC_AUTH_SERVER_URL=http://keycloak-vu5vi:8080/realms/quarkus 
--env KEYCLOAK_REALMS=quarkus 
--env QUARKUS_CONTAINER_IMAGE_BUILD=true 
--env QUARKUS_TEST_CONTAINER_ADDITIONAL_EXPOSED_PORTS_9000=9000 
--env QUARKUS_NATIVE_CONTAINER_BUILD=true 
--env QUARKUS_DEVSERVICES_TIMEOUT=360s 
--env OIDC_USERS=alice=alice,bob=bob 
--env QUARKUS_OIDC_APPLICATION_TYPE=service 
--env QUARKUS_OIDC_CREDENTIALS_SECRET=secret quarkus-oidc-example:1.0.0-SNAPSHOT"

I checked with 3.8 which didn't have hostname v2 by default and the http://keycloak-vu5vi:8080 is not available from host so probably proposed 2nd option is wrong as that url should not be accesible from host machine.

We cant just remove the --hostname-port and add it to the hostname like this addEnv(KEYCLOAK_QUARKUS_HOSTNAME, "http://localhost:" + fixedExposedPort.getAsInt()); it will throw this error:

2025-01-25 17:04:19,119 WARNING [io.qua.ope.run.exp.otl.sen.VertxGrpcSender] (vert.x-eventloop-thread-3) Failed to export TraceRequestMarshalers. The request could not be executed. Full error message: Connection refused: localhost/127.0.0.1:4317
2025-01-25 17:04:19,123 ERROR [io.qua.run.Application] (main) Failed to start application: java.lang.RuntimeException: Failed to start quarkus
	at io.quarkus.runner.ApplicationImpl.doStart(Unknown Source)
	at io.quarkus.runtime.Application.start(Application.java:101)
	at io.quarkus.runtime.ApplicationLifecycleManager.run(ApplicationLifecycleManager.java:121)
	at io.quarkus.runtime.Quarkus.run(Quarkus.java:77)
	at io.quarkus.runtime.Quarkus.run(Quarkus.java:48)
	at io.quarkus.runtime.Quarkus.run(Quarkus.java:137)
	at io.quarkus.runner.GeneratedMain.main(Unknown Source)
Caused by: io.smallrye.mutiny.CompositeException: Multiple exceptions caught:
	[Exception 0] io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:35271
	[Exception 1] io.quarkus.oidc.OIDCException
	at io.smallrye.mutiny.operators.uni.UniOnFailureFlatMap$UniOnFailureFlatMapProcessor.performInnerSubscription(UniOnFailureFlatMap.java:94)
	at io.smallrye.mutiny.operators.uni.UniOnFailureFlatMap$UniOnFailureFlatMapProcessor.dispatch(UniOnFailureFlatMap.java:83)
	at io.smallrye.mutiny.operators.uni.UniOnFailureFlatMap$UniOnFailureFlatMapProcessor.onFailure(UniOnFailureFlatMap.java:60)
	at io.smallrye.mutiny.operators.uni.UniOperatorProcessor.onFailure(UniOperatorProcessor.java:55)
	at io.smallrye.mutiny.operators.uni.UniOperatorProcessor.onFailure(UniOperatorProcessor.java:55)
	at io.smallrye.mutiny.operators.uni.UniOperatorProcessor.onFailure(UniOperatorProcessor.java:55)
	at io.smallrye.mutiny.operators.uni.UniOnFailureFlatMap$UniOnFailureFlatMapProcessor.dispatch(UniOnFailureFlatMap.java:85)
	at io.smallrye.mutiny.operators.uni.UniOnFailureFlatMap$UniOnFailureFlatMapProcessor.onFailure(UniOnFailureFlatMap.java:60)
	at io.smallrye.mutiny.operators.uni.UniOperatorProcessor.onFailure(UniOperatorProcessor.java:55)
	at io.smallrye.mutiny.vertx.AsyncResultUni.lambda$subscribe$1(AsyncResultUni.java:37)
	at io.smallrye.mutiny.vertx.DelegatingHandler.handle(DelegatingHandler.java:25)
	at io.vertx.ext.web.client.impl.HttpContext.handleFailure(HttpContext.java:398)
	at io.vertx.ext.web.client.impl.HttpContext.execute(HttpContext.java:387)
	at io.vertx.ext.web.client.impl.HttpContext.next(HttpContext.java:362)
	at io.vertx.ext.web.client.impl.HttpContext.fire(HttpContext.java:329)
	at io.vertx.ext.web.client.impl.HttpContext.fail(HttpContext.java:310)
	at io.vertx.ext.web.client.impl.HttpContext.lambda$handleCreateRequest$6(HttpContext.java:498)
	Suppressed: io.quarkus.oidc.OIDCException
		at io.quarkus.oidc.runtime.TenantContextFactory$2.apply(TenantContextFactory.java:135)
		at io.quarkus.oidc.runtime.TenantContextFactory$2.apply(TenantContextFactory.java:116)
		at io.smallrye.context.impl.wrappers.SlowContextualFunction.apply(SlowContextualFunction.java:21)
		at io.smallrye.mutiny.groups.UniOnFailure.lambda$recoverWithItem$8(UniOnFailure.java:190)
		at io.smallrye.mutiny.operators.uni.UniOnFailureFlatMap$UniOnFailureFlatMapProcessor.performInnerSubscription(UniOnFailureFlatMap.java:92)
		... 71 more
	Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:35271
	Caused by: java.net.ConnectException: Connection refused
		at [email protected]/sun.nio.ch.Net.pollConnect(Native Method)
		at [email protected]/sun.nio.ch.Net.pollConnectNow(Net.java:682)
		at [email protected]/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:973)
		at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:336)
		at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:339)
		at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776)
		at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
		at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
		at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
		at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
		at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
		at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
		at [email protected]/java.lang.Thread.runWith(Thread.java:1596)
		at [email protected]/java.lang.Thread.run(Thread.java:1583)
		at org.graalvm.nativeimage.builder/com.oracle.svm.core.thread.PlatformThreads.threadStartRoutine(PlatformThreads.java:896)
		at org.graalvm.nativeimage.builder/com.oracle.svm.core.thread.PlatformThreads.threadStartRoutine(PlatformThreads.java:872)
Caused by: [CIRCULAR REFERENCE: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:35271]

@constantin-ungureanu-github
Copy link
Author

Hi @jedla97 ! I updated the repository https://github.com/constantin-ungureanu-github/quarkus-oidc-example as per your suggestions. Instead of using quarkus.http.non-application-root-path I use quarkus.management.root-path and it is indeed working as expected !!

For the original issue, of the integration tests failing due to usage of quarkus-oidc, you guys for sure know better.
It maybe helpful to know that keycloak has a discovery option that should be disabled, as can be seen in the docker-compose.yml file. Maybe you could set up the Keycloak devservice similarly, passing the proper configuration.

I hope to see this fixed soon. Thanks for your replies!

@sberyozkin
Copy link
Member

sberyozkin commented Jan 27, 2025

Hi @jedla97, many thanks for your investigation...

What I do recall is that when mvn clean install -Pnative -Dquarkus.native.container-build=true -Dquarkus.container-image.build=true command is used, the shared network is required (sorry, I mistakenly referred to it as a shared container property as I suggested to @constantin-ungureanu-github above), so it applies only to the case of the shared network, and your 2nd option looks fine to me.

Awhile back we had a dedicated integration test, created by @geoand, I was periodically doing some minor tweaks to get it passing, and then later it was removed due to some cleanup, and unfortunately I never got to restoring it, as far as I recall @iocanel was suggesting we did not need to have an image generated to test this shared network case...

Would you like to have a look at restoring that test in the next couple of weeks and CC @geoand, @iocanel and myself whenever you have it working ?

@jedla97
Copy link
Contributor

jedla97 commented Jan 27, 2025

Hi @sberyozkin I can look at restoring them. Thanks for the links!

For this issue are we want to provide the fix now so we have at least working solution (maybe not for all case but at least fix the startup of Keycloak) or are we want fixt it with test?
From my POV the fix it now make sense as we have already 2 issues and when the restored test found issue it can modified.

@iocanel
Copy link
Contributor

iocanel commented Jan 28, 2025

I don't think that the removed tests are related this issue. The removed tests were maven invoker tests, which pretty much tested the image building.

If the removed modules included Quarkus integration tests that we need, then we could propably restore them, but please not as invoker tests.

@sberyozkin
Copy link
Member

@jedla97 Thanks, my understanding that restoring the actual integration test would require your suggested 2nd option to get this test passing on main and it will likely then resolve both of the issues, but we can double check it later.

@iocanel The removed tests were testing the shared network case related to this issue. So, given that those tests need that image which was built during the maven invoker run, can you recommend how to setup the test so that the shared network case is tested ?

@HerrDerb
Copy link
Contributor

HerrDerb commented Feb 7, 2025

It now seems that with quarkus 3.18.x a keycloak 26.x.x image is required for the dev service (otherwise my application fails to start while testing)

Caused by: java.lang.RuntimeException: Could not obtain configuration from server [http://localhost:52225/realms/other-tenant/.well-known/uma2-configuration].
	at org.keycloak.authorization.client.AuthzClient.<init>(AuthzClient.java:257)
	at org.keycloak.authorization.client.AuthzClient.create(AuthzClient.java:96)
	at org.keycloak.adapters.authorization.PolicyEnforcer.<init>(PolicyEnforcer.java:103)
	at org.keycloak.adapters.authorization.PolicyEnforcer$Builder.build(PolicyEnforcer.java:662)
	at io.quarkus.keycloak.pep.runtime.KeycloakPolicyEnforcerUtil.createPolicyEnforcer(KeycloakPolicyEnforcerUtil.java:93)

With a keycloak image 26.x.x, my test run through, but then my shared network tests fail again.
This means this issue now blocks from updating quarkus, as the workaround with downgrading with a custom image does not work anymore.

I'll try to poke around a little bit today, but it certainly would be appreciated if someone with expertise would have a look, as it is most likely a simple config issue.

@HerrDerb
Copy link
Contributor

HerrDerb commented Feb 7, 2025

@sberyozkin @constantin-ungureanu-github I found that this works for me: #46141 A keycloak expert should probably verify that 🙂

What is still missing is a test that is run with (gradle) quarkusIntTest -Dquarkus.container-image.build=true to ensure future functionality.

@gsmet
Copy link
Member

gsmet commented Feb 11, 2025

My understanding is that the problem was fixed with #46141 ?

Should we close this issue?

@HerrDerb
Copy link
Contributor

@constantin-ungureanu-github should be fixed with 3.18.3. Can you confirm?

@amoscatelli
Copy link
Contributor

Hi everybody,
I am missing something

KEYCLOAK_QUARKUS_HOSTNAME is missing the port, by doing so the issuer is always wrong in tokens generated during integration tests :

{
  "exp": 1741620148,
  "iat": 1741619548,
  "jti": "dae06a44-12ed-413b-b3ef-a55a0b300b76",
  "iss": "http://keycloak-441zi/realms/quarkus",
  "sub": "0450aafc-2e42-4ea3-9e0e-b93ef809cdf7",
  "typ": "Bearer",
  "azp": "quarkus-app",
  "sid": "f80b4c35-4197-4b29-b5d6-c5216accf24f",
  "scope": "microprofile-jwt",
  "upn": "alice",
  "groups": [
    "superadmin"
  ]
}

The correct issuer should be (in my case) http://keycloak-441zi:8080/realms/quarkus

Of course every IT is failing for this ... help ?

@amoscatelli
Copy link
Contributor

Also, this is confimed by documentation :

https://quarkus.io/guides/security-jwt#integration-testing-security-annotation

I expect the correct issue value to be ${keycloak.url}/realms/quarkus, that in my case is http://keycloak-441zi:8080/realms/quarkus

@amoscatelli
Copy link
Contributor

Sorry I forgot to mention this happens in the case of required shared network ...

@HerrDerb
Copy link
Contributor

@amoscatelli I'll have a look, I did work for me without any issues in a shared network. Do you authenticate within the backend?

@amoscatelli
Copy link
Contributor

amoscatelli commented Mar 10, 2025

@HerrDerb thank you.

During my tests I am able to obtain a JWT from Keycloak using KeycloakTestClient (for example the JWT I pasted above).

But the JWT is invalid for the quarkus microservice because the issuer mismatches.

How are you setting the mp.jwt.verify.issuer property in your tests ?

@amoscatelli
Copy link
Contributor

Example code :

package com.infolabtech;

import io.quarkus.test.junit.QuarkusIntegrationTest;
import io.quarkus.test.junit.QuarkusTestProfile;
import io.quarkus.test.junit.TestProfile;
import io.quarkus.test.keycloak.client.KeycloakTestClient;
import io.restassured.RestAssured;
import static io.restassured.RestAssured.oauth2;
import java.util.Map;
import org.eclipse.microprofile.config.ConfigProvider;
import org.eclipse.microprofile.jwt.config.Names;
import org.junit.jupiter.api.Test;

@QuarkusIntegrationTest
@TestProfile(GreetingResourceIT.class)
public class GreetingResourceIT extends AbstractGreetingResourceTest implements QuarkusTestProfile {

    KeycloakTestClient keycloakClient = new KeycloakTestClient();
    
    @Override
    public Map<String, String> getConfigOverrides() {
        return Map.of(
                Names.VERIFIER_PUBLIC_KEY_LOCATION, "${keycloak.url}/realms/quarkus/protocol/openid-connect/certs",
                Names.ISSUER, "${keycloak.url}/realms/quarkus"
        );
    }
    
    @Test
    @Override
    public void authorized() {
        System.out.println("authorized " + keycloakClient.getAuthServerUrl());
        String accessToken = keycloakClient.getAccessToken(
                "alice"
        );
        System.out.println("TEST1 " + accessToken);
        RestAssured.authentication = oauth2(
                accessToken
        );
        super.authorized();
    }
    
    @Test
    @Override
    public void unauthenticated() {
        super.unauthenticated();
    }

    @Test
    @Override
    public void unauthorized() {
        System.out.println("unauthorized " + keycloakClient.getAuthServerUrl());
        String accessToken = keycloakClient.getAccessToken(
                "bob"
        );
        System.out.println("TEST2 " + accessToken);
        RestAssured.authentication = oauth2(
                accessToken
        );
        super.unauthorized();
    }
}

@jedla97
Copy link
Contributor

jedla97 commented Mar 10, 2025

Probably the 2nd option what I mentioned will be needed insted of setting --hostname-backchannel-dynamic

2nd option it replace KEYCLOAK_QUARKUS_HOSTNAME value from "localhost" to "http://" + hostName + ":" + KEYCLOAK_PORT this will expose the generated Keycloak hostname (http://keycloak-vu5vi:8080)

I'll check it later today

@jedla97
Copy link
Contributor

jedla97 commented Mar 10, 2025

Hi @HerrDerb and @amoscatelli I think it should work so I created the #46710

I checked it with reproducer what @constantin-ungureanu-github provided and the test are passing.

Can you try this patch and check if it work on your side?

Also @amoscatelli I checked with your example code which you provided but don't know what happening in AbstractGreetingResourceTest. But don't see any problem why this should not work. If this patch not work for you could you provide small reproducer (as git repo or zip) for me to run it and debug it.

@amoscatelli
Copy link
Contributor

@jedla97 thank you, it seems ok to me :

Image

@HerrDerb
Copy link
Contributor

@jedla97 tested an seems to work for me too. I am actually wondering and a bit confused as I thought I would have tried that too 😅 anyway, it works 😀👍

@quarkus-bot quarkus-bot bot added this to the 3.21 - main milestone Mar 11, 2025
@constantin-ungureanu-github
Copy link
Author

Thanks for working to solve this issue.

I made some updates to the code to be able to use in the tests a different realm than the default one, but the keycloak client is always trying to use the client "quarkus-app", that was not in my realm and I couldn't find a proper way to specify the name of the client. Not sure if there is one, specifying the service name quarkus.keycloak.devservices.service-name doesn't seem to make the keycloak client to use the client with this name. Not a big deal, since I could modify in the realm used for testing the name of the client to be exactly "quarkus-app", but probably devservices keycloak client should have this option to specify the client and secret to use for the custom realm.

However, when I try to build also with native mode enabled it hangs in the native integration tests:

2025-03-11 16:42:36,558 INFO [io.qua.dev.key.KeycloakDevServicesProcessor] (build-6) Dev Services for Keycloak started.
2025-03-11 16:42:38,523 INFO [io.qua.tes.com.DefaultDockerContainerLauncher] (main) Executing "docker run --name quarkus-integration-test-cQQvg -i --rm -p 55062:55062 -p 8444:8444 --net=e3bf225a329e4f13b8c93eb
39b7eb87a73b27a481dc9dd763a196482debd81b2 --env QUARKUS_LOG_CATEGORY__IO_QUARKUS__LEVEL=INFO --env QUARKUS_HTTP_PORT=55062 --env QUARKUS_HTTP_SSL_PORT=8444 --env TEST_URL=http://localhost:0 --env QUARKUS_PROFIL
E=prod --env QUARKUS_OIDC_CLIENT_ID=quarkus-app --env CLIENT_QUARKUS_OIDC_AUTH_SERVER_URL=http://localhost:55045/realms/quarkus --env KEYCLOAK_URL=http://keycloak-tewdr:8080 --env QUARKUS_OIDC_AUTH_SERVER_URL=h
ttp://keycloak-tewdr:8080/realms/quarkus --env KEYCLOAK_REALMS=quarkus --env QUARKUS_CONTAINER_IMAGE_BUILD=true --env QUARKUS_NATIVE_CONTAINER_BUILD=true --env OIDC_USERS= --env QUARKUS_OIDC_APPLICATION_TYPE=se
rvice --env QUARKUS_OIDC_CREDENTIALS_SECRET=secret test/quarkus-oidc-example:1.0.0-SNAPSHOT"
2025-03-11 16:42:38,531 INFO [io.qua.tes.com.DefaultDockerContainerLauncher] (main) Wait for server to start by capturing listening data...
Waited 60 seconds for target\quarkus.log to contain info about the listening port and protocol but no such info was found. Check if the options quarkus.log.level and quarkus.log.file.level are at least INFO (or
more verbose).
Failed to launch the application. The application logs can be found at: D:\Projects\Personal\quarkus-oidc-example\target\quarkus.log
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 95.40 s <<< FAILURE! -- in org.acme.GreetingResourceIT
[ERROR] org.acme.GreetingResourceIT.testHelloEndpoint -- Time elapsed: 0.011 s <<< ERROR!
java.lang.RuntimeException: java.lang.IllegalStateException: Unable to determine the status of the running process. See the above logs for details
at io.quarkus.test.junit.QuarkusIntegrationTestExtension.throwBootFailureException(QuarkusIntegrationTestExtension.java:340)
at io.quarkus.test.junit.QuarkusIntegrationTestExtension.beforeEach(QuarkusIntegrationTestExtension.java:114)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
Caused by: java.lang.IllegalStateException: Unable to determine the status of the running process. See the above logs for details
at io.quarkus.test.common.LauncherUtil.waitForCapturedListeningData(LauncherUtil.java:98)
at io.quarkus.test.common.DefaultDockerContainerLauncher.start(DefaultDockerContainerLauncher.java:303)
at io.quarkus.test.junit.IntegrationTestUtil.startLauncher(IntegrationTestUtil.java:160)
at io.quarkus.test.junit.QuarkusIntegrationTestExtension.doProcessStart(QuarkusIntegrationTestExtension.java:303)
at io.quarkus.test.junit.QuarkusIntegrationTestExtension.ensureStarted(QuarkusIntegrationTestExtension.java:167)
at io.quarkus.test.junit.QuarkusIntegrationTestExtension.beforeAll(QuarkusIntegrationTestExtension.java:127)
... 1 more

I suspected that the issue is that I'm trying to use a custom realm, so alternative is to only use the default configuration. So reverting the usage of custom realm (commenting out #quarkus.keycloak.devservices.realm-path=quarkus-realm.json) but the error still persists.

Commenting all additional settings to keycloak and the error still persists.
#quarkus.keycloak.devservices.image-name=keycloak/keycloak:26.1
#quarkus.keycloak.devservices.shared=false
#quarkus.keycloak.devservices.show-logs=true
#quarkus.keycloak.devservices.realm-path=quarkus-realm.json

One additional detail is that I have activated the swagger/openapi, maybe this has an influence overall.

An additional, more important issue I have observed in the past (can't reproduce it here since the native IT tests fail before) is that when I want to use random management port in tests, it is always 0.
So what a normal URL in an integration test would be something like "http://localhost:54964/health/live", in the native integration tests is "http://localhost:0/health/live".

@jedla97
Copy link
Contributor

jedla97 commented Mar 11, 2025

Hi, @constantin-ungureanu-github looking at this and the problem seems to be that your dockerfiles not use ubi9 images.

for

  • Dockerfile.native-micro it should be quay.io/quarkus/ubi9-quarkus-micro-image
  • Dockerfile.native it should be registry.access.redhat.com/ubi9/ubi-minimal`
  • Dockerfile.jvm and Dockerfile.legacy-jar should be registry.access.redhat.com/ubi9/openjdk-21

I try it with this and it seems to work except that management endpoint whenruning integration tests (LivenessCheckIT#testLivenessProbe)

Also I check if the quarkus update will update the docker files and they still same.

I'll create issues for both of these problems. But in meantime can you try update docker files (https://github.com/constantin-ungureanu-github/quarkus-oidc-example/tree/main/src/main/docker) as I mention above

@constantin-ungureanu-github
Copy link
Author

constantin-ungureanu-github commented Mar 11, 2025

Thanks @jedla97 for the update.
I have made the suggested changes to the dockerfiles and the containers now are starting, exactly as you suggested. When the initial repository was created, there was still UBI8 used as base image.

The keycloak issues indeed seems to have been fixed.

The random management test port issue is now reproducible, I even put a log to see the URL I'm getting for the live probe when trying to use a random management port. I cannot use the default management test port 9001 because this specific port it is blocked on my work machine, so a random port would be exactly fitted for this.

This issue with the random management port on native integration tests it would be nice if it is fixed for the 3.20 LTS version.

[INFO] Running org.acme.LivenessCheckIT
2025-03-11 18:08:08,863 INFO [org.acm.LivenessCheckTest] (main) http://localhost:0/health/live
2025-03-11 18:08:08,891 INFO [io.qua.tes.com.DefaultDockerContainerLauncher] (main) Close the container
2025-03-11 18:08:10,807 INFO [io.qua.dev.key.KeycloakDevServicesProcessor] (docker-java-stream--1426065604) Keycloak:
2025-03-11 18:08:10,807 INFO [io.qua.dev.key.KeycloakDevServicesProcessor] (docker-java-stream--1426065604) Keycloak:
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 2.620 s <<< FAILURE! -- in org.acme.LivenessCheckIT
[ERROR] org.acme.LivenessCheckIT.testLivenessProbe -- Time elapsed: 2.603 s <<< FAILURE!
java.lang.AssertionError:
1 expectation failed.
Expected status code <200> but was <400>.

...

@gsmet
Copy link
Member

gsmet commented Mar 11, 2025

@constantin-ungureanu-github could you clarify how we can reproduce the issue ^. Ideally having a reproducer we can start would help.

@constantin-ungureanu-github
Copy link
Author

@jedla97
Copy link
Contributor

jedla97 commented Mar 11, 2025

@constantin-ungureanu-github @gsmet I create the issue about it #46737

I attached the @constantin-ungureanu-github reproducer and add simple mine.
But tl;dr the test-framework don't exposing the management test port when running containers

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

7 participants