Updating dependencies for GreenMail and Jakarta Mail resulted in "CertificateException: No subject alternative names present" during test execution using DummySSLSocketFactory

[gunalmel]

Test Setup Code (Before):

The following setup code worked with the previous dependencies:

  @BeforeAll
  static void setup() {
    Security.setProperty("ssl.SocketFactory.provider", DummySSLSocketFactory.class.getName());
    mailServer = new GreenMail(new ServerSetup[]{ServerSetupTest.SMTP, ServerSetupTest.SMTPS});
    mailServer.start();
    mailServer.setUser(USER_EMAIL_ADDRESS, USER_NAME, USER_PASSWORD);
}

Previous Dependencies:

implementation 'com.sun.mail:jakarta.mail:2.0.1'
testImplementation 'com.icegreen:greenmail:2.0.1'

Current Dependencies:
After updating to the following dependencies:

implementation 'jakarta.mail:jakarta.mail-api:2.1.3'
implementation 'org.eclipse.angus:jakarta.mail:2.0.3'
testImplementation 'com.icegreen:greenmail-junit5:2.1.2'

the following error occurs during tests:

Caused by: java.security.cert.CertificateException: No subject alternative names present
    at java.base/sun.security.util.HostnameChecker.matchIP(HostnameChecker.java:138)
    at java.base/sun.security.util.HostnameChecker.match(HostnameChecker.java:101)
    at java.base/sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:461)
    at java.base/sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:431)

Code Under Test:
The following method triggers the error:

public void send(EmailMessage emailMsg) {
    Session session;

    if (this.requiresAuth) {
      session = Session.getInstance(buildJavaMailProperties(), buildJavaMailAuthenticator());
    } else {
      session = Session.getInstance(buildJavaMailProperties());
    }

    try {
      var transport = session.getTransport();
      MimeMessage msg = prepareMessage(emailMsg, session);
      transport.connect(emailServer, smtpPort , user, password);
      transport.sendMessage(msg, msg.getRecipients(Message.RecipientType.TO));
      transport.close();
    } catch (Exception ex) {
      LOGGER.error("Exception sending email via java email client with recipients: {} \ncc: {} \nbcc: {} \nsubject: {} \nbody:{}",
                   String.join(", ", emailMsg.getTo()),
                   String.join(", ", emailMsg.getCc()),
                   String.join(", ", emailMsg.getBcc()),
                   emailMsg.getSubject(),
                   emailMsg.getBody());
      throw new SendEmailException(
          emailMsg,
          ex);
    }
  }

Workarounds:

1. If the java mail client uses
   properties.put("mail.smtps.ssl.checkserveridentity", "false");
   then
   Security.setProperty("ssl.SocketFactory.provider", DummySSLSocketFactory.class.getName());
   works fine without the need for java keystore as it was before. That's inconvenient because the java email client has to be modified to use different props when it's running in production.

2. Replacing the use of DummySSLSocketFactory with a keystore resolved the issue.

Keystore Creation:
Removed Security.setProperty("ssl.SocketFactory.provider", DummySSLSocketFactory.class.getName()); Instead created a keystore for the tests and specified it as trust store at the same time.

keytool -genkeypair \
    -alias greenmail \
    -keyalg RSA \
    -keysize 2048 \
    -validity 365 \
    -keystore src/test/resources/greenmail.jks \
    -dname "CN=localhost" \
    -ext "SAN=IP:127.0.0.1,DNS:localhost"

Updated Test Setup (@BeforeAll):

 @BeforeAll
  static void setup() {
    System.setProperty("javax.net.debug", "ssl,handshake");
    System.setProperty("greenmail.tls.keystore.file", "src/test/resources/greenmail.jks");
    System.setProperty("greenmail.tls.keystore.password", "password");
    System.setProperty("javax.net.ssl.trustStore", "src/test/resources/greenmail.jks");
    System.setProperty("javax.net.ssl.trustStorePassword", "password");
    ServerSetup smtps = ServerSetupTest.SMTPS;
    mailServer = new GreenMail(new ServerSetup[]{ServerSetupTest.SMTP, smtps});
    mailServer.start();
    mailServer.setUser(USER_EMAIL_ADDRESS, USER_NAME, USER_PASSWORD);
  }

When the test is run alone the workarounds worked fine. When it's run in a suite with many other tests using gradle it still fails with:

javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

Expected Behavior:
The DummySSLSocketFactory should bypass SSL validation without requiring additional keystore configuration.

Observed Behavior:
Tests fail with java.security.cert.CertificateException: No subject alternative names present.

[marcelmay]

You should not have to tweak SSL settings, as it should be as simple as in this test:
https://github.com/greenmail-mail-test/greenmail/blob/master/greenmail-core/src/test/java/com/icegreen/greenmail/smtp/SmtpSecureServerTest.java

You would also require some more settings probably, as done here:
https://github.com/greenmail-mail-test/greenmail/blob/master/greenmail-core/src/main/java/com/icegreen/greenmail/util/ServerSetup.java#L284

Try to remove the SSL tweaking - does the following work:

@BeforeAll
static void setup() {
    // SHOULD NOT BE REQUIRED:
    // Security.setProperty("ssl.SocketFactory.provider", DummySSLSocketFactory.class.getName());
    mailServer = new GreenMail(new ServerSetup[]{ServerSetupTest.SMTP, ServerSetupTest.SMTPS});
    mailServer.start();
    mailServer.setUser(USER_EMAIL_ADDRESS, USER_NAME, USER_PASSWORD);
}

If this still fails:

1. What JDK are you using
2. Can you provide full stack trace from the error
3. Enable verbose mode

    mailServer = new GreenMail(ServerSetup.verbose(new ServerSetup[]{ServerSetupTest.SMTP, ServerSetup.SMTPS}))``` and provide log

4. Enable full TLS debugging:

   System.setProperty("javax.net.debug", "all")``` and provide log