Bump up Java version to 21 #2682

iamsanjay · 2024-08-30T05:52:39Z

https://issues.apache.org/jira/browse/SOLR-17321

Bump up Java version to 21. Although we have decided to have different versions for Solr and SolrJ to 21 and 17 respectively. Currently in this PR, both of them are pointing to 21 to seek feedback and to test everything. Meanwhile looking for way to set SolrJ to 17.

iamsanjay · 2024-08-30T06:15:27Z

This happens some time!

https://github.com/apache/solr/actions/runs/10627729009/job/29461325476?pr=2682

Task :solr:benchmark:compileJava FAILED
An exception has occurred in the compiler (21.0.4). Please file a bug against the Java compiler via the Java bug reporting page (https://bugreport.java.com/) after checking the Bug Database (https://bugs.java.com/) for duplicates. Include your program, the following diagnostic, and the parameters passed to the Java compiler in your report. Thank you.
java.lang.VirtualMachineError: Out of space in CodeCache for adapters
at jdk.compiler/com.sun.tools.javac.processing.JavacProcessingEnvironment$DiscoveredProcessors$ProcessorStateIterator.next(JavacProcessingEnvironment.java:824)
at jdk.compiler/com.sun.tools.javac.processing.JavacProcessingEnvironment.discoverAndRunProcs(JavacProcessingEnvironment.java:922)
at jdk.compiler/com.sun.tools.javac.processing.JavacProcessingEnvironment$Round.run(JavacProcessingEnvironment.java:1265)
at jdk.compiler/com.sun.tools.javac.processing.JavacProcessingEnvironment.doProcessing(JavacProcessingEnvironment.java:1380)
at jdk.compiler/com.sun.tools.javac.main.JavaCompiler.processAnnotations(JavaCompiler.java:1272)
at jdk.compiler/com.sun.tools.javac.main.JavaCompiler.compile(JavaCompiler.java:946)
at jdk.compiler/com.sun.tools.javac.api.JavacTaskImpl.lambda$doCall$0(JavacTaskImpl.java:104)
at jdk.compiler/com.sun.tools.javac.api.JavacTaskImpl.invocationHelper(JavacTaskImpl.java:152)
at jdk.compiler/com.sun.tools.javac.api.JavacTaskImpl.doCall(JavacTaskImpl.java:100)
at jdk.compiler/com.sun.tools.javac.api.JavacTaskImpl.call(JavacTaskImpl.java:94)
at org.gradle.internal.compiler.java.IncrementalCompileTask.call(IncrementalCompileTask.java:92)
at org.gradle.api.internal.tasks.compile.AnnotationProcessingCompileTask.call(AnnotationProcessingCompileTask.java:94)
at org.gradle.api.internal.tasks.compile.ResourceCleaningCompilationTask.call(ResourceCleaningCompilationTask.java:57)
at org.gradle.api.internal.tasks.compile.JdkJavaCompiler.execute(JdkJavaCompiler.java:55)
at org.gradle.api.internal.tasks.compile.JdkJavaCompiler.execute(JdkJavaCompiler.java:39)
at org.gradle.api.internal.tasks.compile.NormalizingJavaCompiler.delegateAndHandleErrors(NormalizingJavaCompiler.java:98)
at org.gradle.api.internal.tasks.compile.NormalizingJavaCompiler.execute(NormalizingJavaCompiler.java:52)
at org.gradle.api.internal.tasks.compile.NormalizingJavaCompiler.execute(NormalizingJavaCompiler.java:38)
at

build.gradle

dsmiley

BTW we don't have to compile with JDK 17 or use JDK 17 at all for the build. We just need to tell JDK 21 that a few modules for SolrJ are to be compiled with Java 17 language / support, allowing users on JDK 17 to use it. I don't recall the particulars of doing this.

.github/workflows/bin-solr-test.yml

.github/workflows/gradle-precommit.yml

solr/solrj/src/java/org/apache/solr/client/solrj/impl/HttpJdkSolrClient.java

iamsanjay · 2024-10-08T13:00:00Z

./gradlew compileJava seem to work fine. Only If we find a way to set Jdk to 21 for all the test packages. Because currently, test cases from api not able to find classes from other projects that are on higher version, and that's where it's failing.

dsmiley · 2024-10-08T18:19:35Z

Note that "api" module should also be Java Language 17, as it's included with SolrJ.

iamsanjay · 2024-10-09T07:22:45Z

Looks like below config is working. It sets solrj and related projects to Jdk17 and rest of the projects to Jdk21 for main. And, for all the test packages they are using Jdk21.

javac.gradle

    // Use 'release' flag instead of 'source' and 'target'
     tasks.withType(JavaCompile) {
       compileTestJava {
         options.compilerArgs += ["--release", JavaVersion.toVersion(scriptDepVersions['min-java-version']).toString()]
       }
       compileJava {
         options.compilerArgs += ["--release", project.minJavaVersion.toString()]
       }
     }

Now, to test it. I unpack the builds and verify it in MANIFEST.MF.

solrj

Specification-Title: Apache Solr Search Server: solrj
X-Compile-Source-JDK: 17
X-Compile-Target-JDK: 17
X-Build-JDK: 21.0.3 (Oracle Corporation 21.0.3+7-LTS-152)
X-Build-OS: Mac OS X aarch64 14.3

api

Specification-Title: Apache Solr Search Server: api
X-Compile-Source-JDK: 17
X-Compile-Target-JDK: 17
X-Build-JDK: 21.0.3 (Oracle Corporation 21.0.3+7-LTS-152)
X-Build-OS: Mac OS X aarch64 14.3

solr-zookeeper

Specification-Title: Apache Solr Search Server: solrj-zookeeper
X-Compile-Source-JDK: 17
X-Compile-Target-JDK: 17
X-Build-JDK: 21.0.3 (Oracle Corporation 21.0.3+7-LTS-152)
X-Build-OS: Mac OS X aarch64 14.3

solrj-streaming

Specification-Title: Apache Solr Search Server: solrj-streaming
X-Compile-Source-JDK: 17
X-Compile-Target-JDK: 17
X-Build-JDK: 21.0.3 (Oracle Corporation 21.0.3+7-LTS-152)
X-Build-OS: Mac OS X aarch64 14.3

core

Specification-Title: Apache Solr Search Server: core
X-Compile-Source-JDK: 21
X-Compile-Target-JDK: 21
X-Build-JDK: 21.0.3 (Oracle Corporation 21.0.3+7-LTS-152)
X-Build-OS: Mac OS X aarch64 14.3

But I would really appreciate If someone else test it and check If we able to achieve the different version for solr (Jdk21) and solrj(jdk17).

dsmiley · 2024-10-10T01:57:17Z

Nice work. I toyed with it a bit, including tinkering with toolchains. Avoiding toolchains avoids IntelliJ's Gradle's check to tell you that we really should be using a newer Gradle Version:

Your build is currently configured to use incompatible Java 21.0.4 and Gradle 8.4. Cannot sync the project.

We recommend upgrading to Gradle version 8.8.

The minimum compatible Gradle version is 8.5.

The maximum compatible Gradle JVM version is 20.

Possible solutions:
 - Upgrade to Gradle 8.8 and re-sync
 - Upgrade to Gradle 8.5 and re-sync
 - Use Java 20 as Gradle JVM: Open Gradle settings

dsmiley · 2024-10-10T02:01:36Z

Why aren't we using toolchains?: https://docs.gradle.org/current/userguide/toolchains.html

dsmiley · 2024-10-10T03:00:12Z

The Docker build.gradle needs to refer to a JDK 21 base image, not JDK 17.

BTW, when I checked out your branch and before I tinkered with it, I simply tried to get IntelliJ to sync it and then view IntelliJ's module language versions to see if it would get confused on the SolrJ modules, given differing desired language levels for tests. It was confused; it thought the whole module should use JDK 17, even the test sourcepath. I suspect the issue is that we don't use proper metadata (i.e. toolchain) to convey to IntelliJ what's going on.

iamsanjay · 2024-10-10T12:18:40Z

The toolchain cannot be set at the JavaCompile level. Because we are using different Jdk for main and test package. To set at project level, below is the config. But this fails!

allprojects {
  plugins.withType(JavaPlugin) {
    java {
    toolchain {
        languageVersion = JavaLanguageVersion.of(17)
    }
  }
 }
}

Next, we have different toolchain for the tasks. And below is the way to configure different toolchain for compileJava and compileTestJava.

tasks.withType(JavaCompile) {
        compileTestJava {
          javaCompiler = javaToolchains.compilerFor {
            languageVersion = JavaLanguageVersion.of(JavaVersion.toVersion(scriptDepVersions['min-java-version']).toString())
          }
          options.compilerArgs += ["--release", JavaVersion.toVersion(scriptDepVersions['min-java-version']).toString()]
        }
        compileJava {
          javaCompiler = javaToolchains.compilerFor {
            languageVersion = JavaLanguageVersion.of(project.minJavaVersion.toString())
          }
          options.compilerArgs += ["--release", project.minJavaVersion.toString()]
        }
      }

This works! I ran with ./gradlew :solr:api:build --info and I can see different toolchain being loaded

Task ':solr:api:compileJava' is not up-to-date because:
  Output property 'destinationDirectory' file /Users/sanjaydutt/Documents/solr/solr/api/build/classes/java/main has been removed.
  Output property 'destinationDirectory' file /Users/sanjaydutt/Documents/solr/solr/api/build/classes/java/main/org has been removed.
  Output property 'destinationDirectory' file /Users/sanjaydutt/Documents/solr/solr/api/build/classes/java/main/org/apache has been removed.
  Output property 'options.generatedSourceOutputDirectory' file /Users/sanjaydutt/Documents/solr/solr/api/build/generated/sources/annotationProcessor/java/main has been re
moved.
The input changes require a full rebuild for incremental task ':solr:api:compileJava'.
Full recompilation is required because no incremental change information is available. This is usually caused by clean builds or changing compiler arguments.
Compiling with toolchain '/Library/Java/JavaVirtualMachines/jdk-17.jdk/Contents/Home'.
Compiling with JDK Java compiler API.
Class dependency analysis for incremental compilation took 0.003 secs.
Created classpath snapshot for incremental compilation in 0.0 secs.
work action resolve main (project :solr:api) (Thread[#760,Execution worker Thread 3,5,main]) started.
Resolve mutations for :solr:api:processResources (Thread[#760,Execution worker Thread 3,5,main]) started.


> Task :solr:api:compileTestJava
Caching disabled for task ':solr:api:compileTestJava' because:
  Build cache is disabled
Task ':solr:api:compileTestJava' is not up-to-date because:
  Task has failed previously.
The input changes require a full rebuild for incremental task ':solr:api:compileTestJava'.
Full recompilation is required because no incremental change information is available. This is usually caused by clean builds or changing compiler arguments.
Compiling with toolchain '/Library/Java/JavaVirtualMachines/jdk-21.jdk/Contents/Home'.
Compiling with JDK Java compiler API.
Class dependency analysis for incremental compilation took 0.0 secs.
Created classpath snapshot for incremental compilation in 0.003 secs.
Resolve mutations for :solr:api:analyzeTestClassesDependencies (Thread[#757,included builds,5,main]) started.

However, jar task is still, I think ,running with whatever java_home is

Because whenever I open MANIFEST.MF, Build-JDK is 21. I mean we are successfully able to compile files using jdk 17, however whenever jar task is running it's running via 21.

Specification-Title: Apache Solr Search Server: api
X-Compile-Source-JDK: 17
X-Compile-Target-JDK: 17
X-Build-JDK: (21.0.3)
X-Build-OS: Mac OS X aarch64 14.3

I try to add the toolchain config to jar-manifest.gradle, but it's not working

doFirst {
                    def toolchain = javaToolchains.compilerFor {
                        languageVersion = JavaLanguageVersion.of(project.minJavaVersion.toString())
                    }
                }
                manifest {
                    attributes([
                            "Extension-Name"        : implementationTitle,

                            "Implementation-Vendor" : "The Apache Software Foundation",
                            "Implementation-Title"  : implementationTitle,
                            "Implementation-Version": implementationVersion,

                            "Specification-Vendor"  : "The Apache Software Foundation",
                            "Specification-Version" : project.baseVersion,
                            "Specification-Title"   : title,

                            // Evaluate these properties lazily so that the defaults are applied properly.
                            "X-Compile-Source-JDK"  : "${-> project.sourceCompatibility}",
                            "X-Compile-Target-JDK"  : "${-> project.targetCompatibility}",

                            "X-Build-JDK"           : "${System.properties['java.version']} (${System.properties['java.vendor']} ${System.properties['java.vm.version']})",
                            "X-Build-OS"            : "${System.properties['os.name']} ${System.properties['os.arch']} ${System.properties['os.version']}"
                    ])
                }

iamsanjay · 2024-10-10T12:20:52Z

When it comes to Intellij module language level, I am not sure How we will solve that? Given that all solrj and realted modules have two jdk versions now.

iamsanjay · 2024-10-10T12:29:53Z

We use Toolchain to enforce the specific version of the tools we want to use. Here are some use cases:

If our system's JDK is set to 21, but I want to run a project on a different Java version.
If I need a specific Java version from a vendor for all tasks.
If I want to run Gradle and the project on different Java versions.

In our scenario, if the system is configured with JDK 21, that version will be used to run all tasks for Solr and SolrJ. JDK 21 can generate class files with version 61 without issues. However, if we want SolrJ to compile only with JDK 17, Toolchain can accomplish that. Otherwise, the Java version used by Gradle will be applied globally.

iamsanjay · 2024-10-10T12:33:19Z

From Intellij docs:

https://www.jetbrains.com/help/idea/gradle.html#gradle_jvm

Language level: the language level settings are applied for a source root or for a module. If a Gradle project has a single linked project then the project default language level is set to the minimum language level among the module language levels. The module language level is set to sourceCompatibility in the build.gradle file.

iamsanjay · 2024-10-10T12:57:54Z

IMO, There is no way to configure toolchain for jar task like we can do for JavaCompile. Even the JarTask has no mention of compiler like JavaCompiler has.

https://github.com/gradle/gradle/blob/master/platforms/jvm/platform-jvm/src/main/java/org/gradle/jvm/tasks/Jar.java
https://github.com/gradle/gradle/blob/master/platforms/jvm/language-java/src/main/java/org/gradle/api/tasks/compile/JavaCompile.java#L202

My question is that does it even matter which jdk version actually package the class files? Given that we made sure that compileJava task ran only by appropriate jdk.

dsmiley · 2024-10-10T17:57:00Z

However, jar task is still, I think ,running with whatever java_home is

It's normal & acceptable that the entire build itself do JAR packaging etc with the same running JVM, that which Gradle is using -- JDK 21. We shouldn't try to change that or validate the manifest entry for the build JDK. It's just some FYI metadata.

RE IntelliJ -- lots of us use IntelliJ; if IntelliJ will not tolerate Java 21 language features in a SolrJ test, we'll have to use Java 17 for those tests. Okay, not a big deal.

iamsanjay · 2024-10-11T00:39:51Z

gradle Auto-provisioning is disabled.

./gradlew -q javaToolchains                                                             

 + Options
     | Auto-detection:     Enabled
     | Auto-download:      Disabled

Build is failing due to that. It's like now we are basically forcing the version of tools that being used, in this case two Jdk version 17 and 21. Otherwise, If you have both version available then no issues.

> Task :solr:api:compileJava FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':solr:api:compileJava'.
> Error while evaluating property 'javaCompiler' of task ':solr:api:compileJava'.
   > Failed to calculate the value of task ':solr:api:compileJava' property 'javaCompiler'.
      > No matching toolchains found for requested specification: {languageVersion=17, vendor=any, implementation=vendor-specific} for LINUX on x86_64.
         > No locally installed toolchains match and toolchain download repositories have not been configured.

There is a way to enable it via gradle.properties. (I have to check how I can make this file change on crave, it's not picking it up.)

# Disable auto JVM provisioning (we don't use toolchains yet but want no surprises).
org.gradle.java.installations.auto-download=false

There is somewhat related discussion regarding the same in alternative-jdk-support.gradle.

solr/gradle/testing/alternative-jdk-support.gradle

Lines 25 to 35 in a4cc580

    
           // This adds support for compiling and testing against a different Java runtime. 
        
           // 
        
           // I failed to set it up leveraging Gradle's toolchains because 
        
           // a toolchain spec is not flexible enough to provide an exact location of the JVM to be used; 
        
           // if you have two identical JVM lang. versions in auto-discovered JVMs, an arbitrary one is used (?). 
        
           // This situation is not uncommon when debugging low-level stuff (hand-compiled JVM binaries). 
        
           // 
        
           // The code below is a workaround using internal gradle classes. It may stop working in the future 
        
           // but for now it's fine. 
        
           JavaInfo jvmGradle = Jvm.current();

dsmiley · 2024-10-11T04:39:10Z

I might have misunderstood Toolchains; if it means having additional JDKs around then I don't think we want to require that. It's sufficient to have only JDK 21 and configure the language level for certain parts.

dsmiley · 2024-10-15T03:58:41Z

As I mentioned, IntelliJ isn't recognizing the language distinction between tests vs non-tests so I recommend simply using the same language per module. Other than that, I suppose it's ready to merge.

At some point (could be now), we should review all the extra options we pass to java when starting Solr to see if they are still sensible. Some may be obsolete.

iamsanjay · 2024-10-19T06:45:52Z

-XX:-UseBiasedLocking removed in JDK 21. And, due to which Benchmarking is failing. Will be removing it.

solr/bin/solr.in.sh

solr/solr-ref-guide/modules/upgrade-notes/pages/major-changes-in-solr-10.adoc

epugh

LGTM. I love seeing lots of "workarounds" going away! Java21 is going to let us move to the latest OpenNLP and do some cool things with ONNX for ML models..

github-actions bot added client:solrj tool:build start-scripts tests cat:search cat:index scripts module:scripting labels Aug 30, 2024

iamsanjay commented Aug 30, 2024

View reviewed changes

build.gradle Outdated Show resolved Hide resolved

dsmiley reviewed Oct 7, 2024

View reviewed changes

github-actions bot added cat:cli and removed scripts labels Oct 8, 2024

dsmiley mentioned this pull request Oct 10, 2024

force the use jdk21 through Crave #2498

Closed

7 tasks

github-actions bot added the docker Docker image label Oct 10, 2024

iamsanjay added 12 commits October 18, 2024 11:20

Removed explicit cast of HttpClient to AutoClosable

a23be0d

FloatingPointLiteralPrecision from errorProne task

e3af3ae

merge conflicts

e2a666d

merge conflicts

cc96a91

change java version for docker

fb26967

gradle toolchain configured

24fb33c

revert toolchain

58576c7

set source and target at package level

4bb25de

change documentation to reflect JDK 21

46445cb

tidy

c2888ad

suppress warnings

d95ce4d

remove old garbage collector ref

ea75378

iamsanjay force-pushed the Java_Solr21 branch from 5c721f3 to ea75378 Compare October 18, 2024 06:45

github-actions bot removed dependencies Dependency upgrades module:hadoop-auth module:hdfs prometheus-exporter cat:cloud labels Oct 18, 2024

added changes.txt

268a008

dsmiley reviewed Oct 19, 2024

View reviewed changes

solr/bin/solr.in.sh Outdated Show resolved Hide resolved

dsmiley reviewed Oct 19, 2024

View reviewed changes

solr/solr-ref-guide/modules/upgrade-notes/pages/major-changes-in-solr-10.adoc Outdated Show resolved Hide resolved

iamsanjay added 2 commits October 21, 2024 18:51

Updated doc and removed vm arg UseBiasedLocking

c5a6ce7

Merge branch 'main' into Java_Solr21

eaeedad

dsmiley approved these changes Oct 23, 2024

View reviewed changes

epugh approved these changes Oct 23, 2024

View reviewed changes

iamsanjay added 2 commits October 24, 2024 07:32

merge conflicts

6380a21

changes.txt

33f694c

iamsanjay merged commit d840a3b into apache:main Oct 24, 2024
6 checks passed

HoustonPutman mentioned this pull request Nov 5, 2024

Ability to use a java version >21 #2847

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bump up Java version to 21 #2682

Bump up Java version to 21 #2682

iamsanjay commented Aug 30, 2024

iamsanjay commented Aug 30, 2024 •

edited

Loading

dsmiley left a comment

iamsanjay commented Oct 8, 2024

dsmiley commented Oct 8, 2024

iamsanjay commented Oct 9, 2024

dsmiley commented Oct 10, 2024 •

edited

Loading

dsmiley commented Oct 10, 2024

dsmiley commented Oct 10, 2024

iamsanjay commented Oct 10, 2024

iamsanjay commented Oct 10, 2024

iamsanjay commented Oct 10, 2024

iamsanjay commented Oct 10, 2024

iamsanjay commented Oct 10, 2024 •

edited

Loading

dsmiley commented Oct 10, 2024

iamsanjay commented Oct 11, 2024 •

edited

Loading

dsmiley commented Oct 11, 2024

dsmiley commented Oct 15, 2024

iamsanjay commented Oct 19, 2024

epugh left a comment

Bump up Java version to 21 #2682

Bump up Java version to 21 #2682

Conversation

iamsanjay commented Aug 30, 2024

iamsanjay commented Aug 30, 2024 • edited Loading

dsmiley left a comment

Choose a reason for hiding this comment

iamsanjay commented Oct 8, 2024

dsmiley commented Oct 8, 2024

iamsanjay commented Oct 9, 2024

dsmiley commented Oct 10, 2024 • edited Loading

dsmiley commented Oct 10, 2024

dsmiley commented Oct 10, 2024

iamsanjay commented Oct 10, 2024

However, jar task is still, I think ,running with whatever java_home is

iamsanjay commented Oct 10, 2024

iamsanjay commented Oct 10, 2024

iamsanjay commented Oct 10, 2024

iamsanjay commented Oct 10, 2024 • edited Loading

dsmiley commented Oct 10, 2024

iamsanjay commented Oct 11, 2024 • edited Loading

dsmiley commented Oct 11, 2024

dsmiley commented Oct 15, 2024

iamsanjay commented Oct 19, 2024

epugh left a comment

Choose a reason for hiding this comment

iamsanjay commented Aug 30, 2024 •

edited

Loading

dsmiley commented Oct 10, 2024 •

edited

Loading

iamsanjay commented Oct 10, 2024 •

edited

Loading

iamsanjay commented Oct 11, 2024 •

edited

Loading