From Java 11 to Java 17+

After moving from Java 8 to Java 11, it was now time this year to move to Java 17 and 21. The transition was much easier than last year.

The software at work compiles with Java 11, and supports running Java 11 an Java 17 as runtimes (LTS versions). This year, it was time to move to compiling with Java 17 and supporting Java 17 and Java 21 as runtimes.

Java 17

Compilation

The compilation part of Java 17 was much more smoother than for Java 11. By using Gradle toolchains, the change was pretty straightforward and it compiled (and ran!) on the first attempt.

Java Modules

When moving to Java 11, I decided to open all Java Modules at runtime to avoid any Illegal Access to third-party Java modules. It worked really well and we had no errors when running the software.

For Java 17, I wanted to try to modularize our code-base and list modules that could be accessed to have a “cleaner” code-base… I quickly rolled back as opening and explictly listing modules (and transitive modules) we relied on was buggy and would not work on long term.

Locales

Some unit tests check locales and number formatting. The JVM had a change the locale used for french number separator.
Our our side, it just meant updating the unicode separator.

Runtime

The only big issue I encountered at runtime was a change of behavior in interrupting a Thread that was not started.
Some of our unit tests were randomly failing on the CI and always working locally (the usual problems we love …). After some digging (thank you print), I could finally reproduce it with the following test code:

    @Test
    public void testInterruptThread() throws Exception {
        CompletableFuture<Boolean> f = new CompletableFuture<>();
        Thread t = new Thread() {
            public void run() {
                f.complete(isInterrupted());
            }
        };
        t.interrupt();
        t.start();
        assertTrue(f.get());
    }

It returns true (== it works) with Java 17, and false (== it fails) with Java 11. There was a change in behavior on interrupting a thread when it is not alive. As a result, the interrupted flag is kept, and a thread was interrupted before starting.
After a few change in our code-base, it was solved… but it could come back again.

Spark support

Our software still supports different versions of CDP, the oldest one being 7.1.7 which runs up to Java 11.
To support that platform we had to:

  • explicitly force the use of the JVM version via:
    • spark.executorEnv.JAVA_HOME: for Spark executors
    • spark.yarn.appMasterEnv.JAVA_HOME: for the Spark driver
  • open a lot of modules (and more) via:
    • spark.executor.extraJavaOptions: for Spark executors
    • spark.driver.extraJavaOptions: for the Spark driver

Test environment and CI

A lot was done last year when moving from Java 8 to Java 11. So there was little to do this time a part from updating a version number here and there. Gradle did the rest.

Java 21

Java 21 is only supported for the runtime, so no compilation issue here.
Support for Java 21 was added a bit later than Java 17 on a second PR as the two could be split.

Runtime

Unit tests

We relied on an old version of Mockito and ByteBuddy which did not support Java 21. Those libraries were updated.

Jar shading and Lucene

Our software heavily uses shaded Jars to prevent clashes with other runtime libraries as it needs to support a wide range of old and new integration and drivers.

One set of shaded libaries is Lucene. When starting the software with Java 21 for the first time it crashed with:

Caused by: java.lang.LinkageError: MemorySegmentIndexInputProvider is missing in Lucene JAR file

The class was present in the shaded Jar, so I did not really understand the problem. After searching a bit, it turns out that Lucene ships a Multi-Release Jar, for Java 19, 20 and 21. The application started correctly with Java 17 as no code for that release was called.

The Maven Shade Plugin is used to shade dependencies, and it was missing some configuration to relocate Multi-Release classes which are located in META-INF/versions of the source Jars. It was “hackily” fixed by adding a relocation pattern for META-INF/versions on the supported Java version:

  <relocations>
    <relocation>
      <pattern>META-INF/versions/21/org.apache.lucene</pattern>
      <shadedPattern>META-INF/versions/21/shaded.org.apache.lucene</shadedPattern>
    </relocation>
  </relocations>

Spark support

The software supports Spark 3, but only Spark 4 supports Java 21. So if someone wants to run Spark, it has to stick a Java 17.

Test environment and CI

Same as for Java 17, updating a few version number via environment variables and a dedicated pipeline for Java 21 did the job.

Conclusion

The migration from Java 11 to Java 17 was much smoother than from Java 8 to Java 11 but still had a few gotchas… Especially some unexpected runtime behavior change.
It is important to start such a migration early in the release process to ensure to have time to catch unexpected problems, when more and more people will start using the development and beta versions of the software.

This year, that migration was started much earlier than before so we were less in a rush to upgrade, and it went on pretty well after a few weeks of testing running our whole test suite multiple times to ensure to catch any weird runtime bugs.