From Java 8 to Java 11+

For compatibility reason, the product shipped at work is still compiled with Java 8, but can be run with both Java 8 and Java 11. It was now time to move to a new Java version: compile with Java 11 and support running with Java 11 and Java 17.
For ease of transition, it was decided to use a Tick-tock model, where customers can still use the same version of Java to run two versions of the product. So Java 11 can run both the current version compiled with Java 8 and 11 itself.

Java compatibility

As outline and greatly explained in Kinds of Compatibility, Java supports different kind of compatibilities:

  • source: compile to a lower version, and enforce code to be of that version
  • binary: run bytecode from a lower version, maybe mixed with newer versions, but never of the version above the running Java version. I also include behavioral here.

In theory, it means that:

  • Java 11 runtime can run code compiled with Java 11 and below, so in our case compiled with Java 8 and 11
  • Java 17 runtime can run code compiled with Java 17 and below, so in our case compiled with Java 11

In practice…. there are gotchas…

Java 11

Compilation

The first step into the migration was to try and compile the code base with Java 11. As Gradle is used as the build tool, we can leverage its use toolchains to automatically check if the new JVM version was installed on the system, and download it instead.
This helped ensure that all developers would have their dev environment set up with minimal hassle.

A few other things to make stuff compile correctly:

  • _ is now a reserved keyword, so a few renames were necessary. _ was mainly used in our test suite for mocks or try-with-resources. So the usage was limited
  • some dependencies use Scala 2.11, as per their compatibility matrix, only 2.12.12 supports Java 11. As we also want to support Java 17 as runtime, we decided to drop the support for that scala version
  • JVM arguments for logging have changed, (PrintGCTimeStamps for example). Refer to the transition to migrate them
  • a few scripts here and there having explicit Java versions or path hardcoded were also updated

To ensure developers would not use a wrong Java version to compile the product, a guard was also added at runtime so that if the Java version is 8 (which is now totally unsupported), it would fail.
The product can now be compiled and start, it was easy!

Runtime

Unfortunately we were bitten by some unexpected runtime errors:

Most of the runtime errors were caught by unit tests, others with manual testing.

Test environment and CI

Most of the integration tests run within a Docker image. For ease of use, the same image is run for all versions of our product. It was not a problem when only Java 8 was running, but there is now more versions to support and run.
The image first needed to be updated to install the other runtimes for Java.

A few tweaks were made to the CI pipeline in order to select the good Java version based on the branch and the version of the product to build with. So by default, all order versions would still use Java 8, and the newer code would default to Java 11.

Java 17

All the above problems were related to compilation and bug fixed in Java 9+. When the software was first started with Java 17, it did not even start:

java.lang.reflect.InaccessibleObjectException: Unable to make field private final int java.lang.ProcessImpl.pid accessible: module java.base does not "opens java.lang" to unnamed module @145eaa29
	at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:354)
	at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:297)
	at java.base/java.lang.reflect.Field.checkCanSetAccessible(Field.java:178)
	at java.base/java.lang.reflect.Field.setAccessible(Field.java:172)

Java Modules

It was my first real encounter with the Java Platform Module System. And I learned that it was now “strict”.
To sum up, the JVM is now separated in modules, and each module declares what it can access and what it exports explicitly. If some code tries to access a part that is not explitly declared, there will be a runtime error like above.
For compatibility reasons with previous Java versions, all jars in the classpath that are not modules are all put into an “Unnamed” module.

A JVM flag existed, --illegal-access=permit, up to Java 16 to allow access from all modules to all modules. However it was removed from Java 17. As a result, the list of all modules that our code (and third party code!) needs to access needs to be listed explictly as a flag of the JVM via --add-opens.
Because I felt that opening each module one by one when errors occured would not be very safe for the next release, I looked for a way to open all of them at runtime.

I stumbled upon https://github.com/burningwave/core, which allows opening modules, but I did not want to add another dependency to our code base only for that. Fortunately, I found another dedicated class that does the job really well, just by calling the internal method.
A small utility method was added to open all modules the the “Unnamed” one in our code base, and the job was done! … until it crashed again.

When loading a JDBC Snowflake driver, we still got the an illegal access exception:

Caused by: java.lang.reflect.InaccessibleObjectException: Unable to make field long java.nio.Buffer.address accessible: module java.base does not "opens java.nio" to unnamed module @40f9161a
        at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:357)
        at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:297)
        at java.base/java.lang.reflect.Field.checkCanSetAccessible(Field.java:177)
        at java.base/java.lang.reflect.Field.setAccessible(Field.java:171)
        at net.snowflake.client.jdbc.internal.apache.arrow.memory.util.MemoryUtil.<clinit>(MemoryUtil.java:87)

As it is explained in this article

Class Loaders

Our product connects to a lot of external third-party databases. To do that, JDBC drivers are used and loaded into a dedicated class loader so as not to pollute the other classes. However, as I learnt with this error, class loader come with their own unnamed module…

The “fix” to open all modules at runtime only works for the “main” Unnamed module (ClassLoader.getSystemClassLoader().getUnnamedModule()) did not work for Class Loaders. Two choices by then:

  • call the code to open all modules every time when creating a new class loader, this costly and we have to dig through the code base
  • add flags to the JVM startup that allows accessing restricted modules for all code and class loaders, but an explicit list is still required

As it will be lots of trouble and not very future proof to call the code for every Class Loader in our code base, it was decided to add the flags to the JVM. We will then gradually move the the Java Module System for a future next release, and then remove the call to open all modules for that release, when our tests have more time to catch potential illegal access.

Note that there is also ClassLoader.getPlatformClassLoader().getUnnamedModule().

Runtime compilation

In our code base, one feature requires compiling code at runtime, to make use of Snowflake Java UDF. This is done by calling the Java ToolProvider getSystemJavaCompiler() method.
However, as stated in the Snowflake documentation, Java 17 is in prevew and only Java 11 is really supported. When the software is running with Java 17, the System Java Compiler called will be the one from Java 17, that will compile to a Java 17 bytecode by default, which is then incompatible with Snowflake.

Fortunatly, Java can compile to a lower version just by adding the -target or --release flag. So we just added --release to the compiler call.

Test environment and CI

Due to the Java Module System and all surprises coming with it, we went further with the CI.
Everythng was already moved to Java 11, but we added more Java runtime options, to be able to choose Java 17, 11 and 8 (for older supported releases). It will then be useful for the next Java versions (21+).

We then moved the default runtime to Java 17, as we believe it will catch more problems. So now our CI is a mix of Java 17 and Java 11 for some jobs.

The same was done for our Java unit tests, that are still compiled with Java 11, but can now be run with Java 11 and 17.

As most developers use Java 11 by default, there is a pretty good coverage of both Java versions between developers and tests environments.

Remaining parts

Of course there were lots of small parts that also needed checking, upgrading.

Runtime infrastructure

The software also runs on cloud and the different cloud providers. Runtime cloud images were also updated with the new Java version, they were directly upgraded to Java 17.
This is inline with moving all our tests with Java 17 by default.

Support for Spark 2 was also dropped in the process as it was incompatible with the newest Java versions.

Plugins

The software also has extensive supports for plugins, written in a variety of languages, mainly Python. But Java is also a supported language for some part of the stack.
Each plugin was tested with both Java runtime versions to ensure no regression were found.

In the process, we started reviewing how plugins are built to ensure they could also target different Java versions. That is still an ongoing task.

Conclusion

The migration from Java 8 to Java 11 and 17 was much more painful than expected. I thought most of the problems would come from Java 11 and the tooling, compilation around it, but the most painful parts were the JDK bug fixes and the Java Module System, for which we really not prepared.
The transition task involved a few developers, but also part of the QA team to integrate the new Java versions and make sure everything can be parametrized, so thanks to all fo them!

In the long term, our software will move to the Java Module System so as to better export its interfaces, list what needs to be accessible and be more compliant with modern Java practices.

Also have fun skimming through the schemas in The State of the Module System!