From Java 8 to Java 11+

For compatibility reasons, the product shipped at work is still compiled with Java 8, but can be run with both Java 8 and Java 11. It was now time to move to a new Java version: compile with Java 11 and support running with Java 11 and Java 17.
For ease of transition, it was decided to use a Tick-tock model, where customers can still use the same version of Java to run two versions of the product. So Java 11 can run both the current version compiled with Java 8 and 11 itself.

Java compatibility

As outlined and greatly explained in Kinds of Compatibility, Java supports different kinds of compatibility:

  • source: compile to a lower version, and enforce code to be of that version
  • binary: run bytecode from a lower version, possibly mixed with newer versions, but never of a version above the running Java version. I also include behavioral here

In theory, it means that:

  • Java 11 runtime can run code compiled with Java 11 and below, so in our case compiled with Java 8 and 11
  • Java 17 runtime can run code compiled with Java 17 and below, so in our case compiled with Java 11

In practice… there are gotchas…

Java 11

Compilation

The first step in the migration was to try and compile the code base with Java 11. As Gradle is used as the build tool, we can leverage its toolchains to automatically check if the new JVM version is installed on the system, and download it instead.
This helped ensure that all developers would have their development environment set up with minimal hassle.

A few other things to make the code compile correctly:

  • _ is now a reserved keyword, so a few renames were necessary. _ was mainly used in our test suite for mocks or try-with-resources, so the usage was limited
  • some dependencies use Scala 2.11. As per their compatibility matrix, only 2.12.12 supports Java 11. As we also want to support Java 17 as runtime, we decided to drop support for that Scala version
  • JVM arguments for logging have changed (e.g., PrintGCTimeStamps). Refer to the transition to migrate them.
  • a few scripts had hardcoded Java versions or paths, which were also updated

To ensure developers would not use the wrong Java version to compile the product, a guard was added at runtime to fail if Java 8 is detected (now completely unsupported).
The product can now be compiled and started – it was easy!

Runtime

Unfortunately, we encountered some unexpected runtime errors:

Most runtime errors were caught by unit tests, others by manual testing.

Test environment and CI

Most of the integration tests run within a Docker image. For ease of use, the same image is used for all versions of our product. It was not a problem when only Java 8 was running, but there are now more versions to support and run.
The image first needed to be updated to install the other Java runtimes.

A few tweaks were made to the CI pipeline in order to select the correct Java version based on the branch and the version of the product to build. By default, all older versions still use Java 8, and the newer code defaults to Java 11.

Java 17

All the above problems were related to compilation and bugs fixed in Java 9+. When the software was first started with Java 17, it did not even start:

java.lang.reflect.InaccessibleObjectException: Unable to make field private final int java.lang.ProcessImpl.pid accessible: module java.base does not "opens java.lang" to unnamed module @145eaa29
	at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:354)
	at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:297)
	at java.base/java.lang.reflect.Field.checkCanSetAccessible(Field.java:178)
	at java.base/java.lang.reflect.Field.setAccessible(Field.java:172)

Java Modules

It was my first real encounter with the Java Platform Module System. And I learned that it is now “strict”.
To sum up, the JVM is now separated into modules, and each module declares what it can access and what it exports explicitly. If some code tries to access a part that is not explicitly declared, there will be a runtime error like above.
For compatibility reasons with previous Java versions, all jars in the classpath that are not modules are placed into an “Unnamed” module.

A JVM flag existed, --illegal-access=permit, up to Java 16 to allow access from all modules to all modules. However, it was removed from Java 17. As a result, the list of all modules that our code (and third-party code) needs to access must be listed explicitly as a flag of the JVM via --add-opens.
Because I felt that opening each module one by one when errors occurred would not be very safe for the next release, I looked for a way to open all of them at runtime.

I stumbled upon https://github.com/burningwave/core, which allows opening modules, but I did not want to add another dependency to our code base only for that. Fortunately, I found another dedicated class that does the job really well, just by calling the internal method.
A small utility method was added to open all modules to the “Unnamed” one in our code base, and the job was done – until it crashed again.

When loading a JDBC Snowflake driver, we still encountered an illegal access exception:

Caused by: java.lang.reflect.InaccessibleObjectException: Unable to make field long java.nio.Buffer.address accessible: module java.base does not "opens java.nio" to unnamed module @40f9161a
        at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:357)
        at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:297)
        at java.base/java.lang.reflect.Field.checkCanSetAccessible(Field.java:177)
        at java.base/java.lang.reflect.Field.setAccessible(Field.java:171)
        at net.snowflake.client.jdbc.internal.apache.arrow.memory.util.MemoryUtil.<clinit>(MemoryUtil.java:87)

As explained in this article…

Class Loaders

Our product connects to many external third-party databases. To do that, JDBC drivers are used and loaded into a dedicated class loader so as not to pollute the other classes. However, as I learned with this error, class loaders come with their own unnamed module…

The “fix” to open all modules at runtime only works for the “main” Unnamed module (ClassLoader.getSystemClassLoader().getUnnamedModule()). It does not work for other class loaders. Two choices at that point:

  • call the code to open all modules every time a new class loader is created, which is costly and requires digging through the code base
  • add flags to the JVM startup that allow access to restricted modules for all code and class loaders, but an explicit list is still required

As it would be a lot of trouble and not very future-proof to call the code for every class loader in our code base, it was decided to add the flags to the JVM. We will then gradually move to the Java Module System for a future release, and remove the call to open all modules for that release, once our tests have had more time to catch potential illegal access.

Note that there is also ClassLoader.getPlatformClassLoader().getUnnamedModule().

Runtime Compilation

In our code base, one feature requires compiling code at runtime, to make use of Snowflake Java UDF. This is done by calling the Java ToolProvider getSystemJavaCompiler() method.
However, as stated in the Snowflake documentation, Java 17 is in preview and only Java 11 is officially supported. When the software is running with Java 17, the system Java compiler will be the one from Java 17, which compiles to Java 17 bytecode by default, and is therefore incompatible with Snowflake.

Fortunately, Java can compile to a lower version just by adding the -target or --release flag. So we added --release to the compiler call.

Test Environment and CI

Due to the Java Module System and all the surprises that came with it, we extended the CI.
Everything was already moved to Java 11, but we added more Java runtime options, to be able to choose between Java 17, 11, and 8 (for older supported releases). This will be useful for future Java versions (21+).

We then moved the default runtime to Java 17, as we believe it will catch more problems. So now our CI is a mix of Java 17 and Java 11 for some jobs.

The same was done for our Java unit tests, which are still compiled with Java 11, but can now be run with Java 11 and 17.

As most developers use Java 11 by default, there is a good coverage of both Java versions across developer and test environments.

Remaining Parts

Of course, there were many small parts that also needed checking and upgrading.

Runtime Infrastructure

The software also runs on cloud environments and different cloud providers. The runtime cloud images were updated with the new Java version, and were directly upgraded to Java 17.
This is in line with moving all our tests to Java 17 by default.

Support for Spark 2 was also dropped in the process, as it was incompatible with the newest Java versions.

Plugins

The software also has extensive support for plugins, written in a variety of languages, mainly Python. But Java is also a supported language for some parts of the stack.
Each plugin was tested with both Java runtime versions to ensure no regressions were found.

In the process, we started reviewing how plugins are built, to ensure they can also target different Java versions. That is still an ongoing task.

Conclusion

The migration from Java 8 to Java 11 and 17 was much more painful than expected. I thought most of the problems would come from Java 11 and the tooling and compilation around it, but the most painful parts were the JDK bug fixes and the Java Module System, for which we were really not prepared.
The transition task involved a few developers, but also part of the QA team to integrate the new Java versions and make sure everything could be parametrized, so thanks to all of them!

In the long term, our software will move to the Java Module System to better export its interfaces, list what needs to be accessible, and be more compliant with modern Java practices.

Also, have fun skimming through the schemas in The State of the Module System!