After upgrading to Java 17, the third party dependencies that only run on Java 17 could finally be upgraded: Jetty and Spring. The upgrade was much more complex than upgrading the Java version.
Context and Jakarta
Following the tranfer of Java EE
from Oracle to the Eclipse Foundation, the project has been rebranded Jakarta EE
.
This change also came with a library namespace change from javax.*
to jakarta.*
.
Before upgrading the software at work to Java 17, the software as to compile with Java 11, and thus Jetty 10 and Spring 5 were used as dependencies. Following the namespace change, both Jetty and Spring had to be upgraded at the same time.
This also added complexity to the task.
Jetty
Choosing the libraries
The software was first using the following libraries:
"org.eclipse.jetty:jetty-server:10.0.20"
"org.eclipse.jetty:jetty-servlets:10.0.20"
"org.eclipse.jetty:jetty-servlet:10.0.20"
"org.eclipse.jetty:jetty-webapp:10.0.20"
"org.eclipse.jetty.websocket:websocket-jetty-server:10.0.20"
"javax.servlet:javax.servlet-api:3.1.0"
Following this page, dependencies were upgraded to:
"org.eclipse.jetty.ee10:jetty-ee10-servlet:12.0.16"
"org.eclipse.jetty.ee10.websocket:jetty-ee10-websocket-jetty-server:12.0.16"
"jakarta.servlet:jakarta.servlet-api:6.1.0"
The more problematic library to choose and use was the websocket library, as there are several implementations: Standard APIs and Jetty APIs. As the software was heavily using the previous Jetty-based implementation, it was easer to transition using Jetty APIs (more below).
Updating the code base
- the easy part was changing the namespace from
javax.*
tojakarta.*
.
I didn’t useOpenRewrite
or other software, but just went for a brute force find and replace that did the job. - there was a bunch of
getComments()
on cookies, that were removed. The code was not useful and was just removed
Websocket API
One of the most impacting change was the rewrite of the websocket API. The listener implementing WebSocketListener
was migrated to an Annotated based class.
In Jetty 10, the skeleton of the class was:
// Jetty 10 version
public class MyWebSocket implements WebSocketListener {
@Override
public void onWebSocketBinary(byte[] arg0, int arg1, int arg2) {
}
@Override
public void onWebSocketClose(int code, String reason) {
}
@Override
public void onWebSocketConnect(Session session) {
}
@Override
public void onWebSocketError(Throwable arg0) {
}
@Override
public void onWebSocketText(String payload) {
}
}
It was migrated to:
// Jetty 12 version
@WebSocket
public class MyWebSocket {
@OnWebSocketClose
public void onWebSocketClose(int code, String reason) {
}
@OnWebSocketOpen
public void onWebSocketOpen(Session session) {
}
@OnWebSocketError
public void onWebSocketError(Throwable arg0) {
}
@OnWebSocketMessage
public void onWebSocketText(String payload) {
}
}
And creating the Socket went from (not much changed):
// Jetty 10 version
public class WebSocketController extends JettyWebSocketServlet {
@Override
public void configure(JettyWebSocketServletFactory factory) {
factory.setIdleTimeout(timeout);
factory.addMapping("/websocket", new JettyWebSocketCreator() {
@Override
public Object createWebSocket(JettyServerUpgradeRequest servletUpgradeRequest, JettyServerUpgradeResponse servletUpgradeResponse) {
// do some other stuff here for XSRF
return new MyWebSocket();
}
});
}
}
to:
// Jetty 12 version
public class WebSocketController extends JettyWebSocketServlet {
@Override
protected void configure(JettyWebSocketServletFactory factory) {
// Be very idle
factory.setIdleTimeout(timeout);
factory.addMapping("/websocket", (servletUpgradeRequest, servletUpgradeResponse) -> {
// do some other stuff here for XSRF
return new Socket();
});
}
}
In Jetty 10, headers and parameters were available directly via the UpgradableRequest
: session.getUpgradeRequest().getParameterMap()
. But this is not set anymore in Jetty 12, and this was changed to ((WebSocketSession) session).getCoreSession().getParameterMap()
.
Spring
Choosing the libraries
This was much easier than for Jetty. In practice, both libraries were upgraded at the same time. Just go from:
"org.springframework:spring-core:5.3.39"
"org.springframework:spring-web:5.3.39"
"org.springframework:spring-webmvc:5.3.39"
to
"org.springframework:spring-core:6.2.1"
"org.springframework:spring-web:6.2.1"
"org.springframework:spring-webmvc:6.2.1"
and then…
Upgrading the code base
Spring 6 also depends on the jakarta
namespace, which was renamed with Jetty.
- for file uploads,
CommonsMultipartResolver
has been removed fromSpring
and replaced byMultipartResolver
via the use ofStandardServletMultipartResolver
(see docs), and aMultipartConfigElement
LocalVariableTableParameterNameDiscoverer
was removed in favor of the newStandardReflectionParameterNameDiscoverer
- Spring has changed how name were discovered (the above class): so the
-parameter
flag needs to be added to the compiler or all the code need to be changed to specify the name - Spring has changed the default support for trailing
/
and no trailing/
paths. It now defaults tofalse
(see doc). So for the use ofsetUseTrailingSlashMatch(true)
for backward compatibility
Staying backward compatible
Once all the above libraries were upgraded and the code modified. The software started and worked.
There are extensive integration tests to ensure there is no regression between releases, and it higlighted a few problems that also needed to be resolved. Because backward-compatibility is very important for the software, it is always a trade off between deprecating stuff and dropping support, or still supporting to ease the pain of migrating.
URI Compliance
Jetty tries to follow RFCs as strictly as possible. When a behavior diverges from the RFC, it can be modified by adding a few UriCompliance.Violation
. For backward compatibility the software needs to support a lot of legacy URLs and behaviors. For example:
HttpConfiguration config = new HttpConfiguration();
UriCompliance compliance = UriCompliance.from(
EnumSet.of(
// Allow more PATH violations
UriCompliance.Violation.SUSPICIOUS_PATH_CHARACTERS,
UriCompliance.Violation.ILLEGAL_PATH_CHARACTERS,
// Below is from UriCompliance.LEGACY
UriCompliance.Violation.AMBIGUOUS_PATH_SEGMENT,
UriCompliance.Violation.AMBIGUOUS_PATH_SEPARATOR,
UriCompliance.Violation.AMBIGUOUS_PATH_ENCODING,
UriCompliance.Violation.AMBIGUOUS_EMPTY_SEGMENT,
UriCompliance.Violation.UTF16_ENCODINGS,
UriCompliance.Violation.USER_INFO));
config.setUriCompliance(compliance);
However, this was not enough to support ambiguous URIs, and the following code was also needed:
// This is linked to UriCompliance.LEGACY above to allow
// AMBIGUOUS_PATH_ENCODING with Servlet 6 for retro-compatibility
// server is org.eclipse.jetty.server.Server
server.getContainedBeans(ServletHandler.class).forEach(h ->
h.setDecodeAmbiguousURIs(true));
Empty Segments
Jetty 10 in our software allowed multiple empty segments in a URL. That means that http://example.org/bba/cc///b
would be resolved to http://example.org/bba/cc/b
. This was no longer the case with Jetty 12, and it would surely break some customer workflow (as it broke some tests).
It was fixed by ading a specifing handler to rewrite URLs (adated from the issue above) if empty segments are detected:
public class EmptySegmentRewriteHandler extends Handler.Wrapper {
public EmptySegmentRewriteHandler(Handler handler) {
super(handler);
}
@Override
public boolean handle(Request request, Response response, Callback callback) throws Exception {
if (request.getHttpURI().hasViolation(UriCompliance.Violation.AMBIGUOUS_EMPTY_SEGMENT)) {
request = rewriteRequest(request);
}
return super.handle(request, response, callback);
}
Request rewriteRequest(Request request) {
HttpURI rewrite = rewriteHttpURL(request);
return Request.serveAs(request, rewrite);
}
private HttpURI rewriteHttpURL(Request base) {
String param = base.getHttpURI().getParam();
String query = base.getHttpURI().getQuery();
List<String> segments = Arrays
.stream(base.getHttpURI().getPath().split("/"))
.filter(v -> !v.isEmpty())
.toList();
String newPath = "/" + StringUtils.join(segments, "/");
return HttpURI.build(base.getHttpURI(), newPath, param, query).asImmutable();
}
}
Jetty server connection handling
Between Jetty 10 and Jetty 12, the server connection handling was rewritten. Some integration tests were randomly failing after running for some random time.
In some cases, Jetty 10 server was closing the connection by sending a header Connection: Close
. When receiving it, the Http client (Apache HttpClient 4 here) would then reopen a new connection when initiating the next request.
In Jetty 12, there is no more such header, but the server just decides to close the connection. The Http client just failed to query localhost with:
org.apache.http.NoHttpResponseException: 127.0.0.1:41925 failed to respond
The issue could be somehow fixed by adding a RetryHandler
on the Http client to retry the connection when receiving a NoHttpResponseException
error, but that wouldn’t fix the underlying issue if other cliends wanted to connect to the server.
Reproducibility was random and determining the root cause was quite a pain:
- the first try was enabling all
DEBUG
logs on Jetty to try and understand what happens - trying to narrow the issue, a custom version of Jetty was created by adding more logs and debug points to better understand the differences between 10 and 12
- at the end running
tcpdump
finally helped understand the underlying problem
At the end I could reproduce the behavior with a unit test:
package com.example;
import java.io.IOException;
import java.io.OutputStream;
import java.nio.ByteBuffer;
import java.nio.charset.StandardCharsets;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.ExecutionException;
import org.apache.commons.io.IOUtils;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.entity.ContentType;
import org.apache.http.entity.EntityTemplate;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.util.EntityUtils;
import org.eclipse.jetty.client.HttpClient;
import org.eclipse.jetty.client.OutputStreamRequestContent;
import org.eclipse.jetty.io.Content;
import org.eclipse.jetty.server.Handler;
import org.eclipse.jetty.server.Request;
import org.eclipse.jetty.server.Response;
import org.eclipse.jetty.server.Server;
import org.eclipse.jetty.server.ServerConnector;
import org.eclipse.jetty.util.Callback;
import org.junit.jupiter.api.Test;
class JettyServerWithClientTest {
private static Server createServer() {
Server server = new Server(5555);
server.setHandler(new Handler.Abstract() {
@Override
public boolean handle(Request request, Response response, Callback callback) throws IOException {
response.setStatus(200);
response.getHeaders().add("Content-Type", "application/json;charset=utf-8");
String content = "{}";
ByteBuffer buffer = ByteBuffer.wrap(content.getBytes(StandardCharsets.UTF_8));
// Consuming the request makes the client works
// IOUtils.consume(Content.Source.asInputStream(request));
response.write(true, buffer, callback);
return true;
}
});
return server;
}
@Test
public void testJettyClient() throws Exception {
var port = startServer();
var url = "http://localhost:" + port + "/";
try (HttpClient httpClient = new HttpClient()) {
httpClient.start();
for (int i = 0; i < 1000000; i++) {
OutputStreamRequestContent content = new OutputStreamRequestContent();
httpClient.POST(url)
.body(content)
.send(result -> {
});
try (OutputStream output = content.getOutputStream()) {
output.write("{}".getBytes());
}
}
}
}
private static Integer startServer() throws InterruptedException, ExecutionException {
CompletableFuture<Integer> future = new CompletableFuture<>();
var t = new Thread(() -> {
try {
var server = createServer();
server.start();
var port = ((ServerConnector) server.getConnectors()[0]).getLocalPort();
future.complete(port);
server.join();
} catch (Exception e) {
throw new RuntimeException(e);
}
});
t.start();
return future.get();
}
@Test
public void testApacheHttpClient4() throws Exception {
var port = startServer();
var httpClient = HttpClients.createDefault();
var url = "http://localhost:" + port + "/";
for (int i = 0; i < 1000000; i++) {
var post = new HttpPost(url);
var entity = new EntityTemplate(outputStream -> {
outputStream.write("{}".getBytes());
outputStream.flush();
});
entity.setContentType(ContentType.APPLICATION_JSON.toString());
post.setEntity(entity);
var resp = httpClient.execute(post);
try {
assert resp.getStatusLine().getStatusCode() == 200;
assert "{}".equals(IOUtils.toString(resp.getEntity().getContent(), StandardCharsets.UTF_8));
} finally {
EntityUtils.consume(resp.getEntity());
}
}
}
}
When using chunked request (HTTP 1.1), Jetty 12 was answering before the request was consumed on the server side. The client didn’t send the last End of Chunk, on the next request, Jetty 12 would interpret the connection as invalid and close it. The Http client would then fail to connect again and raise the NoHttpResponseException
.
It happened because some of our code answered without consuming the request on a health check endpoint, and the response was sent before the request was entirely consumed.
It resulted in this question on Jetty’s Github.
At the end, the fix on the application side was to consume the request for healthcheck endpoints.
Conclusion
Upgrading both such low level library in an enterprise software was quite a challenge and came with lots of unknown. As usual, it is not because it compiles that it works.
The upgrade was started quite early in the software release process to ensure there was time to fix all runtime unknowns.
At the end, it was much more complex than updating Java versions, and it took a few weeks of work.
Some resources: