After upgrading to Java 17, the third-party dependencies that only run on Java 17 could finally be upgraded: Jetty and Spring. The upgrade was much more complex than upgrading the Java version.

Context and Jakarta

Following the transfer of Java EE from Oracle to the Eclipse Foundation, the project has been rebranded Jakarta EE.
This change also came with a library namespace change from javax.* to jakarta.*.

Before upgrading the software at work to Java 17, it had to compile with Java 11, and thus Jetty 10 and Spring 5 were used as dependencies. Following the namespace change, both Jetty and Spring had to be upgraded at the same time.
This also added complexity to the task.

Jetty

Choosing the libraries

The software was first using the following libraries:

"org.eclipse.jetty:jetty-server:10.0.20"
"org.eclipse.jetty:jetty-servlets:10.0.20"
"org.eclipse.jetty:jetty-servlet:10.0.20"
"org.eclipse.jetty:jetty-webapp:10.0.20"
"org.eclipse.jetty.websocket:websocket-jetty-server:10.0.20"
"javax.servlet:javax.servlet-api:3.1.0"

Following this page, dependencies were upgraded to:

"org.eclipse.jetty.ee10:jetty-ee10-servlet:12.0.16"
"org.eclipse.jetty.ee10.websocket:jetty-ee10-websocket-jetty-server:12.0.16"
"jakarta.servlet:jakarta.servlet-api:6.1.0"

The more problematic library to choose and use was the WebSocket library, as there are several implementations: Standard APIs and Jetty APIs.
As the software was heavily using the previous Jetty-based implementation, it was easier to transition using Jetty APIs (more details below).

Updating the code base

• The easy part was changing the namespace from javax.* to jakarta.*.
  I did not use OpenRewrite or other software, but simply went for a brute-force find and replace, which did the job.

• There were several uses of getComment() on cookies, which were removed. The code was not useful and was simply removed.

WebSocket API

One of the most impactful changes was the rewrite of the WebSocket API.
The listener implementing WebSocketListener was migrated to an annotation-based class.
In Jetty 10, the skeleton of the class was:

// Jetty 10 version
public class MyWebSocket implements WebSocketListener {
  @Override
  public void onWebSocketBinary(byte[] arg0, int arg1, int arg2) {
  }

  @Override
  public void onWebSocketClose(int code, String reason) {
  }

  @Override
  public void onWebSocketConnect(Session session) {
  }

  @Override
  public void onWebSocketError(Throwable arg0) {
  }

  @Override
  public void onWebSocketText(String payload) {
  }
}

It was migrated to:

// Jetty 12 version
@WebSocket
public class MyWebSocket {
  @OnWebSocketClose
  public void onWebSocketClose(int code, String reason) {
  }

  @OnWebSocketOpen
  public void onWebSocketOpen(Session session) {
  }

  @OnWebSocketError
  public void onWebSocketError(Throwable arg0) {
  }

  @OnWebSocketMessage
  public void onWebSocketText(String payload) {
  }
}

And creating the Socket went from (not much changed):

// Jetty 10 version
public class WebSocketController extends JettyWebSocketServlet {
  @Override
  public void configure(JettyWebSocketServletFactory factory) {
    factory.setIdleTimeout(timeout);
    factory.addMapping("/websocket", new JettyWebSocketCreator() {
      @Override
      public Object createWebSocket(JettyServerUpgradeRequest servletUpgradeRequest, JettyServerUpgradeResponse servletUpgradeResponse) {
        // do some other stuff here for XSRF
        return new MyWebSocket();
      }
    });
  }
}

to:

// Jetty 12 version
public class WebSocketController extends JettyWebSocketServlet {
  @Override
  protected void configure(JettyWebSocketServletFactory factory) {
    // Be very idle
    factory.setIdleTimeout(timeout);
    factory.addMapping("/websocket", (servletUpgradeRequest, servletUpgradeResponse) -> {
      // do some other stuff here for XSRF
      return new Socket();
    });
  }
}

In Jetty 10, headers and parameters were available directly via the UpgradableRequest: session.getUpgradeRequest().getParameterMap(). However, this is no longer set in Jetty 12, and it has been changed to ((WebSocketSession) session).getCoreSession().getParameterMap().

Spring

Choosing the libraries

This process was much easier than it was for Jetty. In practice, both libraries were upgraded at the same time. Simply update from:

"org.springframework:spring-core:5.3.39"
"org.springframework:spring-web:5.3.39"
"org.springframework:spring-webmvc:5.3.39"

to

"org.springframework:spring-core:6.2.1"
"org.springframework:spring-web:6.2.1"
"org.springframework:spring-webmvc:6.2.1"

and then…

Upgrading the code base

Spring 6 also depends on the jakarta namespace, which was renamed along with Jetty.

• for file uploads, CommonsMultipartResolver has been removed from Spring and replaced by MultipartResolver through the use of StandardServletMultipartResolver (see docs) and a MultipartConfigElement.
• LocalVariableTableParameterNameDiscoverer was removed in favor of the new StandardReflectionParameterNameDiscoverer
• Spring has changed how parameter names are discovered (see the class above), so the -parameter flag must be added to the compiler, or all code must be updated to specify parameter names explicitly
• Spring has changed the default support for trailing / and non-trailing / paths. It now defaults to false (see doc). Therefore, use setUseTrailingSlashMatch(true) for backward compatibility

Staying backward compatible

Once all the above libraries were upgraded and the code was modified, the software started and functioned correctly.
There are extensive integration tests to ensure that there is no regression between releases, and these tests highlighted a few problems that also needed to be resolved. Because backward compatibility is very important for the software, there is always a trade-off between deprecating features and dropping support, or continuing to support features to ease the migration process.

URI Compliance

Jetty aims to follow RFCs as strictly as possible. When a behavior diverges from the RFC, it can be modified by adding a few UriCompliance.Violation instances. For backward compatibility, the software needs to support many legacy URLs and behaviors. For example:

HttpConfiguration config = new HttpConfiguration();
UriCompliance compliance = UriCompliance.from(
  EnumSet.of(
    // Allow more PATH violations
    UriCompliance.Violation.SUSPICIOUS_PATH_CHARACTERS,
    UriCompliance.Violation.ILLEGAL_PATH_CHARACTERS,
    // Below is from UriCompliance.LEGACY
    UriCompliance.Violation.AMBIGUOUS_PATH_SEGMENT,
    UriCompliance.Violation.AMBIGUOUS_PATH_SEPARATOR,
    UriCompliance.Violation.AMBIGUOUS_PATH_ENCODING,
    UriCompliance.Violation.AMBIGUOUS_EMPTY_SEGMENT,
    UriCompliance.Violation.UTF16_ENCODINGS,
    UriCompliance.Violation.USER_INFO));
config.setUriCompliance(compliance);

However, this was not sufficient to support ambiguous URIs, and the following code was also required:

// This is linked to UriCompliance.LEGACY above to allow 
// AMBIGUOUS_PATH_ENCODING with Servlet 6 for retro-compatibility
// server is org.eclipse.jetty.server.Server
server.getContainedBeans(ServletHandler.class).forEach(h -> 
  h.setDecodeAmbiguousURIs(true));

Empty Segments

Jetty 10 in our software allowed multiple empty segments in a URL. This meant that http://example.org/bba/cc///b would be resolved to http://example.org/bba/cc/b. This is no longer the case with Jetty 12, and it would certainly break some customer workflows (as it broke some tests).

It was fixed by adding a specific handler to rewrite URLs (adapted from the issue above) if empty segments are detected:

public class EmptySegmentRewriteHandler extends Handler.Wrapper {
    public EmptySegmentRewriteHandler(Handler handler) {
        super(handler);
    }

    @Override
    public boolean handle(Request request, Response response, Callback callback) throws Exception {
        if (request.getHttpURI().hasViolation(UriCompliance.Violation.AMBIGUOUS_EMPTY_SEGMENT)) {
            request = rewriteRequest(request);
        }
        return super.handle(request, response, callback);
    }

    Request rewriteRequest(Request request) {
        HttpURI rewrite = rewriteHttpURL(request);
        return Request.serveAs(request, rewrite);
    }

    private HttpURI rewriteHttpURL(Request base) {
        String param = base.getHttpURI().getParam();
        String query = base.getHttpURI().getQuery();
        List<String> segments = Arrays
                .stream(base.getHttpURI().getPath().split("/"))
                .filter(v -> !v.isEmpty())
                .toList();
        String newPath = "/" + StringUtils.join(segments, "/");

        return HttpURI.build(base.getHttpURI(), newPath, param, query).asImmutable();
    }
}

Jetty server connection handling

Between Jetty 10 and Jetty 12, the server connection handling was rewritten. Some integration tests were randomly failing after running for an unpredictable amount of time.
In some cases, the Jetty 10 server was closing the connection by sending a Connection: Close header. Upon receiving this header, the HTTP client (Apache HttpClient 4 in this case) would reopen a new connection when initiating the next request. In Jetty 12, this header is no longer sent; instead, the server simply closes the connection. The HTTP client then fails to query localhost with:

org.apache.http.NoHttpResponseException: 127.0.0.1:41925 failed to respond

The issue could be somehow be addressed by adding a RetryHandler on the HTTP client to retry the connection when a NoHttpResponseException error occurs, but this would not resolve the underlying issue if other clients attempted to connect to the server.

Reproducibility was random, and determining the root cause was quite challenging:

• the first attempt was to enable all DEBUG logs on Jetty to try to understand what was happening
• to further narrow down the issue, a custom version of Jetty was created by adding more logs and debug points to better understand the differences between Jetty 10 and 12
• ultimately, running tcpdump finally helped to identify the underlying problem

Eventually, I was able to reproduce the behavior with a unit test:

package com.example;

import java.io.IOException;
import java.io.OutputStream;
import java.nio.ByteBuffer;
import java.nio.charset.StandardCharsets;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.ExecutionException;

import org.apache.commons.io.IOUtils;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.entity.ContentType;
import org.apache.http.entity.EntityTemplate;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.util.EntityUtils;
import org.eclipse.jetty.client.HttpClient;
import org.eclipse.jetty.client.OutputStreamRequestContent;
import org.eclipse.jetty.io.Content;
import org.eclipse.jetty.server.Handler;
import org.eclipse.jetty.server.Request;
import org.eclipse.jetty.server.Response;
import org.eclipse.jetty.server.Server;
import org.eclipse.jetty.server.ServerConnector;
import org.eclipse.jetty.util.Callback;
import org.junit.jupiter.api.Test;

class JettyServerWithClientTest {
  private static Server createServer() {
    Server server = new Server(5555);
    server.setHandler(new Handler.Abstract() {
      @Override
      public boolean handle(Request request, Response response, Callback callback) throws IOException {
        response.setStatus(200);
        response.getHeaders().add("Content-Type", "application/json;charset=utf-8");
        String content = "{}";
        ByteBuffer buffer = ByteBuffer.wrap(content.getBytes(StandardCharsets.UTF_8));
        // Consuming the request makes the  client works
        // IOUtils.consume(Content.Source.asInputStream(request));
        response.write(true, buffer, callback);
        return true;
      }
    });

  return server;
}

@Test
public void testJettyClient() throws Exception {
  var port = startServer();
  var url = "http://localhost:" + port + "/";

  try (HttpClient httpClient = new HttpClient()) {
      httpClient.start();
      for (int i = 0; i < 1000000; i++) {
        OutputStreamRequestContent content = new OutputStreamRequestContent();
        httpClient.POST(url)
          .body(content)
          .send(result -> {
          });
        try (OutputStream output = content.getOutputStream()) {
          output.write("{}".getBytes());
        }
      }
    }
  }

  private static Integer startServer() throws InterruptedException, ExecutionException {
    CompletableFuture<Integer> future = new CompletableFuture<>();
    var t = new Thread(() -> {
      try {
        var server = createServer();
        server.start();
        var port = ((ServerConnector) server.getConnectors()[0]).getLocalPort();
        future.complete(port);
        server.join();
      } catch (Exception e) {
          throw new RuntimeException(e);
      }
  });
    t.start();
    return future.get();
  }


  @Test
  public void testApacheHttpClient4() throws Exception {
    var port = startServer();

    var httpClient = HttpClients.createDefault();
    var url = "http://localhost:" + port + "/";

    for (int i = 0; i < 1000000; i++) {
      var post = new HttpPost(url);
      var entity = new EntityTemplate(outputStream -> {
          outputStream.write("{}".getBytes());
          outputStream.flush();
      });
      entity.setContentType(ContentType.APPLICATION_JSON.toString());
      post.setEntity(entity);
      var resp = httpClient.execute(post);
      try {
          assert resp.getStatusLine().getStatusCode() == 200;
          assert "{}".equals(IOUtils.toString(resp.getEntity().getContent(), StandardCharsets.UTF_8));
      } finally {
          EntityUtils.consume(resp.getEntity());
      }
    }
  }
}

When using chunked request (HTTP 1.1), Jetty 12 was answering before the request was consumed on the server side. The client did not send the last End of Chunk, and on the next request, Jetty 12 would interpret the connection as invalid and close it. The HTTP client would then fail to connect again and raise the NoHttpResponseException.
It happened because some of our code answered without consuming the request on a health check endpoint, and the response was sent before the request was entirely consumed.

It resulted in this question on Jetty’s Github.

At the end, the fix on the application side was to consume the request for health check endpoints.

Conclusion

Upgrading both such low-level library in an enterprise software was quite a challenge and came with lots of unknowns. As usual, it is not because it compiles that it works.
The upgrade was started quite early in the software release process to ensure there was time to fix all runtime unknowns.

At the end, it was much more complex than updating Java versions, and it took a few weeks of work.

Some resources:

• Jetty: https://jetty.org/docs/jetty/12/programming-guide/migration/11-to-12.html
• Spring: https://github.com/spring-projects/spring-integration/wiki/Spring-Integration-5.x-to-6.0-Migration-Guide