Upgrading Jetty and Spring

After upgrading to Java 17, the third party dependencies that only run on Java 17 could finally be upgraded: Jetty and Spring. The upgrade was much more complex than upgrading the Java version.

Context and Jakarta

Following the tranfer of Java EE from Oracle to the Eclipse Foundation, the project has been rebranded Jakarta EE.
This change also came with a library namespace change from javax.* to jakarta.*.

Before upgrading the software at work to Java 17, the software as to compile with Java 11, and thus Jetty 10 and Spring 5 were used as dependencies. Following the namespace change, both Jetty and Spring had to be upgraded at the same time.
This also added complexity to the task.

Jetty

Choosing the libraries

The software was first using the following libraries:

"org.eclipse.jetty:jetty-server:10.0.20"
"org.eclipse.jetty:jetty-servlets:10.0.20"
"org.eclipse.jetty:jetty-servlet:10.0.20"
"org.eclipse.jetty:jetty-webapp:10.0.20"
"org.eclipse.jetty.websocket:websocket-jetty-server:10.0.20"
"javax.servlet:javax.servlet-api:3.1.0"

Following this page, dependencies were upgraded to:

"org.eclipse.jetty.ee10:jetty-ee10-servlet:12.0.16"
"org.eclipse.jetty.ee10.websocket:jetty-ee10-websocket-jetty-server:12.0.16"
"jakarta.servlet:jakarta.servlet-api:6.1.0"

The more problematic library to choose and use was the websocket library, as there are several implementations: Standard APIs and Jetty APIs. As the software was heavily using the previous Jetty-based implementation, it was easer to transition using Jetty APIs (more below).

Updating the code base

  • the easy part was changing the namespace from javax.* to jakarta.*.
    I didn’t use OpenRewrite or other software, but just went for a brute force find and replace that did the job.
  • there was a bunch of getComments() on cookies, that were removed. The code was not useful and was just removed

Websocket API

One of the most impacting change was the rewrite of the websocket API. The listener implementing WebSocketListener was migrated to an Annotated based class. In Jetty 10, the skeleton of the class was:

// Jetty 10 version
public class MyWebSocket implements WebSocketListener {
  @Override
  public void onWebSocketBinary(byte[] arg0, int arg1, int arg2) {
  }

  @Override
  public void onWebSocketClose(int code, String reason) {
  }

  @Override
  public void onWebSocketConnect(Session session) {
  }

  @Override
  public void onWebSocketError(Throwable arg0) {
  }

  @Override
  public void onWebSocketText(String payload) {
  }
}

It was migrated to:

// Jetty 12 version
@WebSocket
public class MyWebSocket {
  @OnWebSocketClose
  public void onWebSocketClose(int code, String reason) {
  }

  @OnWebSocketOpen
  public void onWebSocketOpen(Session session) {
  }

  @OnWebSocketError
  public void onWebSocketError(Throwable arg0) {
  }

  @OnWebSocketMessage
  public void onWebSocketText(String payload) {
  }
}

And creating the Socket went from (not much changed):

// Jetty 10 version
public class WebSocketController extends JettyWebSocketServlet {
  @Override
  public void configure(JettyWebSocketServletFactory factory) {
    factory.setIdleTimeout(timeout);
    factory.addMapping("/websocket", new JettyWebSocketCreator() {
      @Override
      public Object createWebSocket(JettyServerUpgradeRequest servletUpgradeRequest, JettyServerUpgradeResponse servletUpgradeResponse) {
        // do some other stuff here for XSRF
        return new MyWebSocket();
      }
    });
  }
}

to:

// Jetty 12 version
public class WebSocketController extends JettyWebSocketServlet {
  @Override
  protected void configure(JettyWebSocketServletFactory factory) {
    // Be very idle
    factory.setIdleTimeout(timeout);
    factory.addMapping("/websocket", (servletUpgradeRequest, servletUpgradeResponse) -> {
      // do some other stuff here for XSRF
      return new Socket();
    });
  }
}

In Jetty 10, headers and parameters were available directly via the UpgradableRequest: session.getUpgradeRequest().getParameterMap(). But this is not set anymore in Jetty 12, and this was changed to ((WebSocketSession) session).getCoreSession().getParameterMap().

Spring

Choosing the libraries

This was much easier than for Jetty. In practice, both libraries were upgraded at the same time. Just go from:

"org.springframework:spring-core:5.3.39"
"org.springframework:spring-web:5.3.39"
"org.springframework:spring-webmvc:5.3.39"

to

"org.springframework:spring-core:6.2.1"
"org.springframework:spring-web:6.2.1"
"org.springframework:spring-webmvc:6.2.1"

and then…

Upgrading the code base

Spring 6 also depends on the jakarta namespace, which was renamed with Jetty.

  • for file uploads, CommonsMultipartResolver has been removed from Spring and replaced by MultipartResolver via the use of StandardServletMultipartResolver (see docs), and a MultipartConfigElement
  • LocalVariableTableParameterNameDiscoverer was removed in favor of the new StandardReflectionParameterNameDiscoverer
  • Spring has changed how name were discovered (the above class): so the -parameter flag needs to be added to the compiler or all the code need to be changed to specify the name
  • Spring has changed the default support for trailing / and no trailing / paths. It now defaults to false (see doc). So for the use of setUseTrailingSlashMatch(true) for backward compatibility

Staying backward compatible

Once all the above libraries were upgraded and the code modified. The software started and worked.
There are extensive integration tests to ensure there is no regression between releases, and it higlighted a few problems that also needed to be resolved. Because backward-compatibility is very important for the software, it is always a trade off between deprecating stuff and dropping support, or still supporting to ease the pain of migrating.

URI Compliance

Jetty tries to follow RFCs as strictly as possible. When a behavior diverges from the RFC, it can be modified by adding a few UriCompliance.Violation. For backward compatibility the software needs to support a lot of legacy URLs and behaviors. For example:

HttpConfiguration config = new HttpConfiguration();
UriCompliance compliance = UriCompliance.from(
  EnumSet.of(
    // Allow more PATH violations
    UriCompliance.Violation.SUSPICIOUS_PATH_CHARACTERS,
    UriCompliance.Violation.ILLEGAL_PATH_CHARACTERS,
    // Below is from UriCompliance.LEGACY
    UriCompliance.Violation.AMBIGUOUS_PATH_SEGMENT,
    UriCompliance.Violation.AMBIGUOUS_PATH_SEPARATOR,
    UriCompliance.Violation.AMBIGUOUS_PATH_ENCODING,
    UriCompliance.Violation.AMBIGUOUS_EMPTY_SEGMENT,
    UriCompliance.Violation.UTF16_ENCODINGS,
    UriCompliance.Violation.USER_INFO));
config.setUriCompliance(compliance);

However, this was not enough to support ambiguous URIs, and the following code was also needed:

// This is linked to UriCompliance.LEGACY above to allow 
// AMBIGUOUS_PATH_ENCODING with Servlet 6 for retro-compatibility
// server is org.eclipse.jetty.server.Server
server.getContainedBeans(ServletHandler.class).forEach(h -> 
  h.setDecodeAmbiguousURIs(true));

Empty Segments

Jetty 10 in our software allowed multiple empty segments in a URL. That means that http://example.org/bba/cc///b would be resolved to http://example.org/bba/cc/b. This was no longer the case with Jetty 12, and it would surely break some customer workflow (as it broke some tests).

It was fixed by ading a specifing handler to rewrite URLs (adated from the issue above) if empty segments are detected:

public class EmptySegmentRewriteHandler extends Handler.Wrapper {
    public EmptySegmentRewriteHandler(Handler handler) {
        super(handler);
    }

    @Override
    public boolean handle(Request request, Response response, Callback callback) throws Exception {
        if (request.getHttpURI().hasViolation(UriCompliance.Violation.AMBIGUOUS_EMPTY_SEGMENT)) {
            request = rewriteRequest(request);
        }
        return super.handle(request, response, callback);
    }

    Request rewriteRequest(Request request) {
        HttpURI rewrite = rewriteHttpURL(request);
        return Request.serveAs(request, rewrite);
    }

    private HttpURI rewriteHttpURL(Request base) {
        String param = base.getHttpURI().getParam();
        String query = base.getHttpURI().getQuery();
        List<String> segments = Arrays
                .stream(base.getHttpURI().getPath().split("/"))
                .filter(v -> !v.isEmpty())
                .toList();
        String newPath = "/" + StringUtils.join(segments, "/");

        return HttpURI.build(base.getHttpURI(), newPath, param, query).asImmutable();
    }
}

Jetty server connection handling

Between Jetty 10 and Jetty 12, the server connection handling was rewritten. Some integration tests were randomly failing after running for some random time.

In some cases, Jetty 10 server was closing the connection by sending a header Connection: Close. When receiving it, the Http client (Apache HttpClient 4 here) would then reopen a new connection when initiating the next request. In Jetty 12, there is no more such header, but the server just decides to close the connection. The Http client just failed to query localhost with:

org.apache.http.NoHttpResponseException: 127.0.0.1:41925 failed to respond

The issue could be somehow fixed by adding a RetryHandler on the Http client to retry the connection when receiving a NoHttpResponseException error, but that wouldn’t fix the underlying issue if other cliends wanted to connect to the server.

Reproducibility was random and determining the root cause was quite a pain:

  • the first try was enabling all DEBUG logs on Jetty to try and understand what happens
  • trying to narrow the issue, a custom version of Jetty was created by adding more logs and debug points to better understand the differences between 10 and 12
  • at the end running tcpdump finally helped understand the underlying problem

At the end I could reproduce the behavior with a unit test:

package com.example;

import java.io.IOException;
import java.io.OutputStream;
import java.nio.ByteBuffer;
import java.nio.charset.StandardCharsets;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.ExecutionException;

import org.apache.commons.io.IOUtils;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.entity.ContentType;
import org.apache.http.entity.EntityTemplate;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.util.EntityUtils;
import org.eclipse.jetty.client.HttpClient;
import org.eclipse.jetty.client.OutputStreamRequestContent;
import org.eclipse.jetty.io.Content;
import org.eclipse.jetty.server.Handler;
import org.eclipse.jetty.server.Request;
import org.eclipse.jetty.server.Response;
import org.eclipse.jetty.server.Server;
import org.eclipse.jetty.server.ServerConnector;
import org.eclipse.jetty.util.Callback;
import org.junit.jupiter.api.Test;

class JettyServerWithClientTest {
  private static Server createServer() {
    Server server = new Server(5555);
    server.setHandler(new Handler.Abstract() {
      @Override
      public boolean handle(Request request, Response response, Callback callback) throws IOException {
        response.setStatus(200);
        response.getHeaders().add("Content-Type", "application/json;charset=utf-8");
        String content = "{}";
        ByteBuffer buffer = ByteBuffer.wrap(content.getBytes(StandardCharsets.UTF_8));
        // Consuming the request makes the  client works
        // IOUtils.consume(Content.Source.asInputStream(request));
        response.write(true, buffer, callback);
        return true;
      }
    });

  return server;
}

@Test
public void testJettyClient() throws Exception {
  var port = startServer();
  var url = "http://localhost:" + port + "/";

  try (HttpClient httpClient = new HttpClient()) {
      httpClient.start();
      for (int i = 0; i < 1000000; i++) {
        OutputStreamRequestContent content = new OutputStreamRequestContent();
        httpClient.POST(url)
          .body(content)
          .send(result -> {
          });
        try (OutputStream output = content.getOutputStream()) {
          output.write("{}".getBytes());
        }
      }
    }
  }

  private static Integer startServer() throws InterruptedException, ExecutionException {
    CompletableFuture<Integer> future = new CompletableFuture<>();
    var t = new Thread(() -> {
      try {
        var server = createServer();
        server.start();
        var port = ((ServerConnector) server.getConnectors()[0]).getLocalPort();
        future.complete(port);
        server.join();
      } catch (Exception e) {
          throw new RuntimeException(e);
      }
  });
    t.start();
    return future.get();
  }


  @Test
  public void testApacheHttpClient4() throws Exception {
    var port = startServer();

    var httpClient = HttpClients.createDefault();
    var url = "http://localhost:" + port + "/";

    for (int i = 0; i < 1000000; i++) {
      var post = new HttpPost(url);
      var entity = new EntityTemplate(outputStream -> {
          outputStream.write("{}".getBytes());
          outputStream.flush();
      });
      entity.setContentType(ContentType.APPLICATION_JSON.toString());
      post.setEntity(entity);
      var resp = httpClient.execute(post);
      try {
          assert resp.getStatusLine().getStatusCode() == 200;
          assert "{}".equals(IOUtils.toString(resp.getEntity().getContent(), StandardCharsets.UTF_8));
      } finally {
          EntityUtils.consume(resp.getEntity());
      }
    }
  }
}

When using chunked request (HTTP 1.1), Jetty 12 was answering before the request was consumed on the server side. The client didn’t send the last End of Chunk, on the next request, Jetty 12 would interpret the connection as invalid and close it. The Http client would then fail to connect again and raise the NoHttpResponseException.
It happened because some of our code answered without consuming the request on a health check endpoint, and the response was sent before the request was entirely consumed.

It resulted in this question on Jetty’s Github.

At the end, the fix on the application side was to consume the request for healthcheck endpoints.

Conclusion

Upgrading both such low level library in an enterprise software was quite a challenge and came with lots of unknown. As usual, it is not because it compiles that it works.
The upgrade was started quite early in the software release process to ensure there was time to fix all runtime unknowns.

At the end, it was much more complex than updating Java versions, and it took a few weeks of work.

Some resources: