From JUnit 4 to JUnit 5

Our unit test suite at work was started almost 10 years ago, and is comprised of thousands of tests. A few months ago, there was an initiative to migrate from JUnit 4 to JUnit 5.
Main reasons for the migration were:

  • support for tags, more fine grained that annotations
  • improve test runner, among them much better support for Parameterized tests
  • as a collateral, try reducing the usage of PowerMock which was heavily used

Migration strategy

The code base is comprised of thousands of unit tests. When taking up the task, I never used JUnit 5 before.
After reading a bit of documentation for the different migration steps, I decided to do everything in one go instead of step by step:

  • the JUnit framework provide the JUnit Vintage to run JUnit 4 if needed, but that would mean a mix of test runner in our test suite
  • I did not want two ways of writing tests in the code base with a mix of JUnit 4 and JUnit 5
  • the task seemened doable pretty quickly by using automated tools

Migration

In theory, that was quite simple:

  • add Junit 5 libraries to the build.gradle:
    tests "org.junit.jupiter:junit-jupiter-api:5.9.1"
    tests "org.junit.jupiter:junit-jupiter-engine:5.9.1"
    tests "org.junit.jupiter:junit-jupiter-params:5.9.1"
    tests "org.hamcrest:hamcrest-library:2.2"
    tests "org.mockito:mockito-junit-jupiter:4.11.0" // 5.7.0 requires java 11
    tests "org.mockito:mockito-inline:4.11.0" // 5.7.0 requires java 11
  • remove JUnit 4 libraries:
    tests "junit:junit:4.12"
    tests "org.hamcrest:hamcrest-library:1.3"
    tests "org.mockito:mockito-core:1.10.19"
    tests "org.powermock:powermock-api-mockito:1.6.6"
    tests "org.powermock:powermock-module-junit4:1.6.6"
  • try compiling, look at failing tests… fix them

Because lots of patterns are the same, a few tools could be used to automate the bulk of the migration:

  • using Intellij JUnit 4 to 5 migration inspection. That was a first part that did the bulk of the job
  • a few regexes to substitute:
    • imports, from org.junit.* to org.junit.jupiter.*
    • @RunWith, ExceptionRule and assert* methods
  • we tried using tools like OpenRewrite, but it went out of memory on our code base…
  • but then the rest was done manually to make test pass (it was tedious)

During the migration, I also decided to remove PowerMock, and upgrade our Mockito library to support for mocking static methods. We initially had trouble using PowerMock with JUnit 5, and as newer Mockito supports mocking statics, it was just simpler to upgrade.

Once the main initialization was done, and what could be automated applied, it was time for manual stuff.
A list of tests by import group was then created, and then a list of tasks from the import list. Other developers could then help migrate by taking tasks from that list, to parallelize the job.

Gotchas

Below is a list of main problems that we faced during the migration. Nothing was really tricky, and it was more tedious that anything else. We tried automating as much as possible.
Once a problem was encountered, it was discussed on slack with the best way to fix it. Upon agreement, it was then put in a wiki page so that everybody could use it as a reference for future migration.

Assertions

Failure message arguments have changed: https://junit.org/junit5/docs/current/user-guide/#migrating-from-junit4-failure-message-arguments AssertEquals(“message”, expected, result) became AssertEquals(expected, result, “message”).

A regex was used to invert parameters.

ExceptionRule

@Rule
public ExpectedException thrown = ExpectedException.none();

@Test(expected = ...)

was replaced by assertThrows().

A regex was used to replace the method calls, and then declaration were removed on a second pass.

Parameterized tests

  • @MethodSource was used to migrate tests
  • constructor and private has been removed, and they are now passed as parameters to the tested method
  • a few tests could then be simplified, by merging different classes together

Migrate PowerMock

That was a more tedious part that was hard to automate.
In most cases the code could just be surrounded with a try-with-resources and replace PowerMockito.when with Mockito.when:

  • before
@Test
// some code
PowerMockito.mockStatic(MyClass.class);
PowerMockito.when(MyClass.aStaticMethod()).thenReturn(something);
// some code
  • after
@Test
try (MockedStatic<MyClass> ignored = Mockito.mockStatic(MyClass.class)) {
// some code
Mockito.when(MyClass.aStaticMethod()).thenReturn(something);
// some code
}

Migrating a constructor mock was a bit cumbersome. Here is an example for a Date class:

  • before
// some code
PowerMockito.whenNew(Date.class).withAnyArguments().thenReturn(new Date(1234567L));
  • after
try (MockedConstruction<Date> ignored = Mockito.mockConstruction( // we mock the constructor
    Date.class, // of the Date class
    Mockito.withSettings().defaultAnswer(Answers.CALLS_REAL_METHODS), // with a "real" object (e.g. a spy that calls the real methods of the class)
    (date, context) -> date.setTime(1234567L)) // in order to always set the time of the newly created date
) {
// some code
}

Mockito.any()

Quite a few tests failed without us understanding why. It turns out that Mockito.any() expects a non-null string.

It had to be replaced with Mockito.nullable().
For example: Mockito.nullable(String.class) instead of Mockito.anyString().

System.getEnv()

Some ouf our tests were relying on mocking System.getEnv(), which cannot be statically mocked by Mockito because it is final. A new MockableSystem class has been created in that just wraps the getEnv call to System.

public class MockableSystem {
    public static String getenv(String name) {
        return System.getenv(name);
    }
}

The downside of that, is that our production code now also uses MockableSystem in a few places, but it is acceptable.
The same pattern was applied with Thread, to have a MockableThread.

Validation

This was the easy part:

  • check tests before, that were running with JUnit 4
  • check tests after

Our CI outputs xml files as a result of a test. I just diffed the two lists to ensure all tests were ok.

Conclusion

The migration was more tedious that initially thought. As I estimated it from one week to ten days at most. It was done in two weeks, involving about 3 to 5 people, depending on the other workload. That would be 3 full persons time for the migration.
More than then thousand tests were migrated, comprised of more than thousand files.

All tests were migrated in one go. In the mean time, if new tests were written in JUnit 4, they wouldn’t compile after our modifications, so it was quite easy to spot them and migrate them one by one.

More resources that helped: