From JUnit 4 to JUnit 5

Our unit test suite at work was started almost 10 years ago, and is comprised of thousands of tests. A few months ago, there was an initiative to migrate from JUnit 4 to JUnit 5.
Main reasons for the migration were:

  • support for tags, more fine-grained than annotations
  • improved test runner, including much better support for Parameterized tests
  • as a collateral benefit, try reducing the usage of PowerMock, which was heavily used

Migration strategy

The code base is comprised of thousands of unit tests. When taking up the task, I had never used JUnit 5 before.
After reading some documentation on the different migration steps, I decided to do everything in one go instead of migrating step by step:

  • the JUnit framework provides the JUnit Vintage engine to run JUnit 4 tests if needed, but that would result in a mix of test runners in our test suite
  • I did not want two ways of writing tests in the code base with a mix of JUnit 4 and JUnit 5
  • the task seemed doable fairly quickly by using automated tools

Migration

In theory, it was quite simple:

  • add JUnit 5 libraries to the build.gradle:
    tests "org.junit.jupiter:junit-jupiter-api:5.9.1"
    tests "org.junit.jupiter:junit-jupiter-engine:5.9.1"
    tests "org.junit.jupiter:junit-jupiter-params:5.9.1"
    tests "org.hamcrest:hamcrest-library:2.2"
    tests "org.mockito:mockito-junit-jupiter:4.11.0" // 5.7.0 requires java 11
    tests "org.mockito:mockito-inline:4.11.0" // 5.7.0 requires java 11
  • remove JUnit 4 libraries:
    tests "junit:junit:4.12"
    tests "org.hamcrest:hamcrest-library:1.3"
    tests "org.mockito:mockito-core:1.10.19"
    tests "org.powermock:powermock-api-mockito:1.6.6"
    tests "org.powermock:powermock-module-junit4:1.6.6"
  • try compiling, look at failing tests… fix them

Because lots of patterns were the same, a few tools could be used to automate the bulk of the migration:

  • using IntelliJ JUnit 4 to 5 migration inspection. That was the first step and did the bulk of the job
  • a few regexes to substitute:
    • imports, from org.junit.* to org.junit.jupiter.*
    • @RunWith, ExpectedException, and assert* methods
  • we tried using tools like OpenRewrite, but it went out of memory on our code base…
  • the rest was then done manually to make tests pass (it was tedious)

During the migration, I also decided to remove PowerMock and upgrade our Mockito library to support mocking static methods. We initially had trouble using PowerMock with JUnit 5, and as newer versions of Mockito support mocking statics, it was simpler to upgrade.

Once the main initialization was done, and what could be automated was applied, it was time for the manual work.
A list of tests by import group was created, and then a list of tasks from that import list. Other developers could then help with the migration by picking tasks from that list, to parallelize the work.

Gotchas

Below is a list of the main problems that we faced during the migration. Nothing was really tricky, and it was more tedious than anything else. We tried automating as much as possible.
Once a problem was encountered, it was discussed on Slack to find the best way to fix it. Upon agreement, it was added to a wiki page so that everybody could use it as a reference for the rest of the migration.

Assertions

Failure message arguments have changed:
https://junit.org/junit5/docs/current/user-guide/#migrating-from-junit4-failure-message-arguments
AssertEquals("message", expected, result) became AssertEquals(expected, result, "message").

A regex was used to invert parameters.

ExceptionRule

@Rule
public ExpectedException thrown = ExpectedException.none();

@Test(expected = ...)

was replaced by assertThrows().

A regex was used to replace the method calls, and then declarations were removed on a second pass.

Parameterized tests

  • @MethodSource was used to migrate tests
  • constructors and private have been removed, and values are now passed as parameters to the tested method
  • a few tests could then be simplified by merging different classes together

Migrate PowerMock

That was a more tedious part that was hard to automate.
In most cases, the code could just be surrounded with a try-with-resources and replace PowerMockito.when with Mockito.when:

  • before
@Test
// some code
PowerMockito.mockStatic(MyClass.class);
PowerMockito.when(MyClass.aStaticMethod()).thenReturn(something);
// some code
  • after
@Test
try (MockedStatic<MyClass> ignored = Mockito.mockStatic(MyClass.class)) {
// some code
Mockito.when(MyClass.aStaticMethod()).thenReturn(something);
// some code
}

Migrating a constructor mock was a bit cumbersome. Here is an example for a Date class:

  • before
// some code
PowerMockito.whenNew(Date.class).withAnyArguments().thenReturn(new Date(1234567L));
  • after
try (MockedConstruction<Date> ignored = Mockito.mockConstruction( // we mock the constructor
    Date.class, // of the Date class
    Mockito.withSettings().defaultAnswer(Answers.CALLS_REAL_METHODS), // with a "real" object (e.g. a spy that calls the real methods of the class)
    (date, context) -> date.setTime(1234567L)) // in order to always set the time of the newly created date
) {
// some code
}

Mockito.any()

Quite a few tests failed without us understanding why. It turns out that Mockito.any() expects a non-null string.

It had to be replaced with Mockito.nullable().
For example: Mockito.nullable(String.class) instead of Mockito.anyString().

System.getEnv()

Some of our tests were relying on mocking System.getEnv(), which cannot be statically mocked by Mockito because it is final.
A new MockableSystem class has been created that just wraps the getEnv call to System.

public class MockableSystem {
    public static String getenv(String name) {
        return System.getenv(name);
    }
}

The downside of that is that our production code now also uses MockableSystem in a few places, but it is acceptable.
The same pattern was applied with Thread, to have a MockableThread.

Validation

This was the easy part:

  • check tests before, that were running with JUnit 4
  • check tests after

Our CI outputs XML files as a result of a test. I just diffed the two lists to ensure all tests were OK.

Conclusion

The migration was more tedious than initially thought. I estimated it would take one week to ten days at most. It was done in two weeks, involving about 3 to 5 people, depending on the other workload. That would be 3 full persons’ time for the migration.
More than ten thousand tests were migrated, comprised of more than one thousand files.

All tests were migrated in one go. In the meantime, if new tests were written in JUnit 4, they would not compile after our modifications, so it was quite easy to spot them and migrate them one by one.

More resources that helped: