Detecting conflicting dependencies with Gradle

At work, we use a complex Gradle build shared accross multiple projects. This complexity also brings some redundancies in the way dependencies are computed between different projects. We recently reworked our dependency system to remove those redundancies.

Project setup

The Gradle project uses a multi-project build. Each project declares and uses a set of common and specific dependencies, like for a sub-project:

subproject/build.gradle
dependencies {
    implementation project(path: ":", configuration: "common_libs")
    implementation project(path: ":", configuration: "project_specific_libs")
}

Each configuration then declares the libraries it includes in a parent build.gradle:

build.gradle
configurations {
    common_libs.extendsFrom implementation
    project_specific_libs.extendsFrom implementation
}

dependencies {
    common_libs "commons-io:commons-io:2.15.1"
    project_specific_libs "commons-fileupload:commons-fileupload:1.5"
}

In the above example, restricted to a single project (but there are more) and a common lib, commons-io-2.15.1.jar is required by common_libs and commons-io-2.11.0.jar is added by project_specific_libs as a transitive dependency.
However common_libs has no knowledge of project_specific_libs, and it is the same for project_specific_libs. So commons-io-2.11.0.jar will be fetched for project_specific_libs, and there is now 2 versions of the same library in the classpath.

Dependencies would be resolved as follow:

$ ./gradlew dependencies

> Task :dependencies

common_libs
\--- commons-io:commons-io:2.15.1

project_specific_libs
\--- commons-fileupload:commons-fileupload:1.5
     \--- commons-io:commons-io:2.11.0

Fixing duplicate dependencies

Remove transitive dependencies

One wrong way to fix it is to declare manually all transitive dependencies in commons_libs and use the following configuration in Gradle:

build.gradle
configurations {
    common_libs.extendsFrom implementation
    project_specific_libs.extendsFrom implementation
    project_specific_libs.transitive = false
}

That way project_specific_libs will only depend on direct / declared dependencies and will ignore transitive ones. This is obviously very tedious and moves the burden of managing dependencies and multiple versions of the transitive libraries to the developer, which is a nightmare.

Using configuration inheritance

Another way to fix it is by using configuration inheritance. Gradle allows configuration extension so that its dependencies are resolved knowing the other configuration.

The main config will look like:

build.gradle
configurations {
    common_libs.extendsFrom implementation
    project_specific_libs.extendsFrom common_libs
}

dependencies {
    common_libs "commons-io:commons-io:2.15.1"
    project_specific_libs "commons-fileupload:commons-fileupload:1.5"
}

commons-io:commons-io:2.11.0 will then be replaced by commons-io:commons-io:2.15.1 which is declared in common_libs:

$ ./gradlew dependencies

> Task :dependencies

common_libs
\--- commons-io:commons-io:2.15.1

project_specific_libs
+--- commons-io:commons-io:2.15.1
\--- commons-fileupload:commons-fileupload:1.5
     \--- commons-io:commons-io:2.11.0 -> 2.15.1

Removing duplicate jars

Using configuration inheritance fixes the original dependency problem. However, the build system at work also copy files from configurations into different directories. So that different projects can use different directories in their classpath. Copying is done via specific tasks like:

build.gradle
tasks.register("copy-common-libs", Copy) {
    from configurations.common_libs
    into "libs/common/"
}

tasks.register("copy-project-libs", Copy) {
    from configurations.project_specific_libs
    into "libs/project/"
}

The result is having duplicate jars of the same version in the classpath, which is not as bad as before but still a bit annoying:

$ find libs
libs
libs/project
libs/project/commons-io-2.15.1.jar
libs/project/commons-fileupload-1.5.jar
libs/common
libs/common/commons-io-2.15.1.jar

Substracting configurations

Gradle allows substracting a configuration from another. It is possible to play with it to remove duplicate dependencies, and to modify original copy tasks to use “stripped” versions:

build.gradle
tasks.register("copy-common-libs", Copy) {
    from configurations.common_libs
    into "libs/common/"
}

tasks.register("copy-project-libs", Copy) {
    from configurations.project_specific_libs_stripped
    into "libs/project/"
}

configurations {
    common_libs.extendsFrom implementation
    project_specific_libs.extendsFrom common_libs
    project_specific_libs_stripped
}

dependencies {
    common_libs "commons-io:commons-io:2.15.1"
    project_specific_libs "commons-fileupload:commons-fileupload:1.5"
    project_specific_libs_stripped configurations.project_specific_libs.minus(configurations.common_libs)
}

We will then only have single versions of each library:

$ find libs 
libs
libs/project
libs/project/commons-fileupload-1.5.jar
libs/common
libs/common/commons-io-2.15.1.jar

Refining multiple libraries detection

Caution on version compatibility

Configuration inheritance fixes the problem of having several versions of the same library in the classpath if they can be resolved, meaning that dependencies need to be compatible. The following configuration would still result in having 2 version of commons-io in the classpath:

build.gradle
configurations {
    common_libs.extendsFrom implementation
    project_specific_libs.extendsFrom common_libs
}

dependencies {
    common_libs "commons-io:commons-io:2.10.0"
    project_specific_libs "commons-fileupload:commons-fileupload:1.5"
}
The dependency tree will have a duplicate because commons-fileupload needs to use at least commons-io-2.11:

$ ./gradlew dependencies

> Task :dependencies

common_libs
\--- commons-io:commons-io:2.10.0

project_specific_libs
+--- commons-io:commons-io:2.10.0 -> 2.11.0
\--- commons-fileupload:commons-fileupload:1.5
     \--- commons-io:commons-io:2.11.0

Detecting multiple versions of the same library

To address the above problem, we created a task to scan all configurations and detect conflicting jars:

build.gradle
tasks.register("checkDuplicateRunJars", DefaultTask) {
    doLast {
        List<String> issues = new ArrayList<>();
        getUniqueResolvableConfigurations().each { conf ->
            def parentConfs = conf.extendsFrom
            parentConfs.each { parentConf ->
                if (!parentConf.canBeResolved) {
                    return;
                }
                conf.resolvedConfiguration.resolvedArtifacts.each { artifact ->
                    ModuleVersionIdentifier mvnId = artifact.owner
                    parentConf.resolvedConfiguration.resolvedArtifacts.each { parentArtifact ->
                        ModuleVersionIdentifier parentMvnId = parentArtifact.owner
                        if (mvnId.group.equals(parentMvnId.group) && mvnId.name.equals(parentMvnId.name) && !mvnId.version.equals(parentMvnId.version)
                        ) {
                            issues.add(" - ${conf.name} - ${mvnId.group}:${mvnId.name}:${mvnId.version} <> ${parentMvnId.version} in parent ${parentConf.name}".toString())
                        }
                    }
                }
            }
        }
        if (!issues.isEmpty()) {
            throw new Exception("Multiple versions of the same dependency found\n" + issues.join("\n"))
        }
    }
}

The copy tasks will then depend on that task.
It will find dependencies in all configurations having the same maven group and artifact name, but different versions, and throw an Exception if any is found.

Cleaning up dependencies

At work, a product ships with different applications into a same archive and needs to integrate multiple jar directories. We were sometimes bitten by having multiple versions of the same library in different directories, and thus loaded differently at compile and runtime. Which led to a runtime error while it compiled fine (hint: the runtime lib had a lower version number).

With the above configuration and setup:

  • configuration inheritance
  • configuration substraction
  • dedicated task to detect duplicate libraries of different versions

we ensure that each library is only added once by product, and the dependecy problem is resolved.