At work, we use a complex Gradle build shared across multiple projects. This complexity also introduces some redundancies in how dependencies are computed between different projects. We recently reworked our dependency system to remove those redundancies.

Project setup

The Gradle project uses a multi-project build. Each project declares and uses a set of common and specific dependencies, similar to the following for a sub-project:

dependencies {
    implementation project(path: ":", configuration: "common_libs")
    implementation project(path: ":", configuration: "project_specific_libs")
}

Each configuration then declares the libraries it includes in a parent build.gradle:

configurations {
    common_libs.extendsFrom implementation
    project_specific_libs.extendsFrom implementation
}

dependencies {
    common_libs "commons-io:commons-io:2.15.1"
    project_specific_libs "commons-fileupload:commons-fileupload:1.5"
}

Dependencies would be resolved as follows:

$ ./gradlew dependencies

> Task :dependencies

common_libs
\--- commons-io:commons-io:2.15.1

project_specific_libs
\--- commons-fileupload:commons-fileupload:1.5
     \--- commons-io:commons-io:2.11.0

Fixing duplicate dependencies

Remove transitive dependencies

One wrong way to fix this is to manually declare all transitive dependencies in commons_libs and use the following configuration in Gradle:

configurations {
    common_libs.extendsFrom implementation
    project_specific_libs.extendsFrom implementation
    project_specific_libs.transitive = false
}

Using configuration inheritance

Another way to fix this issue is by using configuration inheritance. Gradle allows configuration extension so that its dependencies are resolved with knowledge of the other configurations.

The main configuration will look like:

configurations {
    common_libs.extendsFrom implementation
    project_specific_libs.extendsFrom common_libs
}

dependencies {
    common_libs "commons-io:commons-io:2.15.1"
    project_specific_libs "commons-fileupload:commons-fileupload:1.5"
}

commons-io:commons-io:2.11.0 will then be replaced by commons-io:commons-io:2.15.1 which is declared in common_libs:

$ ./gradlew dependencies

> Task :dependencies

common_libs
\--- commons-io:commons-io:2.15.1

project_specific_libs
+--- commons-io:commons-io:2.15.1
\--- commons-fileupload:commons-fileupload:1.5
     \--- commons-io:commons-io:2.11.0 -> 2.15.1

Removing duplicate JARs

Using configuration inheritance fixes the original dependency problem. However, the build system at work also copies files from configurations into different directories so that different projects can use different directories in their classpath. Copying is done via specific tasks, such as:

tasks.register("copy-common-libs", Copy) {
    from configurations.common_libs
    into "libs/common/"
}

tasks.register("copy-project-libs", Copy) {
    from configurations.project_specific_libs
    into "libs/project/"
}

The result is having duplicate jars of the same version in the classpath, which is not as bad as before but still a bit annoying:

$ find libs
libs
libs/project
libs/project/commons-io-2.15.1.jar
libs/project/commons-fileupload-1.5.jar
libs/common
libs/common/commons-io-2.15.1.jar

Substracting configurations

Gradle allows subtracting one configuration from another. It is possible to use this feature to remove duplicate dependencies and modify the original copy tasks to use the “stripped” versions:

tasks.register("copy-common-libs", Copy) {
    from configurations.common_libs
    into "libs/common/"
}

tasks.register("copy-project-libs", Copy) {
    from configurations.project_specific_libs_stripped
    into "libs/project/"
}

configurations {
    common_libs.extendsFrom implementation
    project_specific_libs.extendsFrom common_libs
    project_specific_libs_stripped
}

dependencies {
    common_libs "commons-io:commons-io:2.15.1"
    project_specific_libs "commons-fileupload:commons-fileupload:1.5"
    project_specific_libs_stripped configurations.project_specific_libs.minus(configurations.common_libs)
}

We will then only have single versions of each library:

$ find libs 
libs
libs/project
libs/project/commons-fileupload-1.5.jar
libs/common
libs/common/commons-io-2.15.1.jar

Refining multiple libraries detection

Caution on version compatibility

Configuration inheritance resolves the issue of having multiple versions of the same library in the classpath, as long as the dependencies can be reconciled. However, the following configuration would still result in having two versions of commons-io in the classpath:

configurations {
    common_libs.extendsFrom implementation
    project_specific_libs.extendsFrom common_libs
}

dependencies {
    common_libs "commons-io:commons-io:2.10.0"
    project_specific_libs "commons-fileupload:commons-fileupload:1.5"
}

$ ./gradlew dependencies

> Task :dependencies

common_libs
\--- commons-io:commons-io:2.10.0

project_specific_libs
+--- commons-io:commons-io:2.10.0 -> 2.11.0
\--- commons-fileupload:commons-fileupload:1.5
     \--- commons-io:commons-io:2.11.0

Detecting multiple versions of the same library

To address the above problem, we created a task to scan all configurations and detect conflicting JARs:

tasks.register("checkDuplicateRunJars", DefaultTask) {
    doLast {
        List<String> issues = new ArrayList<>();
        getUniqueResolvableConfigurations().each { conf ->
            def parentConfs = conf.extendsFrom
            parentConfs.each { parentConf ->
                if (!parentConf.canBeResolved) {
                    return;
                }
                conf.resolvedConfiguration.resolvedArtifacts.each { artifact ->
                    ModuleVersionIdentifier mvnId = artifact.owner
                    parentConf.resolvedConfiguration.resolvedArtifacts.each { parentArtifact ->
                        ModuleVersionIdentifier parentMvnId = parentArtifact.owner
                        if (mvnId.group.equals(parentMvnId.group) && mvnId.name.equals(parentMvnId.name) && !mvnId.version.equals(parentMvnId.version)
                        ) {
                            issues.add(" - ${conf.name} - ${mvnId.group}:${mvnId.name}:${mvnId.version} <> ${parentMvnId.version} in parent ${parentConf.name}".toString())
                        }
                    }
                }
            }
        }
        if (!issues.isEmpty()) {
            throw new Exception("Multiple versions of the same dependency found\n" + issues.join("\n"))
        }
    }
}

The copy tasks will then depend on that task.
It will find dependencies in all configurations that have the same Maven group and artifact name, but different versions, and throw an exception if any are found.

Cleaning up Dependencies

At work, a product ships with different applications in the same archive and needs to integrate multiple JAR directories. We were sometimes affected by having multiple versions of the same library in different directories, which led to them being loaded differently at compile and runtime. This caused a runtime error, even though it compiled fine (hint: the runtime library had a lower version number).

With the above configuration and setup:

• configuration inheritance
• configuration subtraction
• a dedicated task to detect duplicate libraries with different versions

We ensure that each library is only added once by the product, and the dependency problem is resolved.