Adopting Gradle lockfiles for a project with more than one contributor
Recently a question was raised in our work Slack on why we don't use lockfiles for Java projects. I've already tried to adopt them some time ago, but it didn't work out, and I (to my shame) don't remember why. So I decided to try again and write down the results this time.
🙋 Does Java need lockfiles?
Lockfiles is a widespread practice in the front-end world. The idea is that your build tool generates a text file that lists all dependencies it uses and their versions. This file is then committed to the repository, and it ensures that all developers and CI use the same versions of dependencies.
In the Java world, lockfiles are quite rare. There is a common notion that Java projects are not as fragile as JavaScript ones and don't need such a level of control. If you don't use dynamic versions (*
, [1.0,2.0)
, or 2.+
) in your dependencies, you must be safe. However, not much less important benefit of lockfiles is that they allow you or some scripts to analyze the full dependency graph of your project.
But even the safety of dependency resolution in the Java world is not entirely true. Gradle has quite a mature dependency resolution algorithm, so if you don't declare dynamic versions, the resolution will be deterministic and reproducible. However, if any of your dependencies declare dynamic transitive dependency, you are no longer safe. Gradle offers a locking dependencies feature to mitigate it. Maven, on the other hand, has a less reliable dependency resolution algorithm, but ironically it does not support locking dependencies natively.
📚 Quickstart
If you look at Gradle documentation, you'll see that locking dependencies is trivial. First, you need to add the dependencyLocking
section to the buildfile.
dependencyLocking {
lockAllConfigurations()
}
If you use a multi-project build, you need to add it to every subproject or just once in the root build.gradle
:
subprojects {
dependencyLocking {
lockAllConfigurations()
}
}
Surprisingly, it changes nothing in your build behavior. By default, Gradle validates only using lockfiles that are actually present. If there are no lockfiles, there will be no validation. To change the behavior to more strict:
dependencyLocking {
lockAllConfigurations()
lockMode = LockMode.STRICT
}
Now your build should fail until you generate lockfiles for the first time. Generating lockfiles is actually not that straightforward. The best way to do it is to copy a code snippet from the Gradle documentation:
tasks.register('resolveDependencyLocks') {
notCompatibleWithConfigurationCache("Filters configurations at execution time")
doFirst {
assert gradle.startParameter.writeDependencyLocks
}
doLast {
configurations.findAll {
// Add any custom filtering on the configurations to be resolved
it.canBeResolved
}.each { it.resolve() }
}
}
and then run:
./gradlew resolveDependencyLocks --write-locks
Now you can commit results and enjoy.
🤕 Problems
The first problem you'll face is that Java developers are not very familiar with lockfiles, so they will probably be confused that dependency updates are no longer working. You can improve developer experience a lot if you limit dependency locks to CI only:
// detects if this is a GitLab environment
if (System.getenv('CI') != null) {
dependencyLocking {
lockAllConfigurations()
lockMode = LockMode.STRICT
}
}
This change has no additional risk, as developers shouldn’t publish build artifacts from local machines. If they have changed one of the dependencies' versions locally, they will probably understand that the build no longer matches the original.
Of course, there is still a source of confusion from the fact that the build that passes locally fails on the CI. How to ensure that developers update lockfiles before committing changes (especially given the fact that not everyone reads documentation)? The first thing I suggest is adding the comment to the relevant code. We use the TOML dependency catalog, so any update to the dependencies are located in the same libs.versions.toml
file, which now starts with:
# before committing changes in this file you need to run `./gradlew resolveDependencyLocks --write-locks` command to update dependency lock file
[versions]
android-json = "0.0.20131108.vaadin1"
antlr = "4.9.3"
aopalliance = "1.0"
archaius = "0.7.7"
asm = "9.5"
...
By the way, you may guess now that when you run ./gradlew resolveDependencyLocks --write-locks
locally, you need dependency locks to be enabled, so it is better to modify the previous Gradle snipped as follows:
if (System.getenv('CI') != null || gradle.startParameter.taskRequests.any({ it.args.contains('resolveDependencyLocks') })) {
dependencyLocking {
lockAllConfigurations()
lockMode = LockMode.STRICT
}
}
The second thing you can do is add ./gradlew resolveDependencyLocks --write-locks
to pre-commit hooks. If you use pre-commit, you can add the following to .pre-commit-config.yaml
:
- id: update-dependency-locks
name: Update Gradle Dependency Lock files
description: This hook updates Gradle Dependency Lock files
entry: scripts/pre-commit/update-dependency-locks.sh
language: system
pass_filenames: false
and update-dependency-locks.sh
is:
#!/usr/bin/env bash
set -e
# OSX GUI apps do not pick up environment variables the same way as Terminal apps, and there are no easy solutions,
# especially as Apple changes the GUI app behavior every release (see https://stackoverflow.com/q/135688/483528).
# As a workaround to allow OSX GUI to work, add this (hopefully harmless) setting here
original_path=$PATH
export PATH=$PATH:/usr/local/bin
# If JAVA_HOME is not set (because of the same problem as above) - hope that sdkman is installed
if [ -z "$JAVA_HOME" ] ; then
export JAVA_HOME=~/.sdkman/candidates/java/current
fi
hook_error=0
./gradlew resolveDependencyLocks --write-locks || hook_error=$?
# reset path to the original value
export PATH=$original_path
exit ${hook_error}
Of course, many developers don't use pre-commit hooks because of their slowness and fragility, so the last possible thing is to fail Gradle build on CI with a more descriptive error message in case of lockfile failure. Unfortunately, the Gradle native error message might be confusing, given that not many engineers are familiar with this feature. We already had a CI job that checks dependencies health using autonomousapps/dependency-analysis-android-gradle-plugin, so I've modified the build script for it a bit:
BUILD_HEALTH_OUTPUT=$(./gradlew buildHealth --refresh-dependencies --no-daemon 2>&1; echo $?)
BUILD_HEALTH_EXIT_CODE=$(echo "$BUILD_HEALTH_OUTPUT" | tail -n1)
BUILD_HEALTH_OUTPUT=$(echo "$BUILD_HEALTH_OUTPUT" | head -n-1 | tee /dev/fd/2)
echo "$BUILD_HEALTH_OUTPUT" | grep -qE 'which is (not )?part of the dependency lock state|which has been forced / substituted to a different version|that satisfies the version constraints' && { echo -e "\033[31;1mDependencies do not match lockfile. Please run ./gradlew resolveDependencyLocks --write-locks or setup pre-commit hook as described in https://core.gpages.io/common-api/setup-dev-environment#pre-commit.\033[0;m"; exit 1; }
if [ "$BUILD_HEALTH_EXIT_CODE" -ne 0 ]; then exit $BUILD_HEALTH_EXIT_CODE; fi
And now, instead of:
you will get the following:
🧽 Spotless
Spotless is a Gradle plugin that helps you enforce code style. The interesting thing is that it generates some dynamic dependency configurations, so if you use Spotless, your lockfiles will probably be polluted with records like:
...
com.diffplug.durian:durian-swt.os:4.2.0=spotless-1177740460,spotless-1615032292
com.diffplug.spotless:spotless-eclipse-base:3.5.2=spotless617803313
com.diffplug.spotless:spotless-eclipse-wtp:3.23.0=spotless617803313
...
com.ibm.icu:icu4j:67.1=spotless617803313
com.ibm.icu:icu4j:72.1=spotless-1177740460
...
Spotless does not publish any artifacts, so it would be much easier to exclude its dependencies from locking:
configurations.all {
if (it.name.startsWith('spotless')) {
resolutionStrategy.deactivateDependencyLocking()
}
The same goes for any plugins that generate dynamic dependencies and do not publish any artifacts (for example, jacoco or japicmp).
⚙️ Renovate
It is natural to expect some problems with Renovate because it deals with dependency versions. Renovate does not have full support for lockfiles, as it updates them only for dependencies that it upgrades but ignores their transitive dependencies. It results in errors like the following:
For their issues tracker, it seems that the way to fix it would be to make Renovate run ./gradlew resolveDependencyLocks --write-locks
after every upgrade by adding to the config:
"postUpgradeTasks": {
"commands": [
"./gradlew resolveDependencyLocks --write-locks"
],
"fileFilters": [
'**/gradle.lockfile'
],
"executionMode": "branch"
},
Note that ./gradlew resolveDependencyLocks --write-locks
command should be allow-listed in the global Renovate config.
🧑⚖️ Conclusion
In the Java world, the risks that lockfiles mitigate are quite low. However, they are generally a good way to make using dynamic dependencies versions not only undesirable but also impossible. The most important benefit is that they can be used for security tools analysis.
However, lockfiles by themselves may make the developer experience worse - some tricks should be used to mitigate it.