How to orchestrate version management for corporate SDK using Gradle platform and version-catalog plugins

Prerequisites:
- Gradle 8.2.1 or higher
- JDK 8 or higher

If you are boring to read just go straight to the working demo:

- https://github.com/AlexOreshkevich/sdk-demo
- https://github.com/AlexOreshkevich/library-demo-a
- https://github.com/AlexOreshkevich/version-catalog

When you have a desire to reuse your corporate code it’s pretty common to organize it as artifacts and put to separate git repositories. That in turn would lead to a pretty painful release and version management process which could be managed in a easy way that I would like to show below.

First of all, let’s create our own version catalog in a separate git project. It could look like that:

build.gradle.kts

import net.researchgate.release.ReleaseExtension

plugins {
    `version-catalog`
    `maven-publish`
    id ("net.researchgate.release") version "3.0.2"
}

group = "com.company"

catalog {
    // declare the aliases, bundles and versions in this block
    versionCatalog {
        version("sdk", rootProject.version.toString())

        version("springBoot", "2.1.1.RELEASE")
        version("springCloud", "Finchley.RELEASE") // replace with Greenwich

        version("junit", "5.3.2")
        version("assertj", "3.24.2")
        version("testcontainers", "1.17.3")

        library("javax-inject", "javax.inject:javax.inject:1")

        library("junit-api", "org.junit.jupiter", "junit-jupiter-api").versionRef("junit")
        library("junit-engine", "org.junit.jupiter", "junit-jupiter-engine").versionRef("junit")
        library("assertj", "org.assertj", "assertj-core").versionRef("assertj")

        bundle("junit", listOf("junit-api", "junit-engine", "assertj"))
    }
}

publishing {
    publications {
        create<MavenPublication>("maven") {
            from(components["versionCatalog"])
        }
    }
    repositories {
        maven {
            name = "myRepository"
            val releasesRepoUrl = "https://repo.company.com/repository/internal"
            val snapshotsRepoUrl = "https://repo.company.com/repository/snapshots"
            url = uri(if (project.version.toString().endsWith("SNAPSHOT")) snapshotsRepoUrl else releasesRepoUrl)
            credentials(PasswordCredentials::class)
        }
    }
}

configure<ReleaseExtension> {
    ignoredSnapshotDependencies.set(listOf("net.researchgate:gradle-release"))
    tagTemplate.set(
            "${property("version")}".replace("-SNAPSHOT", "") + "-release"
    )
    failOnUnversionedFiles.set(false)
    failOnSnapshotDependencies.set(true)
    with(git) {
        requireBranch.set("master")
        pushToRemote.set("origin")
    }
}

settings.gradle.kts

rootProject.name = "version-catalog"

dependencyResolutionManagement {
  repositoriesMode.set(RepositoriesMode.FAIL_ON_PROJECT_REPOS)
  repositories {

    maven {
      url = uri("https://repo.company.com/repository/internal")
      mavenContent {
        releasesOnly()
      }
    }

    exclusiveContent {
      forRepository {
        maven {
          url = uri("https://repo.company.com/repository/snapshots")
          mavenContent {
            snapshotsOnly()
          }
        }
      }
      filter {
        includeGroupByRegex("com\\.company.*")
      }
    }
    mavenCentral()
  }
}

See more at https://docs.gradle.org/current/userguide/platforms.html for using version-catalog plugin.

I used `net.researchgate.release` plugin to simplify release management of catalog itself.

Let’s define our SDK as a separate git repository project next. You can create new one using gradle init task.

settings.gradle.kts

rootProject.name = "sdk"

// loads all modules into composite build without hard coding their names
// https://docs.gradle.org/current/samples/sample_composite_builds_hierarchical_multirepo.html#running_multirepo_app_with_dependencies_from_included_builds
file("modules").listFiles()
    ?.filter { !it.name.startsWith(".") }
    ?.forEach { moduleBuild: File ->
      includeBuild(moduleBuild)
    }

dependencyResolutionManagement {

  repositoriesMode.set(RepositoriesMode.FAIL_ON_PROJECT_REPOS)

  repositories {
    mavenLocal()
    mavenCentral()
    maven {
      url = uri("https://repo.company.com/repository/internal")
      mavenContent {
        releasesOnly()
      }
    }
    maven {
      url = uri("https://repo.company.com/repository/snapshots")
      mavenContent {
        snapshotsOnly()
      }
    }
  }

  versionCatalogs {
    create("libs") {
      from("com.company:version-catalog:DEV-SNAPSHOT")
    }
  }
}

Here we consume just published version catalog obtaining version declarations from there. Let’s see how we can use it:

build.gradle.kts

plugins {
  `java-platform`
  `maven-publish`
}

group = "com.company"
version = libs.versions.sdkDev.get()

dependencies {
  constraints {

    val sdkVersion = libs.versions.sdkDev.get()

    api(libs.javax.inject)
    api("com.company:logging:$sdkVersion")
    api("com.company:core-api:${sdkVersion}")
    // .. more constraints goes in here
  }
}

publishing {
  ...// the same as for version-catalog, omitted
}

tasks.clean {
  gradle.includedBuilds.forEach { build ->
    dependsOn(build.task(":lib:clean"))
  }
}

tasks.register("cleanTest") {
  gradle.includedBuilds.forEach { build ->
    dependsOn(build.task(":lib:cleanTest"))
  }
}

tasks.build {
  gradle.includedBuilds.forEach { build ->
    dependsOn(build.task(":lib:build"))
  }
}

tasks.register("publishModules") {
  gradle.includedBuilds.forEach { build ->
    dependsOn(build.task(":lib:publish"))
  }
}

tasks.register("publishModulesToMavenLocal") {
  gradle.includedBuilds.forEach { build ->
    dependsOn(build.task(":lib:publishToMavenLocal"))
  }
}

New tasks like publishModules helps us to call for publish in all modules simultaniosly. If modules depends on each other, you would have to develop custom release pipeline like /bin/release.sh

# publish SDK itself
./gradlew publish

# publish modules that do not depends on any other corporate modules first
./gradlew :independent-module-A:lib:publish

# publish dependent modules next
./gradlew :depends-on-A:lib:publish
./gradlew :depends-on-A-and-above:lib:publish
...

In order to add new module to SDK you have to do the following:

1. Add new module git repository

git submodule add -b dev git@gitlab.company.com:libs/your-library.git modules/module

2. Modify settings.gradle.kts in order to consume version-catalog plugin:

rootProject.name = "your-library"
include("lib")

dependencyResolutionManagement {

  repositoriesMode.set(RepositoriesMode.FAIL_ON_PROJECT_REPOS)

  repositories {
    mavenLocal()
    mavenCentral()
    // ... your corporate repos goes in here
  }

  versionCatalogs {
    create("libs") {
      from("com.company:version-catalog:DEV-SNAPSHOT")
    }
  }
}

3. Modify build.gradle.kts

// use automatic library version from SDK
version = libs.versions.sdk.get()

dependencies {
  api(platform("com.company:sdk:${libs.versions.sdkDev.get()}"))
  
  implementation("com.company:library")
}

Here is the magic. You don’t have to declare all the versions for your libraries ’cause they are managed from SDK

dependencies {
  constraints {
  ...
  }
}

section (see above).

Now you can manage all your libraries all together in order to release them all simultaneously. Don’t forget to enable in your gradle.properties

org.gradle.parallel=true

in order to speed up the build.

The typical Gitlab CI/CD pipeline would look like that:

variables:
  DOCKER_DRIVER: overlay2
  CI_IMAGE: ${CI_REGISTRY}/build-images/gradle:4-jdk8-docker
  GRADLE_OPTS: "-Dhttps.protocols=TLSv1.2,TLSv1.3 -Dorg.gradle.daemon=false -Dorg.gradle.vfs.watch=false -Dorg.gradle.caching=true -Dorg.gradle.jvmargs=-Xmx12288m -XX:+HeapDumpOnOutOfMemoryError"
  JAVA_OPTS: "-Xms6194m -Xmx12288m"
  REPO_DEPLOYER: $REPO_DEPLOYER

stages:
  - build
  - snapshot

default:
  image: ${CI_IMAGE}
  before_script:
    - git submodule sync --recursive
    - git submodule update --init --remote --merge
  cache:
    key:
      files:
        - /home/gradle/.gradle/wrapper/gradle-wrapper.properties 
    paths:
      - /home/gradle/.gradle/caches/ 
      - /home/gradle/.gradle/notifications/ 
      - /home/gradle/.gradle/wrapper/

build:
  stage: build
  script:
    # because modules depends on each other through artifacts (not via classpath) we need to publish them first
    # the order of publications doesn't matter in here, so we can do that in parallel
    - ./gradlew --refresh-dependencies --gradle-user-home /home/gradle/.gradle/ publishToMavenLocal publishModulesToMavenLocal
    
    # now all components of the SDK has the latest versions of each other and could be properly compiled
    - ./gradlew --refresh-dependencies --build-cache --gradle-user-home /home/gradle/.gradle/ build
  rules:
    - if: $CI_PIPELINE_SOURCE == 'merge_request_event'
      when: never
    - if: $CI_COMMIT_BRANCH == "master"
      when: always

SNAPSHOT:
  stage: snapshot
  script: 
    # here release order should be manually controlled, because parallel publication could break dependency tree
    - ./bin/release.sh 
  needs: ["build"]
  rules:
    - if: $CI_COMMIT_BRANCH == "master"
      when: on_success

The first command

git submodule sync --recursive

is used to sync with modules git settings in .gitmodules file which could be modified (branch could be changed, for example):

[submodule "modules/module"]
 path = modules/module
 url = git@gitlab.company.com:company/libs/your-library.git
 branch = dev
[...] // more submodules goes here

Next command

git submodule update --init --remote --merge

helps to pull the latest changes from modules repositories.

Next, before the main build, there is another magic:

/gradlew --refresh-dependencies publishToMavenLocal publishModulesToMavenLocal

Here we are using mavenLocal() repo in order to get current SDK and all modules, because they depend on each other not in a usual way (classpath which is resolved by Gradle during the build), but via binary (as `jar’s`), that’s why we need to deploy them first to local repo.

If your libraries do not depend on each other, you are free to release them all not via manual `./bin/release.sh` but using just

./gradlew publishModules

That’s it.