Related Documents
Document Properties
Kbid252E41
Last Modified03-Feb-2020
Added to KB08-Oct-2013
Public AccessEveryone
StatusOnline
Doc TypeGuidelines, Concepts & Cookbooks
Product
  • Gradle Tools
  • ICM 7.6
  • ICM 7.7
  • ICM 7.8
  • ICM 7.9
  • ICM 7.10

Concept - Gradle Build Tools

1 Introduction

This document gives an overview about the Gradle-based build tools, that are used to create and publish software components to an artifact repository.

1.1 Glossary


Phrase
Meaning
Version Control System (VCS)

Also known as source control, source code management systems (SCM), or revision control systems (RCS). VCS is a mechanism for keeping multiple versions of your files, so that when you modify a file you can still access the previous revisions.

Artifact Repository

Place, where build and package software components are located. Provide a common interface to a dependency management system.

Code AnalysisProcess to analyze source code to calculate metrics, find bugs, etc.
Continuous Delivery PipelineSometimes called Deployment Pipeline, describes the stages, which code artifacts runs through source to production system.
System ComponentA software package of different code artifacts and files, that have to be deployed together.
System Component SetIs a container for system components, that needs to be build and branched together.
AssemblyAn assembly references one or more system components residing in the same or a configured artifact repository in order to deploy or deliver them together.
Build ProcessCompiles and packages files and code artifacts from a source project to deployable artifacts.
Publish ProcessThe process which transfers the deployable artifacts to a configured artifact repository.
Assembly ProcessThis process combines several system components to an assembly.
Deployment ProcessThis process extracts files and code artifacts from an artifact repository and applies the configuration.
Project Gradle DistributionThis is a customized Gradle distribution with the preconfigured artifact repositories and Gradle plugins.
Gradle PluginA Gradle plugin packages up reusable pieces of build logic, which can be used across many different projects and builds.
Project Gradle PluginThis is a Gradle plugin which contains special corporate respectively project settings.
Corporate PluginThe term is used as a synonym for Project Gradle Plugin.
Gradle Extension ObjectJava Bean compliant class holding configurations for Gradle plugins.
Gradle WrapperThe Gradle Wrapper is the preferred way of starting a Gradle build. The wrapper is a batch script on Windows, and a shell script for other operating systems. When you start a Gradle build via the wrapper, Gradle will be automatically downloaded and used to run the build. See for more information The Gradle Wrapper in the Gradle documentation (2.11, 2.7, 2.3, 2.0, 1.8)
Intershop ClusterA number of hosts of different types serving an Intershop 7.
Cluster NodeOne separately deployable part of an Intershop cluster. A host can run multiple nodes of one Intershop cluster.

1.2 References

2 Build Infrastructure and Environment

This section gives an overview about the servers and their configuration, that are needed to run the Intershop build tools in a continuous integration environment. The following figure describes on a very high-level, how these servers interact with each other as well as where the particular build artifacts are located. Please read the Concept - Continuous Delivery Tools -> Infrastructurebefore, if you are not familiar with the overall concept:

BuildInfrastructure

2.1 Build Environment

The Intershop build tools require some tools and environment variables before they can be executed.

  • A Client of the Version Control System is needed to checkout the sources of the system components to be built. There is no dependency to a particular version control system.

  • The Oracle Java Development Kit (JDK) is required by Gradle to execute the Gradle Wrapper and which is in the same version, which is required by the Intershop Application Server. The executables (java, javac, etc.) of this installation should be in the execution path of the according build shell. It is also possible to set the JAVA_HOME environment variable.
  • Gradle stores caches and configuration in a directory called Gradle User Home . The default location is the folder <OS_USER_HOME>/.gradle . Because the cache in the Gradle user home may consume a lot of disk space, it is recommended to override this default value by setting the environment variable GRADLE_USER_HOME. See also   "Appendix D. Gradle Command Line" of the "Gradle User Guide" .

2.2 Project/Corporate Specific Artifacts and Configuration

Gradle comes with a fresh approach for storing environment specific build configuration. Project/corporate specific build configuration is stored / linked from within the VCS with the sources of the project. This way:

  • Starting a Gradle build with the correct configuration for a developer mostly consists of checking out the source project and running Gradle (no installation or configuration required).
  • Developer and CI server use mostly the same build configuration. This way both builds are kept consistent more easily.

This section describes artifacts required to implement this approach. Intershop provides templates from which you can easily create them for your project, see Cookbook - Setup CI Infrastructure (valid to GradleTools 1.1), resp. Cookbook - Setup CI Infrastructure (valid to GradleTools 2.1).

The approach is based on the Gradle webinar "Standardizing and Administering your Enterprise Build Environment".

Terminology: "project" vs. "corporate"

The terms "corporate Gradle distribution" and "project Gradle distribution" are used interchangeably throughout the documentation. The same applies to "corporate Gradle plugin" and "project Gradle plugin". If multiple development projects share the same CI environment, the prefix "corporate" indicates that they must also share these artifact.

2.2.1 Corporate Gradle Plugin

The corporate Gradle plugin handles all necessary repository configurations for the company or project. It contains the URL of the repository server and the directory layout of the repositories.

Since Gradle Tools 2.0 the corporate Gradle plugin is replaced by:

  • The corporate plugin configuration: a single file corporate.properties wrapped into a jar-file containing all company/project specific details of the repository server.
  • An IntershopPlugin defining repositories based on the conventions established in Cookbook - Setup CI Infrastructure (valid to GradleTools 2.1). These conventions are common enough that it is not necessary for every development project/company to duplicate this code. This IntershopPlugin reads the corporate.properties to find company/project specific details.

2.2.2 Corporate Gradle Distribution

A corporate Gradle distribution consists of:

  • a standard Gradle distribution (as provided by gradle.org) plus and
  • a company specific Gradle init script applying project specific plugins and configuration. One of these plugins is the corporate Gradle plugin as described above.

2.2.3 Gradle Wrapper

The Gradle wrapper is a small Java program that downloads a Gradle distribution and starts it. It is stored in the VCS with every source project and contains a URL to the corporate Gradle distribution. By checking out this wrapper and invoking Gradle through it, the project specific configuration is automatically loaded on every Gradle invocation .

This makes the Gradle wrapper the preferred way of starting a Gradle build.

3 Intershop Build Tools

There are different kinds of Gradle plugins in the component oriented build of Intershop. Most plugins come with a default configuration, so for the most system components it is not necessary to create a special configuration. The Intershop Gradle plugins extend the Gradle DSL with their own extensions.

3.1 Gradle Plugin Overview

3.1.1 Plugins for Managing Build Configuration

Plugins for managing the repository chain, Ivy patterns and versions. These plugins are typically applied by the Project Gradle Distribution.

Plugin NameDescription

com.intershop.build.gradle.plugins.VersioningPlugin

Used to determine the version string of particular system components or assemblies
corporate
(com.intershop.build.plugins.IntershopPlugin) 
Used to configure the artifact repository chain.

3.1.2 Plugins for Different Types of Cartridges

Depending on the contents of a cartridge different plugins must be applied for building it. These plugins cover building most common artifacts in Intershop cartridges.

Plugin NameDescription
java-cartridgeCompiles all Java classes of the project
static-cartridgePackages all files in source folder staticfiles/cartridge/*

3.1.3 Plugins for Special Types of Artifacts

The following plugins cover building less common artifacts.

Plugin NameDescription
ish-jaxb-pluginGenerates Java sources from XSD files located in staticfiles/definition
ish-wsdl-pluginGenerates Java sources from WSDL files located in staticfiles/wsdl
ish-javacc-pluginGenerates Java sources for parsers
ish-isml-pluginGenerates JSP files from ISML files located in staticfiles/cartridge/templates
ish-jsp-pluginGenerates Java source files from JSP files
ish-tloc-pluginProvides tasks for the TLoc tool
ish-sonar-pluginProvides tasks to analyze code as well as upload reports
ish-resourcelist-pluginGenerates the resource lists for ORM and Pipelet classes

4 Source Projects

4.1 System Component Sets

Intershop components are organized in system component sets. A system component set is build as a multi-project build (see Build Lifecycle in Concept - Continous Delivery Tools  ).

The system component set itself is the root project. Its project directory is the system component set directory. Components are direct child projects, their project directories are direct subdirectories of the system component set directory. The Gradle settings file in the component set directory determines which subdirectories should be treated as components.

Each component directory contains a build.gradle file for component specific build configuration. Common configuration for all sub projects is consolidated in the  build.gradle of the system component set directory.

The system component set directory also contains a Gradle wrapper (see above).

4.1.1 Multi-project Layout

The system component set directory contains the following files and directories:

Folder / FileDescription
.settingsConfiguration of Intershop Studio plugins
gradleThe Gradle Wrapper and its configuration
<system component>The system components (cartridge) reside beside the Gradle Wrapper directory.
settings.gradleConfiguration of sub projects and build plugins
gradlew.batWindows start script of Gradle Wrapper
gradlewLinux start script of Gradle Wrapper
gradle.propertiesBuild properties of project including JVM settings of build process, version of project, ...
.projectProject file of Intershop Studio
.classpathClass path configuration of project in Intershop Studio

4.1.2 Multi-project Build Configuration

This configuration is used for all components in a component set. This should reduce the overhead when changes on this level in the configuration are necessary.

<component set>/build.gradle
apply plugin: 'component-set'
 
description 'Components Business Platform'
group = 'com.intershop.set'
subprojects {
    apply plugin: 'intershop-base'
    group = 'com.intershop'
    intershop {
        productName = 'Intershop'
        productCopyrightOwner = 'Intershop Communications'
    }
    
    project.plugins.withType(JavaBasePlugin) {
        sourceCompatibility = 1.7
        targetCompatibility = 1.7
    }
	
	project.tasks.withType(Test) {
		test.ignoreFailures = true
	}
}
buildscript {
    dependencies {
        classpath group: 'com.intershop.build.gradle', name: 'ish-component-plugin'
		classpath group: 'com.intershop.build.gradle', name: 'assembly'
    }
}
Line 1Use Intershop build plugin component-set for building the root project
Line 3Description of the root project / component set
Line 5The following configuration steps will be applied to all sub projects
Line 6

All components of this project are Intershop components. Therefore, the Intershop base plugin will be applied to all sub projects of this multi project.

Line 7All components will be published for the group com.intershop.
Lines 8-11The product name ( productName) and the copyright owner ( productCopyrightOwner) are set inside the Intershop DSL extension.
Lines 13-16The source and the target compatibility is set for all sub projects with an applied JavaBasePlugin. The Intershop plugin applies by default the JavaBasePlugin.
Lines 18-20Failed tests should not stop the build. This is a configuration that should only be used during a migration time.
Lines 22-27

This is dependency configuration for the build script. The ish-component-plugin contains plugins for all Intershop component types.

The version of the components in the component set is stored in the gradle.properties.

<component set>/gradle.properties
# project settings
version = 7.4.6.0
 
# build script version
filter.com.intershop.build.set.cartridge-plugins = 1.0.0.0.+

Properties of the form "filter.*" and "version.*" are interpreted by the Versioning plugin, see section Versioning Plugin.

4.2 System Components

The system components reside in a system component set and are part of a multi-project build process as described above. The Intershop Build Tools are based on the standard plugins delivered with Gradle, e.g., the Java Plugin.

The Intershop Build Tools (Gradle build plugins) require a defined layout of particular source project residing in the version control system. The conventions for folder names and contents have two origins:

  1. Most folder names are like in the Ant based Intershop build easing the migrations.
  2. Additionally, some of the default source folders of the Gradle standard plugins are supported.

4.2.1 Project Layout

The following table gives an overview about the project layout of a system component / cartridge:

FolderDescription
deploymentContains custom Gradle scripts customizing the deployment process
edlContains EDL files used to generate code in a development environment
javasource/<package name>Contains Java code and resource files
src/test/javaContains mere JUnit tests
src/test/resourcesContains the resources for the mere JUnit tests

staticfiles/cartridge/components

Contains the component files of the Component Framework
staticfiles/cartridge/configContains the configuration read by the Configuration Framework
staticfiles/cartridge/extensionsContains the extension files of the Extension Framework
staticfiles/cartridge/impexContains configurations for import and export processes
staticfiles/cartridge/libContains external libraries (jars) as well as files, that need to be deployed unpacked into the lib folder of particular cartridge
staticfiles/cartridge/localizationContains the resource bundles for the localization
staticfiles/cartridge/pipelinesContains the pipelines
staticfiles/cartridge/queriesContains the queries
staticfiles/cartridge/static/[default|<language>]Contains the static files of cartridge like icons, images, JavaScript as well as CSS files
staticfiles/cartridge/templates/ [default|<language>] Contains the ISML templates
staticfiles/cartridge/webformsContains the webforms (see: Concept - Webforms)
staticfiles/cartridge/webservicesContains the WSDD files of Axis 1 webservices
staticfiles/definitionXSD and XJB files used to generate Java code via JAXB
staticfiles/general/[root|<folder name>]Files to be deployed to the the local root directory (IS_HOME) or another special folder
staticfiles/share/sitesStatic files of particular sites (images, import files, CSS, branding, ...)
staticfiles/share/systemConfiguration files, that need to be deployed
staticfiles/share/tlocTLoc files (deprecated)
staticfiles/wsdlWSDL files of webservices (see: Web Service Description Language)
build.gradleBuild configuration of system component including required build plugins, dependencies to other system components, etc
.projectProject file of Intershop Studio
.classpathClass path configuration of project in Intershop Studio

4.2.2 Build Configuration

The plugins for different system component types can be combined for an Intershop Cartridge.

Intershop provides a special version plugin, this plugin sets the correct versions from the filter and versions settings in the gradle.properties.

<sub project>/build.gradle
apply plugin: 'java-cartridge'
apply plugin: 'static-cartridge'

intershop {
    displayName = 'Component - SubProject'
	description = 'This is an example'
}

dependencies {
    compile project(':bc_foundation')
    compile project(':bc_mail')
    compile group: 'com.intershop.platform', name: 'core'
    compile group: 'com.intershop.platform', name: 'orm'
    compile group: 'com.intershop.platform', name: 'tools'
    compile group: 'com.intershop.platform', name: 'cache'
    compile group: 'com.google.guava', name: 'guava'
    compile group: 'org.slf4j', name: 'slf4j-api'
}
Lines 1-2The plugins for a cartridge with Java and static content are applied to this build.
Lines 4-6

There are no special configurations necessary for this example cartridge.

Only the displayName is configured with the Intershop DSL extension.

Line 9This is the standard dependency extension of this Gradle project. See

Chapter Dependency Management Basics of the Gradle User Guide or Chapter Dependency Management.

Lines 10,11Dependencies to components in the same component set
Lines 12-15External dependencies to components of other component sets
Lines 16-17

Since Gradle Tools 2.7:

External dependencies to 3rd party libraries can be declared per component. These dependencies will be aggregated by the assembly.

Note

Compile dependencies of a component are not resolved transitively in the Intershop component oriented build.

5 Published Projects

The publishing process is always a part of the build process. Publishing makes built artifacts available for the usage in other builds and for deployment. In the most cases artifacts are published to shared repositories as part of the CI build. Developers publish artifacts always to a local repository.

An Intershop component consists of more than one artifact. Therefore Intershop uses Ivy Publishing for sharing of artifacts.

ArtifactDescription and Usage
<name>-local-<classifier>.zipPackage for the installation in a directory on a special host. The classifier is only used for packages with platform specific software.
<name>-share.zipPackage for the installation in shared file system. This shared files system is used from all application servers.
<name>-cartridge.zipPackage of static content of an Intershop cartridge
<name>.jarJava archive with Java classes. This is only avaiable if the component contains Java sources or contains Java libraries.
deploy.gradleGradle file with special deployment steps. This artifact is optional for cartridges.
ivy.xmlContains the dependency configuration of a component and meta information (name, description, display name etc)

The Intershop base plugin applies the Gradle "ivy-publish" Plugin.

6 Configuring Dependency Versions

6.1 Dependency Versions in  build.gradle Files

There are two types of dependencies:

  • Dependencies between system components inside the same system component set, i.e., within the same Gradle multiproject build. These are called project dependencies.
  • Dependencies between system components in different system component sets. These are called external dependencies.

During build dependencies must be resolved, i.e., to retrieve jar files and other artifacts to build/deploy. Project dependencies are resolved within the source and target folder of the system component set. External dependencies are resolved from one or more repositories, which may contain different versions of the same component. For external dependencies it must therefore be possible to control which of these versions will be used.

It is possible to write dependency version numbers directly into the  build.gradle file, as described by the Gradle User Guide. This however has several disadvantages:

  1. The  build.gradle file is under source control. The developer cannot easily switch dependencies to a locally build version.
  2. The  build.gradle file cannot be edited automatically, like for automatic integration of newer versions when they become available.
  3. Version numbers must be declared separately for each dependency. There is no way to define a common pool of third party versions and apply them with a single declaration.

Using version expressions, like 1.0.0.+, that resolve dynamically to the latest matching version does help with the first two items. It however introduces a strong instability. Executing a command once and then a second time can lead to different results, if in between a new version was published. Also previous build processes cannot be repeated to diagnose problems.

6.2 Versioning Plugin

The Gradle build tools come with a versioning plugin that is applied by the project Gradle distribution. The versioning plugin allows to specify dependency versions via Gradle project properties.

Build plugin versions

The versioning plugin is not only used to specify versions of compile dependencies. It is also used to specify which version of the build plugins have to be used. Examples are the properties filter.com.intershop.build.set.cartridge-plugins and filter.com.intershop.build.set.extension-plugins.

Scope

The versioning plugin controls version information for all build steps including assembly. However, it does not affect the deployment, whose versioning is managed only by the deployed assembly's ivy descriptor.

The versioning plugin recognizes two different kinds of properties:

6.2.1 Version Properties

Version properties have the syntax: version.<component organization>.<component name> = <version number>, e.g., version.com.intershop.core = 7.4.6.1.20131212121212.

They determine the version of a single component. In the above case all dependencies to core are resolved to version 7.4.6.1.20131212121212.

6.2.2 Filter Properties

Filter properties have the syntax: filter.<assembly organization>.<assembly name> = <version number>, e.g., filter.com.intershop.assembly.intershop7= 7.4.6.1.201312121212.

They determine the version of multiple components at once by specifying an assembly or system component set. In the example above all dependencies to components contained in version 7.4.6.1.201312121212 of the assembly intershop7 will be resolved using the versions given in the assembly's ivy.xml file. For example if intershop7 in version 7.4.6.1.201312121212 contains version 2.2.25.0.20131125224905 of com.intershop:3rd_apache, dependencies to 3rd_apache will resolve to this version. Another example using a component set: Specifying filter.com.customer.set.platform-cartridges = 1.1.0.0-local causes a build against local versions for all system components in this set.

Filter assemblies and their own versions

Filter properties determine a version only for all components contained in the specified assembly/system component set, not of the assembly/system component set itself. If - for any reason - the system component set or assembly itself needs to be resolved as dependency:
Set an additional version property.

Refined behaviour since Gradle Tools 2.1

Starting with Gradle Tools 2.1, filter properties do not apply for their own assembly build process anymore, in order to avoid cyclic reproduction of incorrect version information.

Example: Given a filter property filter.com.customer.assembly.customer-assembly = 1.1.0.0-local, this version filter does not apply for the build of customer-assembly itself. The assembly must therefore provide versioning for all of its contained system components via other means, e.g., via a filter of a component set: filter.com.customer.set.customer-cartridges = 1.1.0.0-local.

6.3 Precedence and Conflicts

In certain situations multiple versions may be selected based on the given version and filter properties. For example, you may set filter.com.intershop.assembly.intershop7= 7.4.6.1.201312121212, containing 3rd_tomcat version 2.2.25.0.20131125224905 and version.com.intershop.3rd_tomcat=2.4.13.0.20141125224905.

The following rules are used to determine which version to choose:

  1. If the version of a dependency is given by a filter and a version property, the version property is used. This allows fine grained overriding of filters.
  2. If two filter assemblies contain the same component in different versions:
    1. If one of the contained component versions ends on '-local' it will be used. This allows easy replacement of components with locally built versions for developers.
    2. Otherwise an exception will be thrown indicating which filters are conflicting in which component versions.

Filter conflicts can be solved by disambiguation. You can either:

  1. Remove one of the filters (which is also possible by setting an empty value for the filter property) or
  2. Specify a version property for the conflicted component. This - by above rule - will be used and settle the conflict.

6.4 Physical Locations

Gradle project properties can be set in different physical locations:

  1. Adding them to the gradle.properties file of a system component set or assembly. This file is checked into source control.
  2. Adding them to the  gradle.properties file in the Gradle user home directory. This file can be freely changed by the developer.
  3. Specifying as command line parameter using the syntax -P<property>=<value>.

If different locations contain different values for the same property, they override each other in this order. That is:

  • Command line parameters win over any gradle.properties entry.
  • The gradle.properties in the Gradle User home win over those in the system component set/assembly.

6.5 Best Practices

  1. Omit versions in dependency declarations in build.gradle file, e.g., write:
    compile group: 'com.intershop.platform', name: 'core'

    instead of
    compile group: 'com.intershop.platform', name: 'core', version: '7.5.0.+'
  1. Use fixed dependency versions, not dependency expressions for version and filter properties. E.g., it is technically possible to specify a filter filter.com.customer.set.shop-cartridges=1.0.0.+ which will always use the latest available build of the branch 1.0.0. This is discouraged. As mentioned in the beginning of this chapter this can lead to surprises and timing issues. Instead always give full version and build numbers.
  2. Check-in all version properties that are unlikely to differ between multiple developers or between developers and the CI server into the gradle.properties file of the system component set and assembly. Especially dependencies to components by other vendors and build and deployment plugins. Developers can always still override them if they have to in their Gradle User Home.
  3. Creating or changing version and filter properties can be automated easily , including updating to the latest available version. Differing from version expressions, this is a controlled action and visible in the file - especially if the file is under source control. Specification of filter properties is mostly automated for the developer, see Cookbook - Gradle Developer Workflow (valid to Gradle Tools 2.7). This way developers do not have to manage them, but they may.

The templates provided for the system component set and assembly already follow these practices. See Cookbook - Setup CI Infrastructure (valid to GradleTools 1.1).

7 Summary

Intershop extends the Gradle DSL for building cartridges and components following Intershop conventions. Most Intershop Gradle plugins come with a default configuration.

Since the Intershop component oriented build only extends Gradle, a project build configuration can be extended with third-party or own Gradle plugins. For special requirements, the build configuration can be adapted and expanded to include additional build steps.

Disclaimer

The information provided in the Knowledge Base may not be applicable to all systems and situations. Intershop Communications will not be liable to any party for any direct or indirect damages resulting from the use of the Customer Support section of the Intershop Corporate Web site, including, without limitation, any lost profits, business interruption, loss of programs or other data on your information handling system.

Customer Support
Knowledge Base
Product Resources
Support Tickets