For more detailed concepts and practical recipes please refer to the documents mentioned in the references section.
The following diagram shows a high level overview of the overall process called continuous delivery pipeline:
It starts with the teams working on software components respectively system components, whose source files are checked in a version control system like subversion or git. Each checkin process triggers an automatic process on a continuous integration server like
which checks the system components out, builds them and runs different tests. The report of this process is delivered to the teams to provide feedback in case of problems. The built system components are published to an artifact repository. The subsequent assembly process combines a number of system components in a specific version and publishes different assembly artifacts. The following steps in the delivery pipeline include automatic deployment to clusters with different purposes, execution of different tests like load test, manual tests as well as pre-production tests. At last the new components are deployed to the production system.
The goal of the continuous delivery process is:
|Version Control System (VCS)|
Also known as source control, source code management systems (SCM), or revision control systems (RCS). VCS is a mechanism for keeping multiple versions of your files, so that when you modify a file you can still access the previous revisions.
Place, where build and package software components are located. Provide a common interface to a dependency management system.
|Code Analysis||Process to analyze source code to calculate metrics, find bugs, etc.|
|Continuous Delivery Pipeline||Sometimes called Deployment Pipeline, describes the stages, which code artifacts runs through source to production system.|
|System Component||A software package of different code artifacts and files, that have to be deployed together.|
|System Component Set||Is a container for system components, that needs to be build and branched together.|
|Assembly||An assembly references one or more system components residing in the same or a configured artifact repository in order to deploy or deliver them together.|
|Build Process||Compiles and packages files and code artifacts from a source project to deployable artifacts.|
|Publish Process||The process which transfers the deployable artifacts to a configured artifact repository.|
|Assembly Process||This process combines several system components to an assembly.|
|Deployment Process||This process extracts files and code artifacts from an artifact repository and applies the configuration.|
|Project Gradle Distribution||This is a customized Gradle distribution with the preconfigured artifact repositories and Gradle plugins.|
|Gradle Plugin||A Gradle plugin packages up reusable pieces of build logic, which can be used across many different projects and builds.|
|Project Gradle Plugin||This is a Gradle plugin which contains special corporate respectively project settings.|
|Corporate Plugin||The term is used as a synonym for Project Gradle Plugin.|
|Gradle Extension Object||Java Bean compliant class holding configurations for Gradle plugins.|
|Gradle Wrapper||The Gradle Wrapper is the preferred way of starting a Gradle build. The wrapper is a batch script on Windows, and a shell script for other operating systems. When you start a Gradle build via the wrapper, Gradle will be automatically downloaded and used to run the build. See for more information The Gradle Wrapper in the Gradle documentation (2.11, 2.7, 2.3, 2.0, 1.8)|
|Intershop Cluster||A number of hosts of different types serving an Intershop 7.|
|Cluster Node||One separately deployable part of an Intershop cluster. A host can run multiple nodes of one Intershop cluster.|
If you are interested in an overview of all other Gradle-related documentation available on Intershop Customer Knowledge Base, please refer to Overview - Build, Assembly and Deployment.
Other infrastructure-related documents:
|Concept - Continuous Delivery Tools (valid to 7.10)|
|Concept - Gradle Assembly Tools|
|Concept - Gradle Build Tools|
|Concept - Gradle Deployment Tools|
|Concept - Planning an Intershop 7 Deployment|
|Cookbook - Custom Fix Recipes (valid to 7.4 CI)|
|Cookbook - Deployment of Solr Search Adapters||7.10|
|Cookbook - Deployment Tools ICM 7.10|
|Cookbook - Deployment Tools ICM 7.8|
|Cookbook - Deployment Tools ICM 7.9|
|Cookbook - Encryption||7.10.31|
|Cookbook - Gradle Assembly Tools||2.11.6|
|Cookbook - Gradle Assembly Tools (valid to 7.8)|
|Cookbook - Gradle Build Tools|
|Cookbook - Gradle Deployment Tools (7.4 CI - ICM 7.7)||7.4.6|
|Cookbook - Gradle Developer Workflow|
|Cookbook - Gradle Developer Workflow (valid to Gradle Tools 2.7)|
|Cookbook - Hotfix Recipes|
|Cookbook - Setup CI Infrastructure||2.11|
|Cookbook - Setup CI Infrastructure (valid to Gradle Tools 2.3)|
|Cookbook - Setup CI Infrastructure (valid to Gradle Tools 2.7)|
|Cookbook - Setup CI Infrastructure (valid to GradleTools 1.1)|
|Cookbook - Setup CI Infrastructure (valid to GradleTools 2.1)|
|Cookbook - Solr Cloud Server||7.10|
|Guide - Deployment Solr Cloud Server||7.9|
|Guide - General Database Setup (valid to 7.10)|
|Guide - Quick Database Setup With DBCA Templates||7.0|
|Guide - Quick Project Setup||7.10|
|Guide - Quick Project Setup (valid to Gradle Tools 2.7)|
|Guide - Quick Project Setup (valid to Gradle Tools 2.11)||7.9|
|Guide - Setup Oracle XE as Intershop Development Database (valid to 7.10)|
|Reference - Gradle Assembly Tools|
|Reference - Gradle Build Tools|
|Reference - Gradle Deployment Tools|
More in-depth concepts:
You may also have a look at the release notes of Intershop's Gradle Tools:
|Public Release Notes - Gradle Tools - Version 1.0|
|Public Release Notes - Gradle Tools - Version 1.1|
|Public Release Notes - Gradle Tools - Version 2.0|
|Public Release Notes - Gradle Tools - Version 2.1|
|Public Release Notes - Gradle Tools - Version 2.11||2.11|
|Public Release Notes - Gradle Tools - Version 2.3|
|Public Release Notes - Gradle Tools - Version 2.7|
The Intershop continuous delivery tools support an environment possessing following infrastructure and running illustrated high level processes:
A version control system is required to keep track of:
The Intershop continuous delivery tools do not depend on any specific version control system (VCS). That is why customer projects are free to choose a suitable VCS, e.g:
The continuous integration server takes the central role to schedule, trigger, execute, monitor, queue and distribute the automated processes in the continuous delivery pipeline. Following main processes are managed:
The Intershop continuous delivery tools also do not depend on the used continuous integration server, because most of them are able to execute arbitrary scripts. It may be easier to use a CI servers supporting the Gradle build system, but is not required. Following CI servers are available:
The main purpose of artifact repositories is serving a number of built components (in Apache Ivy: modules) in one or more versions for a particular dependency management system like Apache Ivy or Maven. There are different types of them, that may provide additional functionality.
There are several repository management servers available, that are used to serve artifacts to development teams and provide a centralized approach to manage the built and downloaded software artifacts. Following artifact repository servers work with the Intershop Continuous Delivery Tools:
Local artifact repositories exist on build servers to save recurring network traffic and accelerate the build processes. They are simple file structures following a format pattern, which is defined in the build configuration.
In development environments they contain only temporarly software artifacts, that do not need to be distributed to other persons and systems and that are used in the local development of new versions of the software components.
Intershop 7 (7.4.x and 7.5.x) releases were distributed via DVD containing a file based artifact repository.
Optionally it is possible to execute static code analysis tools on the CI server. The Intershop continuous delivery tools as well as the Intershop Studio provide an integration to Sonarqube, being able to manage execution, rules and reports for different code analyzer like:
The Sonarqube server serves the code quality rules to development environments as well as CI server, which is responsible to regulary execute the code analyzer as well as upload the reports providing an overview about the current code quality.
The Intershop continuous delivery tools are a number of plugins in the Gradle build system:
Knowledge of these is not assumed. Instead this document gives an introduction to the most relevant concepts of these technologies. Gradle bursts with innovative concepts that may be hard to grasp for a novice, but once understood are very powerful (and also very conclusive). If the following sections seem too dense or too abstract for your preferred way of learning, we recommend to work through the first dozen chapters of the Gradle User Guide. Or you follow the deep links to specific chapters scattered across the following text.
Basic knowledge and some practical experience with Ant or Maven is helpful as they are Gradle's evolutionary roots. We provide comparisons to Maven and Ant where appropriate.
You may also consult other resources to get a basic understanding of the Gradle's concepts, that are also applicable to Intershop's continuous delivery tooling.
Multiple books cover Gradle from a variety of angles, some of which are even available as e-book for free.
There is also a free introductory course on Udacity about Gradle covering both the generic fundamentals, as well as Android development in specific. You may just leave out the Android specifics.
Intershop also provides a Technical Training which deals with the DevOps tasks which may arise in the context of continuous integration. These training courses does not exclusively deal with Gradle, but also with best practice approaches for CI project setups.
For more detailed information please refer to the course Intershop 7 - System Administration (IS7-116).
Be aware that the Intershop Technical Training department offers to book a partial training or flexible tailored training sessions.
If you have further questions do not hesitate to contact Intershop's Technical Training department (firstname.lastname@example.org).
Gradle is an automation framework, primarily aimed at building enterprise software. Gradle tries to combine the flexibility of Ant with the power of conventions known by Maven and adds a lot of revised approaches to building software on top of it. While focusing on building software, it is – similar to Ant – still general purpose enough to be suitable for deployment.
Gradle scripts are written in Groovy, a JVM-based scripting language, instead of XML as in Ant or Maven. Groovy has a shallow learning curve for Java developers - most Java source files are also valid Groovy scripts. Compared to Java Groovy loosens up the syntax and type system, allows dynamic augmentation of classes and adds closures as first class citizens (a more powerful flavor of anonymous Java classes). Because of these features Groovy makes it easy to develop custom domain specific languages (DSLs) that can combine declarative with imperative aspects. Gradle leverages this power introducing its own DSL. Being JVM-based Groovy is fully compatible with other JVM-based languages like Java and the large existing set of Java libraries can be used easily in Groovy.
Gradle generally executes in three phases: Gradle User Guide -> Build Lifecycle:
Projectinstance for each of these projects.
gradlecommand and the current directory. Gradle then executes each of the selected tasks.
Gradle tasks (see Gradle User Guide: Using Tasks and More about Tasks) are objects representing chunks of work to be executed. They make up the user interface of Gradle. If you come from Ant, Gradle tasks blend Ant tasks and targets into a single concept. If you know Maven, Gradle tasks replace Mavens phases and goals. Gradle tasks can have properties that determine their input and output, multiple actions that actually perform the work and dependencies to other tasks.
When starting Gradle you typically supply a list of tasks to be executed. During the configuration phase Gradle builds a graph of these tasks and their dependencies and in the second phase - the execution phase - runs the tasks in order (see: Gradle User Guide: Gradle Command Line).
A project and its sub-objects are typically configured by the build.gradle file (see Gradle User Guide: Writing Build Scripts). It mostly replaces property-files used extensively in Ant. You can still use property-files of course, but you should reduce their usage to a minimum. The base location for configuration is the build script. The additional gradle.properties file should only contain properties which you need to override in different environments (CI, developer).
Generally any piece of Groovy or Java code having access to the
Project object during the configuration phase can configure it. A place to store reusable configuration logic are the Gradle plugins- Java/Groovy classes with a single method
apply expecting a
Project as only parameter. It is easy to write custom plugins, especially to turn any piece of Gradle build script into a plugin.
Project is an extensible object – you may add own sub-objects to provide a custom DSL extension or set additional properties. Gradle's extension properties on projects feel very similar to Ant's properties but are arbitrary objects instead of just strings.
Gradle's core is slim and most of the functionality is provided in theform of plugins. Plugins are available from Gradle's developers but also from third party vendors, following Ant's and Maven's best traditions. They cover building projects in a variety of languages and integrating different tools, like static code analysis or custom code generation. Furthermore, as mentioned above, any existing library in Java-byte-code can easily be included and used without wrapping it up in Gradle specific code first (this is necessary for Ant and Maven, since both use XML as their main language).
Besides Gradle build scripts and plugins Gradle knows two other types of scripts: Settings- and Init scripts. Settings scripts play an important role in multi-project builds (see below). Init scripts are executed before build scripts and can be added to an existing build via command line or by placing them in special folders. They are the place to store environment-specific configuration that should not be stored within the project directly.
Gradle offers a multi-project build feature (see: Gradle User Guide -> Build Lifecycle -> Multi-project builds) that can build multiple projects in a single execution of Gradle. In contrast to Ant's subant-feature for iterating over multiple Ant scripts and just execute them sequentially, Gradle's multi-project however truly integrates multiple builds in any imaginable way (see: Gradle User Guide -> Mutli-project Builds).
The most prominent incarnation of the strong integration are dependencies between projects. When declared they will automatically turn into dependencies between (compile) tasks. This influences ordering of builds, enable parallel builds and make sure that changes across projects are integrated.
In a multi-project build the configuration phase is preceded by the initialization phase. In this phase all projects and their build scripts are determined by configuring a DSL-object of type
Settings. Corresponding to build scripts and
Project objects this is typically done by a Gradle settings script, by convention called settings.gradle. From these multiple
Project objects are created and configured.
Projects in a multi-project build can form arbitrary hierarchies. Each project has a name and a unique path, which is formed by appending all its parents and its own name. The path for the root project is simply ':' and that for each direct child of the root project is ':<project_name>'. The direct children of a parent project with path ':<parent>' have the path ':<parent>:<child>' etc. The same notation is used to identify the tasks of a specific project, e.g. ':<parent_project>:<child_project>:<task>'.
The following figure describes the required properties, that are used to manage the dependencies of a software project to other projects or software components:
First of all, each component defines a name, an organization and a revision (version) string, that are used to organize different built versions of this component in an artifact repository. Second, a configuration of available artifact repositories is required, which defines the lookup order. Third, the configuration section defines different build configurations. At the end the particular dependendies to other projects or built software components are defined in context of a given build configuration.
As described above artifact repositories store artifacts with their Ivy module descriptor from any number of components in any number of revisions. They are used from a dependency management system like Apache Ivy to resolve dependencies between these artifacts.There arefour basic operations available on a repository:
These operations form the main workflow when accessing a repository. The dependency management system wraps logic around these operations to resolve a module including its transitive dependencies, recurring from 3. to 2. This is called transitive resolving.
A single resolve process can work with a list of artifact repositories of different type. If multiple revisions are found that match a given version expression, they are compared and the latest returned.
Common to all repositories is that they use patterns to locate meta-data and artifacts. A pattern is a string containing placeholders for organization, module name, version, artifact name, ext and type. The path (file or URL) where an artifact is stored during publication is determined by taking the pattern and replacing placeholders by concrete values.
Misconfigured patterns often results in unresolved artifacts, because the pattern must match to the structure of the connected repository.
See the example below. The component
componentA has the artifact content.zip:
<ivy-module version="2.0"> <info organisation="com.intershop" module="componentA" revision="1.1" /> <publications> <artifact name="content" type="local" ext="zip" conf="runtime"/> </publications> <dependencies/> </ivy-module>
To use a repository in the Intershop continuous delivery tools, you have to configure the repository using Gradle's DSL (see below), including type, root and pattern.
A system component consists of a number of artifacts with different states and meta-data within its lifecycle, which is shown in the following state diagram:
In lifecycle state "source" the system component is available in a version control system. It contains a number of source files, data files, libraries and the build.gradle file containing following meta data:
Intershop 7 consist of more than 200 cartridges, a bunch of tools, servers and libraries, that are system components. An independent lifecycle for each of them would lead to a complex build and test infrastructure as well as to a challenging dependency management with a huge compatibility matrix. That is why we introduced the system component sets.
A system component setexclusively contains one or more system components of different types. It defines a common versioning, branching strategy as well as the same release cycle. Further, following is defined:
The Gradle based Intershop build process converts a system component from lifecycle state "source" to "compiled". Depending on build configuration with configured build plugins different Gradle tasks are executed. For instance the build plugin "java-cartridge" processes following tasks:
The qualified name and version number form the identity of what is called a "module revision" in the Apache Ivy world. In production it is useful to treat module revisions immutable, e.g. once meta-data and contents were published with a certain version and build number, they should not change anymore. (Instead each build should be published with a different build number.) By this way it is possible to refer to this identity at deployment time with absolute certainty.
Artifact types are used to distinguish different kinds of contents that need to be treated differently during deployment, like 'jar' files, content for the shared file system and content for the local file system.
The build process is executed in development environments as well as on the continuous integration server.
The build process is also responsible to publish the system component to an artifact repository, which can be a local folder on the development environment or an artifact repository server. Depending on the purpose of the artifact repository, the version number may differ as explained above.
All version numbers follow the following pattern: <base version>.<build suffix>. The base version is a version declared in the source of a component (build.gradle orgradle.properties) file. Examples are "220.127.116.11" for Intershop 7 components and"18.104.22.168" for build plugins. When being published a suffix is added by the build process automatically depending on the context it runs in.
|22.214.171.124.20140119162517||release (candidate) version|
Built by the Continous Integration server on a schedule (like once a day). Published to a shared repository server, like Nexus or Artifactory.
Running the build again will result in a different timestamp - so each build is uniquely identifiable. Release build are never overwritten in a repository (they are immutable). Asking for the contents of a release version with the same timestamp will always result in the same contents.
Built by the developer. Published to a local repository (a directory on the developer machine).
Running the build locally again will result in the same version number, so old versions will be overwritten (they are mutable). Depending on when you ask for the contents of a local version the contents maytherefore differ.
Built by the Continuous Integration server upon check-in. Published to a shared repository server, like Nexus or Artifactory. (Release versions and snapshot versions are published to different repositories on the same repository server.)
Running the build again will result in the same version number, so old versions will be overwritten (they are mutable). Depending on when you ask for the contents of a snapshot version the contents may therefore differ.
Snapshot versions are created only to speed up the Continuous Integration process and to avoid repository overhead. Since they are mutable, you should not use them in your developer or test environment.
The lifecycle state "deployed" is the result of the deployment process extracting the static files and libraries to the configured location on the target host. Environment specific configurations of the system component are also applied by the Intershop deployment tool.
After a system component is installed it can be started in different runtime environments on the target host. In the Intershop Application Server runtime environment the system components (especially cartridges) take a special role in behavior of particular applications. See detailed information in the Concept - Application Framework (valid to 7.9) as well as Concept - Cartridges (valid to 7.4).
An assembly references a number of system components, that can be deployed or delivered together. The following figure describes the example assembly "A" and its changes from version 1.1 to version 1.2, which are
Also it is possible to copy an assembly and all contained components from one repository to another, or – for that matter – create a repository that solely contains the assembly and its contained components (see: Intershop 7 DVD).
Besides pointing to other components an assembly may contain contents in own artifacts. This is useful for content which can only be created once all contained components are known and aggregates information from them. Examples are database dumps, cross-linked documentation like JavaDoc, or (default) configuration that is specific to the set of contained components.
Assemblies can be created from existing assemblies. This allows to form a delivery chain between teams and vendors, each passing one or more assemblies, which are then modified by adding/removing/replacing components.
The referenced system components can be assigned to internal assembly subsets for different purposes described in the next sections.
An Intershop 7 cluster follows the classical three tier architecture and consists of following cluster nodes:
The Oracle Database is not part of the Intershop 7 delivery, but is required to run an Intershop 7 cluster. The load balancer is an optional node, that is required if several Intershop Web Servers are used.
The Intershop Deployment Tool is responsible to distribute and configure the particular system components to the nodes of the Intershop cluster. In order to reuse deployment configurations of nodes with the same subset of system components the concept of host types is introduced.
Assemblies define one or more host types to reference infrastructure components, that form the runtime environment of a cluster node. A host type is defined by:
The following sections describing the host types delivered with Intershop 7.
The host type "all" is used by the single host deployment to install all nodes of an Intershop cluster on one host. It is implicitly defined by the Intershop assembly plugins.
The Intershop Web Server is responsible among other things to cache and assemble rendered pages, distribute web requests to particular application servers and generate session identifiers. It consists of following parts:
The Intershop Application Server comprises all infrastructure components necessary to create a run-time environment (engine) for Intershop7 cartridges. The components are grouped as:
The Intershop Shared File System is mounted and accessed by all application servers, that are part of the cluster. It consists of three main parts:
(since Intershop version 7.5)
The Intershop Solr Server comprises the infrastructure components necessary to create a run-time environment for the Apache Solr search engine. This host contains:
(until Intershop version 7.5)
In a development project it may be desired to have the complete JavaDoc in one central place for all developers to use. This host type provides:
A published assembly runs through different stages of the continuous delivery pipeline before it is deployed to the production system. The number and characteristics of these stages differ in particular projects and should be freely configurable in the assembly. The assembly subset environment is introduced to fulfill this need of an additional dimension of a deployment configuration. Following sections describe the predefined environments delivered with Intershop 7.
The development environment uses a single node deployment to test the assembly the developer works on. It requires special application server configurations, that:
The test environment configures the application server in production mode and requires all test cartridges of the assembly to run the automated test suite.
The production environment configures the application server in this way, that incoming requests are processed as fast as possible. Further, no test cartridges are installed.
The Intershop continuous delivery tools support following simplified process flow. This section describes the three central processes (build, assemble and deploy), that can be used to create complex continuous delivery scenarios.
The Gradle based Intershop build tools are used to compile and package system components and publish the result to an artifact repository. The tools expect a multi-project structure in the version control system. Depending on purpose there are different types of build processes described in the following sections.
The local build is used by developers to work on particular system components, that are published to a local artifact repository (e.g., a directory in file system). As described above the locally published system components have a version suffix -local. A special feature of the local build process is that the developer gets a finegranular control to the intermediate steps of overall build process by calling sub-tasks of the Gradle project. The Intershop Studio provides an integrated user interface to start these steps. Further, the locally deployed development environment of the Intershop cluster is able to load code artifacts from source folder, that do not need to compile.
Snapshot build processes run on the CI server to provide fast feedback to the developers. The built system components are published to a central artifact repository for further distribution or subsequent build processes. Normally, the process is triggered by checkins of developers. It executes only these intermediate steps of overall build process, that are required to execute different tests like unit tests, DBInit processes or smoke tests.
The release build process is used to produce potential releases of assemblies with their particular system components, that are continuously processed in the continuous delivery pipeline. These processes should execute all steps, that are required to release the particular system components and assemblies. The version suffix ends with a unique identifier, so that each result of a release build process can be non ambiguously identified on particular cluster installations.
The Gradle based assembly tools are used to combine the built system components with some generated files and publish the result to the artifact repository.
In the beginning of the process, the ivy.xml file is generated defining dependencies to the system components in a concrete version. Afterwards the assembly is deployed in order to initialize the database via the DBInit tool (see: Concept - DBMigrate and DBInit (valid to 7.7)). In the end the database dump is created and the generated files are published.
The deployment process expects a published assembly located in an artifact repository. There are two types:
Independent of the deployment process type it is possible to define the environment type defining a particular set of system components with according global configuration. The following figure describes, how one assembly is deployed in different environments:
It is possible to set up more complex continuous delivery scenarios with the Intershop continuous delivery tools involving several teams in a value added chain. Each team is part of the overall continuous delivery process with different deployment pipelines of their produced components. The artifact repositories are used to connect particular development infrastructures and to create the continuous code flow between the teams. The figure below outlines a possible scenario, that connects three continuous integration environments, which continuously integrates the work of several development teams:
The information provided in the Knowledge Base may not be applicable to all systems and situations. Intershop Communications will not be liable to any party for any direct or indirect damages resulting from the use of the Customer Support section of the Intershop Corporate Web site, including, without limitation, any lost profits, business interruption, loss of programs or other data on your information handling system.