Related Documents
Document Properties
Kbid2E5612
Last Modified03-Feb-2020
Added to KB12-Mar-2014
Public AccessEveryone
StatusOnline
Doc TypeGuidelines, Concepts & Cookbooks
Product
  • Gradle Tools
  • ICM 7.6
  • ICM 7.7
  • ICM 7.8
  • ICM 7.9
  • ICM 7.10

Concept - Gradle Assembly Tools

1 Introduction

Technical details about assemblies and the process to create them are presented in this document. Addtionally, it outlines how inheritance chains are built.

Target group of this document and the accompanying cookbook are developers who need to create new assemblies and maintain existing ones. This document assumes that you are familiar with Concept - Continuous Delivery Tools, at least up to chapter Assemblies.

1.1 Glossary


Phrase
Meaning
Version Control System (VCS)

Also known as source control, source code management systems (SCM), or revision control systems (RCS). VCS is a mechanism for keeping multiple versions of your files, so that when you modify a file you can still access the previous revisions.

Artifact Repository

Place, where build and package software components are located. Provide a common interface to a dependency management system.

Code AnalysisProcess to analyze source code to calculate metrics, find bugs, etc.
Continuous Delivery PipelineSometimes called Deployment Pipeline, describes the stages, which code artifacts runs through source to production system.
System ComponentA software package of different code artifacts and files, that have to be deployed together.
System Component SetIs a container for system components, that needs to be build and branched together.
AssemblyAn assembly references one or more system components residing in the same or a configured artifact repository in order to deploy or deliver them together.
Build ProcessCompiles and packages files and code artifacts from a source project to deployable artifacts.
Publish ProcessThe process which transfers the deployable artifacts to a configured artifact repository.
Assembly ProcessThis process combines several system components to an assembly.
Deployment ProcessThis process extracts files and code artifacts from an artifact repository and applies the configuration.
Project Gradle DistributionThis is a customized Gradle distribution with the preconfigured artifact repositories and Gradle plugins.
Gradle PluginA Gradle plugin packages up reusable pieces of build logic, which can be used across many different projects and builds.
Project Gradle PluginThis is a Gradle plugin which contains special corporate respectively project settings.
Corporate PluginThe term is used as a synonym for Project Gradle Plugin.
Gradle Extension ObjectJava Bean compliant class holding configurations for Gradle plugins.
Gradle WrapperThe Gradle Wrapper is the preferred way of starting a Gradle build. The wrapper is a batch script on Windows, and a shell script for other operating systems. When you start a Gradle build via the wrapper, Gradle will be automatically downloaded and used to run the build. See for more information The Gradle Wrapper in the Gradle documentation (2.11, 2.7, 2.3, 2.0, 1.8)
Intershop ClusterA number of hosts of different types serving an Intershop 7.
Cluster NodeOne separately deployable part of an Intershop cluster. A host can run multiple nodes of one Intershop cluster.

1.2 References

2 Built Assembly And Deployment

Like any component the assembly has a source form (located in the version control system) and a built form (located in an artifact repository). Also there is a process called assembly process transforming the sources into the published from.

This chapter describes the published form and how it is interpreted by the deployment. The next chapter covers the process of creating it and the source form.

Each binary assembly has an ivy.xml file. The following example demonstrates basic structure and typical contents of an assembly's ivy.xml file:

ivy.xml of assembly 'primetech'
<ivy-module version="2.0" xmlns:e="http://ant.apache.org/ivy/extra">
  <info organisation="com.intershop.assembly" module="primetech" revision="7.5.0.0.20140207101035" status="integration" 
    publication="20140207124937" e:productName="Intershop 7" e:copyrightOwner="Intershop Communications" e:copyrightFrom="2005">
    <e:hostType name="appserver" includeCartridges="false" includeShare="false" includeLocal="true"/>
    <e:hostType name="share" includeCartridges="true" includeShare="true" includeLocal="false"/>
    <e:hostType name="webserver" includeCartridges="false" includeShare="false" includeLocal="true"/>
    <e:environment name="development"/>
    <e:environment name="production"/>
    <e:environment name="test"/>
  </info>
  <configurations>
    <conf name="appserver" visibility="public"/>
    <conf name="default" visibility="public"/>
    <conf name="deploy-plugins" visibility="public"/>
    <conf name="development-cartridges" visibility="public"/>
    <conf name="init-cartridges" visibility="public"/>
    <conf name="production-cartridges" visibility="public"/>
    <conf name="share" visibility="public"/>
    <conf name="test-cartridges" visibility="public"/>
    <conf name="webserver" visibility="public"/>
  </configurations>
  <publications>
    <artifact name="deploy-settings" type="deploy-settings-gradle" ext="gradle" conf="default"/>
    <artifact name="deploy" type="deploy-gradle" ext="gradle" conf="default"/>
    <artifact name="database-dump" type="dump" ext="dmp" conf="default"/>
    <artifact name="branding" type="share" ext="zip" conf="default"/>
  </publications>
  <dependencies defaultconfmapping="*->;default">
    <dependency org="com.intershop" name="3rd_oracle" rev="11.2.0.3.0.2" conf="appserver, development-cartridges, production-cartridges, test-cartridges" transitive="false"/>
    <dependency org="com.intershop" name="runtime" rev="7.5.0.0.20140207101035" conf="appserver, development-cartridges, production-cartridges, test-cartridges, webserver" transitive="false"/>
    <dependency org="com.intershop" name="tools" rev="7.5.0.0.20140207101035" conf="development-cartridges, production-cartridges, test-cartridges" transitive="false"/>
    <dependency org="com.intershop" name="pf_cartridge" rev="7.5.0.0.20140207101035" conf="development-cartridges, production-cartridges, test-cartridges" transitive="false"/>
    <dependency org="com.intershop" name="pf_objectgraph" rev="7.5.0.0.20140207101035" conf="development-cartridges, production-cartridges, test-cartridges" transitive="false"/>
    ...
  </dependencies>
</ivy-module>

The dependencies element lists all contained components as dependencies. Each dependency declares an exact version number (not version expressions like 7.5.0.0.+).

Also an assembly has a number of named subsets, technically represented by what are called Ivy configurations. Each assembly subset may contain an arbitrary amount of components and a component can be contained in a huge variety of subsets, as indicated by the conf attribute of the dependency element.

Examples of such subsets are components to be deployed on different host types, like components for appserver and webserver hosts. It is possible to use this technique for any aspect in deployment, where it is necessary to mark components for a certain purpose, also custom ones. The (kinds of) subsets predefined by Intershop are described in section Assembly Subsets.

No nesting of assemblies

Built assemblies never depend on other assemblies, i.e., it is not possible to describe a nesting of assemblies. Instead a source assembly may inherit contents from another assembly. (But there are no indications of that in the resulting binary.)

One advantage of this is that the assembly's ivy.xml file completely describes components that can be deployed/are used during deployment. To leverage this the assembly's ivy.xml file are deployed to IS_SHARE and IS_HOME.

Besides dependencies to other components, the assembly's ivy.xml file also describes artifacts of the assembly. Artifacts of the assembly are contained as files near the ivy.xml file in the artifact repository. It is useful to have artifacts not only in infrastructure components and cartridges, but also in assemblies, for artifacts that can only be created once all contained components and their versions are known. An example is the database dump (useful for demo-purposes) of all cartridges. It is possible to add custom artifacts. The (types of) artifacts predefined by Intershop are described in the section Generated Assembly Artifacts.

Lastly the assembly contains additional meta-data using Ivy 'extra attributes' (attributes of the info element) or 'extra elements' (nested elements in the info element). This also leaves room for arbitrary customization. Types of meta-data predefined by Intershop, like copyright information, that are displayed on login screens, are described in the Cookbook - Gradle Assembly Tools (valid to 7.8).

2.1 Cartridges and Infrastructure Components

The assembly distinguishes between two kinds of assigned system components: cartridges and infrastructure components. Whether a system component is considered as a cartridge or an infrastructure component is therefore determined by the assembly subsets it is assigned to. It is neither defined by any meta-data (i.e., the ivy.xml file) in the system component itself nor by any convention or rule.

2.1.1 Cartridges

Cartridges are all these system components that are assigned to one or more environments or the subset init-cartridges (see descriptions below). A system component that is marked as a cartridge in the way it is built and deployed:

  • Using its deploy.gradle artifact, if it has one - i.e., if has a custom deployment script.
  • Using the CartridgeDeploymentPlugin otherwise.

Cartridge location

Until Intershop 7.4, cartridges were always located in IS_SHARE/system/cartridges.

Starting with Intershop 7.5, the location for cartridge deployment can be configured. In this concept, CARTRIDGES_DIR refers to the configured cartridge location.

Also system components marked as cartridges are added to the CARTRIDGES_DIR/cartridgelist.properties file.

Minimum requirements for a cartridge

  1. The deployment script/plugin of a cartridge must at least deploy its ivy.xml file into the cartridge folder (CARTRIDGES_DIR/<cartridge>/release). This is necessary because the cartridge engine tries to look up meta information for all entries of cartridgelist.properties when running the application server or DBInit. The CartridgeDeploymentPlugin already includes this.
  2. A cartridge cannot have dependencies to non-cartridges. This is essential because the cartridge engine tries to look up dependencies of cartridges in cartridgelist.properties.

Besides the ivy.xml file the CartridgeDeploymentPlugin also deploys:

  • all artifacts of type cartridge to the cartridge folder (CARTRIDGES_DIR/<cartridge>)
  • all artifacts of type jar to the cartridge lib folder (CARTRIDGES_DIR/<cartridge>/release/lib)
  • all artifacts of type share to IS_SHARE directory

2.1.2 Infrastructure Components

Infrastructure components are all components that are assigned to one or more host types (see below).

Minimum requirements for an infrastructure component

A component that is marked as a infrastructure component must always supply a deploy.gradle artifact as deployment script.

Most of these scripts apply the InfrastructureDeploymentPlugin which deploys:

  • all artifacts of type local into IS_HOME direcotry
  • all artifacts of type share into IS_SHARE directory

2.2 Assembly Subsets

2.2.1 Environments

Environments are different stages in the life-cycle of an assembly, like development, demo, quality assurance and production. At time of deployment exactly one environment is selected.

An environment is declared using an environment element nested in the info element.

An environment has a name, which is an arbitrary string.

Also it has an associated Ivy configuration by the name <environment>-cartridges. Components assigned to this configuration are only included in the deployment if the environment is selected. Also they are considered cartridges (see above).

There is only one - environment independent - database dump generated for a single assembly. Cartridges containing DBInit prepares should be part of all environments. Or turned the other way around, a cartridge that is special to one environment should not contain DBInit preparers.

2.2.2 Host Types

Host types are types of nodes in a cluster deployment. They are declared using a host-type element nested in the info element of the assemblies ivy.xml file. At time of deployment exactly one host type is selected possessing a name, which is an arbitrary string.

Also it has an associated Ivy configuration with the same name. System components assigned to this configuration are only included in the deployment if the host type is selected. Also they are considered infrastructure components (see above).

Components may be assigned to multiple host types. The host type's flags control indicates which artifacts are to be deployed for each component. There are four flags for each host type:

  • includeShare specifies share artifacts (like sites content) to be deployed into the shared file system (IS_SHARE).
  • includeLocal specifies instance-specific artifacts (like executable files) to be deployed locally (IS_HOME).
  • includeCartridges specifies the cartridge directory's artifacts to be deployed (the cartridge's ivy.xml and artifacts of type cartridge and jar).
  • includeJavadoc specifies the Javadoc will be deployed for all components (including cartridges) as well. (since Gradle Tools 2.0)

See section Cartridges below for what cartridge means in the context of a deployment. Depending on the set flag of the host type (includeCartridges, includeShare or includeJavadoc), the deployment will load cartridges in addition to the host type's own components.

There is an implicit host type all, that can be selected upon deployment despite not being declared in the assemblies ivy.xml file. The host type all is used for the single-machine-installation. It contains all components of all other host types, and has all four flags set to true. (Which also explains why includeShare and includeLocal are independent flags and not neceessarily the opposite of each other.)

It is possible to define custom host types. Predefined are: webserver, appserver, share, javadoc (until 7.5) and solr (since 7.5) (see for the Concept - Continuous Delivery Tools | Assembly Subsset Host Type details). Predefined in this context means, that most assemblies delivered by Intershop declares them.

2.2.3 Ivy Configuration 'init-cartridges'

The Ivy configuration init-cartridges contains system components that should be deployed as cartridges, but only be loaded by the DBInit tool and not by the application server.

This is available as a safety measure against running potentially unsecure DBInit code (like special pipelines) in the application server. If a cartridge contains both production and DBinit code it can instead be added to an environment.

2.2.4 Selected Cartridges and Infrastructure Components

Based on the selected host type and environment, components and cartridges are selected for deployment.

The set of cartridges to be deployed is calculated the following way:

  • Cartridges of the selected environment plus
  • Init-Cartridges

The IS_SHARE/system/cartridges/cartridgelist.properties is generated during the deployment by:

  • Filling the property cartridges (for the application server) with all cartridges of the selected environment
  • Filling the property cartridges.dbinit (for the dbinit) with all cartridges of the selected environment plus all init-cartridges

Non-DBInit-Cartridges

The cartridges in cartridges.dbinit are now always a super-set of those in cartridges. The DBInit automatically detects which of the cartridges in cartridges.dbinit actually contain preparers to be executed.

The set of components to be deployed in total is calculated the following way:

  • All components contained in the selected host-type plus
  • Cartridges as calculated above, if the host-type has the flag includeCartridges set, or no cartridges otherwise.

The following picture shows an assembly with a few subsets, cartridges and infrastructure components:

environments_and_host_types

The next diagram shows, which components get deployed at deployment time:

  • the host-type appserver and
  • the environment production are selected.

(Actually any environment would result in the same deployment here, because the host-type appserver does not include cartridges.)

environments_hosttype_appserver_selected

The last diagram shows, which components get deployed if at deployment time

  • the host-type share and
  • the environment production are selected

hosttypes_environments_share_selected

2.2.5 Ivy Configuration Deploy-Plugins

The Ivy configuration deploy-plugins contains the components needed to run the deployment. They are listed using an exact version (instead of a version expression like 1.0.0.+) to ensure that running a deployment on the same assembly (using the same configuration) will always have the same result.

You should always inherit the configuration deploy-plugins from an assembly supplied by Intershop/your vendor. (You may add additional deploy-plugins, see the  Cookbook - Gradle Assembly Tools (valid to 7.8) for details.)

2.3 Generated Assembly Artifacts

The assembly process generates or copies artifacts, that are required for the deployment process. These artifacts are described in this section.

2.3.1 The Deployment Configuration: deploy-settings.gradle and deploy.gradle

Assemblies must contain two artifacts to control the overall flow of the deployment:

  • deploy-settings.gradle is interpreted like Gradle's settings.gradle files for multi-project builds. It controls which sub-projects are included in the deployment, i.e., which components have to be deployed.
  • deploy.gradle is interpreted like a build.gradle file of the top-level project of a multi-project build. It provides DSL extensions on which the user can configure the deployment and assembly-specific deployment tasks (like generation of the cartridgelist.properties or the deployment of the database dump)

For more information on the contents and roles of these files see the Concept - Gradle Deployment Tools.

2.3.2 Database Dump

Assemblies may contain a database dump that is generated during the assembly process. This dump can be used in certain environments (like demo servers or for developers) to initialize the database and save time compared to running DBInit.

2.3.3 Shared Artifacts

The assembly may contain arbitrary artifacts of type share. These artifacts must be zip-files and are extracted to IS_SHARE directory. An example of such an artifact is a package of branding installations in the <IS_SHARE>/sites folder. This package is generated during the DBInit as a byproduct when running a BrandingPreparer and must be deployed, so a dump import can fully replace a DBInit.

3 Assembly Process

3.1 Steps

The assembly process is a build process, running a build.gradle file. The main Gradle task to run the assembly process is to publish. The following picture shows the steps of the assembly process.

assembly_stages

The steps filled with grey are optional:

  • The step run smoke test is not a dependency of publish and must be triggered explicitly.
  • The step export dump and its dependencies dbinit and deploy server are only executed if the project property releaseWithDump is set to true (default: false).

On continuous Integration servers, the  releaseWithDump property should be set, also some tests should be executed.

In a developer environment  releaseWithDump is typically not set. The remaining two steps when running the task publish are lightweight and fast, reducing the assembly process to the bare minimum. They can still be executed by running dedicated Gradle tasks, see below.

3.1.1 Generate ivy.xml File

The core of the assembly process is to generate the  ivy.xml file of the assembly from declarations in the  build.gradle file. Dependencies and configurations in the Gradle project result in dependencies and configurations in the assembly's ivy.xml file. If transitive dependencies are specified, they are resolved transitively at first and all resulting components are added as dependencies in the  ivy.xml.

You can use Gradle's DSL for dependency management to the full extent to add/modify/exclude dependencies. For most common tasks (like adding cartridges to an environment) we offer a more convenient custom DSL. See the Cookbook - Gradle Assembly Tools (valid to 7.8) for details.

For generating the cartridgelist.properties during deployment, cartridges are sorted in the resulting ivy.xml using a file named cartridge-order.txt.

3.1.2 Deploy Server

As a pre-requisite of later steps the assembly must be deployed. The deployment during the assembly is as close to a deployment in production as it is possible, i.e., it uses the same deployment scripts and plugins like the real deployment does. 

Before running the deployment the assembly process must:

  1. Make the assembly available in a repository. For this a preliminary version of the assembly is published - not containing the database dump, which will be generated in a later step.
  2. Generate a  settings.gradle file for running the deployment. Configuration values that cannot be determined automatically, like ports and the database account, must be provided by the user in an  environment.properties file.

The preliminary version of the assembly is called the build assembly publication. It

  1. has a special version number build(for example com.intershop.assembly:intershop7:build) and
  2. is published to a temporary repository located at <assembly source directory>\target\build-repo.

Developers can trigger this step by running the Gradle task deployServer. The resulting server is sufficient for running most manual and automatic tests during development.

The deployment during the assembly process always specifies host type all, i.e., a single machine deployment.

3.1.3 DBInit and Export Dump

The DBInit is run inside the deployed server. The assembly process observes the console output/log files of the DBInit and fails if any errors occur.

Developers can trigger this step by running the task dbinit.

After running the DBInit the legacy ant export script is used to generate a database dump file.

3.1.4 Smoke Tests

Optionally some some tests can be executed by the test task, including:

  • starting an application server
  • starting a web-server

and observing log-files for errors/warnings. The assembly process fails if an error occurs.

3.1.5 Integration Tests

Starting with Gradle Tools 2.3, the assembly process also executes certain integration tests, that are:

  • Located in a cartridge, that is
    • Newly added by the current assembly (i.e., not inherited)
    • Part of the environment test,
  • Residing in the package tests.unit for test cases, that are starting an embedded server instance.
    These tests are executed by the task ishUnitTest.

Integration test execution is based on Gradle's Test task, so that all standard functionalities also apply for integration tests, e.g., selecting only a subset of tests for execution.

The overall check task depends on all kinds of test tasks. In the future, more types of integration tests may be executable during the assembly process. Call the test task instead of check to execute only smoke tests.

The assembly process fails if an error occurs.

3.1.6 Publish

Finally a non-preliminary ivy.xml file is generated, containing the database dump as an artifact if releaseWithDump is true. The ivy.xml file and all declared artifacts are then uploaded/copied to a repository. From there they can be deployed or have other assemblies inherit from them.

3.2 Assembly Plugins

The steps described above are provided by two Gradle Plugins:

  • Step 1 along with the DSL for host-types, environments and inheritance is contained in com.intershop.build.gradle:assembly. This functionality is very general and not limited to be used to deploy Intershop 7 based projects.
  • All other steps are contained in com.intershop.build.gradle:ish-assembly and are specific to Intershop 7 based projects and runtime architecture.

3.3 Source Files

Typical source files used to build an assembly are:

File NamePurpose
build.gradledescribes build steps and contents of the assembly
gradle.propertiesspecifies frequently/automatically changing configuration for the assembly process, especially version numbers
cartridge-order.txtdescribes the order of cartridges for the cartridgelist.properties file
deploy-settings.gradle, deploy.gradlepublished as artifacts by these names (see section Generated Assembly Artifacts above for details)
environment.properties.sample and environment.propertiestemplate and file filled (should both be ignored by source control) with environment specific values for the deployment inside the assembly process
gradlew, gradlew.bat files, gradle foldereasy execution of Gradle from a working copy (see http://www.gradle.org/docs/current/userguide/gradle_wrapper.html)

3.4 Inheritance

It is possible to inherit declarations/contents from existing binary assemblies automatically. The following things can be inherited:

  • Ivy configurations and their contained components
  • Environments and their contained cartridges
  • Host Types and their contained components
  • Assembly artifacts
  • Database dumps from inherited assemblies as basis for a derived database dump by running an import and a partial DBInit

See the  Cookbook - Gradle Assembly Tools (valid to 7.8) for how the custom DSL for assembly inheritance works.

Disclaimer

The information provided in the Knowledge Base may not be applicable to all systems and situations. Intershop Communications will not be liable to any party for any direct or indirect damages resulting from the use of the Customer Support section of the Intershop Corporate Web site, including, without limitation, any lost profits, business interruption, loss of programs or other data on your information handling system.

Customer Support
Knowledge Base
Product Resources
Support Tickets