Related Documents
Document Properties
Kbid282N89
Last Modified03-Feb-2020
Added to KB08-Jun-2017
Public AccessEveryone
StatusOnline
Doc TypeGuidelines, Concepts & Cookbooks
Product
  • Gradle Tools
  • ICM 7.9
  • ICM 7.10

Cookbook - Gradle Assembly Tools

1 Introduction

This cookbook describes how to configure an assembly at build time. The target audience are developers or build engineers creating new assemblies or maintaining existing ones.

Note

The cookbook is valid for Gradle tools version 2.11.6 and higher.

It is assumed that you are familiar with the Concept - Gradle Assembly Tools.

Topics not covered in this document are:

  1. How to work with an existing (and therefore configured) assembly as part of the developer workflow. This includes execution of the assembly process. 

  2. How to deploy an assembly. This is either part of the developer workflow (see above) or covered in the following documents:

1.1 Glossary


Phrase
Meaning
Version Control System (VCS)

Also known as source control, source code management systems (SCM), or revision control systems (RCS). VCS is a mechanism for keeping multiple versions of your files, so that when you modify a file you can still access the previous revisions.

Artifact Repository

Place, where build and package software components are located. Provide a common interface to a dependency management system.

Code AnalysisProcess to analyze source code to calculate metrics, find bugs, etc.
Continuous Delivery PipelineSometimes called Deployment Pipeline, describes the stages, which code artifacts runs through source to production system.
System ComponentA software package of different code artifacts and files, that have to be deployed together.
System Component SetIs a container for system components, that needs to be build and branched together.
AssemblyAn assembly references one or more system components residing in the same or a configured artifact repository in order to deploy or deliver them together.
Build ProcessCompiles and packages files and code artifacts from a source project to deployable artifacts.
Publish ProcessThe process which transfers the deployable artifacts to a configured artifact repository.
Assembly ProcessThis process combines several system components to an assembly.
Deployment ProcessThis process extracts files and code artifacts from an artifact repository and applies the configuration.
Project Gradle DistributionThis is a customized Gradle distribution with the preconfigured artifact repositories and Gradle plugins.
Gradle PluginA Gradle plugin packages up reusable pieces of build logic, which can be used across many different projects and builds.
Project Gradle PluginThis is a Gradle plugin which contains special corporate respectively project settings.
Corporate PluginThe term is used as a synonym for Project Gradle Plugin.
Gradle Extension ObjectJava Bean compliant class holding configurations for Gradle plugins.
Gradle WrapperThe Gradle Wrapper is the preferred way of starting a Gradle build. The wrapper is a batch script on Windows, and a shell script for other operating systems. When you start a Gradle build via the wrapper, Gradle will be automatically downloaded and used to run the build. See for more information The Gradle Wrapper in the Gradle documentation (2.11, 2.7, 2.3, 2.0, 1.8)
Intershop ClusterA number of hosts of different types serving an Intershop 7.
Cluster NodeOne separately deployable part of an Intershop cluster. A host can run multiple nodes of one Intershop cluster.

1.2 References

2 Recipe: Create a New Assembly Inheriting From an Existing Assembly

2.1 Problem

Developers need to create custom assemblies to deploy custom cartridges or a custom combination of existing cartridges.

An assembly can also be created with the scripts in the development-setup of an Intershop delivery package (See Cookbook - Setup CI Infrastructure).

This recipe provides a more in-depth explanation on how to create an assembly manually.

2.2 Solution

This covers the more common case of creating an assembly that inherits from an existing one.

Create a single root folder and the following files. Add them to your source repository, once you are done. See Concept - Gradle Assembly Tools for the purposes of these files.

fileinitial content
build.gradleApply the 'ish-assembly' plugin and inherit from one of our base assemblies - see discussion.
gradle.propertiesSpecify a version - see discussion.
cartridge-order.txtCopy from inherited assembly. Deprecated for Gradle tools version >= 1.1, see recipe Add Cartridges to an Assembly.
gradlew , gradlew.bat files, gradle folderCopy from other Gradle build process

To verify that your initial assembly works, publish and/or deploy it.

2.3 Discussion

2.3.1 build.gradle

Copy the following minimum build.gradle file and follow through the TODOs:

Gradle tools version 1.0:

build.gradle
// Make the assembly plugin available to the build.gradle script
buildscript { 
    dependencies {
        classpath 'com.intershop.build.gradle:ish-assembly'
		classpath 'com.intershop.build.gradle:plugin-tests'
	 }    
}

// Apply the 'ish-assembly' plugin to the project
apply plugin: 'ish-assembly'

// Specify the namespace in which the assembly is published into the repository. (The name inside this namespace is automatically derived from the folder name.)
// TODO: Change to your desired namespace
group = 'com.example.assembly'

assembly {
	// Inherit from another assembly
	// TODO: Replace by <org>:<name> of the inherited assembly
	inheritFrom('<org>:<name>') {		
		// Inherit assembly deploy.gradle and deploy-settings.gradle scripts
		includeArtifacts type:['deploy-gradle', 'deploy-settings-gradle']

		// By default will also inherit all host-type, environments
		// and the contained components/cartridges
	}

	// Supply additional meta-data that will be displayed in the SMC/at logon screens
    extraAttributes = [
		//TODO: Change to appropriate values
	    'productName': 'Intershop 7',
        'copyrightOwner': 'Intershop Communications',
        'copyrightFrom': '2005'
   ]
}

Gradle tools version >= 1.1:

build.gradle
// Make the assembly plugin available to the build.gradle script
buildscript { 
    dependencies {
        classpath 'com.intershop.build.gradle:ish-assembly'
		classpath 'com.intershop.build.gradle:plugin-tests'
	 }    
}

// Apply the 'ish-assembly' plugin to the project
apply plugin: 'ish-assembly'

// Specify the namespace in which the assembly is published into the repository. (The name inside this namespace is automatically derived from the folder name.)
// TODO: Change to your desired namespace
group = 'com.example.assembly'

assembly {
	// Inherit from another assembly
	// TODO: Replace by <org>:<name> of the inherited assembly
	inheritFrom('<org>:<name>') {		
		// Inherit assembly deploy.gradle and deploy-settings.gradle scripts
		includeArtifacts type:['deploy-gradle', 'deploy-settings-gradle']

		// By default will also inherit all host-type, environments
		// and the contained components/cartridges
	}

	cartridges {
		// Inherit cartridge order from another assembly
		// TODO: Replace by <org>:<name> of the inherited assembly
		order = listFromAssembly('<org>:<name>')
	}

	// Supply additional meta-data that will be displayed in the SMC/at logon screens
    extraAttributes = [
		//TODO: Change to appropriate values
	    'productName': 'Intershop 7',
        'copyrightOwner': 'Intershop Communications',
        'copyrightFrom': '2005'
   ]
}

The ish-assembly plugin extends the Gradle project by:

  • Adding the DSL extension assembly for configuring the assembly's contents. (See other recipes.)
  • Adding the DSL extension assemblyBuild for configuring how a database dump is created during build. (See other recipes.)
  • Adding tasks to
    • publish the assembly
    • deploy the assembly
    • run the DBInit

You have created an assembly that is basically the same like the inherited one, except for a different name. To customize the assembly, see the other recipes.

2.3.2 gradle.properties File

Copy the following minimum gradle.properties file and follow through the TODOs:

# TODO: Provide version number of your assembly (excluding build number/local/snapshot suffix)
version = 7.5.0.0

# build plugin versions
filter.com.intershop.build.set.cartridge-plugins = 1.0.0.0.+
filter.com.intershop.build.set.extension-plugins = 1.0.0.0.+

3 Recipe: Add Custom Deployment Logic to an Assembly

3.1 Problem

I need to add custom deployment logic to my inherited assembly.

When adding my deploy.gradle, I get an exception like:

gradlew publish output
Execution failed for task ':publishBuildAssemblyPublicationToBuildRepository'.
> Failed to publish publication 'buildAssembly' to repository 'build'
   > Invalid publication 'buildAssembly': multiple artifacts with the identical name, extension, type and classifier ('deploy', 'gradle', 'deploy-gradle', 'null').

3.2 Solution

  1. Create the file <ASSEMBLY>/deploy.gradle.
  2. Add the parent assembly's deployment logic to this file.
  3. Add your own custom deployment logic to this file.
  4. Exclude the artifact when inheriting from the parent assembly.

3.3 Discussion

Example until Gradle Deployment Tools 2.7

deploy.gradle until Gradle Tools 2.7
assembly {  
    inheritFrom('com.intershop.assembly:commerce_management_b2x') {
        includeArtifacts
        type:['deploy-settings-gradle']
        //     Exclude the artifact when inheriting from
        the parent assembly....
        //
        includeArtifacts type:['deploy-gradle', 'deploy-settings-gradle']
    }
}
Starting with Intershop Gradle Tools 2.11 it is possible to inherit also the deployment logic of an assembly. All assembly deployment scripts will be executed in alphabetical order, if there is more than one inheritance. Please refer to Cookbook - Setup CI Infrastructure and Public Release Notes - Gradle Tools - Version 2.11 for more detailed information on how to achieve this.

To find the parent assembly's deployment logic, browse your software repository and look for the deploy-gradle artifact of the according assembly.

To exclude the artifact from inheritance, see Recipe: Customize Assembly Inheritance.

4 Recipe: Add Cartridges to an Assembly

4.1 Problem

The developer needs to add cartridges to an assembly.

4.2 Solution

See the discussion for details on the steps.

For Gradle tools version 1.0:

  1. Add cartridges to one or more assembly subsets by using the method assembly.cartridges.include in the  build.gradle file.
  2. Ensure that the added cartridges (and their transitive dependencies, see discussion) are included in the cartridge-order.txt file.
  3. Ensure that a version for the added cartridges is declared in the gradle.properties file of the assembly.

For Gradle tools version >= 1.1:

  1. Add cartridges to one or more assembly subsets by using the method assembly.cartridges.include in the build.gradle file. 
  2. Ensure that the added cartridges (and their transitive dependencies, see discussion) are included in the property assembly.cartridges.order. This ensures that the cartridgelist.properties file is appropriately generated.
  3. Ensure that a version for the added cartridges is declared in the gradle.properties file of the assembly.

4.3 Discussion

4.3.1 Add Cartridges to Subsets in build.gradle File

To add cartridges to one or more environments, add the following declaration:

build.gradle
assembly {
    cartridges {    
		//TODO: Specify organization and name of the cartridges, also choose which environments to include them in   
        include 'com.example:cartridge1', 
				'com.example:cartridge2', 
				in: [production, development, test]
	}
}

Note

Component subsets (host-types, environment, configurations) are available as object literals by their name inside the assembly closure (instead of being passed as strings).

To add a cartridge as init-only-cartridge (see Concept - Gradle Assembly Tools for more details) use the same method, but specify in:init.

build.gradle
assembly {
    cartridges {
		//TODO: Specify organization and name of the cartridges
        include 'com.example:cartridge1',
				'com.example:cartridge2',
				in: init
	}
}

4.3.2 Transitive vs. Intransitive Dependencies

By default adding a cartridge also adds all its transitive dependencies, i.e., all dependencies declared in the configuration default of its ivy.xml file, those cartridges' dependencies etc.

For most applications this makes sense, since you do not want to deploy a cartridge without its dependencies. Also you can minimize declaration effort by including the application suite cartridge. Such a cartridge already has dependencies to all application type cartridges, which in turn should already have dependencies to all cartridges contained in that application.

For more control you may include a cartridge without its dependencies using the following declaration:

build.gradle
assembly {
    cartridges {
        //TODO: Specify organization and name of the cartridges, also choose which environments to include them in
        include(
            'com.example:cartridge1',
            'com.example:cartridge2',
            in: [production, development, test]
        ) {
            transitive = false
        }
    }
}

Note

Dependency Management for Third-Party Libraries

Beware that the dependency management for third-party libraries as implemented by Intershop 7 version 7.6 and newer mandates the use of transitive declaration for cartridge dependencies during assembly.

4.3.3 Ensure Cartridges Are Listed in cartridge-order.txt

(For Gradle tools version 1.0)

To generate the <SHARE>/system/cartridges/cartridgelist.properties file, which serves as basis for look-up precedence of ISML templates, pipelines, queries, etc. and also for the order in which DBInit preparers are executed, the assembly process needs to order cartridges. This is performed by looking up the order of cartridges in the cartridge-order.txt file.

The format of the cartridge-order.txt file is:

  • Each line contains the name of a cartridge (unqualified, e.g., without organization)
  • Empty lines are ignored
  • Lines starting with a hash mark '#' are comments and are ignored as well

The cartridge-order.txt must contain all cartridges that are contained in the assembly, but may contain any number of additional cartridges. It is only used to determine the order of cartridges, not which cartridges to include in the assembly.

If you use multiple assemblies it is therefore recommended to share the  cartridge-order.txt file across them.

When adding cartridges with their transitive dependencies, it is necessary that all these dependencies are also included in cartridge-order.txt.

4.3.4 Ensure Cartridges Are Listed in assembly.cartridges.order

(For Gradle tools version >= 1.1)

To generate the <SHARE>/system/cartridges/cartridgelist.properties file, which serves as basis for look-up precedence of ISML templates, pipelines, queries, etc. and also for the order in which DBInit preparers are executed, the assembly process needs to order cartridges. This is performed by looking up the order of cartridges in the property assembly.cartridges.order.

Order vs. Inclusion

The set of cartridges to include and their order are declared separately. This may seem odd because they are coupled tightly. There are a few reasons for this decision:

  • Sets of dependencies conceptually and technically in Gradle have no notion of order.
  • Inclusion of cartridges in an assembly is declared on subset (environment/host type) level, order is more of a global property.
  • In general using a global order for determining fallback becomes more and more impractical with a rising number of cartridges and contexts in which they are used (like assemblies). Eventually we will overcome this technical restriction in the platform. In the meantime we would like to support ordering cartridges in the best way possible.

See subsection Order from Groovy List Literal for how to specify both included cartridges and their order without redundancies.

The property assembly.cartridges.order is of type List<String>. Each entry of the list should be the unqualified name of a cartridge, e.g., bc_foundation.

Constraints

All cartridges included in the assembly must be contained in the order list exactly once to know how to order them. If a cartridge is included in the assembly, but missing from the order list, an exception is thrown. Another exception is thrown if a cartridge is included in the assembly but occurs multiple times in the order list.

When adding cartridges with their transitive dependencies, it is necessary that all these dependencies are also included in the order list. Also the order list must contain all cartridges inherited from other assemblies.

The order list must contain all cartridges that are contained in the assembly, but may contain any number of additional cartridges.

The property assembly.cartridges.order can be filled in several ways.

4.3.4.1 Best Practice

The following way is considered best practice to fill the assembly.cartridges.order property:

  • Store a list of your own cartridges with the desired sorting
  • Use that list to include the cartridges into the assembly
  • Reuse the order of cartridges from the inherited assembly and concatenate the list of your cartridges
build.gradle
assembly {
    cartridges {
        def productionCartridges = ['customer_cartridge_1', 'customer_cartridge_2']
        
		// TODO: Replace by organization of your cartridges
        include(*(productionCartridges.collect {"<org>:$it"}), in:[development, test, production]) {
            transitive = false
        }

		// TODO: Replace by <org>:<name> of the inherited assembly       
        order = listFromAssembly('<org>:<name>') + productionCartridges
    }
}

4.3.4.2 Order From an Existing Assembly

Read the order of cartridges from an existing assembly using the method listFromAssembly:

build.gradle
assembly {
	cartridges {
		//TODO: Adapt name and organization of assembly
		order = listFromAssembly('com.intershop.assembly:intershop7')
	}
}

4.3.4.3 Order From a File

Read the order from an external file using the method listFromFile:

build.gradle
assembly {
	cartridges {
		//TODO: Adapt name/location of the file. In this example the file is expected in the same directory as the build.gradle file
		order = listFromFile(file('cartridge-order.txt'))
	}
}

The expected file format is:

  • Each line contains the name of a cartridge
  • Leading and trailing spaces are trimmed
  • Empty lines are ignored
  • Lines starting with a hash mark '#' are comments and are ignored as well

4.3.4.4 Order From Groovy List Literal

Create the list from a Groovy list literal directly written in the build.gradle file:

build.gradle
assembly {
	cartridges {
		//TODO: Adapt names of cartridges
		order = ['core', 'bc_foundation', 'bc_marketing']
	}
}

You may also create a list using a list literal once and pass it to both property assembly.cartridges.order and method assembly.cartridges.include. This assumes that you include your cartridges intransitively.

Note

The property expects assembly.cartridges.order unqualified cartridge names, while the assembly.cartridges.include method requires dependency expressions (e.g., qualified names).

build.gradle
assembly {
    cartridges {
        def productionCartridges = ['core', 'bc_foundation']
        
        include(*(productionCartridges.collect {"com.intershop:$it"}), in:[development, test, production]) {
            transitive = false
        }
        
        order = productionCartridges
    }
}

4.3.4.5 Order From Complex Expression

Create the list from any combination of the above, by concatenating multiple other lists, inserting or removing items. You may use any method/operation offered by the JDK or GDK (Groovy extensions to JDK) on java.util.List.

build.gradle
assembly {
    cartridges {
        // Order cartridges:
        // - Cartridge 'customer_java_patches' is first
        // - Then all cartridges inherited from intershop7
        // - Then all cartridges mentioned in 'custom-cartridge-order.txt'
        // - Order cartridge 'storefront_app_patch' directly after cartridge 'storefront_app'
        def intermediaryOrder = ['customer_java_patches'] +  listFromAssembly('com.intershop.assembly:intershop7') + listFromFile(file('custom-cartridge-order.txt'))
        order = intermediaryOrder.plus(intermediaryOrder.find('storefront_app')+1, 'storefront_app_patch')
    }
}

4.3.4.6 Default Order

To be compatible with Gradle tools version 1.0 the property assembly.cartridges.order is filled with the contents of a file cartridge-order.txt if it exists. If it does not it is empty by default.

4.3.5 Declare Version of Added Cartridges

To adjust versions more easily, they should be declared in a properties-file instead of the build.gradle file directly. Alter the gradle.properties file of your assembly by either:

  • Adding a property version.<organization of cartridge>.<name of cartridge>=<version> for each added cartridge, or
  • Adding a property filter.<organization of assembly>.<name of assembly>=<version> of an assembly or component set containing the added cartridges.
    This is already the case for all cartridges in your own component set.

Also add the declaration to the gradle.properties of your component set if your cartridges have dependencies to the added cartridges.

5 Recipe: Add Infrastructure Components to an Assembly

5.1 Problem

The developer needs to add infrastructure components to an assembly.

5.2 Solution

Add infrastructure components to a host-type by using the method assembly.hostTypes.<host-type name>.include.

build.gradle
assembly {
    hostTypes {
		//TODO: replace 'appserver' by the host-type's name
		appserver {
			// TODO: Specify organization and names of the cartridges
            include(
                'com.example:component1',
                'com.example:component2',
            ) {
                transitive = false
            }
		}
	}
}

Note

Order of Declarations

There is a bug in Gradle tools version 1.0-1.1 (ISTOOLS-461): If the build.gradle file of the assembly contains

  1. a declaration for including additional components first, and
  2. after that a declaration for inheriting the same host-type from an existing assembly,

the flags includeShare and includeLocal will not be inherited correctly. This may result in an (almost) empty deployment.

To avoid this error inherit the host-type first and then add additional components.

In Gradle tools version >= 2.0, the order between these two declarations is no longer significant.

5.3 Discussion

The same consideration like in the recipe Add Cartridges to an Assembly regarding transitive vs. intransitive dependencies applies here.

To add infrastructure components including their transitive dependencies use:

build.gradle
assembly {
    hostTypes {
		//TODO: replace 'appserver' by the host-type's name
		appserver {
			// TODO: Specify organization and names of the cartridges
        	include 'com.example:component1',
                    'com.example:component2'
		}
	}
}

Recommendation

In the case of infrastructure components, Intershop recommends using non-transitive dependency inclusion for precise control over the generated assembly.

6 Recipe: Add an Environment to an Assembly

6.1 Problem

To offer a different set of cartridges in a new deployment scenario, the developer needs to define a new environment within an assembly.

6.2 Solution

Use the method assembly.environments.create:

build.gradle
assembly {
	environments {
		//TODO: Specify the name of the new environment
		create 'preproduction'
	}
}

The new environment initially contains no cartridges. Refer to the recipe Add Cartridges to an Assembly for adding cartridges to this environment.

6.3 Discussion

Example: Add a new environment to your assembly that inherits all the cartridges from the production environment

Solution: Add the environment “prelive” to your assembly build.gradle

assembly {   
 …
    environments {
       create ('prelive') {
          production.cartridgesConfiguration.dependencies.all {
             cartridgesConfiguration.dependencies.add(it)
          }
       }
    }
    …
}

7 Recipe: Add a Host Type to an Assembly

7.1 Problem

To offer a different set of infrastructure components in a new deployment scenario, the developer needs to create a new host type in an assembly.

7.2 Solution

Add a declaration in the assembly.hostTypes:

build.gradle
assembly {
	hostTypes {
		//TODO: Replace 'imageServer' by the name of your host-type
		imageServer {
			//TODO: Set the flags (see Concept for meaning of the flags)
			includeLocal = false
			includeShare = false
			includeCartridges = false
			includeJavadoc = false
		}
	}
}

The new host type initially contains no components. Refer to the recipe Add Infrastructure Components to an Assembly for adding components to this environment.

8 Recipe: Customize Handling of Third-Party Libraries

8.1 Problem

I am using dependency management for third-party libraries with Intershop 7 version 7.6 or later, but the default behavior does not work the way I need it.

8.2 Solution

When including cartridges transitively (see Recipe: Add Cartridges to an Assembly), the depending libraries become system components themselves to the assembly. Therefore dealing with libraries equals the dealing with any other system component.

8.3 Discussion

8.3.1 Transitive Dependency Management

When including cartridges transitively, Gradle will not only include their direct dependencies, but also the dependencies of those dependencies and so on.

You can use standard Gradle mechanisms if you need to exclude libraries that come in via this transitive closure:

Assembly's build.gradle (snippet)
configurations.all {
    exclude group:'com.example', module:'excluded-lib'
}

8.3.2 Version Conflicts

The assembly does not allow conflicting versions for any system component. In case some library gets included in multiple versions, you will see an exception like:

Example version conflict
> Could not resolve all dependencies for configuration ':development-cartridges'.
   > A conflict was found between the following modules:
      - com.example:conflicting-lib:1.2.3
      - com.example:conflicting-lib:1.0.0

You may either

  • Resolve the version conflict by using version properties, e.g.:

    Assembly's gradle.properties (snippet)
    version.com.example.conflicting-lib=1.2.3

    Overriding Library Versions

    While it is technically possible to override versions of libraries used by Intershop 7, be advised that the resulting system cannot be supported unless explicitly stated by Intershop.

    or

  • Exclude all but the desired dependency as described in the section Transitive Dependency Management above.

8.3.3 Class Collisions

A new verification has been added, that inspects the Jar files of all libraries in the assembly. In case some Java classes are contained in more than one dependency an error is thrown, e.g.:

Example collision message
> There are class collisions in your dependencies
   > Collision between com.google.code.findbugs:annotations:2.0.0 and com.google.code.findbugs:jsr305:2.0.1
      > javax.annotation.CheckForNull
      > [...]

You can inspect the origin of different dependencies as described below and can use Transitive Dependency Management to exclude unwanted dependencies to resolve collisions.

The Gradle task checkClassCollisions is part of the life cycle task check and usually run during CI build. However, it is not executed automatically in the developer workflow, but can be invoked explicitly:

Invoking the Class Collision Check on an Assembly
gradlew checkClassCollisions

8.3.4 Introspecting Transitive Dependencies

In case you are wondering about the origin of some unwanted dependency, you can use Gradle to get some insight, e.g.:

Invoking Gradle's dependencyInsight on an Assembly
gradlew dependencyInsight --dependency slf4j-api

You do not need to specify the configuration to introspect, as the assembly process defaults to the environment development.

Debugging Version Conflicts

In case of a version conflict, the dependencyInsight task requires you to first resolve the conflict before it can generate its report. The resulting report will still contain information about the origin of conflicting dependencies.

8.3.5 Assembly Inheritance

In general libraries are inherited from the parent assembly just like other system components. See Recipe: Customize Assembly Inheritance for more details.

Libraries excluded in the parent assembly will not be inherited to the child assembly. However, if you add cartridges to your child assembly, that yield the same unwanted transitive dependencies, you have to exclude those dependencies again.

Example

Suppose you have an assembly parent, that includes cartridge A, but excludes the transitive dependency evil.

When the assembly child inherits from parent, evil will not get inherited. However, when child adds cartridge B transitively, this will include the evil dependency again. This state is depicted in the following illustration: example-inherit-exclude

To get rid of lib evil in the child assembly, it must also be excluded there.

The same holds true for version conflict resolutions.They are inherited for the transitive dependencies of inherited system components, but may reappear for newly added system components.

9 Recipe: Customize Assembly Inheritance

9.1 Problem

The developer needs to create an assembly inheriting contents from an existing assembly.

9.2 Solution

Use the method assembly.inheritFrom to declare inheritance from an existing assembly:

build.gradle
assembly {
	inheritFrom('com.example.assembly:base-assembly') {		
		// Control which dependencies/artifacts to inherit, see discussion
		// By default all dependencies in all component subsets and all artifacts are inherited
	}
}

9.3 Discussion

To select which artifacts and dependencies should be inherited, pass a closure to assembly.inheritFrom and inside it call includeArtifacts/ excludeArtifacts as well as includeDependencies/ excludeDependencies any number of times.

This API is largely inspired from Gradle's way to declare which files to copy in a copy operation, called a "copy spec". See the Gradle user guide for drawing parallels. To stay in the naming, the closure passed to the inheritFrom method is called an "assembly inheritance spec".

9.3.1 Selection of Dependencies

Each call to  includeDependencies/ excludeDependencies defines a predicate, i.e., a function that returns true or false given a dependency from the inherited assembly. A certain dependency is inherited if and only if it matches:

  1. Any include and
  2. None of the excludes

As an exception to this rule - if there are no includes declared, all dependencies are considered included. (Excludes are still effective in this case.)

This process is performed for each assembly subset (e.g., host-type, environment, configuration - see Concept - Gradle Assembly Tools). For example, if a cartridge is included in the environments production and development in the inherited assembly, you may select to inherit it only for environment development.

The following listing shows the different ways to define a predicate using different overloadings of includeDependencies. The same overloadings are available for excludeDependencies.

Note

Component subsets (host-types, environment, configurations) are available as object literals by their name inside the inheritFrom closure. Examples in the code below are production and development.

assembly {
	inheritFrom('com.example.assembly:base-assembly') {
		// Include all dependencies in the given component subsets.
		includeDependencies from:[production, development]

		 // Include dependencies by their (unqualified) name in all component subsets they were originally in
	    includeDependencies 'cartridge1', 'cartridge2'

		// Include dependencies by their (unqualified) name, but only in the given component subsets
	    includeDependencies 'cartridge1', 'cartridge2', from:[production, development]

		// Include dependencies based on the return value of a closure for arbitrary complex predicates
		includeDependencies { ModuleVersionIdentifier dependency, componentSubset -> 
			//TODO: Return true or false, based on properties of the dependency.
		} 
	}
}

There are two pitfalls here that may not be obvious:

Pitfall 1: No warning for missing includes

Includes are only a statement for filtering, they do not ensure the existence of a certain dependency. If you specify includeDependencies 'cartridge1' and the base assembly contains 'cartridge1', it is inherited. If it does not however, there is nothing to inherit, but no warning will be given about that. Think of it as how a file filter in a directory copy operation behaves, instead of how a command to copy a specific file would do.

To explicitly check that a certain cartridge is included, add the following line after the inheritFrom closure:

//TODO: Replace 'production' by name of the environment and 'cartridge1' by the name of the of the cartridge
assert project.configurations.'production-cartridges'.dependencies.find { Dependency dependency ->  dependency.name == 'cartridge1' }

Pitfall 2: A single include changes it all

If you do not specify any includes, all dependencies are inherited. Note that if you start to specify any include, they will start to kick in and only explicitly included dependencies are inherited. See the next section for how to overcome this.

9.3.2 Scoping for includeDependencies and excludeDependencies

If you want to declare includes explicitly for some known subset of dependencies, but just inherit everything in others, you can limit the effect of includes and excludes by scopes. This is very similar to Gradle's nesting of copy specs, like providing different include patterns for different sub-directories.

Available scopes are:

  • All component subsets (i.e., host-types, environments, configurations)
  • A special scope by the name of environments, that contains all environments
  • A special scope by the name of hostTypes, that contains all host types

Scopes are opened by calling the method within in the assembly inheritance spec. This method must be passed a closure, which is an assembly inheritance spec itself, i.e., it can also call includeDependencies and  excludeDependencies.

Consider the following examples:

Example A not using scopes
assembly {
	inheritFrom('com.example.assembly:base-assembly') {
		includeDependencies 'cartridge1', from:production
	}
}
Example B using scopes
assembly {
	inheritFrom('com.example.assembly:base-assembly') {
		within(production) {
			 includeDependencies 'cartridge1'
		}
	}
}

In both examples the environment only contains the cartridge1 inherited from the environment production. In example A however, cartridge1 is the only dependency inherited from the assembly. No dependencies are inherited from other environments or host-types.

In example B, the includeDependencies call does not affect any other component subset than production. So - because this is default when there are no explicit includes - other environments, host-types and configurations are inherited with all their dependencies.

Scoping also works for  excludeDependencies calls, but calls within a scope and calls using the  from notation are equivalent.

The following example shows, how to:

  • inherit production and development as is (i.e., with all their dependencies)
  • inherit no other environments
  • in host-type appserver only inherit runtime
  • inherit all other host-types and configurations as is
assembly {
	inheritFrom('com.example.assembly:base-assembly') {
		within(environments) {
			includeDependencies from:[production, development]
		}
 
		within(appserver) {
			includeDependencies 'runtime'
		}
	}
}

9.3.3 Selection of Artifacts

The selection of artifacts works very similar to those of dependencies. Currently all artifacts are associated with the Ivy configuration default, so it is not necessary to look for artifacts by their assembly subset.

The following listing shows the different ways to define a statement using different overloadings of includeArtifacts. The same overloadings are available for excludeArtifacts.

assembly {
	inheritFrom('com.example.assembly:base-assembly') {
		// Include artifact by their type
	    includeArtifacts type:['deploy-gradle', 'deploy-settings-gradle']

		// Include artifacts by their name (excluding their extension)
	    includeArtifacts name:'branding'
		
		// Include artifacts based on the return value of a closure for arbitrary complex predicates
		includeArtifacts { ResolvedArtifact artifact, componentSubset -> 
			//TODO: Return true or false, based on properties of the artifact.
		}
	}
}

9.3.4 Selection of Assembly Subsets

Host-types, environments and configurations that exist in the assembly are automatically copied if any dependency or artifact is inherited from them. If no dependency / artifact is inherited, they are omitted.

Note

If you define a component subset in your assembly which already exists in the inherited assembly, then your subset takes precedence (this specifically applies to host-type attributes).

10 Recipe: Generate a Database Dump

(For Gradle tools version 1.0-1.1, for Gradle tools version >= 2.0 see Recipe: Run DBInit/DBMigrate )

10.1 Problem

The Continuous Integration server should create a database dump during the assembly process and include it as artifact of the assembly, so it can be used for faster setup of demos / developer environments. The maintainer of the assembly needs to declare this.

10.2 Solution

Perform the following steps, see discussion for details:

  1. Include database dump as artifact, which automatically adds its generation to the assembly process.
  2. To speed up the assembly process, optionally configure to generate the dump incrementally based on the dump of another assembly.
  3. Make sure branding installations are handled correctly.

10.3 Discussion

10.3.1 Include Database Dump as Artifact

To include the database dump as an artifact, add the following line to gradle.properties file in your Gradle user home:

releaseWithDump = true

If this property is set, the database dump will be generated before publishing the assembly and the published assembly will include the database dump as artifact of type dump. Typically this is used only in the Gradle user home of the CI server, as publishing assemblies for developers must be fast and developers do not need to generate dumps themselves. You may still set it for a developer temporarily for testing the behavior.

This also includes running the deployment, before the dbinit upon every call of the 'publish' task. (See Concept - Gradle Assembly Tools for an overview of the assembly process.)

You may skip the deployment if you are sure that the last deployment succeeded and nothing relevant has changed. To do so add the project property skipDeployment to your execution. For example:

gradlew publish -PskipDeployment

10.3.2 Generate the Dump Incrementally

You may choose between:

  1. Creating a dump from scratch: This cleans the database, runs a full DBInit and exports the database contents as new dump. This is the default behavior.
  2. Create a dump incrementally: Import a database dump from another assembly, run a partial DBInit over additional cartridges and export the database content as new dump.

A typical use case for incremental dumps is adding demo data in a special assembly with a demo cartridge, that otherwise inherits from a non-demo assembly.

To create a dump incrementally insert the following block in to the build.gradle file of the assembly (note that this uses an assemblyBuild block as root element):

assemblyBuild {
    database {
		//TODO: Replace by <org>:<name> of the assembly you want to inherit database dump from
        inherit('com.example.assembly:base-assembly')
        
		//TODO: Insert list of cartridges for which the partial DBInit should be executed
        initCartridges = ['cartridge1', 'cartridge2']
    }
}

For this to work, you have to find an assembly that contains a suitable dump (e.g., containing a subset of the cartridges in your assembly). This is typically the same assembly that you inherit dependencies from.

No Changes of Locales

One known restriction of incremental database dumps is that you cannot add or remove locales. If you want to reuse the dump, you have to make sure to configure the same locales in <IS_SHARE>/system/config/cluster/localization.properties file as where used for the base dump.

If an incremental generation of the database dump is not possible because of some reason, you can always revert to the slower generation from scratch.

If you inherit from another assembly, that you built locally, make sure that it was built with releaseWithDump enabled (otherwise there is no dump to inherit and the incremental dump generation will fail).

10.3.3 Handle Branding Installations

The  BrandingPreparer in  bc_foundation is used to install branding packages during DBInit. It extracts branding files into the IS_SHARE/sites folder. When initializing a database with a pre-generated dump, instead of running DBInit, these files are missing.

To work around this, the assembly process can also package the extracted branding installations from the sites folder and include them for deployment. There are two cases to consider:

  1. Case: The DBInit in your assembly runs the BrandingPreparer, but the DBInit of the base assembly does not. (Or you do not have a base assembly / perform a full DBInit anyway.) Add the following line to build.gradle:

    build.gradle
     apply plugin: 'ish-assembly-branding'

    This activates packaging of branding installations after generation of the dump and publishs them as an artifact branding of type share. The package will be automatically deployed by standard Intershop 7 build plugins.

  2. Case: The DBInit in your assembly does not run the BrandingPreparer, but the DBInit of the base assembly did. (If unsure, check out the ivy.xml file of the base assembly in the repository for an artifact of name branding and type share.) Add the following line to the build.gradle file (or just add an include to the existing inheritFrom):

    build.gradle
    assembly {
    	// TODO: Replace by <org>:<name> of your inherited assembly
    	inheritFrom('com.example.assembly:base-assembly') {		
    		// Control which dependencies/artifacts to inherit, see discussion
    		// By default all dependencies in all component subsets and all artifacts are inherited
    		includeArtifacts name:'branding'
    	}
    }

    This will inherit the packaged branding installations from the base assembly.


It is currently not possible to run the BrandingPreparer both, in the base assembly and in a partial DBInit in your assembly. In this case you have to revert to a full DBInit.

10.3.4 Encryption

Some of the database content is encrypted. To decrypt the content again:

  1. The previously used encryption must be declared using the same ID in <IS_SHARE>/system/config/cluster/encryption.properties.
  2. Also the keystore file <IS_SHARE>/system/config/cluster/intershop.keystore has to contain the same keys or use  algorithm=PLAIN, e.g., no encryption at all.

Before running the DBInit for creating a dump, the assembly process will automatically modify the encryption.properties to use PLAIN as encryption algorithm:

intershop.encryption.0.id = demo
intershop.encryption.0.algorithm = PLAIN
intershop.encryption.0.passphrase = topsecret
intershop.encryption.0.default=true

11 Recipe: Run DBInit/DBMigrate

(For Gradle tools version >= 2.0, for Gradle tools version 1.0-1.1 see Recipe: Generate a Datatabase Dump)

11.1 Problem

The Continuous Integration server should run DBInit and/or DBMigrate during the assembly process for two reasons:

  1. Test that DBInit can be run successfully or that a dump of an older version can be migrated. (Also use the resulting database content for further tests, like that the application server starts.)
  2. Export a database dump after running DBInit/DBMigrate and publish it as as artifact of the assembly, so it can be used for faster setup of demos / developer environments.

The maintainer of the assembly needs to declare this.

11.2 Solution

  1. Configure how to run DBInit/DBMigrate:
    1. For DBInit choose between running incrementally based on the dump of another assembly, or running a full DBInit over all cartridges.
    2. For DBMigrate specify which dump to import.
    3. Make sure branding installations are handled correctly.
  2. Trigger DBInit/DBMigrate:
    1. Specify to export a database dump and publish it as an artifact by setting the project property releaseWithDump.
    2. Alternatively explicitly run the dbinit or dbmigrate task.

You may configure DBInit and DBMigrate in the same assembly, but never run both in the same Gradle execution.

11.3 Discussion

11.3.1 Configure DBInit

You may choose between:

  1. Creating a dump from scratch: This cleans the database, runs a full DBInit. This is the default behavior.
  2. Create a dump incrementally: Import a database dump from another assembly, run a partial DBInit over additional cartridges.

A typical use case for incremental dumps is adding demo data in a special assembly with a demo cartridge, that otherwise inherits from a non-demo assembly.

To create a dump incrementally insert the following block in to the build.gradle file of the assembly (note that this uses an assemblyBuild block as root element):

assemblyBuild {
    dbinit {
		//TODO: Replace by <org>:<name> of the assembly you want to inherit database dump from
        importFrom('com.example.assembly:base-assembly')
        
		//TODO: Insert list of cartridges for which the partial DBInit should be executed
        initCartridges = ['cartridge1', 'cartridge2']
    }
}

For this to work you have to find an assembly, that contains a suitable dump (e.g., containing a subset of the cartridges in your assembly). This is typically the same assembly that you inherit dependencies from.

No Changes of Locales

One known restriction of incremental database dumps is that you cannot add or remove locales. If you want to reuse the dump, you have to make sure to configure the same locales in <IS_SHARE>/system/config/cluster/localization.properties file as where used for the base dump.

If an incremental generation of the database dump is not possible because of some reason, you can always revert to the slower generation from scratch.

If you inherit from another assembly, that does not include a dump and the project property releaseWithDump is false, a warning is logged and a full dbinit is executed (this is typically the case for developers).

11.3.2 Handle Branding Installations

The  BrandingPreparer in  bc_foundation is used to install branding packages during DBInit. It extracts branding files into the IS_SHARE/sites folder. When initializing a database with a pregenerated dump, instead of running DBInit, these files are missing.

To work around this, the assembly process can also package the extracted branding installations from the sites folder and include them for deployment. There are two cases to consider:

  1. Case: The DBInit in your assembly runs the BrandingPreparer, but the DBInit of the base assembly does not (or you do not have a base assembly / perform a full DBInit anyway). Add the following line to build.gradle:

    build.gradle
     apply plugin: 'ish-assembly-branding'

    This activates packaging of branding installations after generation of the dump and publishs them as an artifact branding of type share. The package will be automatically deployed by standard Intershop 7 build plugins.

  2. Case: The DBInit in your assembly does not run the BrandingPreparer, but the DBInit of the base assembly did (if unsure, check out the ivy.xml file of the base assembly in the repository for an artifact of name branding and type share). Add the following line to the build.gradle file (or just add an include to the existing  inheritFrom):

    build.gradle
    assembly {
    	// TODO: Replace by <org>:<name> of your inherited assembly
    	inheritFrom('com.example.assembly:base-assembly') {		
    		// Control which dependencies/artifacts to inherit, see discussion
    		// By default all dependencies in all component subsets and all artifacts are inherited
    		includeArtifacts name:'branding'
    	}
    }

    This will inherit the packaged branding installations from the base assembly.


It is currently not possible to run the  BrandingPreparer both, in the base assembly and in a partial DBInit in your assembly. In this case you have to revert to a full DBInit.

11.3.3 Configure DBMigrate

Before running DBMigrate a database dump must be imported as start point of the migration.

To import a dump created by the CI server in an older version of the assembly insert the following block in to the build.gradle file of the assembly (note that this uses an assemblyBuild block as root element):

assemblyBuild {
    dbmigrate {
		//TODO: Replace by <org>:<name>:<version> of assembly to import old dump from
        importFrom('com.example.assembly:assembly:1.0.0.0.+')        
    }
}

Alternatively provide a dump file explicitly:

assemblyBuild {
    dbmigrate {
		//TODO: Replace by dump file to import. This example assumes that the dump file is located in the source directory of the assembly
        importFile = file('dump-1.0.0.0.dmp')
    }
}

11.3.4 Include Database Dump as Artifact

To include the database dump as an artifact, decide whether it should originate from running DBInit or DBMigrate.

Add one the following lines to gradle.properties file in your Gradle user home:

# include DBInit dump
releaseWithDump=dbinit
# include DBMigrate dump
releaseWithDump=dbmigrate

If this property is set and you execute the task publish:

  1. The dbinit/dbmigrate task and their dependencies (see next section) are executed.
  2. The database dump is be exported to a file.
  3. The dump file is included as artifact of type dump.

Typically this is used only in the Gradle user home of the CI server. The default value of releaseWithDump is false.

If releaseWithDump is false and you execute the task publish:

  • Neither dbinit nor dbmigrate are executed and
  • The resulting assembly does not contain an artifact.

This way publishing assemblies for developers is fast. You may still set it for a developer temporarily for testing the behavior.

11.3.5 Run DBInit/DBMigrate Explicitly

You may also run dbmigratedbinit tasks explicitly. This will also trigger their dependencies:

  1. Deploy a server
  2. Clean the database
  3. Import a dump if configured/necessary

If any of the prerequisites or dbmigratedbinit itself fail, the build will fail as well. This can be used to test dbinit/dbmigrate.

It will not automatically export the dump, nor include it as an artifact in the assembly.

You may skip the deployment if you are sure that the last deployment succeeded and nothing relevant has changed. To do so add the project property skipDeployment to your execution. For example:

gradlew dbinit -PskipDeployment


12 Recipe: Add Solr Cloud Search Service

Note

This recipe is only valid for the additional Solr6 assembly for 7.9. If this assembly is not used, the original recipe from 7.8 remains valid.

12.1 Problem

To use Solr6-based search in the Commerce Management application and storefront, it is necessary to configure the application server so it can access the Solr6 cluster.

12.2 Solution

Note

Precondition

A Solr cloud cluster has to be set up. For information about the setup see Getting Started with SolrCloud in the Apache Solr Reference Guide.

The Solr6 extension needs to be configured for the host type share.

settings.gradle
...
assembly {
	...
    config{
        solrcloud {
    		zooKeeperHostList = '123.456.789.012:1234';'123.456.789.013:1234'
    		clusterIndexPrefix = 'myHost-1'
		}
    }
}

The SolrCloudDeploymentPlugin configures <IS_SHARE>/system/config/cartridges/ac_search_solr.properties to enable Intershop 7 to connect with the Solr cluster.

Properties relevant for the Solr-based search service are:

PropertyDescriptionExampleDefault Value

solr.zooKeeperHostList

The list of ZooKeeper connection URLs (separated by ';'), which manage the Solr cluster

'localhost:9983' no
solr.clusterIndexPrefixThe prefix to distinguish Solr indexes within the Solr cluster 'myHost-1 '  

<HOSTNAME>-<INSTANCE-ID>

12.3 Discussion

For further information about setting up a Solr cluster and more detailed description, please see Guide - Deployment Solr6.

13 Recipe: Merge Host Types into a New One

13.1 Problem

Two or more host types should be used to form a new one.

13.2 Solution

  • The new host type contains all components of the two previous host types.
  • To obtain the values of the attributes includeJavadoc, includeCartridges, includeShare, includeLocal for the new new host type, the corresponding values of the previous host types are OR linked

The following example refers to the merge of the host types:

  • appserver
  • discoveryserver

to a new host type

  • discoveryappserver
build.gradle (appserver)
assembly {
    hostTypes {
		appserver{
			include (
                'com.intershop:3rd_tomcat',
                'com.intershop:3rd_ant',
                // further components from appserver
            ) {
                transitive = false
            }
            includeLocal = true
            includeCartridges = false          
        }
	}
}
build.gradle (discoveryserver)
assembly {
    hostTypes {
		discoveryserver
		{
			include (
                'com.intershop.microservice:DiscoveryServer',
            ) {
                transitive = false
            }
		
            includeLocal = true
            includeShare = false
            includeCartridges = false     
        }
	}
}
build.gradle (discoveryappserver)
assembly {
    hostTypes {
		discoveryappserver
		{
			include (
				'com.intershop:3rd_tomcat',
                'com.intershop:3rd_ant',
                // further components from appserver
                'com.intershop.microservice:DiscoveryServer',
            ) {
                transitive = false
            }
		
            includeLocal = true
            includeShare = false
            includeCartridges = false      
        }
	}
}

14 Recipe: Add External Library with Newer Version of Dependent Library to Be Deployed

14.1 Problem

The use case is that you want to include a newer version of an external library which is already included (and delivered) by gradle. This example explains this use case with the library hierynomus that depends on bouncycastle.

14.2 Solution

Perform the following stept, see discussion for details:

  1. Tell gradle to use your specific configuration.
  2. Configure build.gradle and create a specific configuration file.
  3. Deploy the system/cartridge.

14.3 Discussion

14.3.1 Requirements

In your project there is a dependency on library hierynomus which has internal depency of bouncycastle. The build of the cartridge and "refresh gradle dependencies" with the new version of bouncycastle work fine. However, during deployment, the old version of bouncycastle is still used and the created jar file is put into the lib directory share\system\cartridges\libs\release\lib. You need to tell gradle to use your specific configuration, which will be created  in the following.

This is how your gradle file looks currently:

build.gradle  Expand source
versionRecommendation {
 
    provider {
 
        // can be activated with a special version
 
        ivy('infrastructure',  'com.intershop.infrastructure:assembly') {}
 
        ivy('platform',  'com.intershop.platform:platform') {}
 
        ivy('content',  'com.intershop.content:assembly') {}
 
        ivy('business',  'com.intershop.business:intershop7') {}
 
        ivy('b2b',  'com.intershop.b2b:b2b') {}
 
 
 
        // default dependencies
 
        ivy('icm-b2x', 'com.intershop.assembly:commerce_management_b2x:7.9.x.x') {}
 
 
 
        // intershop build dependencies
 
        properties('intershopBuild', file('intershopBuild.version')) {}
 
    }

14.3.2 Configure build.gradle and Create Specific Configuration File

To configure gradle you need to tell gradle that there is a separate configuration file for a specific library (bouncycastle in this case) which has to be used.

The configured build.gradle file is shown below:

configured build.gradle  Expand source
versionRecommendation {
    provider {
        properties('overrides', file('overrides.version')) {}
 
        ivy('inspired-b2x', 'com.intershop.responsive:inspired-b2x:x.x.x') {}
 
        // intershop build dependencies
        properties('intershopBuild', file('intershopBuild.version')) {}
   }

In the following file (overrides.version) you specify the version of the library.

overrides.version  Expand source
# override these versions for this project
org.bouncycastle:bcprov-jdk15on = 1.57

14.3.3 Deployment

The build and refresh gradle dependencies still works. The last step is the deployment of the system/cartridge. To make sure that the specific configuration file was picked, you should find the correct version of the file in the lib folder: share\system\cartridges\libs\release\lib.

In the gradle dependy graph you will see, e. g., the following output:

Error rendering macro 'code': Invalid value specified for parameter 'com.atlassian.confluence.ext.code.render.InvalidValueException'
+--- org.bouncycastle:bcprov-jdk15on:1.49 -> 1.57

Disclaimer

The information provided in the Knowledge Base may not be applicable to all systems and situations. Intershop Communications will not be liable to any party for any direct or indirect damages resulting from the use of the Customer Support section of the Intershop Corporate Web site, including, without limitation, any lost profits, business interruption, loss of programs or other data on your information handling system.

Customer Support
Knowledge Base
Product Resources
Support Tickets