Introduction

This cookbook gives practical advise on how to develop using Gradle and Intershop build and deployment tools. The target audience are developers of Intershop customer projects and partners. Knowledge ofConcept - Continuous Delivery Tools (valid to 7.10)is assumed.

This cookbook presents a directory setup and a workflow, that is optimized for a single project.

Preconditions

Before a single developer can set up an environment, a few things have to be set up first for the team. The required team setup is out of scope of this cookbook. Such information is included in:

• Cookbook - Setup of CI Infrastructure- infrastructure setup for continuous integration (especially, setting up a pre-configured Gradle user home- developer configuration - available from the VCS)
• Cookbook - Gradle Build Tools - Setup of a Gradle multi-project

• Cookbook - Gradle Assembly Tools- Create an assembly

Glossary

Recipe: Set up a Development Environment

Problem

I want to develop on Intershop 7.8 CI or later. How do I get a Gradle-based development environment?

Solution

Setting up a development environments consists of the following steps:

1. Set up a Gradle user home.
2. Check out project using your VCS.
3. Build the Gradle multi-project.
4. Deploy a server.
5. Initialize the database.
6. Set up an Intershop Studio workspace.

The following steps provide you with a jump start without further ado. See discussion for further details.

Since command line syntax onCookbook - Gradle Developer Workflow#WindowsandCookbook - Gradle Developer Workflow#Linuxslightly differs, the steps are provided once for Windows and once for Linux. Except for the command line syntax both are equivalent.

Windows

* Commands provided by gradle_environment.bat

Starting services and the application server are out of scope of this document.

Linux

* Commands provided by gradle_environment.sh

Discussion

Gradle User Home

The Gradle user home (<GRADLE_USER_HOME>) is located in its default location, usually <OS_USER_HOME>/.gradle

Project Directory

The project home contains an environment script (gradle_environment.bat | gradle_environment.sh), see next section. The project home is checked into the VCS.

Developer Console

When setting up your CI environment a script gradle_environment.bat | gradle_environment.sh is generated. This script configures the directory layout you work with, by setting a few environment variables.

We refer to a console with these environment variables in place as a developer console. Run all gradlew commands and Intershop Studio from a developer console.

To start a developer console:

1. Open a console or use an existing one.
2. Go to Project home folder.
3. Run
   1. gradle_environment.bat on Windows or
   2. source ./gradle_environment.sh on Linux

The following table lists the most important environment locations provided from gradle_environment.bat | gradle_environment.sh represented by environment variables.

DEVELOPER_BASE

These environment variables are partly for your convenience and partly to configure Gradle/Intershop Studio.

The environment script provides the following alias commands:

Build, Publish and Repositories

Running gradlew publish in <PROJECT> actually consists of of two steps, first build, then publish. It is necessary to not only build components, but also to publish them to a repository (e.g., copy them to a folder with predefined structure) for two reasons:

1. Gradle needs to find dependencies for compiling Java. Gradle knows two kinds of dependencies:
   
   1. External dependencies are all dependencies across different project boundaries (e.g., the cartridge bc_foundation in the project p_business to the cartridge core in the project p_core). They are denoted by compile group: 'com.intershop', name: '<name>' in a component's build.gradle file. They must be resolved from (a local or remote) repository.
   2. Dependencies inside a project (e.g., core to isml, both inside p_core) are called project dependencies and are denoted by compile project(':<name>'). (Project dependencies resolved directly from the target directories)
2. The deployment needs to find all components to deploy in a (local or remote) repository.

Deployment Configuration (environment.properties.sample)

The first call of gradlew in the project will generate a sample file for the deployment configuration. But this will only happen if:

1. No file environment.properties does exist in the assembly project directory.
2. No project property <project name>BuildEnvironmentProperties is specified or the file, specified for this property, does not exist
3. No project property buildEnvironmentProperties is specified or the file, specified for this property, does not exist

If no sample file is generated please check the location of the assembly project, the environment settings and (depending on the workflow) either the

• gradle.properties in your GRADLE_USER_HOME or the
  
• gradle_environment.bat|sh

for some of this configuration properties. Furthermore, there is a property buildEnvironmentPropertiesSample, which defines the location of the sample file. If this property is not set, the sample file can be found in the default location:
<PROJECT>/<assembly name>/target/samples/environment.properties.sample

Initial Deployment

The developer deployment first generates a <SERVER>/settings.gradle file from the environment.properties configuration. Based on this file it then runs standard deployment procedure as described inConcept - Gradle Deployment Toolsspecifying the host type all, e.g., deploying all host types of the assembly. As deployment source uses the repositories declared in your corporate plugin as well as your local repository at <REPO>.

Besides being based on the same deployment scripts, the developer deployment is also tailored for development in a few ways:

• The properties IS_SOURCE in <SERVER>/local/intershop.properties and cartridges.source in <SERVER>/share/system/cartridges/cartridgelist.properties. This ensures that the application server uses the cartridges' checkout directory to load static artifacts and classes.

• Deploy development.properties from <ASSEMBLY>/development.properties to <SERVER>/share/system/config/cluster/development.properties. This way reload and no-preload behavior is pre-configured for the application server.
• Deploy all cartridges exclusively to the assembly environment development (i.e., that are not included in the production environment), like the dev_lilith cartridge.

• Open debug ports in tomcat.bat | tomcat.sh and dbinit.bat | dbinit.sh as configured in <ASSEMBLY>/environment.properties.

The task enableHotCodeReloading will add the following properties to the gradle.properties located in the GRADLE_USER_HOME:

sourceDirectories=<path to the project sources>
sourceCartridges.<project name>=<components of your project>

These settings are necessary for the development environment to load changed files in the server direct from the sources. It is also necessary for other development settings, e.g., the debug port configuration.

Post-install Scripts and Services

The post-install scripts perform all deployment operations for which administrator / root privileges are required. Currently this is only the creation of services. The developer deployment uses the current user as deployment user and – on Windows – also as user for executing the deployed applications / services. Therefore, you have to enter your password for every service to be created.

After the post-install scripts, running services / applications is completely up to you and works mostly as before.

If starting any service fails on Windows, please refer to the Recipe (Windows) Starting a Service Fails in Cookbook - Deployment Tools, according to your ICM version:

• Cookbook - Gradle Deployment Tools (7.4 CI - ICM 7.7)
• Cookbook - Deployment Tools ICM 7.8
• Cookbook - Deployment Tools ICM 7.9
• Cookbook - Deployment Tools ICM 7.10

Intershop Studio

Intershop Studio will find your sources, the deployed server etc. by relying on the environment variables in a developer console. Therefore, start Intershop Studio from a developer console. No more initial configuration in the preference menu is necessary.

To import your project into the workspace:

1. From the menu select File | Import...
   The Import dialog opens.
2. Choose Intershop Studio | Component Set Projects into Workspace (Gradle) wizard and click Next.
   The Import Component Set (Gradle) wizard is started.
3. Browse... for the directory of the project folder.
4. Click Build Model.
5. Select the desired projects (cartridges).
6. Click Select Required to also select all required projects.
7. Click Finish.
8. Wait until the background actions have finished.

To import your assembly into the workspace, use the same steps, but instead select the import wizard Gradle | Gradle Project. I.e:

1. From the menu select File | Import...
   The Import dialog opens.
2. Choose Gradle | Gradle Project wizard and click Next.
   The Import Gradle Project wizard is started.
3. Browse... for the directory of the assembly.
4. Click Build Model.
5. Click Select All.
6. Click Finish.
7. Wait until the background actions have finished.

Process / Data Flow

The following diagram shows how data flows between directories (blue) during the different steps (red).

Recipe: Run Gradle Tasks

Problem

Gradle tasks are the units of work you can trigger. This recipe describes how. For an introduction beyond this recipe, please refer to the Gradle user guide.

Solution

You may run Gradle tasks using Intershop Studio or the console. Generally you have to:

1. Specify which Gradle project to run tasks in.
2. Specify which Gradle task(s) to execute.

The assembly, all cartridges and the multi project itself are Gradle projects. Components and assembly are sub projects of one multi project.

Tasks can have dependencies to other tasks. Executing a certain task will always also execute its dependencies.

You may specify multiple tasks to run in one execution of Gradle. The order of depending tasks is mostly determined automatically by Gradle.

• Depending tasks will always be executed after their dependencies.
• Tasks that do not depend on each other may still have a pre-configured order, if you run them together.

Tasks independent of each other are executed in the order in which you are passing in the tasks.

Running Tasks from Intershop Studio

In Intershop Studio each Gradle project is represented by an Eclipse project.

To execute tasks in a cartridge/assembly or in the root project:

1. Right-click the corresponding project in the Package Explorer.
   The context menu opens.
2. Click Run As | Gradle Build...
   For cartridges you can also reach that dialog by clicking Build Cartridge (Gradle Build) in the menu.
3. In the dialog Edit Configuration select the tasks to be executed.
4. Optionally, click Order to sort the tasks.
5. Click Apply to confirm your settings.
6. Click Run to execute the task(s).

You may also specify additional parameters in the dialog.

To reuse and switch between multiple configurations per project (e.g., different deploy or database tasks in the assembly), give them different names. Thus they can be found in the list of the external tool configurations.

Running Tasks via Console

To execute tasks via Console:

1. Open a developer console.
2. Go to <PROJECT> directory.
3. Run:
   1. On Windows: gradlew :<task1> :<task2>, e.g., gradlew :deployServer :importDump
   2. On Linux: ./gradlew :<task1> :<task2>, e.g., ./gradlew :deployServer :importDump

To execute a task in a certain cartridge (i.e., a child-project), pass in :<cartridge>:<task>, e.g., gradlew :core:clean :core:build.

You can also combine tasks from different cartridges, e.g., gradlew :core:build :bc_foundation:build.

FAILURE:
Could not determine which tasks to execute.

* What went wrong:
Task 'build' not found in root project 'projectName'.

* Try:
Run gradlew
tasks to get a list of available tasks. 

Running Tasks for Remote Tests

If you start remote tests for the assembly the server will be not stopped or started.

For most tasks during development of remote tests it is necessary to have a running server. The situation on a CI server is different. Before a remote test can run, it is necessary to start the server. Regardless of the remote test result it is required to stop the server.

If you want run remote tests on your development machine it is necessary to exclude some stop tasks:

gradlew -x stopNodemanager -x stopWebserver remoteTest

Discussion

The following section is a loose collection of helpful hints to get you started with Gradle. Run all of them in a developer console.

To display all sub projects in the Gradle multi-project execute the following inside <PROJECT>:

gradlew :projects

To display tasks of a (root) project:

gradlew :tasks

To display the tasks of a certain sub-project (e.g., cartridge, assembly) run:

gradlew :<name>:tasks

To find out which tasks are actually going to be executed (e.g., which dependencies are automatically pulled in) and in which order:

gradlew <tasks> --dry-run

Recipe: Initialize the Database

Problem

The database must be initialized before testing (or as part of the test, i.e., to see if the database can be initialized).

Solution

There are two ways to initialize your database.

Import Database Dump of Release

You may import the database dump generated for the released version of your assembly. This is only useful if your local changes have no effect on the initial database state. To import the dump run the task importDump on the assembly.

The release version from which the dump is taken is determined by the property dump.<assembly organization>.<assembly name> in <GRADLE_USER_HOME>/gradle.properties. This way re-running the importDump task always imports the same dump, unless you change the property.

The dump property is automatically created upon first run of the importDump task. To update the property to the latest released version run the updateDumpVersion task.

Execute DBInit

You may run the DBInit through assembly. To do so:

1. Open a developer console and go to <PROJECT>
2. Run
   
   1. On Windows: gradlew dbinit -PskipDeployment
   2. On Linux: ./gradlew dbinit -PskipDeployment

Discussion

Import Database Dump of Release

The dump is downloaded from the release repository and handed over to ant import at <SERVER>/local/tools/misc.

Also branding installations, which are usually created by running DBInit are downloaded from the release repository and extracted into <SERVER>/share/sites.

You can still use ant import at <SERVER>/local/tools/misc to import other dumps.

Execute DBInit

The command first executes ant dbschema-clean at <SERVER>/local/tools/misc to purge your database account. After that it executes dbinit --classic in <SERVER>/local/bin, i.e., a full DBInit over all cartridges without interactions.

You may still run these commands manually if you like - or need to pass in other options.

Recipe: Make Changes Effective on the Server

Problem

After the initial setup, the developer changes the sources of the project. This may happen because of changes made by the present developer or updates to the VCS working copies of a project. The developer must be able to see these changes in the deployed server.

Solution

Depending on the kind of change, different actions are necessary to make the changes visible in the running server.

See discussion below for how to perform these actions.

Discussion

Please, refer to theRecipe: Run Gradle Taskson how to run these tasks. This recipe also mentions how to execute a task on multiple or all cartridges at once.

Rebuild

To rebuild (e.g., recompile) a cartridge, run the build task on that cartridge.

The buildDependents task builds the cartridge itself and all cartridges in your project that depend on it. This is useful to see if Java changes cause compilation errors in depending components.

If not cleaning the build results before building / publishing, Gradle performs an incremental build. Gradle only re-executes Gradle tasks whose inputs have changed (or whose outputs have been tampered with). This is somewhat safer and more reliable than re-executing an Ant build. Using the non-cleaning, incremental build should suffice for developers most of the time.

To execute a clean build run the tasks clean and build on the cartridge(s).

Using these tasks does not touch the contents of your local repository. If you later redeploy your changes, make sure to publish them before.

Publish

To republish a cartridge to your local repository, run the publish task on that cartridge. This overwrites the local version of that cartridge inside your local repository at <REPO>.

Redeploy

All deployment tasks are run on the assembly project. Before you can deploy any change, you must have published it.

To redeploy all cartridges of your project run the task deployCartridges.

To redeploy only a few cartridges:

1. Open a developer console and go to <ASSEMBLY>
2. Run
   
   1. On Windows: gradlew deployCartridges -Pcartridges="<space separated list of cartridges>", e.g., gradlew deployCartridges -Pcartridges="core bc_foundation"
   2. On Linux: ./gradlew deployCartridges -Pcartridges="<space separated list of cartridges>", e.g., ./gradlew deployCartridges -Pcartridges="core bc_foundation"

To redeploy the whole server run the task deployServer.

Running the post-install scripts again is not necessary after redeployment.

Recipe: Run Automatic Tests

Problem

A developer needs to execute and debug tests.

Solution

There are currently three ways to execute tests.

Run Tests Inside Intershop Studio

Before being able to run the system or integration tests, you must set up your IStudio workspace:

1. From the menu select Window | Preferences.
   You may run into a problem when opening the preference dialog the first time, see the boxIssue with Intershop Studio Preferencesabove.
2. Go to preference page Intershop Studio | JUnit Cartridge Test | Classpath.
3. Check both Add Intershop Suite Server entries and All components.
4. Click Apply to confirm your settings.
5. Click Ok.

To run or debug a single test:

1. Select a test class, a Java package containing test classes or a method of a test class
2. Open the context menu and select
   
   1. Run as | JUnit Test for unit tests (located in the tested cartridge, see discussion), or
      Run as | JUnit Cartridge Test for integration or system tests, or
   2. Debug as | JUnit Test for unit tests, or
      Debug as | JUnit Cartridge Test for integration or system tests.

This way of execution applies to all types of tests.

Execute Tests with Gradle

You may run all unit tests of a cartridge (i.e., not a _test-cartridge) by running the Gradle test task on that cartridge. A report is generated per cartridge at <PROJECT>/<cartridge>/target/reports/tests/index.html.

You may also run the Gradle test task on the project. This executes the test tasks of all cartridges and assembly. If your project uses the old project structure with separate component set and assembly projects, the assigned plugin com.intershop.build.gradle.plugins.ComponentSetPlugin (component-set) will generate an overall report of all cartridge tests. The report is then located under <PROJECT>/target/reports/allTests/index.html.

Execute Tests with Test Runner

You may run multiple integration tests or system tests with the Intershop Test-Runner:

1. Open a developer console.
2. If your prompt does not yet start with ES<n>|...(e.g., ES1|D:\is7\trunk\), run <SERVER>/local/bin/environment.bat.
3. Go to <SERVER>/local/tools/misc.
4. Run ant test.

Discussion

There are three kinds of tests that are treated differently.

Unit Tests

Unit tests are tests that require nothing but a properly populated class path to run. Especially, they cannot require a pre-created files or directories (like through deployment) or a database. (They may however start up a pretty complete server in their initialization - as long as this server does not access any file system resources, like configuration files.)

Super classes/annotations for Unit tests are:

• Base class junit.framework.TestCase (legacy test cases written for JUnit version <= 4)
• Classes with @RunWith annotation (newly written test cases starting with JUnit 4)

Unit tests are located in the same cartridge as the functionality they test.

Run unit tests using:

• Intershop Studio, or
• Gradle (developer tools)

Integration Tests

Integration tests require a fully deployed server and are executed within that server, so they have direct access to the classes and objects in the JVM.

Super classes for Integration tests are:

• com.intershop.tools.etest.server.UnitTestCase(sic!)
• com.intershop.tools.etest.remote.CAPITestCase

Integration tests are located in the accompanying test cartridge of the cartridge being tested (bc_pricing_test for bc_pricing).

Run integration tests using:

• Intershop Studio
• Test Runner

System Tests

System tests run outside the server and access it via any remote interface (e.g., via HTTP). Technically they are run like unit tests, but they require additional Java properties to know how to connect to a server.

Run integration tests using:

• Intershop Studio
• Test Runner

Recipe: Create a Cartridge

Problem

Developers need to add new cartridges to an existing multi project.

Solution

To add a new cartridge to an existing project perform the following overall steps:

1. Check out an existing project and import it to Intershop Studio workspace (see according recipes).
2. Create a new cartridge using Intershop Studio.
3. Fill the build.gradle of new cartridge.
4. Add the cartridge to build.gradle file of the assembly and specify the order.
5. Redeploy the assembly

Please see discussion below for details on how to perform these steps.

It is not necessary to register the cartridge inside the multi-project in any way. Instead all sub-folders not explicitly excluded in <PROJECT>/settings.gradle are treated as components and included in the build. This way upon commit they are also included in the build of the continuous integration server automatically.

Other non-Gradle related steps, like adding the cartridge to an application type may still be necessary.

Discussion

Create a New Cartridge Using Intershop Studio

1. From the menu select File | New | Cartridge.
2. Enter cartridge and Java package name.
3. Click Next.
4. Use the template Default (Gradle Support) (should be already selected).
   Properties like organization and version are now specified by the project.
5. Click Finish.

Fill build.gradle

The build.gradle file of a cartridge is both, the build script and the build configuration file for that cartridge. Reusable build functionality comes in Gradle build plugins. Different kinds of artifacts are handled by different plugins. The Gradle build will fail, if you apply a plugin, but there are no matching source files to process.

Intershop Studio generates an initial build.gradle file that is suitable for cartridges containing Java source code and static files. Please refer toCookbook - Gradle Build Tools | Configure the Build of a Component / Cartridgeon how to configure your build.gradle file.

...
Execution failed for task ':<cartridge>:isml2classMain'
> taskdef class com.intershop.beehive.isml.capi.ISML2JSP cannot be found using the classloader AntClassLoader
....

Declare Dependencies

When declaring dependencies you need to distinguish between:

1. Dependencies to cartridges in the same multi-project.
   These are also called project dependencies. Declare them using the syntax compile project(':<cartridge name>') inside the dependencies block.
2. Dependencies to cartridges in another multi-project.
   These are called external dependencies. Declare them using the syntax compile group: '<cartridge organization>', name: '<cartridge name>'

The following example demonstrates the different declarations.

dependencies {
    compile group: 'com.intershop', name: 'tools'
	compile project(':customer_base')
} 

As already used from dependency management with Ivy, the compile classpath is resolved intransitively: If you specify a dependency to the core cartridge you will not automatically have a dependency to tools. So you will not have tools in your classpath unless you declare the dependency explicitly.

Add New Cartridge to Assembly

Please follow the Recipe inCookbook - Gradle Assembly Tools | Add Cartridges to an Assembly.

Publish and Deploy

To let the changes take effect locally, redeploy your assembly, as described inRecipe: Make Changes Effective on the Server.

Recipe: Remove a Development Environment

Problem

The developer needs to remove the development environment when finished to free system resources.

Solution

1. Stop processes running on the deployed server. (Optional for services - as they are stopped automatically later.)
2. Run Gradle task undeploy on the project. This first execution generates an uninstall script for the services.
3. Run as administrator / with root privileges (as requested in the console output): <SERVER>\postInstall\uninstallServices.bat. This stops and removes the services.
4. Run Gradle task undeploy on the project again. This purges your server directory.
5. Delete the directories <WORKSPACE>, your VCS working copies <PROJECT> and finally <GRADLE_USER_HOME>.

Recipe: Clean Gradle Cache

Problem

The Gradle cache located <GRADLE_USER_HOME>/cache currently grows indefinitely and may eventually fill the hard disk.

Solution

On Windows:

1. Close Intershop Studio if open.
2. Open a developer console and go to <PROJECT>.
3. Run gradlew --stop.
4. Delete the folder <GRADLE_USER_HOME>/cache using an OS command.

On Linux:

1. Close IntershopStudio if open.
2. Open a developer console and go to <PROJECT>.
3. Run ./gradlew --stop.
4. Delete the folder <GRADLE_USER_HOME>/cache using an OS command.

Discussion

Cache Contents and Cleaning

The Gradle cache contains:

• Artifacts downloaded from remote repositories, like binaries of cartridges used as dependencies, but also deployment and build plugins. This is by far the largest part.
• Precompiled gradle scripts.

Gradle has no feature for selectively cleaning the artifact cache so far: http://forums.gradle.org/gradle/topics/managing_artifact_cache_growth.

In the meantime, you have to revert to completely deleting the Gradle cache occasionally as described above. The disadvantage of this approach is that all artifacts required for further build and deployment processes will be downloaded again.

Stopping Daemons

Before deleting the cache directory all processes that hold file handles into it must be stopped, i.e., all Gradle daemons. You can do so by calling gradlew --stop.

You can see all running Gradle daemons (not just of the current Gradle user home) by running jps. Example of output:

19452 lilith.jar
22244 Jps
17448 GradleDaemon
13520
15852 GradleDaemon
16772
3468 GradleDaemon
22276