Introduction

Business Purpose

Examples:
Let us assume the Shop Manager of PrimeTech Specials wants to know, whether a Buy Now! banner can increase the customer conversions for placing orders.
Or if the new idea for a great promotion attracts enough attention.
Or he might want to gain first feedback from a small exclusive group of customers about an upcoming visual relaunch of the website by providing them a page variant.

There is now the possibility to define an A/B Test with two or more A/B Test groups. The users of PrimeTech Specials are transparently assigned to one of these groups.
One group only sees the usual storefront, but the other sees the test content that is assigned to that group only.
By monitoring the users main activities, like watching the pages with the test content, applying a test promotion or finally the checkout, the Shop Manager can overview a number of statistics.
These statistics provide feedback, whether an A/B Test group shows significant differences in the users behavior on the store.
Using this approach the Shop Manager gets a fundamental statistical analysis on how to use supporting content and making online marketing decisions.

Glossary

References

This document covers common questions regarding customizing and extending A/B Test functionality on the basis of examples:

• Cookbook - AB Testing

Business Overview

Storefront View (Users)

This section provides an overview of the principle of the Intershop A/B Test functionality.

An A/B test is defined by two or more A/B test groups. Each of them has different assigned test content that can be a set of content components, page variants and promotions.
One of the A/B test groups is the control group. The control group represents a fraction of the targeted users, which only sees the "normal" storefront.

• An A/B test must be accessible to make its content visible to a customer. An A/B test is accessible if it is active (i.e., enabled and running) and the user is part of the target group of the A/B test.
• From these eligible users only a specified fraction is allowed to participate in the A/B test to maintain server capacity (e.g., only 20% of all eligible users shall participate, when the A/B test is active).
• Last but not least all participating users are split and assigned to the several A/B test groups via a specified ratio (e.g., 50% of the participating users shall be assigned to the control group and 50% to the test group x).

Therefore there are the following states for users regarding A/B tests, which are important to differentiate:

1. A/B test not active | A/B test active: Users can only be eligible to participate if the A/B test was active.
2. eligible | not eligible: A user is eligible, if the user matches the target group of the (active) A/B test.
3. participating | not participating: The user is eligible and finally participates or does not participate in the A/B test. If the user participates, he/she will be assigned to an A/B test group.
4. assigned to any A/B test group: The user is eligible and also participating and is therefore assigned to a specific A/B test group.

→ i.e., Not all eligible users participate, but all participating users are assigned to an A/B test group.

Definition of "active":
An A/B test is active, if it is enabled and in status:

• Running
• For running sessions: Closed

It is not active if it is either disabled or in status:

• Pending Start
• For new sessions: Closed
• Completed

(see A/B Test Status)

Participation and A/B test group assignments are stored in a browser cookie, so a returning eligible user will get the same participating status and group assignment again.

Visibility Chart

There are at least two types of A/B test groups. The normal A/B test groups and the control group.

A user of the control group sees the content as if he/she would not participate in the A/B test at all. Therefore, this content's visibility is totally unrelated to the state of the A/B test (even when the A/B test is not active).

There is only one exception: If content is explicitly assigned to the control group, every user that belongs to the control group will see this content and vice versa a user of a test group cannot see this content. This makes it possible to run A/B tests even for already existing content just by adding it to the control group.

If content that is assigned to the control (or a test) group should also be visible to users of a test group, then this content has to be assigned to this test group.

A/B Test Accessibility

The A/B Test knows about its accessibility. Extensions to a Java extension point com.intershop.component.marketing.capi.abtest.ABTestBOAccessibilityHandler-isAccessible can be used to implement the specific rules by which a A/B Test is accessible. For the A/B Test to be considered accessible, there should be no extensions available or at least one extension has to return that the A/B Test is accessible.

In the standard system an accessibility extension exists that checks whether the A/B Test is active (i.e., enabled and running), and the customer is part of the target group.

Inactive ABTest or Not Eligible View

None of the content assigned to the normal A/B test groups is visible. However, the content of the control group is visible based on the standard visibility settings.

Non-participating Users

Precondition: the A/B test must be active
For performance reasons it is recommended for high-traffic websites that they do not assign all eligible users to the A/B test, but only a fraction.
The eligible users, which are not participating in the A/B test, see the same as if the A/B test is inactive or if they were not eligible, i.e., none of the content assigned to the normal A/B test groups is visible. However, the content of the control group is visible based on the standard visibility settings.

Non-participating users are (like not-eligible users) not monitored for this particular A/B test.
This way the particular members of the control group see the same content like the not-participating users, but can still be monitored regarding the base line content like the other A/B test groups.

Nevertheless, it still can happen that a not-participating user of one A/B test is a member of an A/B test group from another active A/B test.

Normal A/B Test Groups View

Precondition: the A/B test must be active
In general: Only the users, which are assigned to a particular (normal) A/B test group can see the content that is assigned to their A/B test group.
Therefore, no one else sees their content (with the exception of the 'Control Group', see below) and, vice versa, they do not see the content of the other A/B test groups of the same A/B test.

Exception: see (Making ABTest Content Visible Outside the ABTest / ABTesting Existing Content)

The activities of the A/B test groups are monitored by logging the triggered events the users are setting off.

Control Group View

The control group is a special A/B test group.
Precondition: the A/B test must be active
Like with all A/B test groups, the assigned content is not visible to the users of the other A/B test group of the same A/B test.
However, for all the others, including the users of the control group, the visibility of this content is only based on the standard visibility settings. Therefore, this content can be visible to everyone (except the users from the other A/B test groups) if the visibility settings are set this way.
This means, its content can ALSO be visible to the non-participating users (i.e., users that are eligible, but are not assigned to an A/B test group).

Exception: see (Making ABTest Content Visible Outside the ABTest / ABTesting Existing Content)

As the control group is still a special A/B test group, it is monitored as well.

Conflicts

Rivaling A/B Tests

A Shop Manager must be aware that multiple A/B tests that are active at the same time can conflict with each other.
Example:

1. Assume a content snippet is assigned to the test groups of two A/B tests active at the same time.
2. A user matches the critera for both A/B tests and is assigned to both tests.
3. In the first test the user is assigned to the test group and thus sees the content.
4. In the second test the user is assigned to the control group and is not allowed to see the content.

In this case the user will not see the content - the do not show condition has priority.

Exception:

In case a conflict occurs between the content of a promotion that is assigned to the first test and content that is explicitly assigned to the second test, the promotion's content has priority in terms of visibility.

Changing Assignment of the User

A Shop Manager must be aware that the customer segment of a user may be changeable.

Example:

1. Assume an already exsiting promotion should be replaced with a new one.
2. The promotion should be tested with unregistered customers only.
3. An unregistered customer assigned to the test group browses the storefront and sees the new promotion.
4. During the checkout process the customer logs in and is now a registered customer which does no longer match the assignment criterion and sees the old promotion.

Making A/B Test Content Visible Outside the A/B Test / A/B Testing of Existing Content

In some situations, especially when already available content is assigned to an A/B Test, it might be useful to have A/B Test content also visible to users outside the A/B Test.
Normally, assigned content is only visible if the A/B Test is active and the users match the criteria above.

Use case 1:
Premium users see a special piece of content, like a red button, that is not visible to others.
The shop manager wants to test if this red button might be good for normal registered users.
Therefore the same content must be visible to a particular A/B Test group (set up for the registered users) and must stay visible to the premium users all the time.
→ Making A/B Test content visible to non-eligible users.

Use case 2:
All users normally see a piece of content, like a 10% promotion.
The shop manager wants to test, whether sales increase if this promotion is changed.
Therefore the already existing content must stay visible as long as the A/B Test is not active. It must be visible according to the A/B Test criteria, if active.
→ Making A/B Test content visible if the A/B Test is not active.

So either the shop manager adds the existing content to the control proup to make this content visible based on the standard visibility settings.
Or he assigns the desired users, consumer groups or affiliate users to the content itself.

• For promotions:
  • Adding the desired users, consumer groups or affiliates to the target group of the promotion itself. (not to the target group of the A/B Test)
• For page variants / components:
  • Adding the desired consumer group to the page variant / component itself.(not to the target group of the A/B Test)

• E.g., use case 1: If consumer group "Premium Users" is added to the button component:
  • It is visible to the premium users all the time.
  • It is only visible to the other users according to the A/B Test criteria.
• E.g., use case 2: If consumer group "Everyone" is added to the target group of the Promotion:
  • It is visible to all users, as long as the A/B Test is inactive.
  • It is visible to all users according to the A/B Test criteria, if it is active.

A/B Test Management (Commerce Management Application)

Running A/B Tests 

The Intershop A/B test feature can be found in the Commerce Management application's Marketing tab.

The main page lists all existing A/B tests of the application.

• It is possible to search the A/B Tests by several criteria.
• Click an A/B test entry to open the preferences of the A/B test.
• It is possible to delete A/B tests, though they have to be either in status Pending start and Completed or being deactivated at least for the time that exceeds the session time-to-live (session.ttl). The parameter session.ttl is set in file /share/system/config/cluster/webadapter.properties and has a default value of 21600 seconds (6 hours).
  This is a security feature to prevent defective behaviour while assigned users are still active on the storefront. However, the shop manager can still circumvent this protection by setting the A/B test to Pending start by adjusting the Start Date. In this case he must be aware, that a deletion or a modification of the content of a running A/B test can cause defective system behavior for the currently active users.
• Click New to create an A/B test.

The following sections describe the detailed preferences.

Test Properties

When creating a new A/B test, the user can set several properties of the test. They can be changed in the tab Properties of the test's detail view later on.

Target Groups

 The target users can be specified as a set of:

• Customer Segments (Everyone, Registered Consumers, Premium Consumers etc) and/or users of
• Affiliate Partners and Affiliate Programs.

The available customer segments are taken from the customer segmentation service. When assigning a customer segment to the A/B test a hidden consumer and user group are created for the given segment if one does not already exist. The customer segment is assigned to the A/B test via that user group.

Only users of these specified user groups are eligible to participate at the A/B test.

Test Groups

This view shows a list of all A/B test groups of the A/B test.

• Click an A/B test group to adjust its preferences.
• A/B test groups can be deleted, however they are subject to the same limitation as the whole A/B test.
• Click New to create an additional AB test group.

• Click the preview button (Action column) to open the Design Preview panel with the particular A/B test group assigned to this session.

  Note

  The preview buttons are only visible if the A/B test is enabled.

Content

The test content of any A/B Test group can be freely assigned.
Each set of assignments can be a collection of content components, page variants and promotions.

Be aware that content has got own target groups.
When assigning content to an A/B Test, the A/B Test group is becoming a target group of the promotion, component or page variant.
Therefore the shop manager has to make sure the content has no other target groups assigned as well. Otherwise the A/B Test might be in conflict with another target group of the content.

Promotions assigned to the control group must have at least one additional target group, e.g., "Everyone", "Registered Customers", "Unregistered Customers", etc. If the promotion has only the control group as target group assigned, then it is not visible on the storefront at all. It must be pointed out that this target group is set within the channel at the promotion itself.
Promotions assigned to a test group should not have additional target groups assigned. If an additional target group is assigned to the promotion, then it is also visible to storefront users that meet the criteria of this target group. But these users might not be participants of the test group.

A/B Test Status

An A/B test follows a sequence of status changes:

1. Pending Start:
   • The A/B test is not started.
   • Intershop recommends to only assign/unassign content and changes properties of A/B test in this phase.
2. In Progress:
   • The A/B test is started.
   • New storefront sessions will be assigned to the A/B test according to the assigning process.
   • Events are logged.
3. Closed:
   • The A/B test is still running.
   • The assigning process is closed, so no new sessions will be assigned to the A/B test.
   • Already assigned sessions are still running until they time out or reach the maximum time to live.
   • Their events are logged.
4. Completed:
   • The A/B test is fully completed and all related sessions are terminated.
   • The calculation is finished.
   • No changes on A/B test will be happening any more.

It is also possible to enable and disable the test in each state.

(see also Status Changes)

Collecting and Viewing Statistical Results

Running the Collection and Cleanup Job

Before the Shop Manager can have a look at the A/B test summary and its resulting statistics, the system needs to aggregate the collected data first.

For this purpose the A/B testing feature provides two jobs, that need to run on a regular base:

Summary Tab

The results of an A/B test are displayed in the tab Summary.

The tab provides a filter to show either:

• All
• Most Important
• Total Values
• Conversion
• Average Values

For every A/B test group the results are shown in plain numbers and in percentage values related to the whole A/B test.
Conversion results also show significance values that indicate whether a A/B test group result significantly differs from the result of the particular control group.
The T-Test is used for conversions with events that hold quantitative data. This means events that log data like the number of Items, that were ordered by a user or his order amount.
The χ²-Test (Chi-square) is used for simple binary events, like the triggering of the checkout event.
If the significance indicator is:

• red (e.g., χ² = R [R for "red"]), the result of the A/B test group significantly differs from the control group only with a probability of less than 90%.
• yellow (e.g., χ² = Y [Y for "yellow"]), the result of the A/B test group significantly differs from the control group with a mediocre probability of 90-95%.
• green (e.g., χ² = G [G for "green"]), the result of the A/B test groupp significantly differs from the control group with a confident probability of more than 95%.
  The higher the basic population, the greater is the confidence of the statistical significance.

When hovering the mouse over the significance indicator, a tooltip shows the detailed significance value.

Problems with Live-Edit- and Multi-Data-Center-Scenarios

There is a limitation with this view.
The statistics of the live and edit systems are separated as everything else. To view the statistics of the productive system you have to log in the live system's Commerce Management application. Otherwise, you will only see the results from the edit system. 

Furthermore, there is no aggregation of the statistics between multiple data centers. Each data center holds its own local statistics and must be viewed by the according Commerce Management application.

 Finishing A/B Tests

After an A/B test has been completed, a decision has to be taken whether the storefront content should be changed (i.e., a test group's content is made visible) or stay unchanged.

For this purpose buttons are enabled at the bottom of an A/B test's summary page when the test is in status Completed. Every test group provides a button Set Content Online whereas the control group provides a button Keep Current Content.

To change the contents of the storefront according to the appearance it to the test group:

1. Click Set Content Online.
   All changes are displayed on an overview page.
2. Make changes according to your wishes.
3. Click Apply.
   The content is published and available for all users matching the criteria.
   

   

To keep the storefront unchanged:

1. Click Keep Current Content.
   The current content of the storefront is equal to the content of the control group.

Once this finishing process is passed, the A/B test is marked as finished and the buttons of the groups are hidden. The summary page also gives information about the taken decision, decision maker and decision time:

Replication

A/B Tests must be replicated from the edit system to the live system. This is done by the usual object replication (Data Replication Tasks) in the Commerce Management application.

Design View

The Design View does not take A/B testing into account. Therefore, it shows the storefront as if there is no A/B test active. A preview of the A/B test groups is available using the preview buttons in the Action column of the test's Test Groups tab (see Test Groups).

Technical and Customization Overview

Cartridges

The general A/B test functionality is located in several cartridges:

• bc_abtest
• bc_marketing
  
  • holds all the basic logic and artifacts like calculations, assignment, retrieval, visibility, manager, POs etc.
• sld_ch_consumer_plugin
  
  • holds the artifacts for the Commerce Management application, like templates and webforms and the A/B Test administration logic.
• app_sf_webshop_b2c and app_sf_webshop
  • holding the logging call logic: when will which event be triggered.
  • holding logic to trigger A/B Test assignment (Prefix pipeline, PGID provider pipeline ProcessPersonalizationGroup-DetermineDynamicPGID...)
  • holding PGID provider pipeline (ProcessPersonalizationGroup-DetermineDynamicPGID...)

User Groups and Dictionary Values

User Groups:

Sessions get assigned to user groups that represent the assigned A/B test and A/B test group:

• The A/B test target group user group is a consolidation of the target groups of an A/B test and therefore represents if a session is eligible to participate at the active A/B test. Its format is: <ABTest-name>_<UUID> of the A/B test.
• The A/B test group user group represents the A/B test group a session got assigned to. Its format is: <ABTest-name>_<UUID> of the A/B test group.

With these user groups it is easy to provide logic for the content visibility and logging events.

Session Dictionary:

Beside the user groups the session dictionary holds two entries:

• T_CurrentABTestGroupUUIDs- Holds all A/B test group UUIDs that are currently assigned to the session.
• T_CurrentAssignedABTestGroupUUIDs- Holds all A/B test group UUIDs that are currently assigned to the session and have actually any content assigned to it. In contrast, T_CurrentABTestGroupUUIDs also holds A/B test groups that have no content assigned.

Furthermore, the pipeline ProcessABTesting-CheckABTestMode checks a session dictionary value called T_ABTestMode. It determines if there are at least existing any potential A/B tests suitable for the current session. It is a general switch you can use to circumvent unnecessary A/B test processing if there are no A/B tests at all.

Calculator Definition

Each cartridge has the possibility to register its own events, calculators and conversions.

The properties scheme always follows the notation:

intershop.marketing.abtest.<calculator|event|conversion>.<unique name for registration>.*

The following examples illustrate the definition of two implemented calculators: Count and TTest.
Each implementation should extend the ABTestCalculator.

First: In Properties Files

The default properties are located in bc_marketing.properties.

format:

intershop.marketing.abtest.calculator.<calculator registry name x>.class                      = <qualified java class name>   //mandatory
intershop.marketing.abtest.calculator.<calculator registry name x>.column.<sql placeholder x> = <field name x>                 //only, if SQL terms in calculator class use placeholders for the database columns
intershop.marketing.abtest.calculator.<calculator registry name x>.column.<sql placeholder y> = <field name y or x>            //only, if SQL terms in calculator class use placeholders for the database columns

• <calculator registration name x>- The registration name of calculator x.
• <qualified java class name>- The complete qualified name of the real class for this calculator
• column properties are only mandatory, if the calculator defines placeholders in its SQL-terms.
  • <sql-placeholder x/y>- If the calculator needs dynamic SQL-fields, these are represented by these placeholders. These are used at the SQL-terms inside the calculator class.
  • <fieldname y/x>- Intermediating field name to match the placeholder with a SQL-field at the event or conversion definition.

# Calculator: Count
## counts events
intershop.marketing.abtest.calculator.count.class                    = com.intershop.component.marketing.capi.abtest.calculators.ABTestCalculatorCount

# Calculator: T-Test
## for more complex value conversions, calculates significance of sum-value
intershop.marketing.abtest.calculator.ttest.class                    = com.intershop.component.marketing.capi.abtest.calculators.ABTestCalculatorTTest
## extends "Count and Sum" class, therefore also its placeholder "columnsum" must be defined:
intershop.marketing.abtest.calculator.ttest.column.columnsum         = SumValue
intershop.marketing.abtest.calculator.ttest.column.columnsumsqu      = SumValue

In the example above, the SQL-terms in the Java class contain two different placeholders: {columnsum} and {columnsumsqu}
In this scenario both of them will temporarily be replaced by the intermediate fieldname SumValue to match the placeholders with corresponding database columns that are defined at the corresponding event and conversions definition.

Second: As Java Class

The Java class representation defines 3 aspects:

• The SQL terms to collect and aggregate the statistics data.
• The post processing, that is executed every time the results are visualized.
• For conversions: The significance calculation, also for result depiction.

Collection of Statistical Data

The following example of the calculator Count, just counts the occurrences of event entries.

public Map<String, String> getSQLFields() {
        HashMap<String, String> map = new HashMap<String, String>();
        map.put("Count", "count(*)");
        return map;
    }

    public int parameterCount()
    {
        return 0;
    }

The method getSQLFields() returns a map consisting of unique keys (SQL names) that are representing the "statisticsmarkers", and a corresponding SQL term, that is appended to the SELECT statement internally.
The SQL term does not have any placeholders. Therefore the method parameterCount returns 0.

The result of each term is stored at the ABTestStatistics_AV table with the name format:

• For events: event-marker or event-currency-marker→ "OrderCreatedEvent-USD-Count"
• For conversions: preevent-postevent-marker or preevent-postevent-currency-marker→ "TestObjectViewedEvent-ItemsOrderedEvent-Sum"

public Map<String, String> getSQLFields()
    {
        HashMap<String, String> map = new HashMap<String, String>();
        map.put("Count", "count(*)");
        map.put("Sum", "sum(nvl({columnsum}, 0))"); // sums up the values of the database column defined by the placeholder
        map.put("TTSumSqu", "sum(power(nvl({columnsumsqu}, 0), 2))");  // takes the square root of the placeholder and sums it up.
        return map;
    }

    public int parameterCount()
    {
        return 2;
    }

As with the counting, here the statisticsmarker Sum is hanged as marker at the back of the event name or conversion.
However, since the contents of a particular field needs to be added up, a corresponding SQL term is formed. Because at this point it is not yet known which database field will be taken particularly, a placeholder is used, which appears in curly brackets {columnsumsqu}.

This placeholder gets a field name, that is defined at the properties file.
In the event and conversion definitions this field name is required to match it with a specific database field. In this example a 2 for parameterCount is returned. It represents the count of all different placeholders inside of all SQL terms of this calculator.

Preparation of Data for Display

Create methods to use the result of such calculators in the template's output:

<isset scope="request" name="myValue" value="#TestGroup:TestStatistics:Calculation:Formula("Count"):Absolute("CheckoutEvent")#">

The Java method getFormula(String calculatorRegistrationName) returns the instance of the wished calculator.
Here a Java Method getAbsolute() is used to get the value from the database into the variable myValue, which can be used in the ISML template:

public Object getAbsolute(String key) {
        return this.getStatistics().getAttribute(key + "-Count");
    }

You can specify any method you want to read and process the data.

Calculating Significance Values

It is recommended to create methods to calculate the significance values of conversion results.
We use the χ²-Test (Chi-Square) for simple count conversions and the T-Test for more complex conversions like sum-values.

Event and Conversion Definition

Definition of Available Events

The available events and executed calculators are part of the configuration. The configuration can be overridden globally.
Each cartridge has the possibility to register its own events, calculators and conversions.

The properties scheme always follows the notation:

intershop.marketing.abtest.<calculator|event|conversion>.<unique name for registration>.*

The default properties are located in bc_marketing.properties. ( \bc_marketing\staticfiles\share\system\config\cartridges\bc_marketing.properties)

format:

intershop.marketing.abtest.event.<event registration  name x>.type                       = <used calculator>
intershop.marketing.abtest.event.<event registration  name x>.column.<database column x> = <used field name of calculator>  //only, if SQL terms in calculator class use placeholders for the database columns
intershop.marketing.abtest.event.<event registration  name x>.column.<database column y> = <used field name of calculator>  //only, if SQL terms in calculator class use placeholders for the database columns

• <event registration name x>- The registration name of the event x. An event can store multiple data at once.
• <used calculator>- The appropriate calculator registration name for processing the results.
• "column" properties are are only mandatory, if the chosen calculator defines placeholders in its SQL terms.
  • <database column x/y>- The column from the database that contains the data to be processed.
  • <used field name of calculator>- The field name from the calculator definition, to match the data of the database column with the SQL placeholders

# Checkout
## simple event count
intershop.marketing.abtest.event.CheckoutEvent.type                   = count

# OrderCreated
## event count and sum up of MoneyValue column (also currency)        // an event is automatically also currency aware, if the logging function references a currency
intershop.marketing.abtest.event.OrderCreatedEvent.type               = countAndSum
intershop.marketing.abtest.event.OrderCreatedEvent.column.MoneyValue  = SumValue

# ItemsOrdered
## sum of DoubleValue column (also currency aware)
intershop.marketing.abtest.event.ItemsOrderedEvent.type               = sum
intershop.marketing.abtest.event.ItemsOrderedEvent.column.DoubleValue = SumValue

Definition of Available Conversions

The available conversions, used events and executed calculators are part of the configuration. The configuration can be overridden globally.

format:

intershop.marketing.abtest.conversion.<conversion registration name x>.type                       = <used calculator>
intershop.marketing.abtest.conversion.<conversion registration name x>.preConditionEvent          = <used pre event>
intershop.marketing.abtest.conversion.<conversion registration name x>.postConditionEvent         = <used post event>
intershop.marketing.abtest.conversion.<conversion registration name x>.column.<database column x> = <used field name of calculator>  //only, if SQL terms in calculator class use placeholders for the database columns
intershop.marketing.abtest.conversion.<conversion registration name x>.column.<database column y> = <used field name of calculator>  //only, if SQL terms in calculator class use placeholders for the database columns

• <conversion registration name x>- The registration name of the conversion x. A conversion stores multiple data at once.
• <used calculator>- The appropriate calculator registration name for processing the results.
• <used pre event>- The event registration name to form the precondition.
• <used post event>- The event registration name to form the post condition.
• "column" properties are are only mandatory, if the chosen calculator defines placeholders in its SQL terms.
  • <database column x/y>- The column from the database that contains the data to be processed.
  • <used field name of calculator>- The field name from the calculator definition, to match the data of the database column with the SQL placeholders

intershop.marketing.abtest.conversion.viewed2clicked.type                    = chiSqu
intershop.marketing.abtest.conversion.viewed2clicked.preConditionEvent       = TestObjectViewedEvent
intershop.marketing.abtest.conversion.viewed2clicked.postConditionEvent      = TestObjectClickedEvent

#(also currency aware)  //a conversion is automatically also currency aware, if the post-condition event is currency aware (see above)
intershop.marketing.abtest.conversion.viewed2ordercount.type                 = chiSqu
intershop.marketing.abtest.conversion.viewed2ordercount.preConditionEvent    = TestObjectViewedEvent
intershop.marketing.abtest.conversion.viewed2ordercount.postConditionEvent   = OrderCreatedEvent

#([also] currency aware)
intershop.marketing.abtest.conversion.viewed2orderamount.type                = ttest
intershop.marketing.abtest.conversion.viewed2orderamount.preConditionEvent   = TestObjectViewedEvent
intershop.marketing.abtest.conversion.viewed2orderamount.postConditionEvent  = OrderCreatedEvent
intershop.marketing.abtest.conversion.viewed2orderamount.column.MoneyValue   = SumValue

Relations

Here you can see the relations between Conversion, Event and Calculator. Also the meaning of the SQL placeholder and the database column is depicted.

Dos and Don'ts

• Do not define multiple conversions with the same preConditionEvent + postConditionEvent + type.
  However, you can define multiple conversions with the same pre- and postcondition, but they have to use separate calculators.
• Do not use registry names more than once.

Event Logging

Logging Conditions and Architecture

Events are logged while a participating user is using the system.
Events are stored at the database table ABTestEvent.
By default it provides the following fields:

• ID - unique ID [not sorted by time] (mandatory)
• DOUBLEVALUE - for values
• MONEYCODE - a currency
• MONEYVALUE - for currency values
• OBJECTTYPE - class of a related object
• OBJECTVALUE - e.g., UUID of the object instance
• PIPELINE - the triggering pipeline
• SESSIONID - the current session ID (mandatory)
• TYPE - event registration name (mandatory)
• USERID - the current user ID
• ABTESTGROUPUUID - the UUID of the A/B test group (mandatory)
• DOMAINID - the current domain (mandatory)
• LASTMODIFIED - date time of creation [or modification] (mandatory, set automatically by database layer)

You need to make sure an event is triggered along with the desired conditions and for all related assigned A/B test groups. It should be logged directly to the database without using the ORM layer to prevent massive overhead caused by this layer (caching and so on).

To make an event currency aware, just pass over the current used currency when logging the event.

Logging Logic

There are several starting points for logging behaviors.

• Log an event only once per session
• Log an event multiple times per session
• Log an event only once in a row
• ...

On default most of the events are logged only once in a row, others are only logged once at all and some are logged every time.
The used pipelets with the logic to log these events are:

• LogABTestClickEvents
• LogABTestEvent (attention for customization: fits only with the default events)

Conversion Logging

Conversions do not need any triggers. It is enough to define the pre- and post-event in the properties.
At the collecting process the statistics collector checks for each definition if the conditions are fulfilled.

Data Storage

Database content:
ABTest
ABTest_AV → for DeactivationTime; see Deletion Logic ("DeactivationTime")
ABTestEvent
ABTestGroup
ABTestGroup_AV → for BufferTime; see Buffer Time Logic
ABTestGroup_PA → for pagelet assignments
ABTestStatistics
ABTestStatistics_AV → for the statistical results; see Concept - AB Testing#UpdateTrackEvents
(DynamicPGID) → for the activation and deactivation of the PGID Provider; see Concept - AB Testing#Page Caching (via PGID Provider)
(UserGroups) → for A/B test target group-user group (<ABTest-name>_<UUID>) and A/B test group-user groups (<AB>_<UUID>)

File system content:
[component]\staticfiles\share\system\config\cartridges\<component>.properties - see Calculator definition and Concept - AB Testing#Event and Conversion definition - bc_marketing is the component for the default A/B test configuration
[component]\staticfiles\cartridge\localizations\abtests_<lang-code/description>.properties - see Concept - AB Testing#Localization - bc_marketing is the component for the default A/B test configuration

Statistics Collector

The statistics collector is a Java class that:

• Collects the event logs from the database and calculates the statistical results
• Removes old and orphaned events
• Sets Closed A/B tests to the Completed status
• Updates the A/B test status

Basically this is done by the AbstractConversionEventStatisticsCollector in package com.intershop.component.marketing.capi.abtest.collectors implementing ABTestEventStatisticsCollector.
If the functionality of the collecting process should be customized, it is recommended to extend the AbstractConversionEventStatisticsCollector.
The collector gets configured and executed by the pipelet ExecuteABTestStatisticsCollectors that runs every time one of the both jobs are started.

Buffer Time Logic

Every user has his/her own session. Every event that gets logged from this user belongs to this session. Sessions can either time out (user is showing no more activity) or exceed its time-to-live (user is active constantly and after some time the session has reached its maximum lifetime).
In order to provide correct conversion statistics, only the events of sessions can be calculated that are not active any more and therefore cannot trigger any more events.

Because no one knows exactly when a session is not in use any more, the worst case has to be assumed. In this case the session gets terminated explicitly after the session time-to-life (session.ttl property - default value 6 hours) has passed.

The BufferTime represents the date and time of the collection process minus the session time-to-live. Hence it is the latest time for newly created sessions, which are are closed or terminated at collection time for sure.

On Event-log processing only events of sessions are considered that have logged the first event before this BufferTime. This ensures that events of active sessions do not get processed only partially.

The latest BufferTime gets stored at the ABTestGroup_AV table for future event processings.

UpdateTrackEvents

Only events of sessions that logged their first event (in best case on session creation) before the current calculated BufferTime, are considered to be processed.
On the other side the old BufferTime is taken into account to prevent already processed events getting calculated twice.
The newly calculated values are aggregated and stored to the database table ABTestStatistics_AV.

• For events: event-marker or event-currency-marker→ e.g., OrderCreatedEvent-USD-Count
• For conversions: preevent-postevent-marker or preevent-postevent-currency-marker→ e.g., TestObjectViewedEvent-ItemsOrderedEvent-Sum

CleanupTrackEvents

The old BufferTime is used to delete all events of sessions that logged their first event (on session creation) before this time. On this way it is assured, that only already calculated events get removed from the database.

Summary Presentation

The Summary page of the A/B test result statistics is grouped into three main categories.

• Total values
  represents total value data collected from single events like the order count and value
• Conversions
  represents data collected by calculating conversion rates, like the number of orders of sessions that have seen the test object before
• Average values
  values that are calculated by bringing two values into relation, like the average number of items per order

The structure of the A/B test summary page is defined at the ISML template ABTestSummary_52 in cartridge bc_marketing. Over here the main structure is an ISModule tag called <issummaryline>:

<ismodule
  template  = "marketing/ABTestSummaryLine_52.isml"
  name 	  = "summaryline"
    attribute = "displaynamekey"
    attribute = "statisticskey1"
    attribute = "statisticmarker1"
    attribute = "statisticskey2"
    attribute = "statisticmarker2"
    attribute = "calculator1"
    attribute = "calculator2"
    attribute = "summarygroup"
    attribute = "type"
    attribute = "currency"
    attribute = "isimportant"
>

Each row of the statistics is defined by one <issummary> tag. Here is the parameter documentation:
All parameters and values are case sensitive.

When a result row is about to render, the ISML template ABTestSummaryLine_52 collects the data, percentage values and significance values (only for conversions) from the calculators and displays each A/B test group in a separate column.

A/B Test Managing Logic

Status Changes

The status of each single ABTest is updated every time only on these events:
Either:

• An eligible users is attempted to become participant, or
• The A/B test gets updated in the Commerce Management application, or
• One of the two event jobs is running.

Until one of this is happening, the status in the Commerce Management application will still show the old value.

As long as the current time has not reached the defined Start Date and Time of an A/B test, the A/B test stays in status Pending Start.
It is highly recommended to change content and A/B test group definition only in this phase.
Removing content while it is still in use by active users of the A/B test group, might cause inconsistent behavior for these users.

The status is automatically changing to Running when the Start Time is reached (and the status gets updated) and stays as long as either the End Time or the Maximum Participants allowed value is not reached.
New sessions are considered to get assigned to the A/B test now and events are monitored.

When one of the ending criteria is reached (and the status gets updated), the status changes to Closed.
Now the assignment process is closed and only the still active sessions are monitored until the session time-to-life (session.ttl of webadapter.properties in share\system\config\cluster) runs out.

With every run of the UpdateTrackEvents or CleanupTrackEEvents job, each closed A/B test is checked for events of sessions that did not pass the session.ttl yet.
If no active assigned session can be retrieved any more, the A/B test is turned to status Completed.

Deletion Logic ("DeactivationTime")

To prevent faulty system behavior on user side, a simple logic is implemented to prevent A/B tests and A/B test groups from being deleted while assigned users might still be active.

If the Shop Manager tries to delete an A/B test or A/B test group a check is performed that provides the following criteria:

• The A/B test has to be either in status Pending Start or Completed.

or

• The A/B test has to be deactived at least for the time of the session.ttl property.

To provide the second check, the ABTestPO.java needs a modification of its generated setEnabled(Boolean aBoolean) method.
Every time the enabled flag is changing, the mapped extensible object attribute DeactivationTime of A/B test is set or removed.
In order to prevent overwriting of the modification by the code generator the @generated tag needs to be removed.

Special case: When the A/B test is generated, it needs to get temporarily enabled in first place. Otherwise a deactivation will not be recognized by the setEnabled(Boolean aBoolean) method, as the enabled flag is not yet initialized.

Workaround: A Shop Manager can still change the start date/time to a future time, so the A/B test will get the status Pending Start again and can be deleted now.

Realtime Participant Count Logic

The A/B test is always aware of the current participant count in real time. In scenarios with application server clusters this has to be ensured centrally by the database without running an elaborate SQL statement each single time.
To solve this the system utilizes the "Number Series" or "Sequences" function of databases. These sequences provide a consistent stream of IDs. Intershop configured the sequence to provide single numbers incremented by 1 each. The cache on database side and application server side must be set to 0. So every new participant triggers a new request for a number increment at the database and returns it to the requesting server. 
This way the system works fail-save if a server might crash.

The sequence is configured as NumberSeriesProvider in the ABTestMgr class.

Participant Assignment Logic

The assignment happens in the pipeline ProcessABTesting-RetrieveABTestGroups.

When a new session is arriving or the context has changed (e.g., at login/logout), the system checks first for running and enabled A/B tests that are matching the assignment of the target groups.
If so, a user is eligible to participate in these A/B tests.

Pipelet AssignABTestToSession:
For each found A/B test the system checks at first for a browser cookie.
See Cookies to see its specification.
The system restores the A/B test assignment out of this cookie, if possible, and logs a new ABTestSessionCreatedEvent.
If no cookie is found, the usual assignment process is started.

The second responsibility of the assignment process is the check for the determination of the end criteria.
If the end date or maximum participants allowed is reached, it sets the A/B test status to Closed.

Both, the pipeline and the pipelet support an optional parameter TestGroupUUIDParams where particular A/B test group UUIDs can be reached in that the session explicitly shall be assigned to. The principle is the same as with the cookie, except for the dash.

Participant Ratio

The participant ratio is a probability value from 0 to 1 (0% to 100% in Commerce Management application view).
The toss up of the user participation is a simple randomized value, that is compared to the participant ratio.
If the value is smaller, the user participates; if larger, does not participate.

A/B Test Group Ratio

The A/B test group ratio is a probability value from 0 to 1 (0% to 100% in Commerce Management application view).
The toss up of the group assignment is a simple randomized value, that is compared to the A/B test group ratios, that are distributed accordingly over the field from 0 to 1.

Content Visibility Logic

General Rule:
Content is always visible as long as no PageletVisibilityFilter prompts the content management system to hide the content.

Every element of content (component or page variant) runs through a content visibility management system to check if an element shall be visible to a user or not.
The CompoundPageletVisibilityFilter calls every implemented filter (here: PageletABTestGroupVisibilityFilter in conjunction with PageletUserGroupVisibilityFilter) and asks if at least one of them permits or denies the visibility.
The A/B test filter checks each content element regarding the assigned A/B test group.
If the content piece is assigned to the control group, it only denies the visibility to users of the other A/B test groups.
If the content piece is assigned to an other A/B test group, it denies the visibility, as long as the user is not assigned to the corresponding A/B test group.
In other words:
Denial by any filter leads to the hiding of the content and the Shop Manager has to make sure the content does not conflict with other A/B test groups or other Intershop features.

When assigning content to an A/B test group the A/B test group becomes the target group of the content.

The visibility of promotions is regulated by the promotions itself. The promotion has to be accessible, and the customer assigned to the related A/B test group, this check returns true.

Localization

The localization of the A/B test summary page is done via ISTEXT.
To provide a simple usability, the ISML template ABTestSummary_52.isml provides a parameter displaynamekey for the ISMODULE-tag <issummaryline>:

<issummaryline      displaynamekey  ="<ISTEXT-key without prefix>"
                    [...]
                        summarygroup    ="<summary group>"  <!-- among other things it defines the prefix -->
                        [...] >

The displaynamekey is an ISTEXT-key (without prefix) which is located at abtests_<lang-code>.properties and abtests_description.properties ([component]\staticfiles\cartridge\localizations\).
For the standard A/B test configuration the cartridge bc_marketing is the containing component.

Each key has a defined format:

### statistics
## total values
# abtest.statistic.total.<eventname[x]>
# where [x] represents &quot;1&quot; for the first value of the event, &quot;2&quot; for the second value and so on
abtest.statistic.total.OrderCreatedEvent1           = Orders
abtest.statistic.total.OrderCreatedEvent2           = Orders Value

## conversions
# abtest.statistic.conversion.<preevent[x]>.<postevent[y]>
# where [x] and [y] represent &quot;1&quot; for the first value of the event, &quot;2&quot; for the second value and so on
abtest.statistic.conversion.TestObjectViewedEvent1.OrderCreatedEvent1       = Test Object Viewed \u21D2 Orders
abtest.statistic.conversion.TestObjectViewedEvent1.OrderCreatedEvent2       = Test Object Viewed \u21D2 Order Value

## average values
# abtest.statistic.average.<numerator>.<denominator>
# where <numerator> and <denominator> can be of type &quot;event[x]&quot; or conversion &quot;preevent[x]_postevent[y]&quot;
# with [x] and [y] representing &quot;1&quot; for the first value of the event, &quot;2&quot; for the second value and so on
abtest.statistic.average.ItemsOrderedEvent1.OrderCreatedEvent1                                                  = Average number of Items per Order
abtest.statistic.average.OrderCreatedEvent2.OrderCreatedEvent1                                                  = Average Order Value
abtest.statistic.average.TestObjectAppliedEvent1_OrderCreatedEvent1.OrderCreatedEvent1                          = Orders that applied a Test Object
abtest.statistic.average.TestObjectAppliedEvent1_OrderCreatedEvent2.TestObjectAppliedEvent1_OrderCreatedEvent1  = ... their average Value

## general phrases
abtest.statistic.general.notYet                 = Not Yet

Because every event can hold multiple data (like count and value), each part-event of a key has got an index starting with 1 (e.g., OrderCreatedEvent1 means the count; OrderCreatedEvent2 means the value).
Every key needs the prefix abtest.statistic. followed by the content group.
The prefix and content group is added automatically to the displaynamekey parameter of <issummaryline>. However, the "general" content group is a special group which can be addressed on various other places.
It is needed to make sure the abtest_description.properties are maintained to provide comprehensive key descriptions.

Page Caching and PGID Provider

The PGID (Personalization Group ID) is a global Intershop concept to handle different groups of web caching scenarios.
In short: Each session gets a PGID based on several factors like the login status, user groups and also the A/B test group assignments.
Each PGID has its own web cache and is served by the Web Adapter separately.

Usually a session gets its PGID after hitting the storefront the first time. However, due to usergroup related test content that must be shown immediately to new sessions at the initial storefront view, A/B testing uses the PGID provider to boost up caching functionality.
If the PGID provider is active, the Web Adapter calls a special pipeline to calculate the PGID before a new session will be served with the initial page. This way the correct PGID will be provided right from the beginning.

Every time an A/B test is created, changes status and/or time, or is deleted, it must be ensured that a DynamicPGID object is created, maintained or removed separately for this A/B test.
A DynamicPGID expects the start date and end date and an informal entry for the origin (here: "by ABTest: " + UUID):

It needs to be set to the corresponding dates:

• As long as the A/B test is in status Enabled + Pending Start/Running/Closed.
• To null as long as the A/B test is in status Disabled or Completed

Cookies

For each A/B test the user is eligible for, the system sets a cookie. The name is preceded by "ab-", followed by the UUID of the particular A/B test.
This cookie can hold either of two mainly different contents.

• Either it contains a dash ("-"), which means the returning user was already chosen to not participate in the A/B test.
• Or it contains the UUID of one of the A/B test groups, which means the returning user was already chosen to be assigned to this A/B test group.
  (see Concept - AB Testing#Participant Assignment Logic)

The cookie lifetime is set to end ca 3 months (90 days) after the end time of the particular A/B test. Doing so guarantees a buffer time if the Shop Manager later on decides to extend the running time of the A/B test.