Introduction

This concept describes business object replication. Business object replication is intended for quick and easy transfer of business objects from the editing system to one or more live systems (see Usage).
For the transfer of large amounts of data, mass data replication has to be used.

Business object replication works by creating transfer entities called "WebObjects" from business objects by the use of configurable WebObject Providers and transferring those WebObjects to the target system(s) via a SOAP WebService. On the target system(s), the objects are validated and merged into the target system's data (see Background).

Glossary

Usage

End User

A user will see the business object replication interface elements on listings and detail pages of supported business objects.

On a business object's detail view, the object can be replicated using the button Publish to Live System. After selecting a target cluster on the next page, a batch process will be created performing the actual replication.

In a listing, one or more objects can be selected and replicated using the button Publish to Live System. The listing also includes a column Published, listing the date(s) and time(s) of the last successful object replication(s) to the target system(s).

After starting the replication, the user is taken to the batch process overview page.

Developer

Information for developers (e.g. on adding object replication support to additional business objects) can be found in the Cookbook - Business Object Replication.

Background

Process Overview

• ORMObject to WebObject transformation: On the editing system, given ORMObjects are transformed to unified WebObjects. That makes it easier to send and receive them with the Axis WebService. For each ORMObject, a WebObjectProvider( WOProvider) is looked up that does the collecting of dependent objects and transforms it. This procedure is repeated for each dependent object too.

• Determine PageCache Searchwords: For each ORMObject, a SearchWordProvider is used (if existent) to determine significant search words for invalidation of the page cache. The search words will only be invalidated on affected sites (because only sites have a page cache and not all sites display the affected object). For instance, if one of the ORMObjects is a product, files of the page cache that contain the UUID, SKU or name of the ProductPO are filed for invalidation for each site that displays the product.

• Determine affected domains for SearchIndex refresh: Using the specific ADProvider(if existent), affected domains for ORMObjects are determined. For instance, the affected domains for a product are the domain that owns the product and the domains the product is shared to.

• Replication BatchJob: After the transformation to WebObjects, determining the affected domains and PageCache SearchWords, the editing system sends the WebObjects via an Axis WebService to the corresponding live systems. The live systems validate and merge the received WebObjects by transforming their corresponding local ORMObjects (if they already exist) to WebObjects and comparing both. In case of a failed validation, the WebObject will not be merged (and will not refresh caches for it); the other WebObjects are not affected by this. The error message (provided by the specific WebObject-Provider) is sent back to the editing system and is shown in the Batch Process Log.

• If the merging was successful, the target system-specific LastPublishDate is written for each object. In this way, the user can see when the object was replicated (e.g. seen in Promotions).

• After merging the changes, the ORMObjects are released from the ORMCache. On the next access of the ORMObjects the database will be queried for the new values.

• If the replicated WebObjects have attached files, the live systems download a compressed archive of them from the editing system and extract them to the local directories.

• After that, the caches are refreshed. The pre-calculated SearchWords and affected domains are filtered by successful WebObject validations (when a WebObject was not merged because of a failed validation, the caches will not be refreshed for it) and sent to the live systems. The live systems refresh the following caches:
  • SearchIndex: by Channel-DomainName
  • PageCache: by Channel-SiteName and their SearchWords

Process Details

Providers

The following diagram shows the process for an example object Sample with two referenced objects, Other and Another:

Example

(1) When the publishing of a Sample is started, the replication framework first resolves all references of the Sample by means of the SampleRefOOProvider(if there is one defined). This results in a Set<Object> containing all referenced sample objects as well as the sample object itself. From the Set an Iterator is created within which, according to the PublishOrder defined in SampleRefOOProvider, the Sample is the first/last element.

Now the elements of the iterator are replicated one by one in chunks of default size 15 where for each object the corresponding <object_type>WOProvider is used.

(2) WebObjects for the referenced objects are created using the ReferencedOtherWOProvider and the ReferencedAnotherWOProvider, respectively.

(3) WebObjects for Sample and its attributes are created by means of the SampleWOProvider.

(4) The WebObjects are transferred to the live system.

(5) When receiving the objects, the validate method of the matching providers will be called, where the referenced objects of Sample are validated and merged before the Sample web object itself. The validation itself depends on (i.e. is implemented in) the WOProvider. The provider can also perform some actions depending on the outcome of the validation. If the validation fails, an error will be logged.

(6) Whenever the validation of a web object succeeds, the merge method of the matching provider is called. This method is responsible for creating/updating business objects in the live system to match the transferred object.

Page Cache Handling

For each ORMObject a SearchWordProvider is used (if existent) to determine significant SearchWords for invalidation of the PageCache.
The SearchWords will only be invalidated on affected sites (because only sites have a PageCache and not all sites display the affected object).
For example, if one of the ORMObjects is a product, files of the PageCache that contain the UUID, SKU or name of the ProductPO are filed for invalidation for each site that displays the product.

Search Index Handling

Using the specific ADProvider(if existent), affected domains for ORMObjects are determined.
For instance, the affected domains for a product are the domain that owns the product and the domains the product is shared to.

Published Objects

ORMObjects That Are Published During Campaign Publishing

Special Cases That Require Some Post-Processing

The current campaign publishing is organized in such a way that promotions and pagelets assigned to a campaign are published before the campaign is published. So, if the final live side validation of the campaign fails, then the campaign itself is not published, whereas assigned promotions and pagelets at that moment are already updated. This requires post-processing for each assigned promotion and pagelet:

• assigned promotions: If the campaign to be published does not yet exist in the live system, set the promotions's UseCampaignTargetGroup flag to false and disable the promotion. This procedure is complete because as soon as the promotion is published by means of the ReferencedPromotionWOProvider, its campaignUUID attribute is set to null in order to avoid the situation of the referenced promotion being assigned to another - no longer supported - campaign on live side. Now, if the campaign to be published does already exist in the live system, then two cases have to be distinguished:
  • If the promotions's UseCampaignTargetGroup flag is true and if the current campaign target group is no direct child of the promotion target group, make the current campaign's target group a direct child of the promotion's target group again.
  • If the promotion's UseCampaignTargetGroup flag is false and if the current campaign's target group is a direct child of the promotion's target group, remove this direct child relation.
• assigned pagelets: If the campaign to be published currently does not exist in the live system or if a pagelet assigned to the campaign to be published currently has no user group assignment in the live system, then the pagelet is set "offline", i.e., "not published". In order to understand this procedure, one has to know that the assignment of a pagelet to an arbitrary campaign or promotion is represented by a special user group assignment where a campaign/promotion user group ID starts with CAMPAIGN_/ Promotion_. A pagelet that has no user group assignment must not be displayed in the storefront.

Now assume that the campaign validation is successful. If the campaign did not exist in the live system before, then the assigned and published content is fine. Otherwise there may be a difference between former and new assigned pagelets that has to be dealt with. This is done as follows:

• All pagelets that have been assigned to the campaign before and hence to no other campaign or promotion, but will no longer be assigned to the campaign to be published, are deactivated.
• All pagelet user group assignments to other campaigns or promotions among those pagelets that are assigned to the campaign to be published are deleted, because there must not be more than one such user group assignment for each pagelet.

The promotion post-processing in case of successful campaign validation is straightforward: No longer valid assignments to other campaigns are removed and the new assignment to the campaign to be published is confirmed.

ORMObjects That Are Published During Pagelet Publishing

ORMObjects That Are Published During Promotion Publishing

Special Cases That Require Some Post-Processing

In analogy to campaign publishing, the current promotion publishing is organized in such a way that pagelets assigned to a promotion are published before the promotion is published. So, if the final live side validation of the promotion fails, then the promotion itself is not published whereas assigned pagelets at that moment are already in the state "published". This requires a post-processing for each assigned pagelet which is done in the same way as for pagelets assigned to a campaign in the case of campaign validation failure.

The pagelet post-processing in case of successful promotion validation is the same as in case of successful campaign validation.

Mass Data / Performance

Business object replication is not meant for mass data. For replicating mass data, staging is used which copies content from the editing-database-scheme to the live-system database-scheme.

The replication is done by sending and receiving serialized objects via web services. If a huge object (with several megabytes) is sent, the web services could massively slow down.

If the user plans to replicate a lot of objects (e.g. 10000 products), these objects will be sent in several loops, 15 objects each loop (this value can be set by staging.objects.chunksize in <enfinity>/share/system/config/cluster/staging.properties) and the cache refresh is started when all objects have been sent and merged.