Introduction
A workflow to export data and use them in SPARQUE AI is probably a much needed configuration for ICM. This guide shows one way to automate this workflow. For this automation, the following steps are necessary (find more detailed instructions for these steps in the main part of this document, also find the attached ZIP file with the files mentioned in this guide sparqueExportAndProcessC…):
Create a product export job
Create a catalog export job (for each catalog)
Create a file destination in Azure DevOps
Create a transport configuration
Create a process chain for automatic export
Create all job configurations via DBInit
Configure SPARQUE to read from export destination
References
Steps for Automation
Create Product Export Job
To automate the export of all products, create a job that runs the product export:
# Name of job configuration RunProductExport.Name=RunProductExport RunProductExport.Description=RunProductExport #RunProductExport.Date=2010.11.01 at 00:00:00 #RunProductExport.Interval=1440 RunProductExport.PipelineName=ProcessImpexJob RunProductExport.PipelineStartNode=Start RunProductExport.EnableJob=true RunProductExport.ApplicationSite=inSPIRED-Site RunProductExport.ApplicationURLIdentifier=inTRONICS # add custom attributes (keypair with AttributeName<Number> = AttributeValue<Number>) RunProductExport.AttributeName1=DomainName RunProductExport.AttributeValue1=inSPIRED-inTRONICS RunProductExport.AttributeName2=ExportDirectory RunProductExport.AttributeValue2=sparque RunProductExport.AttributeName3=JobName RunProductExport.AttributeValue3=ProcessCatalogImpex RunProductExport.AttributeName4=ProcessPipelineName RunProductExport.AttributeValue4=ProcessProductExport RunProductExport.AttributeName5=ProcessPipelineStartNode RunProductExport.AttributeValue5=Export RunProductExport.AttributeName6=SelectedFile RunProductExport.AttributeValue6=exportFromProcessChain.xml
Create Catalog Export Job
Catalogs need to be exported separately, one export per catalog. This can also be done via a job configuration, similar to the following:
# Name of job configuration RunCatalogCamerasExport.Name=RunCatalogCamerasExport RunCatalogCamerasExport.Description=RunCatalogCamerasExport #RunCatalogCamerasExport.Date=2010.11.01 at 00:00:00 #RunCatalogCamerasExport.Interval=1440 RunCatalogCamerasExport.PipelineName=ProcessImpexJob RunCatalogCamerasExport.PipelineStartNode=Start RunCatalogCamerasExport.EnableJob=true RunCatalogCamerasExport.ApplicationSite=inSPIRED-Site RunCatalogCamerasExport.ApplicationURLIdentifier=inTRONICS # add custom attributes (keypair with AttributeName<Number> = AttributeValue<Number>) RunCatalogCamerasExport.AttributeName1=DomainName RunCatalogCamerasExport.AttributeValue1=inSPIRED-inTRONICS RunCatalogCamerasExport.AttributeName2=ExportDirectory RunCatalogCamerasExport.AttributeValue2=sparque RunCatalogCamerasExport.AttributeName3=CatalogID RunCatalogCamerasExport.AttributeValue3=Cameras-Camcorders RunCatalogCamerasExport.AttributeName4=ProcessPipelineName RunCatalogCamerasExport.AttributeValue4=ProcessCatalogExport RunCatalogCamerasExport.AttributeName5=ProcessPipelineStartNode RunCatalogCamerasExport.AttributeValue5=Export RunCatalogCamerasExport.AttributeName6=SelectedFile RunCatalogCamerasExport.AttributeValue6=exportCameras.xml
Create File Destination in Azure DevOps
To create a file destination in Azure DevOps, perform the following steps:
Go to Microsoft Azure.
Create a storage account or use an existing one.
Create a new file share. In this example we will call it sparque.
Create an access key, it will be required in the next step.
Create Transport Configuration
For the full transport, create a transport configuration as shown below:
domain=inSPIRED-inTRONICS process.id=SparqueTransport process.displayname=SparqueTransport process.type=AZURE location.local=<path to shared file system>/sites/inSPIRED-inTRONICS-Site/units/inSPIRED-inTRONICS/impex/export/sparque account.key=<previously created access key> account.name=<storage account name> file.share=<previously created folder in fileshare, e.g. sparque> process.direction=PUSH process.delete=0
The transport can then be automated using a job.
ExecuteSparqueTransport.Name=ExecuteSparqueTransport ExecuteSparqueTransport.Description=ExecuteSparqueTransport #ExecuteSparqueTransport.Date=2010.11.01 at 00:00:00 #ExecuteSparqueTransport.Interval=1440 ExecuteSparqueTransport.PipelineName=FileTransportJob ExecuteSparqueTransport.PipelineStartNode=Start ExecuteSparqueTransport.EnableJob=true # add custom attributes (keypair with AttributeName<Number> = AttributeValue<Number>) ExecuteSparqueTransport.AttributeName1=TransportProcessID ExecuteSparqueTransport.AttributeValue1=SparqueTransport
Create and Execute Process Chain for Automatic Export
The process chain contains all previous exports and the transport configuration. Timeouts should be adjusted for projects. Also, depending on the number of products/categories, it may be more efficient/faster to run the exports concurrently.
For details on all of the process chain options, see Concept - Process Chains (valid to 11.x).
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<chain xmlns="http://www.intershop.com/xml/ns/enfinity/6.4/core/processchain" name="Chain 1" timeout="90">
<sequence name="Chain 1.1 - Sequence" timeout="90">
<job job="RunProductExport" domain="inSPIRED-inTRONICS" name="Chain 1.1.1 - Job" timeout="60"/>
<job job="RunCatalogCamerasExport" domain="inSPIRED-inTRONICS" name="Chain 1.1.2 - Job" timeout="60"/>
<!--- more catalog exports i.e.<job job="RunCatalogSpecialsExport" domain="inSPIRED-inTRONICS" name="Chain 1.1.3 - Job" timeout="60"/>--->
<job job="ExecuteSparqueTransport" domain="inSPIRED-inTRONICS" name="Chain 1.1.4 - Job" timeout="30"/>
</sequence>
</chain>
A process chain can be triggered manually in the back office, the automation approach would be to create a job configuration for this as well:
# Name of job configuration ExecuteSparqueProcessChain.Name=ExecuteSparqueProcessChain ExecuteSparqueProcessChain.Description=ExecuteSparqueProcessChain #ExecuteSparqueProcessChain.Date=2010.11.01 at 00:00:00 #ExecuteSparqueProcessChain.Interval=1440 ExecuteSparqueProcessChain.PipelineName=ExecuteProcessChain ExecuteSparqueProcessChain.PipelineStartNode=Start ExecuteSparqueProcessChain.EnableJob=true # add custom attributes (keypair with AttributeName<Number> = AttributeValue<Number>) ExecuteSparqueProcessChain.AttributeName1=XmlFileName ExecuteSparqueProcessChain.AttributeValue1=system\\config\\cluster\\ExportAndTransportProducts.xml
Create All Job Configurations via DBInit
Job configurations and transport configurations can be created through DBInit using the PrepareTransportConfiguration and PrepareJobConfigurations preparers:
Class1500 = com.intershop.component.transport.dbinit.PrepareTransportConfiguration \
com.intershop.demo.responsive.dbinit.data.job.TransportConfiguration
Class1505 = com.intershop.beehive.core.dbinit.preparer.job.PrepareJobConfigurations \
inSPIRED-inTRONICS \
com.intershop.demo.responsive.dbinit.data.job.JobConfigurations
Configure SPARQUE to Read from Export Destination
Create Shared Access Signature
To access the created file in the file share, a Shared Access Signature must be created.
Navigate to Security + networking | Shared access signature:
Settings:
Allowed services: File
Allowed resource types: Service, Container, Object
Allowed permissions: Read, List
Allowed IP addresses: Add if necessary
Define the expiry date/time - Select a proper date in the future. Make sure you refresh the signature after expiration date.
Signing Key: Use the same access key as above.
Click on Generate SAS and connection string.
Copy the string of SAS token.
See also Microsoft | Grant limited access to Azure Storage resources using shared access signatures (SAS).
Alternatively, if the Azure Portal is not available for the task, you can create a SAS token using the Azure CLI as follows:
az storage share generate-sas --name <share-name> --account-name <storage-account-name> --permissions rl --https-only --expiry 2028-01-01T00:00Z --account-key <storage-account-key>
Use File Service SAS Token in Sparque
After running the export jobs and the transport configuration, the exported and transported files are in the located created file share. SPARQUE.AI can use this file share as a base for a dataset. To use this function, configure a dataset source to Fetch a file from URL and enter the path to the file share along with the access key. This allows SPARQUE.AI to fetch data from this data source.
Example: <https://<storageaccount>.file.core.windows.net/<filesharename>/<exportfile>?<SAS token>
