Personal Library

Guide - Migration Solr4 to SolrCloud Adapter

Introduction

This guide describes the differences in the handling of Solr configuration between the Solr4 and SolrCloud adapter and helps to migrate existing projects that currently use the Solr4 search adapter to use the newer SolrCloud Search Adapter.

References

General Steps

The following steps outline the general approach to updating from Solr4 to SolrCloud Adapter:

Deploy SolrCloud Server(s) See Guide - Deployment Solr Cloud Server
Determine and migrate any custom changes (optional)
Change ICM deployment and configuration according to Cookbook - Deployment of Solr Search Adapters (Recipe: Use Solr Cloud Adapter)
Deploy the changes
Rebuild search indexes to re-create Indexes at the SolrCloud Server Job - Rebuild Search Indexes
Delete old Solr4 indexes (data directory) and Solr4 configuration (conf directory) from the shared file system.

To understand the differences in the configuration of Solr4 and SolrCloud adapter we describe these configurations in more detail in the following section.

Cluster Configuration

The Solr Search Adapters both come with a general configuration that is used to configure indexes when these indexes are created in the system. This configuration is stored in sub folders of the shared file system in share/system/config/cluster. The following sections describe the handling of this configuration during the index creation.

Solr4 Search Adapter

share/system/config/cluster contains a sub-folder solr with the general configuration. Additionally there is a solr.xml that keeps track of the indexes that exist in the Solr server. This file is maintained by the Solr server. When there is an index created in the backoffice, the Solr search adapter creates a core in the Solr server by copying the share/system/conf folder to the individual index directory. Depending on the locale of the created index it additionally copies language specific files to the conf folder that overwrite the general configuration. Thereafter the adapter synchronizes the field list of the individual schema.xml in the file system with the search index attribute list contained in the ISH-Config.xml. A core is created at the Solr Server with the Core Admin API.

SolrCloud Search Adapter

share/system/config/cluster contains a sub-folder solrcloud that contains the general configuration. This configuration is initially uploaded to the SolrCloud Server when an index (a collection in terms of SolrCloud) is to be created and the SolrCloud Adapter does not find a configuration set named <prefix>_ish_config at the given SolrCloud Server.

When a search index is created in the backoffice, the SolrCloud search adapter checks the existence of an according collection at the SolrCloud Server. If there is no such collection, a configuration set with a name equally to the collection is created from the common configuration set. If the common configuration set does not exist, it is uploaded to the SolrCloud Server as config set ish_config. The important difference with regard to the Solr4 adapter is the upload of the common configuration to the SolrCloud server. Changes that are made to the share/system/config/cluster/solrcloud configuration in the file system only have an effect if the common configuration set is changed and the collection is recreated. If the original configuration has to be changed, refer to Change Configuration Solr at Cloud Server.

Synonyms and stop word lists are maintained as managed resources at the SolrCloud server. While the synonyms are also stored in the ISH-Config.xml, the stop word lists are maintained only at the SolrCloud server.

Migration of Solr Configuration

All the Solr4 indexes consist of an index directory in the shared file system that holds the ISH-Config.xml, the Solr configuration files in the conf folder and the index data in the data directory.

The SolrCloud Adapter also uses the index directory containing the ISH-Config.xml but maintains the Solr configuration at the SolrCloud Server with the Solr APIs for the schema, config sets and collections. If there is initially no collection at the SolrCloud server for an index at the application server side, the collection and the configuration (config set) is created from the data of the ISH-Config.xml and from the general configuration of the ish_config config-set. Once the configurations from the ISH-Config.xml are synced with the SolrCloud Server, there will be an additional lastSchemaUpdate.properties file that stores the date of the last schema synchronization between the ISH-Config.xml and the configuration at the SolrCloud Server.

Migrating Custom Configuration Changes

If your project changed the Solr configuration files e.g. solrconfig.xml or schema.xml then you need to migrate and test these changes to be included into the SolrCloud server manually.

Apply Custom Changes to the General Configuration

Apply the custom changes to the configuration that is stored in the share/system/config/cluster/solrcloud directory. Please be aware that these changes are uploaded only initially at the first creation of a collection by the SolrCloud adapter. You may use deployment utilities to apply these changes during the deployment.

Apply Custom Changes to Individual Indexes

Changes to individual indexes can be applied once the collection and the associated config set is created in the SolrCloud server. The "Rebuild Search Indexes" schedule from SLDSystem can be used to create all the existing indexes at the SolrCloud server. Download the configuration with the Solr command line tool, apply custom changes to this index configuration and re-upload the changes to the SolrCloud server.

Migration of Synonyms

The Solr4 synonyms that are stored in the ISH-Config.xml are synchronized at the creation of the collection at the SolrCloud server and whenever the synonyms are changed and saved in the ICM back office.

Migration of Stop Word Lists

If there are stop word lists from Solr4 indexes available in the index directory in the shared file system, the initial creation of the collection at the SolrCloud server also submits the content of these stop word lists (stopwords.txt and querystopwords.txt) to the SolrCloud server. Subsequent changes to the stop word lists in the ICM back-office are applied to the managed resources of the SolrCloud server only.

Change Configuration at SolrCloud Server

The initial configuration at share/system/config/solrcloud can be changed by deployment tools (Cookbook - Deployment Tools ICM 7.10). However, this configuration is only uploaded if there is no configuration on the Solr server. To maintain and change the configurations that is already available on the SolrCloud Server, the Solr distribution provides tooling to download/upload the configuration sets. Please refer to the Solr Documentation for available command line utilities, especially the solr zk downconfig/upconfig commands. Of course it is also possible to use the Solr server REST APIs to change specific parts of the configuration.

Info

Be aware that the configuration (e.g. the list of fields) is synchronized from the SolrCloud adapter with the existing ISH-Config.xml.

Solr 8 Migration

In general the migration of Solr4 to the newer SolrCloud Adapter for Solr8 works in the same way as the migration described above.

Migration from Solr 6 to Solr 8 of the default configuration supplied with the adapter has introduced only 3 main changes:

All Trie* fields have changed to use the newer Point* datatypes
ManagedSynonymFilterFactory was replaced by ManagedSynonymGraphFilterFactory
The luceneMatchVersion in the solrconfig.xml

If your Solr configuration used any custom changes, you should check these custom changes according to the Solr Upgrade Notes. Pay special attention to the major changes that were done for Solr 7 and Solr 8 to see if your custom configuration is affected by the upgrade.

Upgrading the SolrCloud Server from 6 to 8

You may follow the upgrade procedure described in the official documentation: Upgrading a Solr Cluster.

You may also set up a new Solr8 cluster aside the Solr6 cluster and switch to the new cluster by connecting ICM to that new cluster. This will upload the initial configurations set and recreate all collections at the first index build as described above.

Data Replication

The data replication of indexes has changed. The standard ICM file replication is not used anymore to replicate the index data. Instead, the adapter comes with its own replication decorator that uses the backup/restore collection API functionality of Solr. Please see Guide - Solr Replication for the description and configuration of the index replication with the SolrCloud server.

Data replication with a SolrCloud cluster requires to mount a shared file system at the same path on each Solr node. The ICM must configure the property solr.cloudBackupLocation to point to this shared location.

Solr Server version 8.6.3 or higher in conjunction with data replication requires setting the backup location as an allowed path for the Solr server cluster. You can specify the allowed paths by setting a system property via environment in the startup script:

solr.in.sh

SOLR_OPTS="$SOLR_OPTS -Dsolr.allowPaths=/mnt/solr_backup"

Disclaimer

The information provided in the Knowledge Base may not be applicable to all systems and situations. Intershop Communications will not be liable to any party for any direct or indirect damages resulting from the use of the Customer Support section of the Intershop Corporate Web site, including, without limitation, any lost profits, business interruption, loss of programs or other data on your information handling system.

Table of Contents