Guide - Migration Solr4 to Solr Cloud Adapter

1 Introduction

This guide describes the differences in the handling of Solr configuration between the Solr4 and Solr Cloud adapter and helps to migrate existing projects that currently use the Solr4 search adapter to use the newer Solr Cloud Search Adapter.

2 References

2.1 General Steps

The following steps outline the general approach to updating from Solr4 to Solr Cloud Adapter:

To understand the differences in the configuration of Solr4 and Solr Cloud adapter we describe these configurations in more detail in the following section.

3 Cluster Configuration

The Solr Search Adapters both come with a general configuration that is used to configure indexes when these indexes are created in the system. This configuration is stored in sub folders of the shared file system in share/system/config/cluster. The following sections describe the handling of this configuration during the index creation.

3.1 Solr4 Search Adapter

share/system/config/cluster contains a sub-folder solr with the general configuration. Additionally there is a solr.xml that keeps track of the indexes that exist in the Solr server. This file is maintained by the Solr server. When there is an index created in the backoffice, the Solr search adapter creates a core in the Solr server by copying the share/system/conf folder to the individual index directory. Depending on the locale of the created index it additionally copies language specific files to the conf folder that overwrite the general configuration. Thereafter the adapter synchronizes the field list of the individual schema.xml in the file system with the search index attribute list contained in the ISH-Config.xml. A core is created at the Solr Server with the Core Admin API.

3.2 Solr Cloud Search Adapter

share/system/config/cluster contains a sub-folder solrcloud that contains the general configuration. This configuration is initially uploaded to the Solr Cloud Server when an index (a collection in terms of Solr Cloud) is to be created and the Solr Cloud Adapter does not find a configuration set named <prefix>_ish_config at the given Solr Cloud Server.

When a search index is created in the backoffice, the Solr Cloud search adapter checks the existence of an according collection at the Solr Cloud Server. If there is no such collection, a configuration set with a name equally to the collection is created from the common configuration set. If the common configuration set does not exist, it is uploaded to the Solr Cloud Server as config set ish_config.  The important difference with regard to the Solr4 adapter is the upload of the common configuration to the Solr Cloud server. Changes that are made to the share/system/config/cluster/solrcloud configuration in the file system only have an effect if the common configuration set is changed and the collection is recreated.

Synonyms and stop word lists are maintained as managed resources at the Solr Cloud server. While the synonyms are also stored in the ISH-Config.xml, the stop word lists are maintained only at the Solr Cloud server.

4 Migration of Solr Configuration

All the Solr4 indexes consist of an index directory in the shared file system that holds the ISH-Config.xml, the Solr configuration files in the conf folder and the index data in the data directory.

The Solr Cloud Adapter also uses the index directory containing the ISH-Config.xml but maintains the Solr configuration at the Solr Cloud Server with the Solr APIs for the schema, config sets and collections. If there is initially no collection at the Solr Cloud server for an index at the application server side, the collection and the configuration (config set) is created from the data of the ISH-Config.xml and from the general configuration of the ish_config config-set. Once the configurations from the ISH-Config.xml are synced with the Solr Cloud Server, there will be an additional lastSchemaUpdate.properties file that stores the date of the last schema synchronization between the ISH-Config.xml and the configuration at the Solr Cloud Server.

4.1 Migrating Custom Configuration Changes

If your project changed the Solr configuration files e.g. solrconfig.xml or schema.xml then you need to migrate and test these changes to be included into the Solr Cloud server manually.

4.1.1 Apply Custom Changes to the General Configuration

Apply the custom changes to the configuration that is stored in the share/system/config/cluster/solrcloud directory. Please be aware that these changes are uploaded only initially at the first creation of a collection by the Solr Cloud adapter. You may use deployment utilities to apply these changes during the deployment.

4.1.2 Apply Custom Changes to Individual Indexes

Changes to individual indexes can be applied once the collection and the associated config set is created in the Solr Cloud server. The "Rebuild Search Indexes" schedule from SLDSystem can be used to create all the existing indexes at the Solr Cloud server. Download the configuration with the Solr command line tool, apply custom changes to this index configuration and re-upload the changes to the Solr Cloud server.

4.2 Migration of Synonyms

The Solr4 synonyms that are stored in the ISH-Config.xml are synchronized at the creation of the collection at the Solr Cloud server and whenever the synonyms are changed and saved in the ICM back office.

4.3 Migration of Stop Word Lists

If there are stop word lists from Solr4 indexes available in the index directory in the shared file system, the initial creation of the collection at the Solr Cloud server also submits the content of these stop word lists (stopwords.txt and querystopwords.txt) to the Solr Cloud server. Subsequent changes to the stop word lists in the ICM back-office are applied to the managed resources of the Solr Cloud server only.

4.4 Change Configuration at Solr Cloud Server

To maintain and change the configurations that reside at the Solr Cloud Server, the Solr distribution provides tooling to download/upload the configuration sets. Please refer to the Solr Documentation about the available command line utilities, especially the solr zk downconfig/upconfig commands. Of course it is also possible to use the Solr servers REST-APIs to change specific parts of the configuration.

Info

Be aware that the configuration (e.g. the list of fields) is synchronized from the Solr Cloud adapter with the existing ISH-Config.xml.

5 Solr 8 Migration

In general the migration of Solr4 to the newer Solr Cloud Adapter for Solr8 works in the same way as the migration described above.

Migration from Solr 6 to Solr 8 of the default configuration supplied with the adapter has introduced only 3 main changes:

  • All Trie* fields have changed to use the newer Point* datatypes
  • ManagedSynonymFilterFactory was replaced by ManagedSynonymGraphFilterFactory
  • The luceneMatchVersion in the solrconfig.xml

If your Solr configuration used any custom changes, you should check these custom changes according to the Solr Upgrade Notes. Pay special attention to the major changes that were done for Solr 7 and Solr 8 to see if your custom configuration is affected by the upgrade.

5.1 Upgrading the Solr Cloud Server from 6 to 8

You may follow the upgrade procedure described in the official documentation: Upgrading a Solr Cluster.

You may also set up a new Solr8 cluster aside the Solr6 cluster and switch to the new cluster by connecting ICM to that new cluster. This will upload the initial configurations set and recreate all collections at the first index build as described above.

Disclaimer

The information provided in the Knowledge Base may not be applicable to all systems and situations. Intershop Communications will not be liable to any party for any direct or indirect damages resulting from the use of the Customer Support section of the Intershop Corporate Web site, including, without limitation, any lost profits, business interruption, loss of programs or other data on your information handling system.

Customer Support
Knowledge Base
Product Resources
Tickets