Related Documents
Document Properties
Kbid
29T787
Last Modified
19-Apr-2021
Added to KB
04-Feb-2021
Public Access
Everyone
Status
Online
Doc Type
Guidelines, Concepts & Cookbooks
Product
ICM 7.10

Cookbook - Solr Cloud Server

1 Introduction

This cookbook contains recipes for frequently asked questions about the Solr server deployment.

1.1 References

2 Recipe: Zookeeper IOExceptions "Unreasonable Length" or "Len Error"

2.1 Problem

There are Zookeeper IOExceptions similar to the following:

IOException example
org.apache.zookeeper.ClientCnxn; Session 0x10166d8b6020002 for server xxxxxxxxxx:2181, unexpected error, closing socket connection and attempting reconnect => java.io.IOException: Unreasonable length = 1491656
        at org.apache.jute.BinaryInputArchive.checkLength(BinaryInputArchive.java:127)
java.io.IOException: Unreasonable length = 1491656
        at org.apache.jute.BinaryInputArchive.checkLength(BinaryInputArchive.java:127) 
        at org.apache.jute.BinaryInputArchive.readBuffer(BinaryInputArchive.java:92)
        at org.apache.zookeeper.proto.GetDataResponse.deserialize(GetDataResponse.java:56) 
        at org.apache.zookeeper.ClientCnxn$SendThread.readResponse(ClientCnxn.java:919)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:101) 
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:363)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) 

2.2 Solution

The exceptions occur if configuration files are too big to be stored in Zookeeper by default. The default size is 1MB.

To increase the size, set the jute.maxbuffer system property for the startup of the Zookeeper server, the Solr server, and the ICM application server.

2.3 Discussion

It is important to set the system property consistently in all servers and clients that upload or download such configuration files. The value of the system property specifies the maximum size in bytes.

  1. Add JVMFLAGS to zookeeper-env.sh for the Zookeeper server:

    zookeeper-env.sh
    JVMFLAGS=-Djute.maxbuffer=2097152
  2. Add the SOLR_OPTS to solr.in.sh for the Solr server:

    /etc/default/solr.in.sh
    SOLR_OPTS="$SOLR_OPTS -Djute.maxbuffer=2097152

    Note

    This setting applies to the Solr server startup only. It does not apply to the solr zk command line utility.
  3. If the added configuration files are part of the default configuration in share/system/config/cluster/solrcloud which is initially uploaded from the ICM application server, you have to add the jute.maxbuffer system property to the ICM application server(s):

    settings.gradle
    	jvmArgs {
        	additionalJvmArgs = [ '-Djute.maxbuffer=2097152' ]
    	}

3 Recipe: Use Basic Authentication for ICM - Solr Cloud Server Communication

3.1 Problem

During index builds the following error might occur: "Can't create a configset with an unauthenticated request from a trusted baseConfigSet".

How do I solve this error?

3.2 Solution

Starting with Solr 8.6.3, the Solr server refuses to create a config set from a base config set if the user that communicates with the Solr server is an unauthenticated user. To solve this problem, you can switch to basic authentication.

It is also possible to solve this issue by disabling the required authentication of the config set upload by setting the system property solr.disableConfigSetsCreateAuthChecks=true at the Solr server startup.

3.3 Discussion

The setup of basic authentication requires the following steps:

  1. Add (upload) a security.json to the Solr server.
    There is a security-sample.json provided in the installation of the Solr Cloud Adapter that is located in share/system/config/cluster/solrcloud. In that sample file, there are two users: solr with password SolrRocks and intershop with password !InterShop00!
    To upload the security.json, you can use the zkcli.sh/zkcli.bat script provided with the Solr server installation located in server/scripts/cloud-scripts:

    zkcli putfile
    zkcli.sh -zkhost localhost:2181/solr8 -cmd putfile /security.json security-sample.json
  2. Configure ICM to use basic authentication when communicating with the Solr server:

    settings.gradle
    	jvmArgs {
    		additionalJvmArgs = ['"-Dsolr.httpclient.builder.factory=org.apache.solr.client.solrj.impl.PreemptiveBasicAuthClientBuilderFactory" "-Dsolr.httpclient.config=%IS_SHARE%/system/config/cluster/solrcloud/solr-authentification.properties"']
        }

    This configuration example uses the provided solr-authentification.properties to use the user intershop with password !InterShop00! that corresponds with the user that is contained in the security.json.

  3. Change the default passwords, see Solr Ref Guide 6.6 | Editing Authentication Plugin Configuration.

4 Recipe: Change Default Solr Configuration Files

4.1 Problem

I want to add custom configurations to the Solr configuration files, e.g. adding a new field type to the managed-schema.

4.2 Solution

Change the default configuration deployed to the share/system/config/cluster/solrcloud by using a deployment content filter in a customization cartridge.

4.3 Discussion

Although the schema and the config can be changed directly in the Solr cloud server with the Solr Schema REST API, it is often better to use a deployment script so that the customization can be applied to installations consistently.

The cluster configuration in the shared file system is applied to the Solr server if the ish_config is not already existing. Please see chapter 3.2 Solr Cloud Search Adapter of the Guide - Migration Solr4 to Solr Cloud Adapter.

The following example adds an additional field type textASCIIFolded that is defined in the customTypesText to the managed-schema file directly before the list of fields. The example uses the existing fullContent filter to manipulate the deployed content in favor of the xmlContentFilter to keep the existing file content as it is. But be aware that the result must form valid XML and especially a valid managed-schema definition. Otherwise the creation of indexes or the indexing will fail!
  1. Create a deploy.gradle in the deployment folder of the customization cartridge

    custom-cartridge/deployment/deploy.gradle
    apply plugin: com.intershop.deploy.cartridge.CartridgeDeploymentPlugin
    
    deployment {
    
    	def customTypesText = '''
      <!-- custom field types inserted during deployment -->
      <fieldType class="solr.TextField" name="textASCIIFolded">
    	<analyzer>
    	  <tokenizer class="solr.KeywordTokenizerFactory"/>
    	  <filter class="solr.LowerCaseFilterFactory"/>
    	  <filter class="solr.ASCIIFoldingFilterFactory"/>
    	</analyzer>
      </fieldType>
      <!-- end custom field types -->
    
    '''
    
        filters {
          
    	  fullContent('addCustomTypesToSchema') {
            dir = target.shareDirectory
            include 'system/config/cluster/solrcloud/default/conf/managed-schema'
            action { StringBuilder content -> 
    			content.insert(content.indexOf('  <field name="'), customTypesText)
    		}
    	  }
    	}
    }
  2. Adapt the desired custom content
  3. Execute the deployment
  4. Re-create the Solr configuration and index by deleting the ish_config config set, the collections, and the config sets (Job - Cleanup Search Indexes or Job - ProcessChain.InitIndexesProductAssignments)


Be aware that the mentioned jobs Job - Cleanup Search Indexes or Job - ProcessChain.InitIndexesProductAssignment delete all configurations and collections that are prefixed by the configured solr.ClusterIndexPrefix of the ICM installation.  No storefront search will be available during that process.


5 Recipe: Add Language-Specific Solr Configuration Files

5.1 Problem

I have a new locale added to ICM and want to have an additional language-specific field-type available in the Solr schema configuration.

5.2 Solution

Add a schema_<language>.json in share/system/config/cluster/solrcloud/lang and additional files to share/system/config/cluster/solrcloud/default/lang to the deployment

5.3 Discussion

The creation of new indexes is looking for a language-specific schema file in the share/system/config/cluster/solrcloud/lang folder with the language identifier taken from the index locale of the created index.

Example: Creating an index in the locale ja_JP will look for a schema_ja.json file

schema_ja.json
{
    "add-field-type":{
        "name":"text_ja",
        "class":"solr.TextField",
        "autoGeneratePhraseQueries":"false",
        "positionIncrementGap":"100",
        "analyzer":{
            "tokenizer":{
                "class":"solr.JapaneseTokenizerFactory",
                "mode":"search"
            },
            "filters":[
                {
                    "class":"solr.JapaneseBaseFormFilterFactory"
                },
                {
                    "class":"solr.JapanesePartOfSpeechStopFilterFactory",
                    "tags":"lang/stoptags_ja.txt"
                },
                {
                    "class":"solr.CJKWidthFilterFactory"
                },
                {
                    "class":"solr.ManagedStopFilterFactory",
                    "managed":"default"
                },
                {
                    "class":"solr.JapaneseKatakanaStemFilterFactory",
                    "minimumLength":"4"
                },
                {
                    "class":"solr.LowerCaseFilterFactory"
                }
            ]
        }
    }
}

This json file can contain multiple valid Solr Schema REST API commands. Please see Solr documentation about the Schema REST API.


The example references a static file for lang/stoptags_ja.txt that needs to be added to the static default configuration in the share/system/config/cluster/solrcloud/default/lang folder.

In contrast, the solr.ManagedStopFilterFactory, a managed resource is used for the stop words. This enables the built-in management of stop words directly from the ICM back-office.



Disclaimer

The information provided in the Knowledge Base may not be applicable to all systems and situations. Intershop Communications will not be liable to any party for any direct or indirect damages resulting from the use of the Customer Support section of the Intershop Corporate Web site, including, without limitation, any lost profits, business interruption, loss of programs or other data on your information handling system.

Customer Support
Knowledge Base
Product Resources
Tickets