Introduction

References

• Guide - Deployment Solr Cloud Server
• Guide - Migration Solr4 to SolrCloud Adapter

Recipe: Zookeeper IOExceptions "Unreasonable Length" or "Len Error"

Problem

There are Zookeeper IOExceptions similar to the following:

org.apache.zookeeper.ClientCnxn; Session 0x10166d8b6020002 for server xxxxxxxxxx:2181, unexpected error, closing socket connection and attempting reconnect => java.io.IOException: Unreasonable length = 1491656
        at org.apache.jute.BinaryInputArchive.checkLength(BinaryInputArchive.java:127)
java.io.IOException: Unreasonable length = 1491656
        at org.apache.jute.BinaryInputArchive.checkLength(BinaryInputArchive.java:127) 
        at org.apache.jute.BinaryInputArchive.readBuffer(BinaryInputArchive.java:92)
        at org.apache.zookeeper.proto.GetDataResponse.deserialize(GetDataResponse.java:56) 
        at org.apache.zookeeper.ClientCnxn$SendThread.readResponse(ClientCnxn.java:919)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:101) 
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:363)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) 

Solution

The exceptions occur if configuration files are too big to be stored in Zookeeper by default. The default size is 1MB.

To increase the size, set the jute.maxbuffer system property for the startup of the Zookeeper server, the Solr server, and the ICM application server.

Discussion

It is important to set the system property consistently in all servers and clients that upload or download such configuration files. The value of the system property specifies the maximum size in bytes.

1. Add JVMFLAGS to zookeeper-env.sh for the Zookeeper server:

   zookeeper-env.sh

   JVMFLAGS=-Djute.maxbuffer=2097152

2. Add the SOLR_OPTS to solr.in.sh for the Solr server:

   /etc/default/solr.in.sh

   SOLR_OPTS="$SOLR_OPTS -Djute.maxbuffer=2097152

   Note

   This setting applies to the Solr server startup only. It does not apply to the solr zk command line utility.

3. If the added configuration files are part of the default configuration in share/system/config/cluster/solrcloud which is initially uploaded from the ICM application server, you have to add the jute.maxbuffer system property to the ICM application server(s):

   settings.gradle

   	jvmArgs {
       	additionalJvmArgs = [ '-Djute.maxbuffer=2097152' ]
   	}

Recipe: Use Basic Authentication for ICM - Solr Cloud Server Communication

Problem

During index builds the following error might occur: "Can't create a configset with an unauthenticated request from a trusted baseConfigSet".

How do I solve this error?

Solution

Starting with Solr 8.6.3, the Solr server does not create a config set from a base config set if the user that communicates with the Solr server is an unauthenticated user. To solve this problem, you can switch to basic authentication.

It is also possible to solve this issue by disabling the required authentication of the config set upload by setting the system property solr.disableConfigSetsCreateAuthChecks=true at the Solr server startup.

Discussion

The setup of basic authentication requires the following steps:

1. Add (upload) a security.json to the Solr server.
   There is a security-sample.json provided in the installation of the Solr Cloud Adapter that is located in share/system/config/cluster/solrcloud. In that sample file, there are two users: solr with password SolrRocks and intershop with password !InterShop00!
To upload the security.json, you can use the zkcli.sh/zkcli.bat script provided with the Solr server installation located in server/scripts/cloud-scripts:

   zkcli putfile

   zkcli.sh -zkhost localhost:2181/solr8 -cmd putfile /security.json security-sample.json

   To enable the Solr Cloud authentication and authorization framework inside a Windows-based cloud cluster, you need to upload the ICM json file.

   A basic command line for this is:

   zkcli putfile

   d:\solr-8.9.0\bin>solr zk cp D:\CI\710385\live\share\system\config\cluster\solrcloud\security-sample.json zk:/security.json -z localhost:9983

   In this example the ICM security-sample.json is uploaded in the Solr Cloud environment to enable authentication.

2. Configure ICM to use basic authentication when communicating with the Solr server:

   settings.gradle

   	jvmArgs {
   		additionalJvmArgs = ['"-Dsolr.httpclient.builder.factory=org.apache.solr.client.solrj.impl.PreemptiveBasicAuthClientBuilderFactory" "-Dsolr.httpclient.config=%IS_SHARE%/system/config/cluster/solrcloud/solr-authentification.properties"']
       }

   This configuration example uses the provided solr-authentification.properties to use the user intershop with password !InterShop00! that corresponds with the user that is contained in the security.json.

   
   

3. Change the default passwords, see Solr Ref Guide 6.6 | Editing Authentication Plugin Configuration.

Recipe: Change Default Solr Configuration Files

Problem

I want to add custom configurations to the Solr configuration files, e.g. adding a new field type to the managed-schema.

Solution

Change the default configuration deployed to the share/system/config/cluster/solrcloud by using a deployment content filter in a customization cartridge.

Discussion

Although the schema and the config can be changed directly in the Solr cloud server with the Solr Schema REST API, it is often better to use a deployment script so that the customization can be applied to installations consistently.

The cluster configuration in the shared file system is applied to the Solr server if the ish_config is not already existing. Please see chapter 3.2 Solr Cloud Search Adapter of the Guide - Migration Solr4 to SolrCloud Adapter.

1. Create a deploy.gradle in the deployment folder of the customization cartridge

   custom-cartridge/deployment/deploy.gradle

   apply plugin: com.intershop.deploy.cartridge.CartridgeDeploymentPlugin
   
   deployment {
   
   	def customTypesText = '''
     <!-- custom field types inserted during deployment -->
     <fieldType class="solr.TextField" name="textASCIIFolded">
   	<analyzer>
   	  <tokenizer class="solr.KeywordTokenizerFactory"/>
   	  <filter class="solr.LowerCaseFilterFactory"/>
   	  <filter class="solr.ASCIIFoldingFilterFactory"/>
   	</analyzer>
     </fieldType>
     <!-- end custom field types -->
   
   '''
   
       filters {
         
   	  fullContent('addCustomTypesToSchema') {
           dir = target.shareDirectory
           include 'system/config/cluster/solrcloud/default/conf/managed-schema'
           action { StringBuilder content -> 
   			content.insert(content.indexOf('  <field name="'), customTypesText)
   		}
   	  }
   	}
   }

2. Adapt the desired custom content
3. Execute the deployment
4. Re-create the Solr configuration and index by deleting the ish_config config set, the collections, and the config sets (Job - Cleanup Search Indexes or Job - ProcessChain.InitIndexesProductAssignments)

Recipe: Add Language-Specific Solr Configuration Files

Problem

I have a new locale added to ICM and want to have an additional language-specific field-type available in the Solr schema configuration.

Solution

Add a schema_<language>.json in share/system/config/cluster/solrcloud/lang and additional files to share/system/config/cluster/solrcloud/default/lang to the deployment

Discussion

The creation of new indexes is looking for a language-specific schema file in the share/system/config/cluster/solrcloud/lang folder with the language identifier taken from the index locale of the created index.

Example: Creating an index in the locale ja_JP will look for a schema_ja.json file

{
    "add-field-type":{
        "name":"text_ja",
        "class":"solr.TextField",
        "autoGeneratePhraseQueries":"false",
        "positionIncrementGap":"100",
        "analyzer":{
            "tokenizer":{
                "class":"solr.JapaneseTokenizerFactory",
                "mode":"search"
            },
            "filters":[
                {
                    "class":"solr.JapaneseBaseFormFilterFactory"
                },
                {
                    "class":"solr.JapanesePartOfSpeechStopFilterFactory",
                    "tags":"lang/stoptags_ja.txt"
                },
                {
                    "class":"solr.CJKWidthFilterFactory"
                },
                {
                    "class":"solr.ManagedStopFilterFactory",
                    "managed":"default"
                },
                {
                    "class":"solr.JapaneseKatakanaStemFilterFactory",
                    "minimumLength":"4"
                },
                {
                    "class":"solr.LowerCaseFilterFactory"
                }
            ]
        }
    }
}

This json file can contain multiple valid Solr Schema REST API commands. Please see Solr documentation about the Schema REST API.