Introduction

The present guide is addressed to administrators and software developers who want to operate IOM 3. It enables them to understand what the components of IOM are, how to configure them, and to run processes like installations and updates.

For a technical overview please see References.

Glossary

References

• System Requirements - Intershop Order Management 3.0
• System Requirements - Intershop Order Management 3.1
• System Requirements - Intershop Order Management 3.3
• System Requirements - Intershop Order Management 3.4
• System Requirements - Intershop Order Management 3.5
• System Requirements - Intershop Order Management 3.6
• System Requirements - Intershop Order Management 3.7
• Overview - IOM 2.17 - 3.0 Migration
• Guide - Intershop Order Management - Technical Overview

Additional References

• Kubernetes
• Helm
• PostgreSQL 12
• WildFly
• Adopt Open JDK
• JSON Web Tokens
• JSON Web Algorithms (JWA) | 3.2. HMAC with SHA-2 Functions

Prerequisites

Production systems for Intershop Order Management (IOM) are usually provided as a service in the Azure Cloud. This service is part of the corresponding Intershop Commerce Platform. Non-production environments require separate agreements with Intershop.
For the purpose of adapting the software to specific customer requirements and/or customer-specific environments, it is also possible to operate IOM (for example for corresponding CI environments, test systems etc) outside the Azure Cloud and independently of Azure Kubernetes (AKS). In support of this, this document is intended for IOM administrators and software developers.

Kubernetes

The exact required version of Kubernetes can be found in the system requirements (see References).

IOM requires a Kubernetes runtime environment. Intershop cannot provide support on how to set up, maintain or operate a Kubernetes runtime environment.

When using the Intershop Commerce Platform, Kubernetes is included. In this case, Intershop is fully responsible to set up, maintain, and operate the Kubernetes cluster as part of Intershop Commerce Platform.

Helm

The exact required version of Helm can be found in the system requirements (see References).

IOM requires Helm to be operated in a Kubernetes environment. Intershop cannot provide support on how to set up and use Helm properly.

When using the Intershop Commerce Platform, Helm is included. In this case, Intershop is fully responsible to set up and use Helm as part of Intershop Commerce Platform.

Mail Server

The exact requirements of the mail server can be found in the system requirements (see References).

IOM requires an existing mail server that processes e-mails sent from IOM via the SMTP protocol. Intershop cannot provide support on how to set up, maintain, or operate a mail server. A mail server is not part of the Intershop Commerce Platform.

PostgreSQL Server

Exact requirements of PostgreSQL server can be found in the system requirements (see References).

IOM requires a PostgreSQL database hosted by a PostgreSQL database server. Intershop cannot provide support on how to set up and operate a PostgreSQL server. Some configuration hints will be given as part of this document in section PostgreSQL Server Configuration.

When using the Intershop Commerce Platform, a PostgreSQL database is included. In this case, Intershop is fully responsible to set up and maintain the database as well as set up and operate the according PostgreSQL server as part of Intershop Commerce Platform.

Tools & Concepts

In order to understand this document, it is essential to know some basic concepts and tools. It is not the goal of this document to teach you all these tools and concepts. However, it is intended to provide an insight into how these tools and concepts are used in the context of Intershop Order Management.

Kubernetes

Kubernetes is an open-source container-orchestration system for automating computer application deployment, scaling, and management. Many cloud services offer a Kubernetes-based platform or infrastructure as a service (PaaS or IaaS) on which Kubernetes can be deployed as a platform-providing service. Many vendors also provide their own branded Kubernetes distributions. (https://en.wikipedia.org/wiki/Kubernetes).

Since Kubernetes is a standard for cloud operations, using it for IOM promises the best compatibility with a wide range of cloud providers. Nevertheless, functionality is guaranteed for Microsoft Azure Kubernetes service as part of the Intershop Commerce Platform only. You can use other environments at your own risk.

A full description of Kubernetes can be found at https://kubernetes.io/docs/home/.

Kubectl

Kubectl is a command-line interface to control Kubernetes clusters. It is part of Kubernetes, see https://kubernetes.io/docs/reference/kubectl/overview/.

Since it is a client which runs on the machine used to control the Kubernetes-cluster, it has to be installed separately. For this reason, it is listed as a separate tool. In the narrow sense, it is not required to operate IOM, but it is used in this document within the section Examples, to view the status of Kubernetes-objects.

Helm

Helm (https://helm.sh) sits on top of Kubernetes. Helm is a tool to manage the life cycle (install, upgrade, rollback, uninstall) of complex Kubernetes applications. To do so, it enables the development and provision of so-called Helm charts, which are basically descriptions of Kubernetes objects, combined by a template and scripting language.

IOM Docker Images

IOM is provided in form of Docker-images. These images can be used directly, as shown in section Examples of this document, or can be the base for further customization in the context of projects.

The images are available at:

• docker.intershop.de/intershophub/iom-dbaccount:1.3.0.0
• docker.intershop.de/intershophub/iom-config:3.6.0.0
• docker.intershop.de/intershophub/iom-app:3.6.0.0

docker.intershop.de is a private docker registry. Private Docker registries require authentication and sufficient rights to pull images from them. The according authentication data can be passed in a Kubernetes secret object, which has to be set using the Helm parameter imagePullSecrets.

The document Pull an Image from a Private Registry from Kubernetes documentation explains in general how to create Kubernetes secret objects, suitable to authenticate at a private Docker registry. Pull images from an Azure container registry to a Kubernetes cluster from Microsoft Azure documentation explains how to apply this concept to private Azure Container Registries.

The following box shows an example of how to create a Kubernetes secret to be used to access the private Docker Registry docker.intershop.de. The name of the newly created secret is intershop-pull-secret, which has to be passed to Helm parameter imagePullSecrets. It has to reside within the same Kubernetes namespace as the IOM cluster which uses the secret.

kubectl create secret docker-registry intershop-pull-secret \
    --docker-server=docker.intershop.de \
    --docker-username='<user name>' \
    --docker-password='<password>' \
    -n <kubernetes namespace>

IOM Helm-charts

IOM Helm-charts is a package containing the description of all Kubernetes-objects required to run IOM in Kubernetes. IOM Helm-charts are provided by Intershop at https://repository.intershop.de/helm. To use IOM Helm-charts, you have to execute the following commands (you may also have to pass credentials, which is not shown in the example).

# Add all Intershop charts
helm repo add intershop https://repository.intershop.de/helm

# Now the repo can be used to install IOM.
# The following command was taken from the examples section. Without the preconditions described there, it will not work.
# It is shown here only for demonstration how to reference the IOM Helm-chart after adding the according repository.
helm install demo intershop/iom --values=values.yaml --namespace iom --timeout 20m0s --wait

The following illustration shows the most important components and personas when operating IOM with Helm. The project owner has to define a values file (available configuration parameters are explained in section Parameters), which can be used along with IOM Helm-charts to install, upgrade, rollback, and uninstall IOM within a Kubernetes runtime environment.

This is a very generalized view, which has some restrictions when used with IOM. The next section explains these restrictions in detail.

Restrictions on Rollback

IOM uses a database that is constantly evolving along with new releases of IOM. For this reason, every version of IOM brings its own migration scripts, which are lifting the database to the new level. in general, old versions of the IOM database are not compatible with new versions of IOM application servers and vice versa. Also, projects change the database, when rolling out new or changed project configurations.

Helm does not know anything about changes inside the database. When rolling back a release, only the changes in values and IOM Helm-packages are rolled back. To avoid inconsistencies and failures (e.g. rollback to an old IOM application server version after updating the database structures to the new version), it is strongly recommended to avoid rollback in general.

Restrictions on Upgrade

The same reasons that make the rollback process problematic also limit the upgrade process. 

When executing the upgrade process, the standard behavior of Helm is to keep the application always online. The different IOM application servers are updated one after another. In case of incompatible database changes, this would lead to problems, since one of the following cases is unavoidable: an old IOM application server tries to work with an already updated IOM database or vice versa.

To overcome this problem, IOM Helm-charts provide the parameter downtime, which controls the behavior of the upgrade process. If downtime is set to true, the whole IOM cluster will be stopped during the upgrade process. The IOM database will be upgraded first and after that, the IOM application servers are started again. This setting should always be used when upgrading to a new IOM version, unless stated otherwise.

Within the context of projects, many changes can be applied to the running IOM cluster without requiring a downtime. In this case, the value of downtime has to be set to false before starting the upgrade process.

For security reasons, the default value of downtime is true to avoid any inconsistencies. Once you have understood the concept of the downtime parameter, you should set it to false to avoid downtimes as often as possible, and only set it to true when really required.

Intershop Commerce Platform

The previous section IOM Helm charts gave you a general view on Helm, the IOM Helm-charts, and the according processes. The Intershop Commerce Platform environment modifies this concept a little bit, as shown in the following illustration.

Project owners are not able to trigger any processes directly. They can only manage a sub-set of values to be applied along with the IOM Helm-chart. The processes are triggered by a flux-controller that observes the Git repository holding the values files. Depending on the type of IOM installation (INT, Pre-PROD, PROD, etc.) processes might need to be triggered manually by Intershop Operations. Intershop Operations maintains a values file too, which has higher precedence than the file of the project owner. This way it is ensured that the project owner is not able to change any critical settings. Which ones are affected, depends on the type of IOM installation (INT, Pre-PROD, PROD, etc.). For example, a project owner should never be able to set log-level to DEBUG or TRACE on PROD environments.

Examples

Despite the fact that Kubernetes and IOM Helm-charts make it very easy to set up and upgrade IOM installations, a reference to all the exiting parameters that are available to control IOM Helm-charts is a very uncomfortable starting point. For this reason, three typical usage scenarios were chosen to provide an easy-to-understand entry point into IOM Helm-charts. All examples were designed in a way that Intershop Commerce Platform is not required. The following examples strictly follow the concept described in section IOM Helm-Charts.

In order to understand the optional and required components defined in IOM Helm-charts, it is strongly recommended to read Guide - Intershop Order Management - Technical Overview first.

Local Demo System Running in Docker Desktop on Mac OS X 

Preconditions

• Mac computer: Mac OS X >= v.10.15
• Sufficient hardware resources: >= 16 GB main memory, multicore CPU
• Installation of Docker Desktop: >= v.2.3
  • See: https://www.docker.com/products/docker-desktop 
  • >= 12 GB memory and >= 2 CPUs have to be assigned (Preferences | Resources | Advanced)
  • Enable Kubernetes (Preferences | Kubernetes)
  • Directories used to hold persistent data have to be shared with Docker-Desktop (Preferences | Resources | File Sharing)
• Installation of Helm: >= v3.2
  • See: https://helm.sh/docs/intro/install/
• Access to IOM docker images
• Access to IOM Helm-charts

Requirements and Characteristics of IOM Installation

Requirements and characteristics are numbered. You will find these numbers also in the values file listed below in order to see the relation between requirement and current configuration.

1. Usage of integrated PostgreSQL server.
2. PostgreSQL data stored persistently to safe main memory.
3. No reset of PostgreSQL data during the installation process.
4. Usage of the integrated SMTP server (Mailhog).
5. Web access to the GUI of Mailhog.
6. The shared file system of IOM stored persistently.
7. Local access to the shared file system of IOM.
8. Due to limited resources, only one IOM application server should run.
9. Usage of the integrated NGINX controller for direct access to GUIs of IOM and Mailhog.
10. Due to limited resources, only one instance of the integrated NGINX controller should run.
11. No access from another computer required.

Values File

This values file cannot be copied as it is. Before it can be used, persistence.hostPath and postgres.persistence.hostPath have to be changed to existing paths, which are shared with Docker Desktop.

The values file contains minimal settings only, except dbaccount.resetData, which was listed explicitly, even if it contains the default value only.

# use one IOM server only (requirement #8).
replicaCount: 1

imagePullSecrets:
  - intershop-pull-secret

image:
  repository: "docker.intershop.de/intershophub/iom-app"
  tag: "3.6.0.0"

# configure ingress to forward requests for host "localhost" to IOM (requirements #9, #11).
# since integrated NGINX controller should be used, its class has to be set explicitly.
ingress:
  enabled: true
  className: nginx-iom
  hosts:
    - host: localhost
      paths: 
        - path: "/"
          pathType: Prefix

# IOM has to know its own public URL
oms:
  publicUrl: "https://localhost/"

# store data of shared-FS into local directory (requirement #6, #7)
persistence:
  hostPath: /Users/username/iom-share

config:
  image:
    repository: "docker.intershop.de/intershophub/iom-config"
    tag: "3.6.0.0"

# create IOM database and according database user before starting IOM. 
# do not reset existing data during installation (requirement #3)
dbaccount:
  enabled: true
  resetData: false # optional, since false is default
  image:
    repository: "docker.intershop.de/intershophub/iom-dbaccount"
    tag: "1.3.0.0"

# use integrated PostgreSQL server (requirement #1).
# store database data persistently into local directory (requirement #2).
postgres:
  enabled: true
  persistence:
    enabled: true
    hostPath: /Users/username/pgdata

# enable integrated NGINX ingress controller.
# this controller should not act proxy (requirement #9).
nginx:
  enabled: true
  proxy:
    enabled: false

# configure integrated NGINX ingress controller.
# one instance of NGINX is sufficient for demo scenario (requirement #10).
# set type to LoadBalancer to be accessible from public network (requirement #9).
ingress-nginx:
  controller:
    replicaCount: 1
    service:
      type: LoadBalancer

# enable integrated SMTP server (requirement #4).
# configure ingress to forward requests for any host to mailhog GUI (requirements #9).
# since ingress for IOM defined a more specific rule, mailhog GUI can be reached using any hostname except localhost.
# since integrated NGINX controller should be used, its class has to be set explicitly.
mailhog:
  enabled: true
  ingress:
    enabled: true
    className: nginx-iom
    hosts:
      - host:
        paths:
          - path: "/"
            pathType: Prefix

# create docker volume "iom-pgdata"
docker volume create —name=iom-pgdata -d local

# get mount-point of newly created docker volume
# use mount-point as value for helm-parameter postgres.persistence.hostPath
docker volume inspect —format='{{.Mountpoint}}' iom-pgdata
/var/lib/docker/volumes/iom-pgdata/_data

# to remove docker volume, execute the following command
docker volume rm iom-pgdata

Installation of IOM

Create a file values.yaml,and fill it with the content listed above. Adapt the settings of persistence.hostPath and postgres.persistence.hostPath to point to directories on your computer, which is shared with Docker Desktop. After that, the installation process of IOM can be started.

# create namespace "iom"
kubectl create namespace iom

# install IOM into namespace "iom"
helm install demo intershop/iom --values=values.yaml --namespace iom --timeout 20m0s --wait

This installation process will now take some minutes to finish. In the meantime, the progress of the installation process can be observed within a second terminal window. Using kubectl you can see the status of every Kubernetes object. For simplicity, the following example is showing the status of pods only.

Just open a second terminal window and enter the following commands.

# A few seconds after start of IOM, only the integrated Postgres server is in "Init" phase. All other
# pods are in earlier phases.
kubectl get pods -n iom
NAME                                                  READY   STATUS              RESTARTS   AGE
demo-iom-0                                            0/1     Pending             0          2s
demo-mailhog-5dd4565b98-jphkm                         0/1     ContainerCreating   0          2s
demo-ingress-nginx-controller-f5bf56d64-cp9b5         0/1     ContainerCreating   0          2s
demo-postgres-7b796887fb-j4hdr                        0/1     Init:0/1            0          2s

# After some seconds all pods except IOM are "Running" and READY (integrated Postgresql server, integrated 
# SMTP server, intergrated NGINX). IOM is in Init-phase, which means the init-containers are currently executed.
kubectl get pods -n iom
NAME                                                  READY   STATUS     RESTARTS   AGE
demo-iom-0                                            0/1     Init:1/3   0          38s
demo-mailhog-5dd4565b98-jphkm                         1/1     Running    0          38s
demo-ingress-nginx-controller-f5bf56d64-cp9b5         1/1     Running    0          38s
demo-postgres-7b796887fb-j4hdr                        1/1     Running    0          38s

# The first init-container executed in iom-pod is dbaccount. Log messages can be seen
# by executing the following command. If everything works well, the last message will announce the
# successful execution of create_dbaccount.sh script.
kubectl logs demo-iom-0 -n iom -f -c dbaccount
...
{"tenant":"company-name","environment":"system-name","logHost":"demo-iom-0","logVersion":"1.0","appName":"iom-dbaccount","appVersion":"1.3.0.0","logType":"script","timestamp":"2020-08-06T11:33:17+00:00","level":"INFO","processName":"create_dbaccount.sh","message":"success","configName":null}

# The second init-container executed by iom-pod is config, which fills the database and applies 
# migrations and configurations. The last message of config container will announce successful execution
# of load_dbmigrate.sh script.
kubectl logs demo-iom-0 -n iom -f -c config
...
{"tenant":"company-name","environment":"system-name","logHost":"demo-iom-0","logVersion":"1.0","appName":"iom-config","appVersion":"3.6.0.0","logType":"script","timestamp":"2020-08-06T11:35:51+00:00","level":"INFO","processName":"load_dbmigrate.sh","message":"success","configName":"env-name"}

# If init-containers have finished successfully, the iom-pod is now in "Running" state, too. But it is not "READY"
# yet. Now the IOM applications and project customizations are deployed into the Wildfly application server.
kubectl get pods -n iom
NAME                                                  READY   STATUS    RESTARTS   AGE
demo-iom-0                                            0/1     Running   0          3m50s
demo-mailhog-5dd4565b98-jphkm                         1/1     Running   0          3m50s
demo-ingress-nginx-controller-f5bf56d64-cp9b5         1/1     Running   0          3m50s
demo-postgres-7b796887fb-j4hdr                        1/1     Running   0          3m50s

# If all pods are "Running" and "READY" the installation process of IOM is finished.
kubectl get pods -n iom
NAME                                                  READY   STATUS    RESTARTS   AGE
demo-iom-0                                            1/1     Running   0          7m20s
demo-mailhog-5dd4565b98-jphkm                         1/1     Running   0          7m20s
demo-ingress-nginx-controller-f5bf56d64-cp9b5         1/1     Running   0          7m20s
demo-postgres-7b796887fb-j4hdr                        1/1     Running   0          7m20s

If all pods are Running and Ready, the installation process is finished. You should check the first terminal window, where the installation process was running.

Now we can access the web GUI of the new IOM installation. In fact, there are two Web GUIs, one for IOM and one for Mailhog. According to our configuration, all requests dedicated to localhost will be forwarded to the IOM application server, any other requests are meant for an integrated SMTP server (Mailhog). Just open the URL https://localhost/omt in a web browser on your Mac. After accepting the self-signed certificate (the configuration did not include a valid certificate), you will see the login page of IOM. Login as admin/!InterShop00! to proceed.

Any other request that is not dedicated to localhost will be forwarded to Mailhog. To access the web-GUI of Mailhog, just open the URL https://127.0.0.1/ in your web browser. Once again you have to accept the self-signed certificate and after that, you will see the MailhogGUI.

Upgrade IOM

From a Helm perspective, the rollout of any change in values or charts is an upgrade process. The process is identical, no matter if only a simple value is changed or new docker images of a new IOM release are rolled out. The example shown here will demonstrate how to change the log-level of the Quartz subsystem, running in the WildFly application server.

Before the start, keep the restrictions on upgrade in mind. A change of a log-level is an uncritical change that can be applied without downtime. But we have decided to use a single IOM application server only (see Requirement #8). When using a single IOM application server only, an upgrade process with downtime is inevitable. Hence, we do not have to think about the setting of parameter downtime.

1. Modify values.yaml by adding the following lines to the file:

   change values.yaml

   log:
     level:
       quartz: INFO
   

   These changes are now rolled out by running Helm's upgrade process to the existing IOM installation.

2. Start the upgrade process within a terminal window.

   upgrade IOM

   helm upgrade demo intershop/iom --values=values.yaml --namespace iom --timeout 20m0s --wait
   

   The upgrade process will take some minutes before finished. 

3. Enter the following commands in a second terminal window to watch the progress.
   As we did it for the installation process before, this example is restricted to the status of pods only.

   observe progress

   # Only the Kubernetes object of IOM has changed. Therefore Helm only upgrades IOM, the integrated SMTP server,
   # integrated postgresql server and integrated NGINX are running unchanged. A few seconds after starting the
   # upgrade process, the only existing iom-pod is stopped.
   kubectl get pods -n iom
   NAME                                                  READY   STATUS        RESTARTS   AGE
   demo-iom-0                                            1/1     Terminating   0          40m
   demo-mailhog-5dd4565b98-jphkm                         1/1     Running       0          40m
   demo-ingress-nginx-controller-f5bf56d64-cp9b5         1/1     Running       0          40m
   demo-postgres-7b796887fb-j4hdr                        1/1     Running       0          40m
   
   # After the iom-pod is terminated, a new iom-pod is started with new configuration. Init containers are 
   # partially executed again.
   kubectl get pods -n iom
   NAME                                                  READY   STATUS     RESTARTS   AGE
   demo-iom-0                                            0/1     Init:2/3   0          6s
   demo-mailhog-5dd4565b98-jphkm                         1/1     Running    0          41m
   demo-ingress-nginx-controller-f5bf56d64-cp9b5         1/1     Running    0          41m
   demo-postgres-7b796887fb-j4hdr                        1/1     Running    0          41m
   
   # Finally the pod is "Running" and "READY" again, which means, IOM is up again.
   kubectl get pods -n iom
   NAME                                                  READY   STATUS    RESTARTS   AGE
   demo-iom-0                                            1/1     Running   0          5m4s
   demo-mailhog-5dd4565b98-jphkm                         1/1     Running   0          46m
   demo-ingress-nginx-controller-f5bf56d64-cp9b5         1/1     Running   0          46m
   demo-postgres-7b796887fb-j4hdr                        1/1     Running   0          46m

Uninstall IOM

The last process demonstrates how to uninstall IOM.

helm uninstall demo -n iom
release "demo" uninstalled

kubectl delete namespace iom
namespace "iom" deleted

Since database data and shared file system of IOM were stored in local directories of the current host, they still exist after uninstalling IOM. In fact, this data represents the complete state of IOM. If we would install IOM again, with the same directories for shared file system and database data, the old IOM installation would be reincarnated.

CI System Running in Minikube on Virtualized Linux

Preconditions

• Linux VM, capable to run Minikube (e.g. CentOS 7)
• Sufficient hardware resources: >= 24 GB main memory, multiple CPU cores
• Installation of Docker, >= v.19.03
  • see: https://docs.docker.com/install/linux/docker-ce/centos/
• Installation of Minikube, >= v.1.12
  • see: https://kubernetes.io/de/docs/tasks/tools/install-minikube/
• Installation of kubectl, >=v.1.18
  • see: https://kubernetes.io/docs/tasks/tools/install-kubectl/#install-kubectl-on-linux
• Installation of Helm, >= v.3.2
  • see: https://helm.sh/docs/intro/install/
• Access to IOM docker images
• Access to IOM Helm Charts

Docker, Minikube, kubectl, and Helm are external tools. This document does not cover how to install and use these tools.

If there are any traps or pitfalls you need to know, they will be explained here.

Minikube Traps, Pitfalls, Restrictions

Minikube provides an easy way to set up a local Kubernetes cluster. Local means that the whole cluster is running on a single machine only. Therefore, it is a good playground for developers or for the setup of the demo- and CI-installations, but of course not for any kind of serious service.

Running Minikube in an already virtualized environment raises the bar a bit higher. For such type of environment, originally the none driver was recommended (Minikube Documentation | Drivers | none), which was at the time of writing replaced by docker driver (Minikube Documentation | Drivers | docker). Therefore, the docker driver was chosen for the following examples.

Access from Public Network

Minikube supports access to applications through services of type LoadBalancer. However, a service of this type cannot be directly accessed from outside the Minikube Cluster. To do so, the execution of an additional command is required (minikube tunnel), see: Minikube Documentation | Handbook | Accessing apps | LoadBalancer access. As long as minikube tunnel is not running, the external IP of the LoadBalancer service remains in the state pending. This has an impact on the installation process of IOM. The installation process of IOM has to be started with command line argument --wait (see parameter downtime). But when using --wait, the IOM installation will not finish until the external IP is available, too, which means that the installation process will run into a timeout.

Thus, for this example, another way was chosen to make IOM and Mailhog-GUI accessible from the public network. Instead of using the combination of service of type LoadBalancer and minikube tunnel to provide access to services, service type ClusterIP will be used and after IOM has started, kubectl port-forward will enable access from outside. 

Finally, to get access to the port providing web-GUIs of IOM and Mailhog, you have to configure firewalld.

Persistent Data

Minikube supports persistent data in general but has restrictions for some drivers, e.g. for docker driver, which is used in our example: Minikube Documentation | Handbook | Persistent Volumes. When using a docker driver, we could use the mount command to get access to a local directory within Minikube. This requires the '9P' file system kernel extension, which is not available on all systems and is missing on our test systems, too.

In summary, this means: for our example, we can use persistent data, but we cannot access the according directories from the host directly. Instead, these persistent volumes are hidden somewhere within the internal Docker/Minikube data structures. According to our requirements, this is fully sufficient.

Requirements and Characteristics of IOM Installation

Requirements and characteristics are numbered again. You will find these numbers in the values-file, which is listed below, in order to see the relation between requirement and current configuration.

1. At least two IOM application servers must run in parallel to test a distributed system.
2. Usage of integrated PostgreSQL server.
3. PostgreSQL data are stored persistently in safe main memory.
4. No reset of PostgreSQL data during the installation process. Due to limitations of persistent data, a reset of PostgreSQL data during the installation process would work, but would not be sufficient, since the shared file system of IOM cannot be reset as easy. To delete all persistent data between CI runs, the Minikube cluster has been deleted.
5. Usage of an integrated SMTP server (Mailhog).
6. The shared file-system of IOM must be stored persistently. Required if more than one IOM application server is running.
7. Usage of integrated NGINX controller for direct access to GUIs of IOM and Mailhog.
8. At least two instances of integrated NGINX controller should run.
9. Access from other computers to IOM is required.
10. The system should be able to be upgraded without downtime.

Execution of Tests

The requirements listed above do not show any traces of test executions. That is contradictory to the goals of CI systems in general. IOM Helm charts support the execution of tests, but Intershop does not deliver any ready-to-run test images. It is up to you to develop images for testing your specific IOM project.

In order to enable you to reproduce the installation of the CI system, the execution of tests was skipped in the example. Nevertheless, this section will give you a short overview of how the integration of tests could look like.

For the execution of tests, the following things and information are required:

• A docker image that executes the tests and that collects and provides the results. The easiest way to access the results would be to provide them via HTTP.
• The docker image used for the test may need the following information for test execution:
  • Base URL of IOM to be able to send HTTP requests.
  • Database connection information to prepare test data for customizations.
  • Base URL of Mailhog to check the results of mail-tests.

Since test-images have to be provided in the project context, the integration into IOM Helm charts is very generic. IOM Helm charts simply provide a default deployment created by Helm. 

Please be aware that the following extract of a values-file is purely fictitious. Names of docker images and names of environment variables fully depend on the project-specific docker images. Resource usage is realistic, it is oriented to our own tests, using Geb tests in combination with Firefox.

iom-tests:
  enabled: true
  imagePullSecrets:
    - name: intershop-pull-secret
  image:
    repository: mycompany/iom-tests
    tag: 1.0.0.0
    pullPolicy: Always
  env:
    # name of service of integrated NGINX controller
    - name: IOM_HOST
      value: ci-ingress-iom-ingress-nginx-controller
    # name of service of integrated PostgreSQL server
    - name: DB_HOST
      value: ci-postgres
    - name: DB_PORT
      value: '5432'
    - name: OMS_DB_NAME
      value: oms_db
    - name: OMS_DB_USER
      value: oms_user
    - name: OMS_DB_PASSWD
      value: OmsDB
    # name of service of integrated mailhog server
    - name: MAILHOG_HOST
      value: ci-mailhog
    - name: MAILHOG_PORT
      value: 1025
  containerPort: 8080
  livenessProbe:
    httpGet:
      path: /
      port: http
  readinessProbe:
    httpGet:
      path: /
      port: http
  resources:
    limits:
      cpu: 2
      memory: 11000Mi
    requests:
      cpu: 2
      memory: 11000Mi
  ingress:
    enabled: true
    annotations:
      nginx.ingress.kubernetes.io/ssl-redirect: "false"
    hosts:
      # hostname to be used to get access to test results
      - host: tests.iomci.com
        paths:
          - path: /
            pathType: Prefix

Values File

This values file cannot be copied as it is. Before it can be used, all occurrences of jdevoms11.rnd.j.intershop.de have to be replaced with the hostname of your CI-system.

# start 2 IOM application servers (requirement #1)
replicaCount: 2

# run upgrade processes without downtime (requirement #10)
downtime: false

imagePullSecrets:
  - intershop-pull-secret

image:
  repository: "docker.intershop.de/intershophub/iom-app"
  tag: "3.6.0.0"

# configure ingress to forward requests to IOM, which were sent to jdevoms11.rnd.j.intershop.de (requirement #9).
# since integrated NGNIX controller should be used, its class has to be specified (requirement #7)
ingress:
  enabled: true
  className: nginx-iom
  hosts:
    - host: jdevoms11.rnd.j.intershop.de
      paths: 
        - path: /
          pathType: Prefix

# IOM has to know its own public URL
oms:
  publicUrl: "https://jdevoms11.rnd.j.intershop.de:8443/"

# store data of shared file system into local directory (requirement #6)
persistence:
  hostPath: /mnt/share

config:
  image:
    repository: "docker.intershop.de/intershophub/iom-config"
    tag: "3.6.0.0"

# create IOM database and according user before starting IOM
dbaccount:
  enabled: true
  image:
    repository: docker.intershop.de/intershophub/iom-dbaccount
    tag: "1.3.0.0"

# use integrated PostgreSQL server
# store database data persistently into local directory (requirements #2, #3)
postgres:
  enabled: true
  persistence:
    enabled: true
    hostPath: /mnt/pgdata

# enable integrated NGINX ingress controller
# this controller should not act proxy (requirement #7)
nginx:
  enabled: true
  proxy:
    enabled: false

# enable integrated SMTP server
# allow access to Web-GUI of mailhog. all requests should be sent to Web-GUI of mailhog,
# unless a more specific rule exists. (requirement #5)
# since integrated NGNIX controller should be used, its class has to be specified (requirement #7).
mailhog:
  enabled: true
  ingress:
    enabled: true
    className: nginx-iom
    hosts:
      - host:
        paths:
          - path: /
            pathType: Prefix

Installation of IOM

According to requirement #4, the Minikube cluster has to be deleted after each test run. For this reason, the creation of the Minikube cluster was added to the current section, too.

# Minikube, using vm-driver "docker", must run in user space.
# Hence, an according user has to be created.
sudo useradd -m -U oms

# Get permission to access docker daemon socket
sudo usermod -aG docker oms && newgrp docker
 
# Change user.
# All commands, also from following sections, have to be executed by user oms!
su - oms
 
# Start minikube as oms user.
minikube start --vm-driver=docker

Create a file values.yaml and fill it with the content listed above. Adapt the hostname in values.yaml.

Once the Minikube cluster is running and the values file is prepared, the installation of IOM can be started.

# create namespace iom
kubectl create namespace iom

# install IOM into namespace iom
helm install ci intershop/iom --values=values.yaml --namespace iom --timeout 20m0s --wait

# make port 443 of integrated NGNIX controller available on all interfaces at port 8443
kubectl port-forward service/ci-ingress-nginx-controller 8443:443 -n iom --address 0.0.0.0

This installation process will now take some minutes to finish. In the meantime, the progress of the installation process can be observed within a second terminal window. Using kubectl you can see the status of every Kubernetes object. But for simplicity, the following example shows the status of pods only.

Just open a second terminal window and enter the following commands:

# A few seconds after starting IOM, only the integrated postgresql is in "Init" phase. All other
# pods are in earlier phases.
kubectl get pods -n iom
NAME                                                READY   STATUS              RESTARTS   AGE
ci-iom-0                                            0/1     Pending             0          3s
ci-mailhog-5744886bb8-x78l7                         0/1     ContainerCreating   0          3s
ci-ingress-nginx-controller-749c5f8f4-7fk6j         0/1     ContainerCreating   0          3s
ci-ingress-nginx-controller-749c5f8f4-d4htz         0/1     ContainerCreating   0          3s
ci-postgres-575976b886-ph6s4                        0/1     Init:0/1            0          3s

# A few seconds later, the IOM pod is in "Init" phase as well, which means the init-containers are
# currently executed.
kubectl get pods -n iom
NAME                                                READY   STATUS              RESTARTS   AGE
ci-iom-0                                            0/1     Init:0/3            0          16s
ci-mailhog-5744886bb8-x78l7                         1/1     Running             0          16s
ci-ingress-nginx-controller-749c5f8f4-7fk6j         0/1     ContainerCreating   0          16s
ci-ingress-nginx-controller-749c5f8f4-d4htz         0/1     ContainerCreating   0          16s
ci-postgres-575976b886-ph6s4                        0/1     PodInitializing     0          16s

# The first init-container executed in iom-pod is dbaccount. Log messages of this init-container can be seen
# by executing the following command. If everything works well, the last message will announce the successful execution
# of create_dbaccount.sh script.
kubectl logs ci-iom-0 -c dbaccount -f -n iom
...
{"tenant":"company-name","environment":"system-name","logHost":"ci-iom-0","logVersion":"1.0","appName":"iom-dbaccount","appVersion":"1.3.0.0","logType":"script","timestamp":"2020-08-07T10:41:15+00:00","level":"INFO","processName":"create_dbaccount.sh","message":"success","configName":null}

# The second init-container executed by iom pod is config, which is filling the database, applying
# migrations and configurations. The last message of config container will announce successful execution
# of load_dbmigrate.sh script.
kubectl logs ci-iom-0 -c config -f -n iom
...
{"tenant":"company-name","environment":"system-name","logHost":"ci-iom-0","logVersion":"1.0","appName":"iom-config","appVersion":"3.6.0.0","logType":"script","timestamp":"2020-08-07T10:42:03+00:00","level":"INFO","processName":"load_dbmigrate.sh","message":"success","configName":"env-name"}

# When init-containers have finished successfully, the iom-pod is now in "Running" state too. But it is not "READY"
# yet. Now the IOM applications and project customizations are deployed into the Wildfly application server.
kubectl get pods -n iom
NAME                                                READY   STATUS    RESTARTS   AGE
ci-iom-0                                            0/1     Running   0          2m26s
ci-mailhog-5744886bb8-x78l7                         1/1     Running   0          2m26s
ci-ingress-nginx-controller-749c5f8f4-7fk6j         1/1     Running   0          2m26s
ci-ingress-nginx-controller-749c5f8f4-d4htz         1/1     Running   0          2m26s
ci-postgres-575976b886-ph6s4                        1/1     Running   0          2m26s

# When the first iom-pod is "Running" and "READY", the second IOM pod will be started. From
# now on, the IOM is accessible from outside.
kubectl get pods -n iom
NAME                                                READY   STATUS     RESTARTS   AGE
ci-iom-0                                            1/1     Running    0          4m52s
ci-iom-1                                            0/1     Init:0/3   0          2s
ci-mailhog-5744886bb8-x78l7                         1/1     Running    0          4m52s
ci-ingress-nginx-controller-749c5f8f4-7fk6j         1/1     Running    0          4m52s
ci-ingress-nginx-controller-749c5f8f4-d4htz         1/1     Running    0          4m52s
ci-postgres-575976b886-ph6s4                        1/1     Running    0          4m52s

# When all pods are "Running" and "READY", the installation process of IOM has finished.
kubectl get pods -n iom
NAME                                                READY   STATUS    RESTARTS   AGE
ci-iom-0                                            1/1     Running   0          6m15s
ci-iom-1                                            1/1     Running   0          85s
ci-mailhog-5744886bb8-x78l7                         1/1     Running   0          6m15s
ci-ingress-nginx-controller-749c5f8f4-7fk6j         1/1     Running   0          6m15s
ci-ingress-nginx-controller-749c5f8f4-d4htz         1/1     Running   0          6m15s
ci-postgres-575976b886-ph6s4                        1/1     Running   0          6m15s

When all pods are Running and Ready, the installation process has finished. You should check the first terminal window, where the installation process was running. As the last step of the IOM installation, the port-forwarding of NGINX service has to be done.

Now we can access the web GUI of the new IOM installation. As already shown in the first example, there are two web-GUIs, one of IOM and one of Mailhog. According to the configuration, all requests dedicated to jdevoms11.rnd.j.intershop.de (please replace it with the hostname of your CI system) will be forwarded to the IOM application server. Any other requests are meant for the integrated SMTP server (Mailhog). Just open the URL https://jdevoms11.rnd.j.intershop.de:8443/omt in a web browser. After accepting the self-signed certificate (the configuration did not include a valid certificate), you will see the login page of IOM. Login as admin/!InterShop00! to proceed.

Any other request that is not dedicated to jdevoms11.rnd.j.intershop.de, will be forwarded to Mailhog. To access the web-GUI of Mailhog, just use the IP instead of the hostname. Open URL https://10.0.29.69:8443/ (please replace it with the IP of your CI system) in your web browser. Once again you have to accept the self-signed certificate and after that, you will see the Mailhog-GUI.

Upgrade of IOM

Now we repeat the upgrade process which was already shown in the previous example. This simple example was chosen since from a Helm perspective, the rollout of any change in values or charts is an upgrade process. The process is identical, no matter if only a simple value is changed or new docker images of a new IOM release are rolled out. 

This example includes setting the downtime parameter (see: Restrictions on upgrade). A change of log-level options is an uncritical change that can be applied without downtime. Since there is more than one IOM application server now, the upgrade process can be executed without downtime.

For upgrading IOM, values.yaml has to be changed. Just add the following lines to the file:

log:
  level:
    quartz: INFO

These changes are now being rolled out by running the Helms upgrade process to the existing IOM installation. Start the process within a terminal window.

helm upgrade ci intershop/iom --values=values.yaml --namespace iom --timeout 20m0s --wait

The upgrade process will take some minutes before it is finished. In the meantime, we can watch the progress. As we did it in the installation process before, this example is restricted to the status of pods only. Just enter the following commands in a second terminal window.

# Right away after starting the upgrade process, the first iom-pod is terminated.
kubectl get pods -n iom
NAME                                                READY   STATUS        RESTARTS   AGE
ci-iom-0                                            1/1     Running       0          12m
ci-iom-1                                            1/1     Terminating   0          7m44s
ci-mailhog-5744886bb8-x78l7                         1/1     Running       0          12m
ci-ingress-nginx-controller-749c5f8f4-7fk6j         1/1     Running       0          12m
ci-ingress-nginx-controller-749c5f8f4-d4htz         1/1     Running       0          12m
ci-postgres-575976b886-ph6s4                        1/1     Running       0          12m

# When this pod has finished, it will be started again, now using the new configuration.
# Initialization is mostly identical to the install process.
kubectl get pods -n iom
NAME                                                READY   STATUS     RESTARTS   AGE
ci-iom-0                                            1/1     Running    0          13m
ci-iom-1                                            0/1     Init:0/3   0          2s
ci-mailhog-5744886bb8-x78l7                         1/1     Running    0          13m
ci-ingress-nginx-controller-749c5f8f4-7fk6j         1/1     Running    0          13m
ci-ingress-nginx-controller-749c5f8f4-d4htz         1/1     Running    0          13m
ci-postgres-575976b886-ph6s4                        1/1     Running    0          13m

# When initialization has finished, deployment of IOM- and customization-apps is executed.
# During this time, the pod is "Running" but not "READY".
kubectl get pods -n iom
NAME                                                READY   STATUS    RESTARTS   AGE
ci-iom-0                                            1/1     Running   0          13m
ci-iom-1                                            0/1     Running   0          15s
ci-mailhog-5744886bb8-x78l7                         1/1     Running   0          13m
ci-ingress-nginx-controller-749c5f8f4-7fk6j         1/1     Running   0          13m
ci-ingress-nginx-controller-749c5f8f4-d4htz         1/1     Running   0          13m
ci-postgres-575976b886-ph6s4                        1/1     Running   0          13m

# When the pod is "Running" and "READY", it is able to handle incoming requests. The other
# iom-pod can now be terminated.
kubectl get pods -n iom
NAME                                                READY   STATUS        RESTARTS   AGE
ci-iom-0                                            1/1     Terminating   0          14m
ci-iom-1                                            1/1     Running       0          88s
ci-mailhog-5744886bb8-x78l7                         1/1     Running       0          14m
ci-ingress-nginx-controller-749c5f8f4-7fk6j         1/1     Running       0          14m
ci-ingress-nginx-controller-749c5f8f4-d4htz         1/1     Running       0          14m
ci-postgres-575976b886-ph6s4                        1/1     Running       0          14m

# After termination, it is started again with new configuration. Init containers are
# mostly executed as during installation process.
kubectl get pods -n iom
NAME                                                READY   STATUS     RESTARTS   AGE
ci-iom-0                                            0/1     Init:0/3   0          0s
ci-iom-1                                            1/1     Running    0          119s
ci-mailhog-5744886bb8-x78l7                         1/1     Running    0          15m
ci-ingress-nginx-controller-749c5f8f4-7fk6j         1/1     Running    0          15m
ci-ingress-nginx-controller-749c5f8f4-d4htz         1/1     Running    0          15m
ci-postgres-575976b886-ph6s4                        1/1     Running    0          15m

# IOM- and customization-apps are deployed.
kubectl get pods -n iom
NAME                                                READY   STATUS    RESTARTS   AGE
ci-iom-0                                            0/1     Running   0          13s
ci-iom-1                                            1/1     Running   0          2m12s
ci-mailhog-5744886bb8-x78l7                         1/1     Running   0          15m
ci-ingress-nginx-controller-749c5f8f4-7fk6j         1/1     Running   0          15m
ci-ingress-nginx-controller-749c5f8f4-d4htz         1/1     Running   0          15m
ci-postgres-575976b886-ph6s4                        1/1     Running   0          15m

# Both IOM pods are "Running" and "READY" again, the upgrade process is finished now.
kubectl get pods -n iom
NAME                                                READY   STATUS    RESTARTS   AGE
ci-iom-0                                            1/1     Running   0          2m38s
ci-iom-1                                            1/1     Running   0          4m37s
ci-mailhog-5744886bb8-x78l7                         1/1     Running   0          17m
ci-ingress-nginx-controller-749c5f8f4-7fk6j         1/1     Running   0          17m
ci-ingress-nginx-controller-749c5f8f4-d4htz         1/1     Running   0          17m
ci-postgres-575976b886-ph6s4                        1/1     Running   0          17m

Uninstall IOM

The last process demonstrates how to uninstall IOM. In order to get rid of persistent data, too (see requirement #4), the whole Minikube cluster has to be deleted. In fact, it would be sufficient to only delete the Minikube cluster, but for completeness, the other commands are listed as well.

# uninstall IOM release
helm uninstall ci -n iom
release "ci" uninstalled
 
# delete Kubernetes namespace used for IOM
kubectl delete namespace iom
namespace "iom" deleted

# delete whole Minikube cluster
minikube delete

Production System Running in Azure Cloud

Preconditions

Please keep in mind that these preconditions reflect the use case described in section IOM Helm-charts. When using Intershop Commerce Platform, these preconditions are all covered by Intershop.

• Azure Kubernetes Service (AKS) and virtual machines of sufficient size. 
• Access to a PostgreSQL server preferred as "Azure Database for PostgreSQL servers".
• Access to a File service (e.g. Azure Files)
• Running Ingress controller in AKS
• Secrets for database access, tls for ingress, etc.
• Installation of kubectl on the machine, that is used for installation, >=v.1.18
  • see: https://kubernetes.io/docs/tasks/tools/install-kubectl/#install-kubectl-on-linux
• Installation of Helm on the machine, that is used for installation, >= v.3.2
  • see: https://helm.sh/docs/intro/install/
• Access to AKS from the machine used for installation
• Access to IOM docker images
• Access to IOM Helm-charts

Requirements and Characteristics of IOM Installation

Requirements and characteristics are numbered again. You will find these numbers in the values-file listed below in order to see the relation between requirement and current configuration.

1. Two IOM application servers must run in parallel.
2. Usage of an external PostgreSQL server (Azure Database for PostgreSQL server).
3. No reset of PostgreSQL data during the installation process. 
4. Usage of an external SMTP server.
5. Shared file system of IOM located on externally provided resources.
6. Usage of an external Ingress controller, which is not based on NGINX. In this case, the integrated NGINX ingress controller has to act as a proxy, which is providing load-balancing and sticky sessions.
7. Two instances of integrated NGINX controller must run.
8. The system should be able to be upgraded without downtime.

Values File

The values file shown below reflects the requirements of the straight Helm approach, as described in section IOM Helm-charts. When running on Intershop Commerce Platform, most settings are made by Intershop Operations within the second values file. On the customer's side, only few parameters remain that have to be set. These are downtime, image, config, and oms.smtp and maybe some other parameters, depending on the type of installation.

Of course, this values file cannot be copied as it is. It references external resources and external services, which do not exist in your environment, additionally the hostname iom.mycompany.com does also not match your requirements.

# start 2 IOM application servers (requirement #1)
replicaCount: 2

# run upgrade processes without downtime (requirement #8)
downtime: false

imagePullSecrets:
  - intershop-pull-secret

image:
  repository: "docker.intershop.de/intershophub/iom-app"
  tag: "3.6.0.0"

# configure ingress to forward requests to IOM, which are sent to 
# to host iom.mycompany.com. (requirement #6)
ingress:
  enabled: true
  hosts:
    - host: iom.mycompany.com
      paths: 
        - path: /
          pathType: Prefix
  tls:
    - secretName: mycompany-com-tls
      hosts:
        - iom.mycompany.com

# information about external postgresql service (requirement #2)
pg:
  host: postgres-prod.postgres.database.azure.com
  port: 5432
  userConnectionSuffix: "@postgres-prod"

  # root-database and superuser information. The very first installation initializes
  # the database of IOM. After that, these information should be removed from values
  # file completely (and dbaccount should be disabled/removed too)
  user: postgres
  passwdSecretKeyRef:
    name: mycompany-prod-secrets
    key: pgpasswd
  db: postgres

# IOM has to know its own public URL
oms:
  publicUrl: "https://iom.mycompany.com/"
  db:
    name: oms_db
    user: oms_user
    passwdSecretKeyRef:
      name: mycompany-prod-secrets
      key: dbpasswd
  # configuration of external smtp server (requirement #4)
  smtp:
    host: smpt.external-provider.com
    port: 25
    user: my-company-prod
    passwdSecretKeyRef:
      name: mycompany-prod-secrets
      key: smtppasswd

log:
  metadata:
    tenant: mycompany
    environment: prod

caas:
  envName: prod

resources:
  limits:
    cpu: 1000m
    memory: 3000Mi
  requests:
    cpu: 1000m
    memory: 3000Mi

# store data of shared file system at azurefile service (requirement #5)
persistence:
  storageClass: azurefile
  storageSize: 60G

config:
  image:
    repository: "docker.intershop.de/intershophub/iom-config"
    tag: "3.6.0.0"

# Create IOM database and according user before starting IOM. Creates IOM database
# while running install process. After that, dbaccount should be completely removed
# from the values file. Without set explicitly, data are not reset during start
# (requirement #3).
dbaccount:
  enabled: true
  image:
    repository: docker.intershop.de/intershophub/iom-dbaccount
    tag: "1.3.0.0"

# enable integrated NGINX ingress controller. Without any further configuration, 
# it acts as a proxy (requirement #6).
nginx:
  enabled: true

Installation of IOM

Create a file values.yaml and fill it with the content listed above. Adapt all the changes to the file that are required by your environment. After that, the installation process can be started.

# create namespace mycompany-iom
kubectl create namespace mycompany-iom
 
# install IOM into namespace mycompany-iom
helm install ci intershop/iom --values=values.yaml --namespace mycompany-iom --timeout 20m0s --wait

This installation process will now take some minutes to finish. In the meantime, the progress of the installation process can be observed within a second terminal window. Using kubectl, you can see the status of every Kubernetes object. For simplicity, the following example shows the status of pods only.

Just open a second terminal window and enter the following commands.

# One second after start, all pods are in very early phases.
kubectl get pods -n mycompany-iom
NAME                                                 READY   STATUS              RESTARTS   AGE
prod-iom-0                                           0/1     Pending             0          1s
prod-ingress-nginx-controller-76db7cfc6d-2h4w9       0/1     ContainerCreating   0          1s
prod-ingress-nginx-controller-76db7cfc6d-tzzsl       0/1     ContainerCreating   0          1s

# Little bit later, the integrated NGINX ingress is "Running" and "READY". IOM is in initialization phase,
# which means the init-containers are currently executed.
kubectl get pods -n mycompany-iom
NAME                                                 READY   STATUS     RESTARTS   AGE
prod-iom-0                                           0/1     Init:0/2   0          24s
prod-ingress-nginx-controller-76db7cfc6d-2h4w9       1/1     Running    0          24s
prod-ingress-nginx-controller-76db7cfc6d-tzzsl       1/1     Running    0          24s

# After a few minutes IOM is "Running", but not "READY" yet. The init-containers are finished
# now and the IOM- and project-applications are currently deployed into the Wildfly application server.
kubectl get pods -n mycompany-iom
NAME                                                 READY   STATUS    RESTARTS   AGE
prod-iom-0                                           0/1     Running   0          4m43s
prod-ingress-nginx-controller-76db7cfc6d-2h4w9       1/1     Running   0          4m43s
prod-ingress-nginx-controller-76db7cfc6d-tzzsl       1/1     Running   0          4m43s

# The first iom-pod is "Running" and "READY", which means the IOM System is usable now.
# The second iom-pod has just started and is currently initialized.
kubectl get pods -n mycompany-iom
NAME                                                 READY   STATUS     RESTARTS   AGE
prod-iom-0                                           1/1     Running    0          9m35s
prod-iom-1                                           0/1     Init:0/2   0          2s
prod-ingress-nginx-controller-76db7cfc6d-2h4w9       1/1     Running    0          9m35s
prod-ingress-nginx-controller-76db7cfc6d-tzzsl       1/1     Running    0          9m35s

# Both iom-pods are "Running" and "READY". Installation of IOM is finished.
kubectl get pods -n mycompany-iom
NAME                                                 READY   STATUS    RESTARTS   AGE
prod-iom-0                                           1/1     Running   0          15m
prod-iom-1                                           1/1     Running   0          5m49s
prod-ingress-nginx-controller-76db7cfc6d-2h4w9       1/1     Running   0          15m
prod-ingress-nginx-controller-76db7cfc6d-tzzsl       1/1     Running   0          15m

When all pods are Running and Ready, the installation process has finished. You should check the first terminal window, where the installation process was running.

Upgrade of IOM

Now we repeat the upgrade process, which was already shown in the previous example. This simple example was chosen since from a Helm perspective, the rollout of any change in values or charts is an upgrade process. The process is identical, no matter if only a simple value is changed or if new docker images of a new IOM release are rolled out. 

Also setting the downtime parameter (see: Restrictions on upgrade) is considered. A change of a log-level is an uncritical change which can be applied without downtime. Since we have more than one IOM application server, the upgrade process can now be executed without downtime.

Add the following lines to the values.yaml:

log:
  level:
    quartz: INFO

These changes are now rolled out by running the Helm upgrade process to the existing IOM installation. Start the process within a terminal window.

helm upgrade ci intershop/iom --values=values.yaml --namespace mycompany-iom --timeout 20m0s --wait

The upgrade process will take some minutes before being finished.

In the previous section you might have noticed that the behavior of pods during the installation process is identical no matter which Kubernetes environment was used (Docker Desktop, Minikube). The same applies to the upgrade process. For this reason, the box "Observe progress" will be skipped in the current section.

Uninstall IOM

The last process demonstrates how to uninstall IOM. Please keep in mind that the uninstall process only covers the objects defined in IOM Helm-charts. In the current production example many external resources and external services are referenced. These resources and services remain untouched by the uninstall process of IOM.

# uninstall IOM release
helm uninstall prod -n mycompany-iom
release "prod" uninstalled
  
# delete Kubernetes namespace used for IOM
kubectl delete namespace mycompany-iom
namespace "mycompany-iom" deleted

Parameters

Each example shown in the Examples section before used a values file to define the specific setup of IOM (see: demo, ci, prod). The current section now describes each parameter you have already used before in detail. There are also many more parameters that were not used in the examples.

In the Examples section, you have already learned that IOM Helm-charts also provide optional components: integrated PostgreSQL server, integrated SMTP server, integrated NGNIX controller, and support for the execution of tests. These optional components are covered by separate sections.

IOM

Integrated SMTP Server

A complete list of parameters can be found here: https://github.com/codecentric/helm-charts/tree/master/charts/mailhog

The table below only lists parameters that have to be changed for different operation options of IOM.

Integrated NGINX Ingress Controller

A complete list of parameters can be found here: https://github.com/kubernetes/ingress-nginx/tree/master/charts/ingress-nginx

The table below only lists parameters that have to be changed for different operation options of IOM and also those that must not be changed at all.

Integrated PostgreSQL Server

IOM Tests

The iom-tests sub-chart provides a very generic way to run tests on an IOM installation. The sub-chart and the according parameters are simply the pure skeleton, resulting from a helm create call. The section Execution of tests, which is part of the example CI System running in Minikube on virtualized Linux, demonstrates how this could be used.

Selected Configuration Aspects in Depth

References to Kubernetes Secrets

All parameters ending with SecretKeyRef serve as an alternative way to provide secret information. Instead of storing entries as plain text in values file, these parameters allow reference entries within Kubernetes secrets. For more information about secrets, see public Kubernetes documentation.

SecretKeyRef parameters require a hash structure, consisting of two entries with the following hash-keys:

• name: is the name of the Kubernetes secret, containing the referenced key
• key: is the name of the entry within the secret

The following two boxes show an example, which consists of two parts:

• The definition of the Kubernetes secret, which contains entries for different secret values, and
• The values file, which references these values.

apiVersion: v1
kind: Secret
metadata:
  name: pgsecrets
type: Opaque
data:
  pguser:   cG9zdGdyZXM=
  pgpasswd: ZGJ1c2VycGFzc3dk

...
# general postgres settings, required to connect to postgres server
# and root db.
pg:
  userSecretKeyRef:
    name: pgsecrets
    key:  pguser
  passwdSecretKeyRef:
    name: pgsecrets
    key:  pgpasswd
  db:                 postgres
  sslMode:            prefer
  sslCompression:     "1"
  sslRootCert:
...

Metrics & Monitoring

To improve the observability of IOM in cloud environments, a number of metrics provided by the application server as well as custom metrics are exposed by WildFly in Prometheus format. They can be collected / scraped from the “http://[pod]:9990/metrics” endpoint. From an “Ops” perspective, the most important metrics can be found in the “base” and “application” namespaces - i.e. metrics prefixed by “base_” or “application_”.

Base Metrics

Base metrics exposed by WildFly correspond to the “Required Metrics” as defined by the microprofile-metrics specification. This includes basic JVM parameters such as uptime, thread counts and garbage collector statistics.

Example - Base Metrics

# HELP base_memory_usedHeap_bytes Displays the amount of used memory.
# TYPE base_memory_usedHeap_bytes gauge
base_memory_usedHeap_bytes 1.39456648E9
# HELP base_memory_usedNonHeap_bytes Displays the amount of used memory.
# TYPE base_memory_usedNonHeap_bytes gauge
base_memory_usedNonHeap_bytes 4.90891528E8
# HELP base_thread_count Number of currently deployed threads
# TYPE base_thread_count gauge
base_thread_count 310.0

Application Metrics

Application metrics are “custom” metrics exposed by the IOM platform or project code. The list below includes metrics that can be used to monitor the application/infrastructure in case there are no feasible alternatives for external monitoring.

Example - Application Metrics

# TYPE application_iom_shared_fs_disk_total_bytes gauge
application_iom_shared_fs_disk_total_bytes 8.07992815616E11
# TYPE application_iom_shared_fs_disk_usage_ratio gauge
application_iom_shared_fs_disk_usage_ratio 0.4600122290006161
# TYPE application_iom_shared_fs_disk_used_bytes gauge
application_iom_shared_fs_disk_used_bytes 3.71686576128E11

PostgreSQL Server Configuration

To achieve the best performance, almost all the required data (tables and indexes) for the ongoing workload should be able to reside within the file system cache. Monitoring the I/O activity will help to identify insufficient memory resources.

The IOM is built with Hibernate as an API between the application logic and the database. This mainly results in a strong OLTP activity, with a large number of tiny SQL statements. Larger statements occur during import/export jobs and for some OMT search requests.

The following main parameters in $PGDATA/data/postgresql.conf should be adapted, see PostgreSQL 12 | Chapter 19. Server Configuration.

You can consider PGConfig 2.0 as a guideline (using the OLTP Model).

Some aspects of data reliability are discussed in PostgreSQL 12 | Chapter 29. Reliability and the Write-Ahead Log. Understanding VACUUM is also essential when configuring/monitoring Postgres, see PostgreSQL 12 | Chapter 24. Routine Database Maintenance Tasks.

Options and Requirements of IOM Database

Tablespace

The database initialization made by dbaccount image is creating a user and database, which uses the system-wide default tablespace pg_default. If you want to use a custom tablespace, you have to create it prior to the database initialization, see PostgreSQL: Documentation: 12: CREATE TABLESPACE.

To make the database initialization process aware of this newly created tablespace, the parameter dbaccount.tablespace has to be set to its name. If this is done, this tablespace will be set as default tablespace for the IOM database user and for the IOM database during the initialization process.

Timezone

All database clients and the IOM database have to use the same timezone. For this reason, all IOM Docker images are configured on OS-level to use timezone Etc/UTC. The process, executed by dbaccount init-image, sets this timezone for the IOM database user as well.

Locale/Encoding

The locale of database clients and the locale of the IOM database have to be identical. For this reason, all IOM Docker images are setting environment variable LANG to en_US.utf8.

Accordingly, the setting on the database is made by dbaccount init-image. Using parameter dbaccount.options, it is possible to configure this process.

When creating the IOM database by dbaccount init-image, using the wrong Encoding, Collate or Ctype is the most common reason for failed initialization of the IOM database. The according values have to be exactly identical to the values used by template databases. Hence, if there are any problems with Encoding, Collate, or Ctype when creating the IOM database, the existing databases should be listed to get the correct values. To do so, just use psql database client with parameter -l to list them.

The following box shows how to do this after an initialization error if IOM is running on Docker-Desktop.

# get name of PostgreSQL pod
kubectl get pods -n iom
NAME                                             READY   STATUS       RESTARTS   AGE
demo-ingress-nginx-controller-6c6f5b88cc-6wsfh   1/1     Running      0          67s
demo-iom-0                                       0/1     Init:Error   3          67s
demo-mailhog-5d7677c7c5-zl8gl                    1/1     Running      0          67s
demo-postgres-96676f4b-mt8nl                     1/1     Running      0          67s

# execute psql -U postgres -l within PostgreSQL pod
kubectl exec demo-postgres-96676f4b-mt8nl -n iom -t -- psql -U postgres -l
                                 List of databases
   Name    |  Owner   | Encoding |  Collate   |   Ctype    |   Access privileges
-----------+----------+----------+------------+------------+-----------------------
 postgres  | postgres | UTF8     | en_US.utf8 | en_US.utf8 |
 template0 | postgres | UTF8     | en_US.utf8 | en_US.utf8 | =c/postgres          +
           |          |          |            |            | postgres=CTc/postgres
 template1 | postgres | UTF8     | en_US.utf8 | en_US.utf8 | =c/postgres          +
           |          |          |            |            | postgres=CTc/postgres
(3 rows)

Search-Path

In some circumstances, the search path for database objects has to be extended. Search-Path is set by dbaccount init-image. This process can be configured by parameter dbaccount.searchPath.