Process

The process Process - Update OMS Node 1.0 is a rather heavy weight process, since it completely deletes and reinstalls IOM. In case smaller configuration changes, like changing Java options or logging configuration, etc., need to be applied, reconfigure_oms_node provides a more lightweight process. The drawback of the process reconfigure_oms_node is that it does not cover all configuration changes. For details about the applicable changes in each OMS process see Reference - Ansible4IOM Variables - OMS. Since reconfiguration does not make any changes to the database content, the process might run without forcing a downtime of the whole system, if the process is not executed for all nodes at the same time.

ANSIBLE_LIBRARY=<path to Ansible4IOM>/modules/ \
ANSIBLE_ROLES_PATH=<path to Ansible4IOM>/roles/ \
ansible-playbook -i <path to inventory file>/inventory <path to Ansible4IOM>/processes/reconfigure_oms_node.yml

Steps

The process consists of the following steps:

1. ftpd_cleanupservice
   1. Disable/stop pure-ftpd service in order to avoid successful health-checks during reconfiguration process.
2. oms_cleanupservice
   1. Stop/disable IOM service.
3. oms_reconfig_for_detach_from_cluster (backend server of groups oms_ha_node and oms_azure_node only)
   1. Increment setting of port-offset in installation.properties by one.
   2. Set thread count to 0 in quartz-cluster.properties.
4. (java_install)
5. oms_installation_properties
   1. Update installation.properties file with values found in the inventory.
6. oms_cluster_properties
   1. Update cluster.properties file with values found the in inventory.
7. oms_service
   1. Setup and start OMS service directly (not by watchdog).
8. oms_configuration
   1. Execute pre_oms_configuration_hook.
   2. Apply system.std.*.properties.
   3. Apply JMS load balancing.
   4. Apply cluster.properties.
   5. Execute post_oms_configuration_hook.
9. oms_deploy
   
   1. Execute pre_oms_deploy_hook.
   2. Redeploy all artifacts and wait for Wildfly to get ready again.
   3. Execute post_oms_deploy_hook.
10. oms_reconfig_for_attach_to_cluster (backend server of groups oms_ha_node and oms_azure_node only)
    
    1. Restore original port-offset in installation.properties.
    2. Restore original thread count in quartz-cluster.properties.
11. oms_watchdog
    1. Update watchdog.properties.
    2. Stop OMS service.
    3. Update systemd unit file of watchdog service.
    4. Reload, enable, start watchdog service.
12. ftpd_service
    1. Execute pre_ftpd_start_hook
    2. Start/enable FTPd-service.

Background Information

Reconfiguring the process is intended to change simple configuration settings without reinstallation of IOM. This limits the settings to be changed. Reference - Ansible4IOM Variables 1.0 gives an overview about possible changes. Please see also Process - Setup OMS Node 1.0, to get a general overview about configuration of IOM installations.

Reconfiguration Process - Avoid Side Effects of Parallel Execution of Backend Servers

During the reconfigure process, the load balancer must not forward any requests to the IOM application server currently reconfigured. Since the load balancer uses health-check requests sent to the application servers to determine the application servers to use, the goal can be reached by marking the according node as unhealthy. This is realized by stopping FTPd-service during the reconfiguration process.

During the reconfiguration process, backend servers are not under control of the watchdog. During this time, the currently reconfigured backend server might run in parallel to other backend servers. It has to be ensured that the currently reconfigured backend server must not execute jobs and must not receive and handle JMS messages.

Technically, this behavior is realized by the roles oms_reconfig_for_attach_to_cluster and oms_reconfig_for_detach_from_cluster. When detached from the cluster, the backend server will not run any jobs and will not receive JMS messages. Even if it is running in parallel to other backend servers, the detached backend server will not influence the cluster at all.

Examples

Use Different Java Options for Backend and Frontend Servers

The variable JBOSS_JAVA_OPTS, defined in roles/oms_config/defaults, is a hash. The variable cannot be set in the inventory file directly. The inventory file supports setting of simple variables only. To set complex variables, use files located at directory group_vars. Depending on the installation type that is used, a file within group_vars has to be created. It must be named exactly as the server group assignment: use (oms_single_node, oms_ha_node, oms_azure_node).

The following example shows the necessary settings to change Java options of an Azure Cloud installation of IOM. Frontend and backend server will use different configurations.

JBOSS_JAVA_OPTS: {
  backend: "-Xms512M -Xmx4096M -XX:+UseG1GC -XX:ParallelGCThreads=8 -XX:+UseAdaptiveSizePolicy -XX:TargetSurvivorRatio=40 -XX:+PrintGCDetails -XX:+PrintGCDateStamps",
  frontend: "-Xms512M -Xmx2048M",
}

Now, the process reconfigure_oms_node can be executed. To avoid a downtime of the system, both HA-nodes of the IOM system are updated after each other.

ANSIBLE_LIBRARY=<path to Ansible4IOM>/modules/ \
ANSIBLE_ROLES_PATH=<path to Ansible4IOM>/roles/ \
ansible-playbook -i <path to inventory file>/inventory <path to Ansible4IOM>/processes/reconfigure_oms_node.yml --limit iomha1
 
ANSIBLE_LIBRARY=<path to Ansible4IOM>/modules/ \
ANSIBLE_ROLES_PATH=<path to Ansible4IOM>/roles/ \
ansible-playbook -i <path to inventory file>/inventory <path to Ansible4IOM>/processes/reconfigure_oms_node.yml --limit iomha2

Change is.oms.db.hostlist in cluster.properties

When adding a new PostgreSQL HA-node, all IOM application servers have to be reconfigured, to enable failover to the new node. Therefore the list of DB-hosts in property is.oms.db.hostlist has to be extended and cluster.properties have to be loaded into application servers.

Edit the inventory file and add the new server to variable is_oms_db_hostlist (see roles/oms_config/defaults).

...
[all:vars]

is_oms_db_hostlist="10.10.10.1:5432,10.10.10.2:5432,10.10.10.3:5432"
...

Now the process reconfigure_oms_node can be executed. To avoid a downtime of the system, both HA-nodes of the IOM system are updated after each other.

ANSIBLE_LIBRARY=<path to Ansible4IOM>/modules/ \
ANSIBLE_ROLES_PATH=<path to Ansible4IOM>/roles/ \
ansible-playbook -i <path to inventory file>/inventory <path to Ansible4IOM>/processes/reconfigure_oms_node.yml --limit iomha1
 
ANSIBLE_LIBRARY=<path to Ansible4IOM>/modules/ \
ANSIBLE_ROLES_PATH=<path to Ansible4IOM>/roles/ \
ansible-playbook -i <path to inventory file>/inventory <path to Ansible4IOM>/processes/reconfigure_oms_node.yml --limit iomha2

Change Log-Level of IOM Backend Servers to ERROR

The following property defines the default setting of the BAKERY file-handler:

• /subsystem=logging/periodic-rotating-file-handler=BAKERY: level="WARN"

The property is part of the system.std.backend.properties file. Ansible4IOM does not reflect any property of system.std.*.properties in own variables. Hence, Ansible4IOM does not support an update of system.std.*.properties directly. Instead, custom Ansible code has to be placed to pre_oms_configuration_hook in order to make changes to system.std.*.properties. Also see Process - Setup OMS Node 1.0.

Depending on the scope of the hook (project or installation), the file pre_oms_configuration_hook.yml has to be stored in the directory global_hooks or installation_hooks. Also see Concept - Ansible4IOM Server Configuration Management 1.0 - 1.1.

It is important to use the variables OMS_SERVER_TYPE, OMS_ETC[OMS_SERVER_TYPE] and OMS_USER (see roles/oms_config/defaults) for a robust implementation of the hook. This way, the custom configuration will still work, even if the installation of IOM will be customized far more.

- name: update Log-Level of BAKERY file-handler in system.std.backend.properties
  lineinfile:
    dest: "{{OMS_ETC[OMS_SERVER_TYPE]}}/system.std.{{OMS_SERVER_TYPE}}.properties"
    regexp: '^[ \t]*/subsystem=logging/periodic-rotating-file-handler=BAKERY:'
    line: '/subsystem=logging/periodic-rotating-file-handler=BAKERY: level="ERROR"'
    state: present
  when:
    - OMS_SERVER_TYPE == "backend"
  become: true
  become_user: "{{OMS_USER}}"

Now the process reconfigure_oms_node can be executed. To avoid a downtime of the system, both HA-nodes of the IOM system are updated after each other.

ANSIBLE_LIBRARY=<path to Ansible4IOM>/modules/ \
ANSIBLE_ROLES_PATH=<path to Ansible4IOM>/roles/ \
ansible-playbook -i <path to inventory file>/inventory <path to Ansible4IOM>/processes/reconfigure_oms_node.yml --limit iomha1
 
ANSIBLE_LIBRARY=<path to Ansible4IOM>/modules/ \
ANSIBLE_ROLES_PATH=<path to Ansible4IOM>/roles/ \
ansible-playbook -i <path to inventory file>/inventory <path to Ansible4IOM>/processes/reconfigure_oms_node.yml --limit iomha2