Update and upgrade JBoss EAP with Ansible

In this follow-up to Automate and deploy a JBoss EAP cluster with Ansible, we will explain how to maintain and keep those instances updated, again in a fully automated manner, leveraging Ansible and the Ansible collection for Red Hat JBoss Enterprise Application Platform (EAP).

Indeed, it is critical to ensure that all JEE application server instances always be up to date, especially in regard to security fixes. Therefore, we’ll discuss not only how to apply patches to update the server but also how to perform an upgrade to migrate to a new major version.

Prerequisites

In the previous article, we used the following playbook to install the WildFly cluster using Ansible on one single host. However, in this article, we’ll use only one instance of JBoss EAP (and no longer WildFly) for simplicity's sake.

You can use the following playbook to install an instance of JBoss EAP:

---
- name: Install EAP 7
  hosts: all
  collections:
    - redhat.eap
  roles:
    - eap_install
    - eap_systemd

Note: The playbook above uses the redhat.eap certified collection instead of the upstream middleware_automation.wildfly instance. The previous article used WildFly (the community version of the JEE application server) instead of JBoss EAP (the product supported by Red Hat). However, as updates are rarely produced for the upstream version (because of the fast release cycle, it’s easier to update to the next version), this article will focus on JBoss EAP, as product users often have to manage such updates.
For the playbook above to function as expected, you will need to install the collections redhat.eap on the Ansible controller:

# ansible-galaxy collection install redhat.eap
Starting galaxy collection install process
Process install dependency map
Starting collection install process
Downloading https://console.redhat.com/api/automation-hub/v3/plugin/ansible/content/published/collections/artifacts/redhat-eap-1.3.1.tar.gz to /root/.ansible/tmp/ansible-local-344ezcdnc0/tmpc0c7yq1u/redhat-eap-1.3.1-b5h7g9vf
Downloading https://console.redhat.com/api/automation-hub/v3/plugin/ansible/content/published/collections/artifacts/redhat-redhat_csp_download-1.2.2.tar.gz to /root/.ansible/tmp/ansible-local-344ezcdnc0/tmpc0c7yq1u/redhat-redhat_csp_download-1.2.2-2kmr5p0m
Installing 'redhat.eap:1.3.1' to '/root/.ansible/collections/ansible_collections/redhat/eap'
redhat.eap:1.3.1 was installed successfully
Installing 'redhat.redhat_csp_download:1.2.2' to '/root/.ansible/collections/ansible_collections/redhat/redhat_csp_download'
redhat.redhat_csp_download:1.2.2 was installed successfully

As redhat.eap is a certified collection only available to Red Hat customers, you need to configure the ansible.cfg file to use Ansible Automation Hub so that the collection can be retrieved:

# cat ansible.cfg
[defaults]
host_key_checking = False
retry_files_enabled = False
nocows = 1

[inventory]
# fail more helpfully when the inventory file does not parse (Ansible 2.4+)
unparsed_is_failed=true

[galaxy]
server_list = automation_hub, galaxy

[galaxy_server.galaxy]
url=https://galaxy.ansible.com/

[galaxy_server.automation_hub]
url=https://cloud.redhat.com/api/automation-hub/

auth_url=https://sso.redhat.com/auth/realms/redhat-external/protocol/openid-connect/token

token=<your-token>

A bit of context first

Before we continue, let's define what we mean (in this article) by update versus upgrade.

Updates

An update to an existing JBoss EAP instance consists of the deployment of a series of fixes to the product’s code. Updates are provided to Red Hat customers through the Red Hat Customer Portal. They contain a set of changes made against the JEE server code, either to fix an issue or address a security concern (or both). You need to use the JBoss CLI tool to deploy an update. However, as we’ll see soon, the redhat.eap collection will take care of this for us.

It’s important to note that such an update only brings fixes to the server. No functionality changes (unless required to resolve a problem) nor API changes are performed. Therefore, an update does not change the major version of JBoss EAP—only the minor version. For instance, bringing JBoss EAP 7.4.0 to 7.4.6 (but not to 7.5).

Because an update to a more recent minor version only includes small changes, they rarely require modification to the applications hosted by JBoss EAP. They should be performed as quickly as possible in order to ensure the server can not be compromised by a known security issue.

Upgrades

An upgrade is a more involved operation. Indeed, it comes with API changes and new features, meaning that, before being performed, the applications hosted by JBoss EAP should be tested against the new version and potentially adapted to work in this new context.

Also, an upgrade to JBoss EAP is a change of major version (EAP 7.3 to 7.4). This cannot be achieved by updating the files of the currently installed software. A complete, new installation needs to be performed, and the configuration, along with the hosted apps, needs to be migrated to this new root folder.

Note: This article focuses on the update and upgrade of the app server itself and, purposely, does not discuss configuration changes and application migration. On this front, too, the collection redhat.eap provides some help by leveraging the JBoss migration tool. This scenario will be the topic of a follow-up article.

Applying a cumulative patch

Let’s consider the following scenario:

A series of JBoss EAP 7.3.0 instances were freshly installed in order to perform tests before production. The tests were successful, and the team now wants to promote those servers from testing to preprod. The main requirement (regarding JBoss EAP) is to run the latest version of the server (7.3.10).

Here is the playbook used to fully automate the installation process:

---
- name: "Update EAP to latest {{ eap_patch_version }}"
  hosts: eap_servers
  vars:
    eap_version: 7.3.0
    eap_apply_cp: True
    eap_patch_version: 7.3.10
    eap_instance_name: eap73
  collections:
    - redhat.eap
  roles:
    - eap_install
    - eap_systemd

This playbook relies entirely on the two roles provided by the redhat.eap collection, to install EAP and start the associated service. The only information required is the server (major) version provided by the variable eap_version. We also configured the collection to update this minor version (eap_patch_apply set to true) to the latest available minor version (7.3.10). We also added a variable to change the name of the service running the server to eap73.

Once this playbook has run successfully and Ansible has started the server, we can see that we are now running the latest version available of JBoss EAP 7.3, as proven by the following line of the server.log:

# tail -f /opt/jboss_eap/jboss-eap-7.3/standalone/log/server.log
2023-03-01 11:25:29,413 INFO  [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-6) WFLYJCA0001: Bound data source [java:jboss/datasources/ExampleDS]
2023-03-01 11:25:29,483 INFO  [org.jboss.as.patching] (MSC service thread 1-2) WFLYPAT0050: JBoss EAP cumulative patch ID is: jboss-eap-7.3.10.CP, one-off patches include: none
2023-03-01 11:25:29,494 WARN  [org.jboss.as.domain.management.security] (MSC service thread 1-2) WFLYDM0111: Keystore /opt/jboss_eap/jboss-eap-7.3/standalone/configuration/application.keystore not found, it will be auto generated on first use with a self signed certificate for host localhost
2023-03-01 11:25:29,499 INFO  [org.jboss.as.server.deployment.scanner] (MSC service thread 1-7) WFLYDS0013: Started FileSystemDeploymentService for directory /opt/jboss_eap/jboss-eap-7.3/standalone/deployments
2023-03-01 11:25:29,544 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-3) WFLYUT0006: Undertow HTTPS listener https listening on [0:0:0:0:0:0:0:0]:8443
2023-03-01 11:25:29,593 INFO  [org.jboss.ws.common.management] (MSC service thread 1-5) JBWS022052: Starting JBossWS 5.3.0.Final-redhat-00001 (Apache CXF 3.3.12.redhat-00001)
2023-03-01 11:25:29,674 INFO  [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0212: Resuming server
2023-03-01 11:25:29,676 INFO  [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: JBoss EAP 7.3.10.GA (WildFly Core 10.1.25.Final-redhat-00001) started in 2300ms - Started 306 of 560 services (355 services are lazy, passive or on-demand)
…

Notes:

• By default, the collection installs the latest major version of JBoss EAP (7.4.0). As we will discuss upgrading to the next major version in the second part of this article, we have purposely installed the previous major version of the JEE server.
• If the playbook is run again, no changes are reported, as the collection ensures that the setup of JBoss EAP is idempotent.

Upgrading to the next major version of JBoss EAP

Minor upgrades contain only fixes. The actual changes are minimal. Features themselves are not modified, and no new ones are added unless a security fix requires it. In short, this means users can perform such updates on their system without fearing side effects or issues for the hosted applications. However, upgrading JBoss EAP to the next major version is a completely different scenario.

Let’s consider the following requirement. The JBoss EAP instances previously installed using Ansible have been targeted to be upgraded to 7.4 with the latest available minor version (7.4.10). Teams behind the hosted apps have tested and confirmed that no code changes are required; however, a plan is already in place if something goes wrong during the upgrade. Any instance having issues during the upgrade needs to downgrade and resume running the previous version.

Let’s build a playbook implementing such a strategy. First, it will stop the currently running instance of JBoss EAP on the target, then install and start the new version, using the same configuration template as the previous server. And, if anything goes wrong during this process, the existing service will be started.

Here is the playbook we will use to perform this migration:

---
- name: "Update EAP to latest {{ eap_patch_version }}"
  hosts: eap_servers
  vars:
    eap_offline_install: True
    eap_apply_cp: True
    eap_patch_version: 7.4.9
    eap_config_base: 'standalone.xml'
    eap_instance_name: eap74
  collections:
    - redhat.eap
  roles:
    - eap_systemd
  pre_tasks:
    - name: "Ensure previous version of EAP is not running"
      ansible.builtin.service:
        name: eap73
       state: stopped
  tasks:

    - block:
     - name: "Perform EAP {{ eap_patch_version }} installation"
       ansible.builtin.include_role:
         name: eap_install

     - name: "Ensure EAP service is deployed and running."
       ansible.builtin.include_role:
         name: eap_systemd

     - name: "Ensure EAP service is functional"
       ansible.builtin.include_role:
         name: eap_validation

     rescue:

     - name: "EAP upgrade failed, fallback to previous version"
       ansible.builtin.service:
         name: eap73
         state: running

The playbook above implements the migration process we described above. First, it stops the existing JBoss EAP server running on the target. Then, it leverages the redhat.eap collection again to install the new version and start it as a service. To ensure that this instance is functional, it runs the eap_validation role, also provided by the collection, that performs some basic sanity checks against the service.

Assuming nothing has failed during those three steps, the new server is running, and the migration has been completed successfully. If anything goes wrong, the new server is shut down, and the previous instance is restarted.

Note: This playbook is not idempotent and is aimed at being run only for migration purposes. This is both for the sake of simplicity and readability.

Performing the upgrade and update without downtime

Most of the time, updating and upgrading JBoss EAP will require restarting the service, which implies some downtime. However, if there is more than one instance of JBoss EAP to update, it is possible to use a powerful Ansible feature to avoid any downtime (assuming there is a smart proxy in front of the JBoss EAP farm, such as httpd using mod_cluster).

By default, Ansible will try to connect to any hosts belonging to the eap_servers group and perform the update and the upgrade in parallel. If everything goes according to plan, there is still a risk of downtime, as the server might be restarting at the same time. If anything goes wrong, most, if not all, will need to roll back to the previous version, leading to even more chances of downtime.

However, this execution strategy can be configured by adding the keyword serial to the playbook. This will configure how Ansible will operate on the list of servers to connect and execute by batch, only a subset of them at once. With this feature, we can implement the following approach:

• Update (or upgrade) on target;
• Update (or upgrade) one-third of the entire farm;
• Update (or upgrade) the rest of the targets.

Such a strategy offers a lot of peace of mind; if the first machine to be targeted fails to update or upgrade, you can stop the process here and investigate what went wrong. The rest of the servers are still running without any risk of downtime. The same applies to the next batch. If something goes wrong with the first third of the servers, then most of the farm is still running the old version, uninterrupted.

To implement this strategy in our upgrade playbook, we only need to add the attribute serial. We valorize it using a list containing three values, as shown below:

---
- name: "Update EAP to latest {{ eap_patch_version }}"
  hosts: all
  serial:
    - 1
    - 30%
    - 70%
  vars:
    eap_offline_install: True
    eap_apply_cp: True
    eap_patch_version: 7.4.9
    eap_instance_name: eap74
  collections:
    - redhat.eap
  roles:
    - eap_install
    - eap_systemd
  pre_tasks:
    - name: "Ensure previous version of EAP is not running"
      ansible.builtin.service:
        name: eap73
      state: stopped
  tasks:
    - block:
        - name: "Perform EAP {{ eap_patch_version }} installation"
          ansible.builtin.include_role:
            name: eap_install
        - name: "Ensure EAP service is deployed and running."
          ansible.builtin.include_role:
            name: eap_systemd
        - name: "Ensure EAP service is functional"
          ansible.builtin.include_role:
            name: eap_validation
      rescue:
        - name: "EAP upgrade failed, fallback to previous version"
          ansible.builtin.service:
            name: eap73
            state: running

Note that the values within the serial property can indeed be a mix of integers (1) and percentages.

Conclusion

With the heavy lifting of the server installation managed by the redhat.eap collection and the primitives provided by Ansible, we have implemented a sound strategy to perform updates and upgrades on a potentially very large farm of instances.

The two playbooks we displayed in this article are both short and simple to understand. Their content focuses only on the environment specificity (version executed, execution strategy), and the inner workings of EAP are fully encapsulated inside the collection, which provides peace of mind in what would typically be a complex operation and process.