Set up an Active-Active Cluster

This topic describes how to set up an active-active cluster for Deploy with multiple master and multiple external workers.

HA Active-Active Setup

In a HA cluster configuration, there are two types of cluster setups: Active-Active and Active Hot Standby.
Digital.ai Deploy is recommended to be set up in a multi-node Active-Active cluster, as it provides realtime load balancing capability.
In a multi-node setup, you can have multiple masters and multiple workers, with the workers connected to each of the masters.
A master node is a Deploy server you log on to in order to access the Deploy user interface and create configuration items, such as infrastructure, environments, applications, and so on.
The multiple masters ensure continuous functioning in case of failover and the multiple workers ensure uninterrupted task execution and scalability.
Deployment tasks and Control tasks are assigned to the Deploy worker that performs the task execution. For more information, see Tasks, rules, and configuration Items.
The multiple masters are controlled by a load balancer.

Important: The Deploy masters and workers must have similar configuration (database, ActiveMQ, plugins, and so on). A worker cannot run the tasks assigned by the master if the configuration differs.

Requirements to Set Up an Active-Active Cluster

Two or more Deploy stateless master nodes
Two or more external Deploy worker nodes
An external database server, for example PostgreSQL
A load balancer that receives HTTP(S) traffic and forwards that to the Deploy master nodes. For more information, see HAProxy load balancer documentation.
ActiveMQ
A shared drive location to store exported CIs and reports.

Important: It is always recommended to have a multi-node setup with odd number of nodes to facilitate high fault tolerance in production environments. It is also recommended not to have a cluster with more than five nodes to prevent database latency issues. You can, however, with some database configuration tuning, have a cluster with more than five nodes. Contact Digital.ai Support for more information about setting up a cluster with more than five nodes.

About the Setup

Let us consider the following multi-node Deploy setup for illustrative purposes:

Three Deploy masters (Master 1, Master 2, and Master 3)
Three workers (Worker 1, Worker 2, and Worker 3)
One load balancer
One ActiveMQ server (JMS broker)
One database server—PostgreSQL

Step 1: Download and extract the latest Deploy application to Master 1 Server

Create a folder for installation from where you will execute the installation tasks. This will be the root directory.
Download the Deploy ZIP package from the Deploy Software Distribution site (requires customer login).

Step 2: Install the Deploy license

Download the Deploy license file from the Deploy Software Distribution site
Copy your license file to the conf directory of the root directory.

Step 3: Configure the external Postgres database node

You must set up a database for the Deploy to store data. Create and configure an empty database before you start Deploy installation. During the installation, Deploy creates the database schema on the database you created. Optionally, you can have separate database for the operational and reporting databases. For more information about creating two separate databases, see Separate Databases for Reporting and Repository in Deploy.

Use a industrial-grade external database server, for example PostgreSQL, for production use. For more information, see Configure the Database and Artifacts Repository.

For more information about how to create Postgres database, see the PostgreSQL documentation.

Step 4: Configure the Postgres JDBC driver

Download the Postgres JDBC driver. See PostgreSQL JDBC driver.
Copy the JAR file to the lib folder of the root directory.

Step 5: Update the `deploy-repository.yaml` file with the external database details

In the root folder, go to centralConfiguration folder and open the deploy-repository.yaml file.
Configure the parameters to point to the database schema as shown in the following sample configuration:

xl:
  repository:
    database:
     db-driver-class-name: "org.postgresql.Driver"
     db-password: "samplepassword"
     db-url: "jdbc:postgresql://<IP address of the external database server>/postgres"
     db-username: "sample-user"
     max-pool-size: 10
   artifacts.root: "build/artifacts"

Step 6: Download the ActiveMQ client

Download the JMS message broker—ActiveMQ client.
Add JMS .jar file—org.apache.activemq:activemq-client to thelib folder of the root directory.

Step 7: Set up the ActiveMQ node

Run ActiveMQ in the Docker container.
From the centralConfiguration folder of the root directory, open deploy-task.yaml file.
Set the in-process-worker parameter to false.

Update the JMS details as shown in the sample configuration:

deploy:
queue:
 external:
   jms-driver-classname: org.apache.ActiveMQConnectionFactory
     jms-url: tcp://<IP address of the ActiveMQ server node>:61616
   jms-username: admin
     jms-password: admin

Step 8: Set up the Load Balancer node

To install the HA proxy (load balancer), run:
yum install -y haproxy
Enable the HA proxy by running the following command:
systemctl enable haproxy
To start the HA proxy, run:
systemct start haproxy
Run the following command to check the status:
systemct status haproxy

Step 9: Update the hostname of the Master 1 node

From the conf folder of the root directory, open the deployit.conf file.
Update the server.hostname parameter with the hostname of the master server.
Update the deploy.cluster.node.port parameter with the port number. For example, deploy.cluster.node.port=25520—default port number is 25520.

Step 10: Update the `deploy-cluster.yaml` file with the cluster mode

Digital.ai Deploy 22.3 brings you the following changes related to the high availability (HA) cluster setup:

You can no longer set mode to default in the deploy-cluster.yaml file and have Digital.ai Deploy run in HA active-active cluster mode.
You must set mode to full in the deploy-cluster.yaml file if you want to run Deploy in HA active-active cluster mode.

Here’s an example deploy-cluster.yaml configuration file with mode set to full.

deploy:
  cluster:
    akka:
      actor:
        loggers:
        - akka.event.slf4j.Slf4jLogger
        loglevel: INFO
        provider: akka.cluster.ClusterActorRefProvider
      cluster:
        auto-down-unreachable-after: 15s
        custom-downing:
          down-removal-margin: 10s
          stable-after: 10s
        downing-provider-class: ''
    membership:
      heartbeat: 10 seconds
      jdbc:
        connection-timeout: 30 seconds
        idle-timeout: 10 minutes
        leak-connection-threshold: 15 seconds
        max-life-time: 30 minutes
        max-pool-size: 1
        minimum-idle: 1
        password: '{cipher}gfdqswdksahgdksahgdkas'
        pool-name: ClusterPool
        url: ''
        username: ''
      ttl: 60 seconds
    mode: full
    name: xld-active-cluster