The configuration of the XL Deploy master instance running with multiple local or external workers is currently available as a BETA feature. The fully supported default configuration for version 8.6 is the XL Deploy master instance with an in process worker running in the same process as the master. The other custom configurations with multiple workers are subject to change in future releases and are currently not recommended for production use.

In XL Deploy, deployment tasks and control tasks are executed by the task execution engine. Based on your deployment task, the XL Deploy planner generates a deployment plan with steps that XL Deploy will carry out to deploy an application. You can manually modify the plan. When the plan is ready, XL Deploy generates a DeploymentTask that is sent to the task execution engine.

As of version 8.6, the task execution engine can be separated from the core XL Deploy server called the master instance. The tasks in XL Deploy are executed by a process called worker. The current default configuration contains a worker that is part of the master and runs in the same process. This is called an in process worker and it is unique. Local and external workers run in separate processes and can be located on different machines from the master or on the same machine as the master. Local workers are workers that run in separate processes but are located in the same installation directory as the XL Deploy master.

The master-worker setup enables you to scale XL Deploy by adding multiple workers connected to a master to execute more tasks from XL Deploy. The task is created on the master instance and sent to a worker to be executed. A worker can have multiple tasks assigned to the execution queue. Each task from the master can be assigned to only one worker.

When you install XL Deploy, the default configuration is to execute the tasks by an in process worker that runs in the same process as the master. You can change the default configuration and use multiple workers that run in different processes from the master. You cannot use the in process worker and other workers in the same configuration.

Default configuration with in process worker

As of version 8.6.0, when you install an XL Deploy instance, the default configuration is to execute the tasks on an internal worker that runs in the same process as the master. This setting is defined in the xl.task.in-process-worker=true property in the xl-deploy.conf file.

In-process worker

This default setup provides identical functionality as the XL Deploy default configuration for pre 8.6.0 versions.

Configure XL Deploy to use multiple workers

This is a BETA feature.

The worker is a task executor that is running in a different java process from the XL Deploy master process. To start a worker, it must have an identical configuration with the master: the folder structure, contents, and configuration files must be identical and it must use the same database.

Custom configurations with workers

  • Master and workers in one folder location

    You can start a separate java process for each worker from the same folder location as the master. These are called local workers.

    Local worker

    This setup provides improved availability and a faster XL Deploy restart procedure for simple configuration changes such as a new plugin or configuration files updates. You can use the startlocalworker script to quickly add new workers. Ensure that you allocate for each worker the same resources required for the master.

  • Master and workers in separate folder location on the same machine

    You must copy the installation directory of the master to a different location on the same machine for each worker and start a process from the new location.

    External worker

    This setup supports a faster restart procedure of XL Deploy for advanced configuration changes such as replacing or removing a plugin. Ensure that you allocate for each worker the same resources required for the master.

  • Master on one machine and workers on separate locations on different machines

    You must copy the installation directory of the master to a location on the different machine for each worker and start a process from the new location. The master and the registered workers must all exist in the same subnetwork.

    Remote worker

    This setup supports high availability and better scalability for XL Deploy by using multiple machines to run workers connected to one or more masters.

System Requirements

If you are running XL Deploy with an in process worker configuration, see Requirements for installing XL Deploy.

If you are running XL Deploy with multiple workers on the same machine, ensure that you allocate the same resources for each worker as for the master. A worker process has the same resource requirements as the master process.

Network connectivity

If you are running XL Deploy in a configuration with multiple workers on different machines, the master and the workers must reside in the same subnetwork.

Security

To configure secure communication between master and workers, refer to Configure secure communication with workers and satellites.

Restarting XL Deploy in the master-worker setup

The master-worker setup allows for high availability of XL Deploy. XL Deploy needs to be restarted for configuration, plugin, type system changes, or upgrades. With the master-worker setup, you can safely restart XL Deploy without waiting for all the running tasks to be completed. The tasks are executed by the workers and will continue to run on the existing configuration while the master is restarted.

After XL Deploy is restarted with the new configuration, you must manually synchronize the configuration for the existing workers. The master will not assign new tasks to workers that have the old configuration. You can either register new workers to the master, or free up workers before restarting the XL Deploy master.

Important: Upgrades to XL Deploy that include database changes or breaking type system changes still require a complete XL Deploy restart and cannot be performed while parts of the system are running.

Deployments with the master-worker setup

When you perform a deployment, the deployment mapping and the planning are created on the master instance. The deployment execution task, which consists of executing the steps of the plan, is sent to a worker. Each task from the master is assigned to one worker. The XL Deploy master uses a round-robin task scheduling method to assign tasks to workers. The method consist of assigning tasks one by one, to each worker in the list.

Important: The master assigns tasks only if there are available workers registered with the master that have an identical configuration.

Recovery on worker level

Task recovery occurs on the worker level. Each worker has a recovery space reserved on the disk where it writes the task information. When the system goes down, the tasks can be recovered and reinitiated from the worker recovery space on the disk.

Master-worker and satellites

These are the main characteristics that differentiate workers and XL Satellites:

  Worker Satellite
Network topology Workers should reside on the same machine or in the same network as the master for stability, speed, and reduced latency. Satellites can exist in a data center away from the master or workers.
Network stability Workers and master must reside in the same stable network. Satellites can be connected by stable networks.
Task execution One task is assigned to one worker. Satellites execute a part of a task containing block of steps. One deployment task can involve multiple satellites. The host that you deploy to determines which satellites execute the block.
Functionality Full functionality and identical to the master. Limited functionality, does not require access to all resources. Satellites are CIs that exist in the XL Deploy repository and cannot execute all step types.

Satellites can be used with all master-worker configurations.

For more information about XL Satellites, refer to Getting started with the satellite module.

Communication between master and worker

Communication between the master process and the worker process when sending data is done through two channels.

From master to worker

  1. When you start a new worker, you must specify the address of the master. The worker registers with the master.
  2. The master selects a worker and sends a task to execute it.
  3. The master sends instructions to the worker such as starting the ta§sk, pausing the task, etc.

All the command information is pulled from the master to the worker. These tasks are communicated through one command channel.

From worker to master

The second channel of communication is in the other direction, from the worker to the master, through the REST API. This is used when a worker must perform an action that is only available on the master.

To set up the communication, you must specify the locations for these channels when you start a worker. For more information, see Add, start, and use workers

Overview of workers in XL Deploy

In the Explorer, workers are shown in the Monitoring section under Workers. The list contains the following information:

ID A unique (technical) identifier of the worker.
Name The user assigned name of the worker (not unique).
Address The (technical) unique address of the worker containing hostname and port.
State The state that the worker is in.
# Deployment tasks The number of deployment tasks assigned to the worker.
# Control tasks The number of control tasks assigned to the worker.

The state of a worker can be:

Connected The worker is connected to the master and can be used to execute tasks.
Incompatible The worker is connected to the master, but is on a different configuration. The master will not send new tasks to this worker. This is a temporary state, while the system is being updated.
Draining The worker is shutting down once all tasks are completed (cancelled or archived). The master will not send new tasks to this worker.
Disconnected The worker is not connected to the master. This can occur due to network interruptions, a failure of the machine, or an issue in the worker. The master displays the number of deployment or control tasks that are running on it, but these tasks cannot be managed.

Master worker setup in active/hot-standby

When you are using XL Deploy in an active/hot-standby setup, the database is shared by all the XL Deploy master nodes. The workers you register with an XL Deploy master will also connect to the same database. The communication between the workers and the active XL Deploy master node is done through the load balancer.

As of version 8.6.0, the default setup is to have an in process worker in each of the hot-standby master nodes. To disable the in process worker and add other workers, set xl.task.in-process-worker=false in the XL_DEPLOY_SERVER_HOME/conf/xl-deploy.conf file for each master node.

To create a worker:

  • for the -api flag, specify the REST endpoint of the load balancer.
  • point one -master flag to each of the hot-standby master nodes

The contents of the installation directories must be identical between all the master nodes and all the workers. Ensure all the conf, plugins, and ext folders contain are synchronized in all location.

To ensure high availability, add workers across multiple locations in the same subnetwork. All the workers can communicate with any active master node. If the communication between a master and a worker fails and the worker cannot be reached, the system will display the worker in an disconnected state tasks in unknown state.

Important: The master and workers require access to the artifacts in XL Deploy. You must store the artifacts in an external system such as Nexus or Artefactory, in a central database, or using a file share for the storage location.