Production environment install

This topic is a step-by-step guide on how to set up XL Deploy in a production-ready environment. It describes how to configure the product, environment, and server resources to get the most out of the product. This topic is structured in three sections:

  • Preparation: Describes the prerequisites for the recommended XL Deploy setup.
  • Installation: Covers setup and configuration procedures for each of the components.
  • Administration/Operation: Provides an overview of best practices to maintain and administer the system once it is in production.

Important: Proper configuration of I/O subsystems (database, file system, network) is critical for the optimal performance and operation of XL Deploy. This guide provides best practices and sizing recommendations.

XL Deploy is active/active capable, and external workers are fully supported. Both are recommended features. The load balancer fronting each of the XL Deploy servers is required to support sticky sessions based on session cookies when running in active/active mode. Finally, the XL_DEPLOY_SERVER_HOME/exports/ folder should be a shared file system between each of the master nodes.

Note: Hostname resolution between masters and workers is based on DNS lookups. This means localhost and other /etc/hosts-based lookups will fail by default. Although using localhost is considered bad practice, if your installation relies on it you can reinstate the behavior by including a setting akka.io.dns.resolver = inet-address in xl-deploy.conf.

Production environment setup

XL Deploy Production Configuration

Preparation

In the first phase of setting up a production environment, you need to determine the correct hardware requirements and obtain the necessary prerequisites. A production-ready XL Deploy setup is a clustered, multi-node active/active setup with multiple external workers. As such, you will need multiple machines co-located in the same network segment.

Alternatives for the recommended setup are to use local workers, or try out a Kubernetes-based setup.

Obtaining XL Deploy servers

The Requirements for installing XL Deploy topic describes the minimum system requirements. Here are the recommended requirements for each XL Deploy production machine (both masters and workers):

  • 3+ Ghz 2 CPU quad-core machine (amounting to 8 cores) or better
  • 4 GB RAM or more
  • 100 GB hard disk space

Note: All of the XL Deploy cluster nodes must reside in the same network segment. This is required for the clustering protocol to optimally function. For best performance with minimize network latency, it is also recommended that your database server be located in the same network segment.

Obtaining the XL Deploy distribution

Download the XL Deploy ZIP package from the XebiaLabs Software Distribution site (requires customer log-in).

For information about the supported versions of XL Deploy, see Supported XebiaLabs product versions.

Choosing a database server

A production setup requires an external clustered database to store the XL Deploy data. The supported external databases are described in Configure the XL Deploy SQL repository.

For more information about hardware requirements, see the database server provider documentation.

Artifacts storage location

You can configure XL Deploy to store and retrieve artifacts in three different storage repository formats:

  • Repository Manager: Artifacts are managed by tools such as Nexus and Artifactory
  • Database: The artifacts are stored in and retrieved from a relational database management system (RDBMS)
  • File system: The artifacts are stored on and retrieved from the file system

XebiaLabs recommendations are, in order of preference:

  1. A repository management system, with artifacts referenced using the fileUri property on the artifacts.
  2. A clustered database, if the RDBMS supports the size of the artifacts.
  3. On a shared file system.

XL Deploy can only use one local artifact repository at any time. The configuration option xl.repository.artifacts.type can be set to either file or db to select the storage repository.

Choosing a load balancer

The recommended XL Deploy production setup is an active/active one, but each user session resides on a single node. As a result, you need to front the installation with a load balancer that supports sticky sessions based on session cookies.

This guide uses HAProxy as an example. You can use any HTTP(s) load balancer that supports the following features:

  • SSL offloading
  • Checking a custom HTTP endpoint for node availability
  • Sticky sessions based on session cookies

Here are examples of load balancers that support this feature set:

Choosing an authentication provider

XL Deploy supports a number of single-sign on (SSO) authentication providers, including Secure LDAP (LDAPS) and OIDC providers. Many cloud providers support authentication through OIDC:

If you do not want to use a cloud provider, or if your SSO solution is not compatible with OIDC, you can integrate your SSO with Keycloak, which is an OIDC bridge.

For more information, see Configure OpenID Connect (OIDC) Authentication for XL Deploy or Connect XL Deploy to your LDAP or Active Directory

Choosing a monitoring and alerting solution

For a production installation, make sure you set up a monitoring system to monitor the system and product performance for the components comprising your installation. XL Deploy exposes internal and system metrics over Java Management Extensions (JMX). Any monitoring system that can read JMX data can be used to monitor the installation.

Common monitoring and alerting tools include:

Choosing a forensic data gathering toolchain

In addition to monitoring, ensure that system-related data gathering is active and available. You can analyze the gathered forensic data at a later point in time and perform root cause analysis for outages. You can also use forensic data to determine usage and peak load patterns.

For this kind of monitoring, you can use a time series database. Common tools include:

You can graph and analyze the gathered data using a tool such as Grafana.

It is also recommended that you set up log file monitoring. The industry-standard toolchain for log file monitoring is the ELK stack:

These tools allow log files to be read and indexed while they are being written, so you can monitor for errant behavior during operation and perform analysis after outages.

Database server configuration

The basic database setup procedure, including schemas and privileges, is described in Configure the XL Deploy SQL repository. For various databases, additional configuration options are required to use them with XL Deploy or for a better performance.

MySQL or MariaDB

Important: MariaDB is not an officially supported database for XL Deploy, but you can use it as a drop in replacement for MySQL.

The default installation of MySQL is not tuned to run on a dedicated high-end machine. It is recommended that you change the following MySQL settings to improve its performance. These settings can be set in the MySQL options file. See the MySQL documentation to locate this file on your operating system.

Setting Value
innodb_buffer_pool_size Set this to maximum 70-75% of the available RAM of the database server. This setting controls the size of the database structure that can be kept in memory. Larger size provides better performance for the application due to caching at the database level.
innodb_log_file_size Set this to 256M. This sets the size of redo logs that MySQL keeps. If you set this to a large value, MySQL can process peak loads by keeping the transactions in the redo log.
innodb_thread_concurrency Set this to 2 * CPU cores of the database server. For example: for a 2 CPU quad-core machine, the optimal setting is 2 CPU * 4 Cores * 2 = 16.
max_allowed_packet Set this to 16M. This represents the maximum size of the packet transmitted from the server to the client. As the XL Deploy database for some columns contains BLOBs, this setting is better than the default of 1M.
open_files_limit XebiaLabs recommends setting this value to 10000 for large installations. This setting controls the number of file descriptors the MySQL database can keep open. This setting cannot be set to a higher value than the output of ulimit -n on a Linux/Unix system. If this limit is lower than the recommended value, see the documentation for your operating system.
innodb_flush_log_at_trx_commit Advanced: The default setting of this option is 1. Every transaction is always flushed to disk on commit, ensuring full ACID compliance. Setting this to either 0 (only flush the transaction buffer once per second to the transaction log), or 2 (directly write the transaction to the transaction log, flush the log once per second to disk), can produce transaction loss of up to a second worth of data. Note: When using a battery-backed disk cache, this setting can be set to 2 to prevent direct flushes to disk. The battery-backed disk cache ensures that the cache is flushed to disk before the power fails.

PostgreSQL

There are a number of settings in a default installation of PostgreSQL that can be tuned to better perform on higher end systems. These configuration options can be set in the PostgreSQL configuration file. See the PostgreSQL documentation to locate this file on your operating system.

Setting Value
shared_buffers Set to 30% of the available RAM of the database server. This setting controls the size of the memory allocated to PostgreSQL for caching data.
effective_cache_size Set to 50% of the available RAM of the database server. This setting provides an estimate of the memory size available for disk caching. The PostgreSQL query planner uses this setting to figure out if the query plan results can fit in the memory or not.
checkpoint_segments Set to 64. This setting controls how often the Write Ahead Log (WAL) is check-pointed. The WAL is written in 16MB segments. If you set this to 64, the WAL is check-pointed once every 64 * 16MB = 1024MB or once per 5 minutes, whichever is reached first.
default_statistics_target Set to 250. This setting controls the amount of information stored in the statistics tables for optimizing query execution.
work_mem Set to 0.2% of the available RAM of the database server. This setting controls the memory size available per connection for performing memory sorts and joins of query results. In a 100 connection scenario, this will be 20% of the available RAM in total.
maintenance_work_mem Set to 2% of the available RAM. This setting controls the amount of memory available to PostgreSQL for maintenance operations such as VACUUM and ANALYZE.
synchronous_commit Advanced: The default setting of this option is on. This guarantees full ACID compliance and no data loss on power failure. If you have a battery-backed disk cache, you can switch this setting to off to produce an increase in transactions per second.

Security settings

It is important to harden the XL Deploy environment from abuse. There are many industry-standard practices to ensure that an application runs in a sandboxed environment. You should minimally take the following actions:

  1. Run XL Deploy in a VM or a container. There are officially supported docker images for XL Deploy.
  2. Run XL Deploy on a read-only file system. XL Deploy needs to write to several directories during operation, specifically its conf/, export/, log/, repository/, and work/ subdirectories. The rest of the file system can be made read-only. (The export/ directory needs to be shared between XL Deploy master instances)
  3. Enable SSL on JMX on XL Deploy server and satellites. See Using JMX counters for XL Satellite and conf/xl-deploy.conf.example.
  4. Configure secure communications between XL Deploy and satellites.
  5. Change the default Derby database to a production-ready database as described above.
  6. Do not enable SSL since the load balancer will offload SSL as described in Finalize the node configuration and start the server.

Operating system

XL Deploy supports running on any commercially supported Microsoft Windows Server version (under Mainstream Support), or any Linux/Unix operating systems. Ensure that you maintain these systems with the latest security updates.

Java version

Important: XL Deploy requires Java SE 8 or Java SE 11. Running XL Deploy on non-LTS Java versions is not supported. See the Java SE support roadmap for details and dates.

XL Deploy can run on the Oracle JDK or JRE, as well as OpenJDK. Always run the latest patch level of the JDK or JRE, unless otherwise instructed. For more information on Java requirements see Requirements for installing XL Deploy.

Installation and execution

The XL Deploy installation procedure for both master and worker nodes is the same. To install XL Deploy on the machines with the minimum required permissions:

Linux/Unix operating systems

  1. Create a dedicated unprivileged non-root user called xl-deploy. This ensures that you can lock down the operating system and prevent accidental privilege escalations.
  2. Create a directory under /opt called xebialabs, where the xl-deploy user has read access.
  3. Extract the downloaded version of XL Deploy in the /opt/xebialabs directory.
  4. Change the ownership of the installed product to xl-deploy and grant the user read access to the installation directory.
  5. Grant the xl-deploy user write access underneath the /opt/xebialabs/xl-deploy-<version>-server/ folder to the conf/ and log/ subdirectories. For password encryption purposes, the user should have write access to conf/deployit.conf and conf/xl-deploy.conf. The user should also be able to create new subdirectories, or have write access to the export/, repository/, work/ subdirectories. Finally, ensure that all files in the log/ subfolder have write permissions. The server will start without them, but no logging will be available. The export/ subdirectory should be a shared filesystem across all masters and workers.
  6. Copy your license file to the /opt/xebialabs/xl-deploy-<version>-server/conf directory. You can download your license file from the XebiaLabs Software Distribution site (requires customer log-in).

Microsoft Windows operating systems

  1. Create a dedicated unprivileged non-Administrator account called xl-deploy. This ensures that you can lock down the operating system and prevent accidental privilege escalations.
  2. Create a directory C:\xebialabs, where the xl-deploy user has default Read, List folder contents and Read & Execute access.
  3. Extract the downloaded version of XL Deploy into the C:\xebialabs\xl-deployit-<version>-server directory.
  4. Grant the xl-deploy user Read, Read & execute, List folder contents, and Write permissions to this installation directory so XL Deploy can add and modify necessary files and create subdirectories. Alternatively, grant Write permission to the conf\ and log\ folders, and to the newly to be created export, repository, and work subdirectories. (In case of workers, the work directory will be called differently.)
  5. Copy your license file to the C:\xebialabs\xl-deploy-<version>-server\conf directory. You can download your license file from the XebiaLabs Software Distribution site (requires customer log-in).

Configure the SQL repository

For a clustered production setup, XL Deploy requires an external database, as described in Configure the XL Deploy SQL repository.

Configure XL Deploy master-worker connectivity

The suggested production setup uses external workers to execute deployments and control tasks. On each node, edit conf/xl-deploy.conf to include a setting xl.task.in-process-worker = false. On each of the nodes running XL Deploy as a worker instance, the -master <address>:<port> flags’ addresses will be resolved against DNS, so you need to make sure that your DNS server resolves each as an A record or an SRV record listing each of the XL Deploy master instances. This address will be resolved periodically (as determined by the xl.worker.connect.interval setting in the conf/xl-deploy.conf file), and the worker will adjust the addresses it connects to.

Configure user authentication

Set up a secure process for authenticating users. For production setups, you can use an OIDC provider, use Keycloak as an OIDC bridge, or use an LDAP directory system over the LDAPS protocol.

For more information, see:

Configure XL Deploy Java virtual machine (JVM) options

To optimize XL Deploy performance, you can adjust JVM options to modify the runtime configuration of XL Deploy. To increase performance, add or change the following settings in the conf/xld-wrapper-linux.conf or the conf\xld-wrapper-windows.conf file.

Setting Value
-server Instructs the JVM to run in the server profile.
-Xms8192m Instructs the JVM to reserve a minimum of 8 GB of heap space. If you have less than 10GB memory dedicated to your (virtual or physical) machine, lower this number accordingly. (Note that the operating system itself also needs memory)
-Xmx8192m Max heap size. Set this to the same number as the previous setting to get optimal performance from the G1 garbage collector (enabled below)
-XX:MaxMetaspaceSize=1024m Instructs the JVM to assign 1 GB of memory to the metaspace region (off-heap memory region for loading classes and native libraries).
-Xss1024k Instructs the JVM to limit the stack size to 1 MB.
-XX:+UseG1GC (Java 8 only) instructs the JVM to use the new G1 (Garbage First) garbage collector which deals with large stacks (like XL Deploy’s) more efficiently than the default one. On Java 11, G1 is already the default garbage collector.
-Dsun.net.inetaddr.ttl=60 Limits DNS caching to 60s.
-XX:+HeapDumpOnOutOfMemoryError Instructs the JVM to dump the heap to a file in case of an OutOfMemoryError. This is useful for debugging purposes if the XL Deploy process crashes.
-XX:HeapDumpPath=log/ Instructs the JVM to store generated heap dumps in the log/ directory of the XL Deploy server.

Configure the task execution engine

Deployment tasks are executed by the XL Deploy workers’ task execution engines. Based on your deployment task, one of the XL Deploy master instances generates a deployment plan that contains steps that one of the XL Deploy workers will carry out to deploy the application. You can tune the XL Deploy worker’s task execution engine with the settings described in Configure the task execution engine. Note that a worker is only eligible for executing a deployment task when its configuration matches the master’s that generated the deployment plan.

Finalize the node configuration and start the server

After the node(s) are configured for production use, you can finalize the configuration.

Run the /opt/xebialabs/xl-deploy-<version>-server/bin/run.sh or C:\xebialabs\xl-deploy-<version>-server\bin\run.cmd script on a single node to start the XL Deploy server.

Because this is the initial installation, XL Deploy prompts a series of questions. See the table below for the questions, recommended responses, and considerations.

Question Answer Explanation
Do you want to use the simple setup no Some properties need to be changed for production scenarios.
Please enter the admin password Choose a strong and secure admin password.
Do you want to generate a new password encryption key yes You should generate a random unique password encryption key for the production environment.
Please enter the password you wish to use for the password encryption key If you want to start XL Deploy as a service on system boot, do not add a password to the password encryption key. Adding a password to the encryption key prevents automated start. If your enterprise security compliance demands it, you can add a password in this step.
Would you like to enable SSL no SSL offloading is done on the load balancer. In this scenario, it is not required to enable SSL on the XL Deploy servers.
What HTTP bind address would you like the server to listen to 0.0.0.0 Add this address to listen on all interfaces. If you only want to listen on a single IP address/interface, specify that one.
What HTTP port number would you like the server to listen on 4516 This is the default port; you can enter a different port number.
Enter the web context root where XL Deploy will run / By default, XL Deploy runs on the / context root (in the root of the server).
Enter the public URL to access XL Deploy https://LOADBALANCER_HOSTNAME For XL Deploy to correctly rewrite all the URLs, it must know how it can be reached. Enter the IP address or hostname configured on the load balancer, instead of the IP address (and port) of the XL Deploy server itself. The protocol is https.
Enter the minimum number of threads for the HTTP server 30 Unless otherwise instructed, the default value can be used.
Enter the maximum number of threads for the HTTP server 150 Start with the default value. If the monitoring points to thread pool saturation, this number can be increased.
Do you agree with these settings yes Type yes after reviewing all settings.

After you answer yes to the final question, the XL Deploy server will boot up. During the initialization sequence, it will initialize the database schemas and display the following message:

You can now point your browser to https://<IP_OF_LOADBALANCER>/

Stop the XL Deploy server. Edit the conf/deployit.conf file and change these configuration options to a hardened setting:

Option Value Explanation
hide.internals true Hides exception messages from end users and only shows a key. The XL Deploy administrator can use this key find the exception.
client.session.timeout.minutes 20 Defines the session idle timeout. Set this to the number of minutes that is defined by your enterprise security compliance officer.

Copy the conf/repository-keystore.jceks and conf/deployit.conf to the other nodes so that they run on the same settings.

All nodes are now fully configured and can be booted up.

Boot sequence

Start the nodes:

  1. Start the master nodes.
  2. Wait until each of the nodes is reachable at http://<node_ip_address>:4516/.
  3. Check that each node reports success on a GET request to http://<node_ip_address>:4516/ha/health
  4. Start each of the worker nodes.
  5. Check that each worker node connects successfully to each of the master nodes, e.g. by using the CLI command workers.list()

Running as a service

Once you are satisfied that the XL Deploy master instances run without issue, you can install XL Deploy as a service by executing the bin/install-service.sh or bin\install-service.cmd script.

Configure the additional tools

Set up the load balancer

This example shows a minimal HAProxy 2.0 load balancer configuration. The sections below show how to set up the routing and health checks for the load balancer.

defaults
  mode http   # <0>

frontend xl-http   # <1>
  bind 0.0.0.0:80
  http-request redirect scheme https if !{ ssl_fc }   # <2>

frontend xl-https   # <3>
  bind 0.0.0.0:443 ssl crt /path/to/certificate.pem   # <4>
  default_backend xld-backend   # <5>

backend xld-backend   # <6>
  option httpchk GET /deployit/ha/health   # <7>
  balance roundrobin   # <8>
  cookie SESSION_XLD prefix   # <9>
  server xld-1 xld-server-1.example.com:4516 check cookie abc   # <10>
  server xld-2 xld-server-2.example.com:4516 check cookie def
  server xld-3 xld-server-3.example.com:4516 check cookie ghi
  • <0> Enable HTTP traffic analysis so we may e.g. redirect traffic (<2>) and manipulate cookies (<9>, <10>)
  • <1> The xl-http front end handles all HTTP requests on port 80
  • <2> Redirect all requests to HTTPS if the front connection was not made using an SSL transport layer.
  • <3> The xl-https front end will handle all incoming SSL requests on port 443.
  • <4> Ensure you have a properly signed certificate to ensure a hardened configuration.
  • <5> Every incoming request on HTTPS will be routed to the xld-backend back end.
  • <6> The xld-backend will handle the actual load balancing for the XL Deploy nodes.
  • <7> Every XL Deploy node is checked on the /deployit/ha/health endpoint to verify whether it is up. If this endpoint returns a non-success status code, the node is taken out of the load balancer until it is back up.
  • <8> New user sessions are distributed round-robin
  • <9> Sticky sessions are supported by putting a prefix on the SESSION_XLD cookie
  • <10> Enable health check and use the abc prefix on the SESSION_XLD cookie to arrange sticky sessions bound to master node xld-1, prefix def for xld-2, etc.

Administration and operation

This section describes how to maintain the running system and what to do if monitoring shows any issues in the system.

Back up XL Deploy

To prevent inadvertent loss of data, ensure you regularly back up your production database as described in Back up XL Deploy.

Set up monitoring

Set up the desired metrics

Ensure that you monitor the following statistics for the systems that comprise your XL Deploy environment including the load balancer, your XL Deploy nodes, and database servers:

  • Network I/O
  • Disk I/O
  • RAM usage
  • CPU usage

Add monitoring to XL Deploy

You can remotely monitor JMX, add a Java agent such as the Dynatrace agent, or use a tool such as collectd to push the monitoring statistics to a central collectd server.

In general, it is not recommended that you add Java agents to the Java process. Testing has shown that the Java agents can adversely influence the performance characteristics of the XL Deploy system. You should also make sure to not expose insecure or unauthenticated JMX over the network, as it can be used to execute remote procedure calls on the JVM.

The optimal solution is to set up collectd to aggregate the statistics on the XL Deploy server and push them to a central collecting server that can graph them. To do this, you must install the following tools on the XL Deploy server:

After these tools are installed, you can download this sample collectd.conf file, which is preconfigured to monitor relevant XL Deploy application and system statistics. To use this file, add two configuration values to the configuration:

  • IP_ADDRESS_HERE: Enter the IP address of the central collectd server
  • NETWORK_INTERFACE_HERE: Enter the network interface over which XL Deploy communicates

Connectivity to middleware

This section reviews how XL Deploy will traverse your network to communicate with middleware application servers to perform deployment operations. Since XL Deploy is agentless, communication is done using standard SSH or WinRM protocols.

Standard XL Deploy connectivity

In this example, XL Deploy, using the Overthere plugin, connects to the target server using either SSH or WinRM.

XL Deploy connects using SSH or WinRM

For more information, review the following:

Standard XL Deploy connectivity using Jumpstation

  1. XL Deploy, using the Overthere plugin, connects to the jumpstation target server using SSH. Nothing is installed on the jumpstation server.
  2. Connection is made from jumpstation, using SSH or WinRM, to the target server.

XL Deploy connects using jumpstation to target server

For more information, see Jumpstation details and Connect XL Deploy through an SSH jumpstation or HTTP proxy

Standard XL Deploy connectivity using Satellite

How it works:

  1. XL Deploy communicates to XL Satellite application using TCP.
  2. Deployment workload is moved from XL Deploy JVM to XL Satellite.
  3. XL Satellite, using the Overthere plugin, connects to the Target server using SSH or WinRM.

XL Deploy, first moves workload to XL Satellite through TCP, then SSH or WinRM to target server

For more information, see getting started with the satellite module.

Communication protocols and capabilities

XL Deploy Outbound Connection Type Outbound Connections to Target Servers Deployment Tasks Control Tasks UI Extensions
XL Deploy Server - SSH, WinRM Y Y Y
XL Satellite TCP SSH, WinRM Y N N
Jumpstation SSH SSH, WinRM N/A N/A N/A

Jumpstation details

Jumpstation:

  • Uses standard SSH encryption
  • Can use PKI or User credentials
  • Uses a TLS-encrypted satellite connection
  • Will create a tunnel to allow direct communication to a target host
  • Port (configurable): 22 default
  • Port only open during communications. Pipe closed after task executed
  • Is bi-directional during activity

XL Deploy, connecting to DMZ using SSH, connects to Jumpstation

With this set up, you then need to:

  1. Establish connectivity method (credentials, or signed certificate).
  2. Define a firewall port.
  3. Generate IP Table rules.

For more details, see connecting to jumpstation.