Configure secure communication with workers and satellites

The XL Deploy master communicates with workers and satellite servers over a secure communication channel using TLS/SSL technology to encrypt data. This algorithm relies on certificate checking and data encryption using asymmetric keys.

TLS in a nutshell

TLS is based on the notion of public and private keys. The server contains a private key and a public certificate. In the Java world, they are stored in a keystore. The private key must be hidden and can be protected with a passphrase. This key must not be given out or communicated.

When a client tries to reach a server, it authenticates the destination. The server must prove its identity. To achieve this, the client gets a list of trusted certificates. This is the truststore. It contains public certificates that are verified by a trusted authority.

When a client tries to reach a server, there is a negotiation phase. During this phase, the client challenges the server to authenticate it. Once identified, every bit of data transferred between each side of the communication is encrypted.

With this technology, an external process that you do not manage cannot pretend to be a satellite of yours, and external processes cannot listen to the secure communication.

Configure secure communication

To configure secure communication between XL Deploy master, workers, and satellites:

  1. Generate a key and certificate for the master, each worker, and each satellite server.
  2. Add all certificates to a truststore.
  3. Configure each component to use their own key and the truststore containing all the other certificates.

To ensure that communication is fully secure, each key should be unique.

Sample keystore and truststore creation

This example shows how to create keys and self-signed certificates using the keytool utility.

Configure the truststore with generated keys for all master, worker, and satellite components:

  1. Generate a key for a master, worker, or satellite:

    keytool -genkey -alias component_name -keyalg RSA -keypass k3yp@ss -storepass st0r3p@ss -keystore component_name.p12 -storetype pkcs12 -validity 360 -keysize 1024
  2. Export the public certificate from the keystore file and import the certificate into the truststore:

    keytool -export -keystore component_name.p12 -storepass st0r3p@ss -alias component_name | keytool -import -noprompt -alias component_name -storepass st0r3p@ss -keystore xld-truststore.p12 -storetype pkcs12

Repeat this procedure for each worker and each satellite.

Configuration example

This is an example of a configuration script for two master nodes in active/hot-standby, four workers, and three satellites.

keytool -genkey -alias master-1 -keyalg RSA -storepass st0r3p@ss -keystore master-1.p12 -storetype pkcs12 -validity 360 -keysize 1024
keytool -genkey -alias master-2 -keyalg RSA -storepass st0r3p@ss -keystore master-2.p12 -storetype pkcs12 -validity 360 -keysize 1024

keytool -genkey -alias worker-1 -keyalg RSA -storepass st0r3p@ss -keystore worker-1.p12 -storetype pkcs12 -validity 360 -keysize 1024
keytool -genkey -alias worker-2 -keyalg RSA -storepass st0r3p@ss -keystore worker-2.p12 -storetype pkcs12 -validity 360 -keysize 1024
keytool -genkey -alias worker-3 -keyalg RSA -storepass st0r3p@ss -keystore worker-3.p12 -storetype pkcs12 -validity 360 -keysize 1024
keytool -genkey -alias worker-4 -keyalg RSA -storepass st0r3p@ss -keystore worker-4.p12 -storetype pkcs12 -validity 360 -keysize 1024

keytool -genkey -alias satellite-1 -keyalg RSA -storepass st0r3p@ss -keystore satellite-1.p12 -storetype pkcs12 -validity 360 -keysize 1024
keytool -genkey -alias satellite-2 -keyalg RSA -storepass st0r3p@ss -keystore satellite-2.p12 -storetype pkcs12 -validity 360 -keysize 1024
keytool -genkey -alias satellite-3 -keyalg RSA -storepass st0r3p@ss -keystore satellite-3.p12 -storetype pkcs12 -validity 360 -keysize 1024

keytool -export -keystore master-1.p12 -storepass st0r3p@ss -alias master-1 | keytool -import -noprompt -alias master-1 -storepass st0r3p@ss -keystore xld-truststore.p12 -storetype pkcs12
keytool -export -keystore master-2.p12 -storepass st0r3p@ss -alias master-2 | keytool -import -noprompt  -alias master-2 -storepass st0r3p@ss -keystore xld-truststore.p12 -storetype pkcs12
keytool -export -keystore worker-1.p12 -storepass st0r3p@ss -alias worker-1 | keytool -import -noprompt -alias worker-1 -storepass st0r3p@ss -keystore xld-truststore.p12 -storetype pkcs12
keytool -export -keystore worker-2.p12 -storepass st0r3p@ss -alias worker-2 | keytool -import -noprompt -alias worker-2 -storepass st0r3p@ss -keystore xld-truststore.p12 -storetype pkcs12
keytool -export -keystore worker-3.p12 -storepass st0r3p@ss -alias worker-3 | keytool -import -noprompt -alias worker-3 -storepass st0r3p@ss -keystore xld-truststore.p12 -storetype pkcs12
keytool -export -keystore worker-4.p12 -storepass st0r3p@ss -alias worker-4 | keytool -import -noprompt -alias worker-4 -storepass st0r3p@ss -keystore xld-truststore.p12 -storetype pkcs12
keytool -export -keystore satellite-1.p12 -storepass st0r3p@ss -alias satellite-1 | keytool -import -noprompt -alias satellite-1 -storepass st0r3p@ss -keystore xld-truststore.p12 -storetype pkcs12
keytool -export -keystore satellite-2.p12 -storepass st0r3p@ss -alias satellite-2 | keytool -import -noprompt -alias satellite-2 -storepass st0r3p@ss -keystore xld-truststore.p12 -storetype pkcs12
keytool -export -keystore satellite-3.p12 -storepass st0r3p@ss -alias satellite-3 | keytool -import -noprompt -alias satellite-3 -storepass st0r3p@ss -keystore xld-truststore.p12 -storetype pkcs12

Important: When configuring secure communication for your custom scenario, make sure you use strong and unique passwords.

Configure satellites

After you have configured the truststore on a satellite, update the SATELLITE_HOME/conf/satellite.conf configuration file to enable SSL. For example:

xl {
  server {
    hostname = ""
    bind-hostname = "0.0.0.0"
    port = 8380
    streaming {…}
    ssl {
      enabled = yes
      key-store = "keystore"
      key-password = "changeme"
      key-store-password = "changeme"
      trust-store = "truststore"
      trust-store-password = "changeme"
      random-number-generator = "AES128CounterSecureRNG"
    }
  }
  satellite {
    streaming {…}
    timeout {…}
    directory {
      work = "workdir"
      recovery = "recovery"
    }
    metrics {…}
    maintenance {…}
    tasks {
      akka {
        jvm-exit-on-fatal-error = on
      }
    }
  }
  jmx {
    enabled = no
    port = 1099
    ssl {
      enabled = no
    }
  }
}

Configure XL Deploy

This is a sample ssl section of a XL_DEPLOY_SERVER_HOME/conf/xl-deploy.conf file:

xl {
  server {
    # The public hostname of this system.
    # Must be resolvable for all satellites and workers.
    # If empty, the name of the system will be resolved and the returned address will be used.
    # For the worker setup this will be the hostname of a worker.
    # If master and worker are using the same folder, `xl-worker.conf` should be used to configure worker-specific properties (e.g. hostname, port, ssl settings).
    #
    // hostname = ""
    # The host name to listen on.
    # The default setting will listen on all network interfaces.
    # A specific interface can be targeted by name or address.
    #
    // bind-hostname = "0.0.0.0"

    # Use this property to specify worker port
    // port = 8180

    ssl {
      enabled = yes
      # For the worker ssl-enabled setup use this one to set up keystore settings
      key-store = "/Users/sampleuser/xl-software/conf/xld.jks"
      key-password = "xldstorepass"
      key-store-password = "xldkeypass"

      trust-store = "/Users/sampleuser/xl-software/conf/xld-truststore.jks"
      trust-store-password = "st0r3p@ss"

      # Protocol to use for SSL encryption, choose from:
      # Java 7:
      #   'TLSv1.1', 'TLSv1.2'
      // protocol = "TLSv1.2"

      # Example: ["TLS_RSA_WITH_AES_128_CBC_SHA", "TLS_RSA_WITH_AES_256_CBC_SHA"]
      # You need to install the JCE Unlimited Strength Jurisdiction Policy
      # Files to use AES 256.
      # More info here:
      # http://docs.oracle.com/javase/7/docs/technotes/guides/security/SunProviders.html#SunJCEP
      // enabled-algorithms = ["TLS_RSA_WITH_AES_128_CBC_SHA"]

      # (Docs taken from https://doc.akka.io/docs/akka/2.5.16/general/configuration.html)
      # There are two options, and the default SecureRandom is recommended:
      # "" or "SecureRandom" => (default)
      # "SHA1PRNG" => Can be slow because of blocking issues on Linux
      #
      # "AES128CounterSecureRNG" and "AES256CounterSecureRNG" were deprecated in Akka
      # 2.5.16 and if these configuration values are used it will use the default
      # SecureRandom anyway and log a warning. The deprecated implementations are
      # not recommended, but they can be enabled by using configuration values
      # "DeprecatedAES128CounterSecureRNG" or "DeprecatedAES256CounterSecureRNG"
      #
      # Setting a value here may require you to supply the appropriate cipher
      # suite (see enabled-algorithms section above)
      // random-number-generator = ""
    }

    # Timeout for gathering replies from workers
    aggregation-timeout = 5 seconds
  }
}

Enable logging for satellites

To enable logging of secure communications on a satellite, set the SATELLITE_OPTS environment variable before starting the satellite:

export SATELLITE_OPTS="$SATELLITE_OPTS -Djavax.net.debug=all"

To enable logging on XL Deploy, set the DEPLOYIT_SERVER_OPTS variable:

export DEPLOYIT_SERVER_OPTS="$DEPLOYIT_SERVER_OPTS -Djavax.net.debug=all"

Configure satellite for step logs monitoring

You can use two different configurations to setup monitoring of step logs with Elastic Stack while using a satellite.

  1. By streaming of step logs from the satellite

To achieve streaming of step logs from the satellite, you must include the xl.satellite.steplog.type = external parameter in conf/satellite.conf. The default value of xl.satellite.steplog.type is internal. Additionally, you may setup these values to customize initialization-time and steplog queue size respectively:

xl.satellite.steplog.initialization-time=10 seconds
xl.satellite.steplog.queue-size=10000

This ensures that the step logs are immediately streamed to an XL Deploy worker and consequently stored in the Elastic Stack.

  1. Without streaming step logs from the satellite

You can have a different configuration than the one specified above, by choosing not to stream the step log as it is executed.

Note: However, this is not recommended, as in this case XL Deploy will store the step log in memory and send the logs to the XL Deploy server only when the block of steps completes. The value of of xl.satellite.steplog.type should be set to internal. As a result, unlike monitoring the step log using GUI, you will not be able to view the step log information during execution.