Improve file.Folder deployment performance

Suppose you are doing a deployment where one of the deployables is a file.Folder or any type derived from this. As part of the deployment, placeholders will be replaced in each of the files contained in the folder, and then the files are transferred to a temporary directory on the target host before moving them to their final deployment destination.

The legacy way of doing this is by copying each of the files one-by-one. While this works well, it can be slow when there are many files to copy since each file has some connection overhead. As of 9.7, XL Deploy provides additional copy strategies to speed up this proces.

Copy strategies

These are the available strategies:

  • OneByOne - the default, legacy strategy of copying files one by one
  • Tar - before copying the files, bundle them up into a tarball, copy over the tarball to the target system in one go, and untar it on the remote system by invoking tar -xf <tarball> -C <tempdir>
  • ZipWindows - before copying the files, bundle them up into a zip archive, copy over the zip archive to the target system in one go, and unzip it using built-in powershell capabilities.

    Note: The ZipWindows strategy uses ExtractArchive, which is part of PowerShell 5.0 and higher. If you are running a Windows Server version older than Windows Server 2016, we recommend you upgrade to PowerShell 5.0 to take advantage of this feature. More information can be found here: Windows Powershell System Requirements.

  • ZipUnix - similar to ZipWindows, but unzip it using a command-line invocation unzip <archive> -d <tempdir>

Selecting a copy strategy

In order to be backwards compatible, XL Deploy will default to the legacy OneByOne strategy.

This behaviour can however be overridden on a per-host basis. Any overthere.Host CI has a new property called Copy Strategy inside a new Zip section, that allows you to select which strategy is to be used for deploying file.Folder CIs to this host. Note that this is an optional value.

Selecting a copy strategy

Detection of unzip capabilities

XL Deploy can detect which unzip/untar capabilities the target host has. This behaviour is turned off by default since this incurs a small detection overhead, but it can be enabled by setting xl.task.artifact-copy-strategy.autodetect = true in conf/xl-deploy.conf.

Enabling autodetection

When set to true, copy strategies are tried one by one, until one succeeds. For Windows target hosts, the try order is: ZipWindows, Tar, ZipUnix, OneByOne. For Unix hosts, the try order is: Tar, ZipUnix, ZipWindows, OneByOne. A test zip or tar archive will be copied to a temparary directory on the target host and the respective unzip/untar commands are tried. If these fail, the next strategy is tried; if it succeeds then the current strategy under test is picked for the deployment of the file.Folder.

Logging

To have a better look under the hood, configure conf/logback.xml to enable DEBUG logging on the com.xebialabs.deployit.io namespace

Enabling logging

Example log output looks like this (with autodetect enabled, deploying to a Windows host):

Example log output