Manage deployments in XL UP (BETA)

This topic will enable you to perform functions such as upgrading and destroying instances of the platform that were deployed from XL UP.

The following actions can be done on active deployments from XL UP:

Updating a deployment

XL UP will automatically detect during the setup if you have an existing instance of the DevOps Platform, and will ask if you want to update it, and noting that any changes will be irreversible.

? DevOps Platform is already deployed on this cluster. Are you sure you want to update? This cannot be undone (y/N) y

You will then be asked if you want to change the existing configurations. If you select No, you can only upgrade the platform version.

If you select Yes, you will then be asked if you want to use a custom Docker Registry and custom images:

Next, select the version of Deploy and then Release to upgrade. See Upgrading versions for more information on upgrading.

The setup will ask you to provide new values for every parameter that can be updated, and will also show you the current value for that parameter:

? Enter the number of Deploy master to spin up 2 ? Enter the desired RAM requests for the Deploy Master pods [? for help] (1600Mi)

For more information, see Parameters that can be updated.

Confirm the updates and press Enter. The platform will un-deploy and re-deploy with the new parameter values, while preserving your data.

Note that the updated parameters can also be supplied with an answer file. If you provide an answer file in the command, you will be presented with the values and asked if you want to proceed to deployment with them.

Parameters that can be updated

The following parameters can be updated:

  • UseCustomRegistry
  • RegistryURL
  • DockerUser
  • DockerPass
  • InstallXLD (only if Deploy is not already deployed)
  • XldVersion
  • XldTag
  • InstallXLR (only if Release is not already deployed)
  • XlrVersion
  • XlrTag
  • XldMasterCount
  • XldMasterRAMRequest
  • XldMasterRAMLimit
  • XldMasterCPURequest
  • XldMasterCPULimit
  • XldWorkerCount
  • XldWorkerRAMRequest
  • XldWorkerRAMLimit
  • XldWorkerCPURequest
  • XldWorkerCPULimit
  • XldLic
  • RabbitMQUrl
  • RabbitMQDriver
  • RabbitMQUsername
  • RabbitMQPassword
  • XlrLic
  • XlrDbUrl
  • XlrDbName
  • XlrDbUser
  • XlrDbPass
  • XlrReportDbUrl
  • XlrReportDbName
  • XlrReportDbUser
  • XlrReportDbPass
  • XlrRAMRequest
  • XlrRAMLimit
  • XlrCPURequest
  • XlrCPULimit
  • MonitoringInstall(monitoring uninstalling is not possible with the update)
  • MonitoringDataRetention

If you are using an external database you can update these parameters:

  • XldDbUrl
  • XldDbUser
  • XldDbPass
  • XlrDbUrl
  • XlrDbUser
  • XldDbPass
  • XlrReportDbUrl
  • XlrReportDbUser
  • XlrReportDbPass

If you are using an external JMS queue you can update these parameters:

  • RabbitMQUrl
  • RabbitMQDriver
  • RabbitMQUsername
  • RabbitMQPassword

Rolling updates

By default, XL UP undeploys Deploy and Release before an update for any changes made to the configuration. This behaviour can be changed by passing the --rolling-update flag in the command line. This will perform a rolling update for Deploy/Release without any downtime, and without undeploying and redeploying the instances.

Rolling updates can only be done in the following circumstances, as it could cause issues when there are schema changes between version updates:

  • Version updates without any schema changes between versions. Please refer to the release notes for the product version you want to upgrade to, to make sure that there are no schema changes in the new version.
  • Scaling up workers/masters without changing versions in Deploy
  • Scaling up Release without changing versions
  • Switching to custom containers for the same version with additional plugins/configurations.

Upgrading versions

The xl up steps include an option to select the version of each product. If your current version is behind the newest available version in the CLI, you can select a newer version and run the deployment again. This will upgrade the instances of Deploy or Release to the version specified, while preserving your data.

Note: the upgrade is not a rolling upgrade, but undeploys the prior version and deploys the newer version over it, while preserving the data in the database. Therefore you should plan for downtime while the upgrade is in process.

Troubleshooting upgrade issues

If something goes wrong during an upgrade you may be left with a corrupted environment.

  • If both Deploy and Release are deployed, and Deploy fails during upgrade, then both Deploy and Release will be removed.
  • If both Deploy and Release are deployed, and Deploy succeeds but Release fails during upgrade, then Deploy will remain active but Release will be removed.

Once you have investigated the problem, you will need to rerun XL UP using the same upgrade versions that previously failed. You will no longer be able to roll back to the previously working version, and the command line will only provide the later versions for deployment.

Undeploying instances

The --undeploy CLI flag will remove the current deployment. The command will ask you to specify the Kubernetes cluster to be uninstalled, and will completely remove it from the target environment. Here is an example of an undeployment from a local Docker instance:

Example

➜ xl up --undeploy
? Select the Kubernetes setup where the DevOps Platform will be installed: LocalK8S [Local K8s from Docker Desktop for Mac/Windows]
? Do you want to use Kubernetes' current-context from ~/.kube/config? Yes
? DevOps Platform is already deployed on this cluster. Are you sure you want to undeploy everything? This cannot be undone Yes
Deleting namespace...
	Resource "Namespace" still deleting
	Resource "Namespace" still deleting
	Resource "Namespace" still deleting
	Resource "Namespace" still deleting
	Resource "Namespace" still deleting
	Resource "Namespace" deleted
Namespace deleted
Deleting storage classes...
	Resource "StorageClasses" deleted
Storage classes deleted
Deleting cluster role bindings...
	Resource "ClusterRoleBindings" deleted
Cluster role bindings deleted
Deleting cluster roles...
	Resource "ClusterRoles" deleted
Cluster roles deleted
Everything has been undeployed!

In cases where a deployment was unsuccessfully completed, the --undeploy flag will also remove all deployed items from the target environment. It will still ask you to specify the Kubernetes cluster and remove all remaining artefacts from it.

Note: When undeploying, some data volumes (PVCs) will be left on the Kubernetes cluster for backup purposes. This is because XL UP uses reclaimPolicy: Retain for all storage classes that XL UP creates. This might be different if user-defined storage classes were used during the deployment.

These will remain on your system until you actively remove them. For more information, see Persistent Volumes - Retain.

Monitoring

XL UP automatically provisions a set of reporting tools during deployment to provide visibility into the health of your systems. Monitoring tools are only provided for AWS EKS, Google Kubernetes Engine, and multinode Kubernetes.

The dashboards will have the same access credentials supplied during the XL UP setup. At the end of the deployment from XL UP, the CLI will give the address of everything that was successfully deployed, including the monitoring tools. For example, you will see something like the following:

***********************************************************************
XL-DEPLOY URL - https://SOME_IP:30443/deploy/
XL-RELEASE URL - https://SOME_IP:30443/xl-release/
KIBANA URL - https://SOME_IP:30443/kibana/
GRAFANA URL - https://SOME_IP:30443/grafana/
***********************************************************************

The set of monitoring tools installed with XL UP is:

For monitoring

Kibana with EFK Stack and Elasticsearch, Fluentd.

The Kibana dashboard contains all system logs for the deployed modules. You can filter for each module and pod to view the logs for that component.

Kibana dashboard

Querying from Kibana

The simplest form of queries in Kibana is called a Lucene Query. For example, to find Deploy logs for a failed deployment, you can execute the following query directly in Kibana:

log:"com.xebialabs.platform.script.jython.JythonException: Error while executing script" AND kubernetes.namespace_name.keyword:"xebialabs"

This can be added to the Search bar at the top of the default dashboard, or on the Discover tab, as pictured below. You can open one of the results, and then optionally click on View surrounding documents to get more context about a specific log entry.

search bar

To find logs for a specific time, for example the last hour, click on the link in the top right corner of the dashboard (see image below), and select Last 1 Hour.

time search

Of course, Elasticsearch is much more powerful and you can tailor dashboards and searches by using the more advanced query/filter options available in the Elasticsearch Query DSL

For metrics

Grafana dashboard, consisting of Grafana, Prometheus, and Kube State Metrics.

The Grafana dashboard displays a real-time view of the resource usage of Deploy and XL Resource. You can view the CPU and ram usage, the threads used, and the PODs and containers.

Grafana dashboard

Querying from Grafana

The Grafana implementation provided by the xl up command hooks into the deployed Prometheus instance to aggregate and visualize metric data coming from the XL DevOps Platform.

The default XebiaLabs Monitoring dashboard gives insights on the JVM and OS level metrics coming from Deploy and Release, as well as additional metrics on the provisioned resources. Feel free to customize this dashboard according to your needs.

As with Kibana, Grafana can also show statistics for a certain time period, as pictured below.

grafana time search

Refer to the documentation for the monitoring tools for more information on how to use them: