Upgrades and Patches

Performs database and load-balancer-related package upgrades.

ClusterControl upgrades database and load balancer packages by utilizing the vendor's package repository. The upgrade process depends on the availability of updated packages in this repository. If the vendor repository is outdated or inaccessible (due to network restrictions or proxy configurations), ClusterControl cannot perform the upgrade as new packages will not be available in the server's repository list.

Warning

A database upgrade is a risky operation and ClusterControl does not support an upgrade rollback operation. Please perform this activity on a test/staging cluster first before proceeding with the actual upgrade. Upgrades should only be performed when there is as little traffic as possible on the cluster. Performing a full database backup (or taking a full VM snapshot) before an upgrade will increase the chance of recovery in case a rollback is needed. Taking a backup (before the upgrade) or restoring a backup (to rollback) is not part of this particular upgrade job.

Upgrade procedures

Database cluster upgrade procedures vary based on the cluster's topology, as detailed below.

Primary-replica replication

ClusterControl upgrades primary-replica setups by updating replica/secondary nodes first (in random order), followed by the primary node. Expect a service interruption during the primary node upgrade until it restarts.

Attention

ClusterControl does not perform automatic switchover before upgrading the primary node, which is always upgraded last. Always test upgrade procedures in a non-production environment before applying them to production databases.

Multi-primary

In a multi-primary setup, ClusterControl upgrades database nodes randomly, one node at a time. If an upgrade fails, the entire process stops, necessitating manual intervention for node recovery or reinstallation. Consequently, a suitable maintenance window with minimal cluster traffic is crucial for upgrades.

Tip

If database connections are load-balanced, performing the entire upgrade without database service downtime may be possible. Always test upgrade procedures in a test environment before applying them to a production database.

Minor upgrade

Performs minor version software upgrades for database and load balancer software. Minor upgrade version formatting is notably different depending on the database type and vendor (from x to y):

MySQL/Percona Server for MySQL – 8.0.x to 8.0.y
MariaDB – 11.4.x to 11.4.y
MongoDB – 7.0.x to 7.0.y
PostgreSQL/TimescaleDB/EntepriseDB – 16.x to 16.y
Redis Cluster/Sentinel – 7.x to 7.y
ProxySQL – 2.6.x to 2.6.y
PgBouncer – 1.22.x to 1.22.y
HAProxy – 2.1.x to 2.1.y
MariaDB MaxScale – 22.08.x to 22.08.y

Checking for available upgrades

To determine if a newer version is available for your database or load balancer, follow the instructions below for either the ClusterControl GUI or CLI.

ClusterControl GUIClusterControl CLI

Navigate to ClusterControl GUI → Clusters
Select the desired cluster from the list.
Go to Actions (...) → Upgrades → Minor upgrades.
ClusterControl will display a list of database and load balancer nodes within the cluster that can be upgraded.
Click Check for upgrades. This initiates a background job to query the package repositories of the database and load balancer nodes for the latest available versions.
Once the job is complete, verify if a new version is available.

To initiate a check for newer package versions on cluster ID 17 and wait for the process to finish, execute the following command:
```
s9s cluster --check-pkg-upgrades --cluster-id=17 --wait
```

Performing minor version upgrade

Minor upgrades are performed online on the selected node(s), one node at a time. The node will be stopped, then the software will be updated, and then the node will be started again. If a node fails to upgrade, the upgrade process is aborted and manual intervention is required to recover or reinstall the node.

To perform a minor version upgrade, either for the database cluster or load balancer, or both, follow the instructions below for either the ClusterControl GUI or CLI.

ClusterControl GUIClusterControl CLI

Navigate to ClusterControl GUI → Clusters
Select the desired cluster from the list.
Go to Actions (...) → Upgrades → Minor upgrades.
ClusterControl will display a list of database and load balancer nodes within the cluster that can be upgraded.
Check all the boxes for the nodes that you want to upgrade. Clicking on the top checkbox will automatically check all the boxes underneath it.
Select Upgrade. ClusterControl will start an upgrade process even if a newer version is not available. If no upgrade is needed, the job will check package versions and report "Nothing to upgrade."
Monitor the job log messages closely.

Update database and load balancer packages to the newest minor version across all nodes in cluster ID 1:
```
s9s cluster \
    --upgrade-cluster \
    --cluster-id=1 \
    --wait
```

Update database packages to the newest minor version for all MySQL database nodes in cluster ID 1:

s9s cluster \
    --upgrade-cluster \
    --cluster-id=1 \
    --nodes="192.168.0.81:3306;192.168.0.82:3306;192.168.0.83:3306" \
    --wait

Major upgrade

Info

ClusterControl only supports PostgreSQL-based clusters for major version upgrade, for example from PostgreSQL 16.x to PostgreSQL 17.x.

Performs major software upgrades for the database software. Only one major version upgrade is supported at a time, for example, from PostgreSQL 13.x to 14.x. At the moment, only PostgreSQL-based clusters are supported. If you want to upgrade from PostgreSQL 12.x to 14.x, you have to upgrade stage-by-stage to PostgreSQL 13.x first, followed by PostgreSQL 14.x.

For a primary-replica replication setup, ClusterControl will perform the upgrade starting from the replica/secondary nodes and will eventually perform the upgrade on the primary node of a cluster (secondary first, primary last). During the eventual primary node upgrade, expect a service disruption to the database cluster service until the primary node is online again after restarting. If a node fails to be upgraded, the upgrade job is aborted and manual intervention is required to recover or reinstall the node. Due to this fact, it is important to have a proper maintenance window, which should only be performed when there is as little traffic as possible to the cluster.

Performing major version upgrade

Major upgrades are performed online on all database nodes in the cluster, one node at a time. The node will be stopped, then the software will be updated, and then the node will be started again. If a node fails to upgrade, the upgrade process is aborted and manual intervention is required to recover or reinstall the node.

Attention

Depending on the upgrade method chosen, you might require the storage capacity to hold 2 data sets.

To perform a major version upgrade for the database cluster, follow the instructions below for either the ClusterControl GUI or CLI.

ClusterControl GUIClusterControl CLI

Navigate to ClusterControl GUI → Clusters
Select the desired cluster from the list. Only PostgreSQL-based clusters are supported.
Go to Actions (...) → Upgrades → Major upgrades.
Under the Configuration tab, ClusterControl will display the database vendor, current major version and the upgrade major version. This is informational and not configurable. ClusterControl only allows one major version upgrade at a time.
Choose one of the upgrade method from the dropdown:
- copy – This is the default method with pg_upgrade. The --copy option is used which will copy "files at a time" which is faster than doing regular backup and restore. This requires disk storage capacity to hold both the old and the new files. Note that data stored in user-defined tablespaces is not being copied to the new cluster during an upgrade. It remains in its original file system location but is duplicated into a subdirectory indicating the version number of the new cluster. To manually relocate files that are stored in a tablespace after upgrading, move the files to the new location and update the symbolic links to point to the files.
- link – This method uses the --link option with pg_ugprade. Hard links will be used instead of copying (--copy) files which is faster and requires no extra disk storage. It is crucial to refrain from starting the old cluster after this step to avert any potential data corruption with the new cluster since they share the data.
- pgdumpall – This is an alternative method to pg_upgrade and uses pgdumpall to copy files by performing a backup on the old cluster and then restoring it on the new cluster. This requires disk storage capacity to hold the backup for the old and new files.
Accept the disclaimer to proceed with the upgrade.
Under the Advanced tab, configure the following:
- Temporary master port - The temporary PostgreSQL port of the older version. ClusterControl will reconfigure the PostgreSQL port of the older instances to this port for the duration of the upgrade, since the common port 5432, will be taken over by the newer PostgreSQL instance after the upgrading job completes.
- Temporary upgrade candidate port - The temporary PostgreSQL port of the newer version while the upgrade job is running. ClusterControl distinguishes every node with a “hostname:port” pair. This provides a temporary entity for the newer database instance for the duration of the upgrade before it takes over the actual PostgreSQL port. This port will be released after the upgrade job is completed.
- Delete old node - Toggle on to delete the older PostgreSQL instance data if the upgrade succeeds. Note that all previous data will be deleted and this action cannot be rolled back.
- Preserve configuration settings - When enabled, important PostgreSQL configuration parameters will be preserved during major version upgrades instead of using default values for the new version.
Proceed to the Preview section and click Submit. ClusterControl will trigger a major upgrade job and the progress can be monitored under ClusterControl GUI → Activity Center → Jobs.

Perform major upgrade from PostgreSQL 14 to PostgreSQL 15 for cluster ID 21:

s9s cluster \
    --upgrade-cluster \
    --upgrade-to-version=15 \
    --cluster-id=21 \
    --log

Perform major upgrade from PostgreSQL 15 to 16 for cluster ID 1, preserve the configuration and use upgrade method --link:

s9s cluster \
    --upgrade-cluster \
    --upgrade-to-version=16 \
    --upgrade-method=link \
    --preserve-configuration \
    --cluster-id=1 \
    --log