Secondary Standby

ClusterControl can be deployed in a couple of different ways for redundancy and high availability:

Secondary standby - Acts as a hot standby in case the primary ClusterControl host goes down.
CMON Controller Pool - A scalable pool of controllers designed for demanding, multi-site and high-growth environments.
ClusterControl Replication - A high-availability deployment using replication with a primary-secondary architecture.

Secondary Standby

It is possible to have more than one ClusterControl server to monitor the same cluster. This is useful if you have a multi-datacenter cluster and you may need to have ClusterControl on the remote site to monitor and manage the alive nodes if the connection between them goes down. However, ClusterControl servers must be configured to be working in active-passive mode to avoid race conditions when recovering failed nodes or clusters.

In the active mode, the ClusterControl node acts as a primary controller, where it performs automatic recovery and management activities. Therefore the primary controller Cluster/Node Auto Recovery must be set to on. The secondary ClusterControl node however must be configured with Cluster/Node Auto Recovery turned off.

Installing standby server

The steps described in this section must be performed on the secondary ClusterControl server.

Install ClusterControl as explained on the Quickstart.
Import the same cluster via ClusterControl GUI → Deploy a cluster → Import a database cluster. Ensure to toggle off Cluster auto-recovery and Node auto-recovery in the Node configuration section. Repeat this step if you want to import more than one cluster.

Example
Set up the cluster configuration accordingly to follow similar settings with the primary ClusterControl (backup schedules, alerting configuration, user roles, etc).

At this point, the architecture should be looking like this:

flowchart TD
    U[/DBA/<br>SysAdmin/] --> A[ClusterControl A<br><i><strong>Active</strong></i><br>#40;auto-recovery enabled#41;]
    B[ClusterControl B<br><i><strong>Standby</strong></i><br>#40;auto-recovery disabled#41;]
    A ==> |manage & monitor|D[[Database/<br>Load balancer nodes]]
    B ==> |monitor|D[[Database/<br>Load balancer nodes]]

Nothing should be performed on the primary side. The primary ClusterControl server shall perform automatic recovery in case of node or cluster failure. Use the secondary ClusterControl server for monitoring purposes only. For management and recovery purposes like rebuilding the replication, resyncing the node, backup and restore, perform those activities on the primary ClusterControl server.

Info

You don't need an additional ClusterControl license for multiple ClusterControl instances. You can apply the same license as your primary ClusterControl server onto the secondary server. The license is bounded on the number of database/load balancer nodes it manages.

Activating the secondary standby

If you want to make the standby server runs in the active mode, you must do the following:

If the primary ClusterControl server is still alive, stop the primary ClusterControl controller services, or shutdown the server. To stop all ClusterControl processes, run the following command on the primary ClusterControl server:

systemctl stop cmon cmon-cloud cmon-ssh cmon-events

Toggle on Cluster auto-recovery and Node auto recovery on the secondary ClusterControl server.

Example

At this point, the standby server has taken over the primary role and you can perform the management activities on the database nodes or clusters. We can visualize the current topology as below:

flowchart TD
    A[ClusterControl A<br><i><strong>Offline</strong></i><br>#40;auto-recovery disabled#41;]
    U[/DBA/<br>SysAdmin/] --> B[ClusterControl B<br><i><strong>Active</strong></i><br>#40;auto-recovery enabled#41;]
    A ==> D[[Database/<br>Load balancer nodes]]
    B ==> |manage & monitor|D[[Database/<br>Load balancer nodes]]

Attention

Do not let two or more ClusterControl instances perform automatic recovery to the same cluster at a given time.