Add Replica Node

In ClusterControl, scaling an existing cluster to add more nodes for high availability can be done by adding a replica. Adding a replica in ClusterControl is a common task for improving the resilience, read scalability, and availability of your database systems.

ClusterControl simplifies this process significantly eliminating hassle and manual work. This page will guide you add replica node based on the database cluster that ClusterControl supports.

Support Matrix

The following database cluster types, vendors and topology are supported:

Database	Vendor	Topology
MySQL	Percona, Percona Server for MySQL Pro, Oracle	Standalone, replication
MariaDB	MariaDB	Standalone, replication
Galera Cluster	MariaDB, Percona	Galera certification-based replication
PostgreSQL	PostgreSQL, EnterpriseDB	Standalone, streaming replication
TimescaleDB	TimescaleDB	Standalone, streaming replication
Redis	Redis, Valkey	Sentinel, cluster

Prerequisites

Same as deploying a new cluster, nodes needs to be provisioned based on the requirements that ClusterControl requires in order to have a deployment for adding node successful. See list below:

Make sure the target database nodes are running on a supported architecture platform and operating system. See Hardware and Operating System.
Passwordless SSH (SSH using key-based authentication) is configured from the ClusterControl node to all database nodes. See SSH Key-based Authentication.
Make sure that the OS user you setup for passwordless login is the same user your ClusterControl and its cluster used during deployment.
Verify that sudo is working properly if you are using a non-root user. See Operating System User.

Percona Server for MySQL Pro - 8.0.

Make sure you have acquired the Percona Client ID and Percona Access Token.

PostgreSQL Entperise DB (EDB)

Make sure you have acquired a valid EDB repository token.

MySQL Replication

The following vendors and versions are supported for adding replica node:

Oracle MySQL - 8.0.
Percona Server for MySQL - 8.0.
Percona Server for MySQL Pro - 8.0.
MariaDB Server - 10.4, 10.5, 10.6 (LTS), 10.11 (LTS) and 11.4 (LTS).

Adding a replication requires a running and healthy primary node in the course of action in order to complete the job successfully. When the job is in progress, ClusterControl will copy the data from the primary but this shall not lock the primary node since ClusterControl uses binary backup as explained below:

For Oracle MySQL and Percona Server (either Community edition or Pro), it will use Percona Xtrabackup to stream data from its primary to the target replica node.
For MariaDB Server, it will use Mariabackup to stream data from its primary to the target replica node.

If you are concerned of the traffic and/or the data to be copied from your Primary node is large, it is recommended to use it during low peak period or rebuild your replica node using backup created and taken using ClusterControl. This shall be discussed on how to rebuild from backup in the following section below.

Default configuration

By default, ClusterControl deploys MySQL/MariaDB replication with the following configurations:

ClusterControl uses netcat or nc to stream data using a binary backup (xtrabackup ormariabackup`). It uses port 9999, or if used, it will search between 9990-9998 range and will use the next port available.
MySQL GTID with log_slave_updates enabled (Oracle MySQL and Percona only).
MariaDB GTID with log_slave_updates enabled (MariaDB only).
Replica node shall be configured with read_only=ON and super_read_only=ON (if supported) for protection against accidental write.
Semi-sync replication is enabled
performance_schema is enabled
The generated account credentials are stored inside secrets-backup.cnf under the MySQL configuration directory.
Since ClusterControl will copy its data from the Primary node which means users will be copied as well, ClusterControl will add one user with its IP address or hostname to grant necessary privileges for replication:

Database user Purpose

rpl_user MySQL/MariaDB replication.

Note

If you would like to customize the above configurations, modify the template base file to suit your needs before proceeding to the deployment. See Configuration Template.

Attention

ClusterControl sets read_only=ON on all slaves but a privileged user (SUPER) can still write to a slave (except for MySQL versions that support super_read_only).

Deployment steps

ClusterControl GUIClusterControl CLI

Go to the Clusters dashboard. Choose your target source cluster and click the ellipsis button (...)
Choose Add new → Replica node
Choose Create a replica node
First step is Node configuration.
1. Specify the Port of your target replica node. Default is set to 3306.
2. Data directory is set to /var/lib/mysql. If you have prepared a separate path or on a different volume, replace the value with your custom path.
3. Configuration template is set to my.cnf.repl84. If you have your own custom template, ClusterControl will show the templates that has been added or configured in /etc/cmon/templates. For more details see Configuration template
4. Install software is set to On. You can disable this if you do not want to install the software if the package is already installed. If set to On, ClusterControl will uninstall any existing package found and install the right package and version of the database instead.
5. Security configuration to either disable firewall or SELinux/AppArmor. Leave check if you need to Disable firewall (default). Leave check if you need to Disable SELinux/AppArmor (default). Otherwise uncheck if any of the two needs not be disabled. If you have uncheck Disable SELinux/AppArmor, make sure that you have configured properly to allow creating files and opening sockets for the database to run properly.
Click Continue.
Second step is Advanced Settings.
1. Rebuild from a backup is disabled. If you desire to rebuild your data instead of streaming from the Primary node, toggle it to On. This shall reveal a drop-down list with all full-backup created using Clustercontrol.
2. Delay the replica node is disabled. If you desire to delay the replica especially for data recovery purposes, switch this to On.
3. Include in LoadBalancer set (if exists) is disabled. Switch it to *On if you want ClusterControl to discover any existence of load balancer that ClusterControl supports.
4. Semi-synchronous replication is enabled by default. Switch to Off if you only need the traditional asynchronous replication.
Click Continue.
Third step is Add node.
1. Choose and select your Primary node.
2. In the Node field, this is where you have to specify and input the IP address or hostname of your target replica node. If you intend to add another one, this will just overwrite what was previously inputted and added since it only allows one node at a time.
Click Continue to proceed the last step Preview.
Fourth and the last step is Preview, this is the last step where it shows all the values and choosen options for the ClusterControl to use for setting up the node.
Click Finish button to deploy the job.

To be added soon

Recommended Next Steps

Galera Cluster

The following vendors and versions are supported for adding replica node:

Percona XtraDB Cluster - 8.0.
MariaDB Galera - 10.4, 10.5, 10.6 (LTS), 10.11 (LTS) and 11.4 (LTS).

Adding a replication requires a running and healthy primary node in the course of action in order to complete the job successfully. A galera cluster can have multiple primary nodes so you can choose and assign which is the desired primary node to be used as a primary of the target replica node. When the job is in progress, ClusterControl will copy the data from the primary but this shall not lock the primary node since ClusterControl uses binary backup as explained below:

For Percona XtraDB Cluster, it will use Percona Xtrabackup to stream data from its primary to the target replica node.
For MariaDB Server, it will use Mariabackup to stream data from its primary to the target replica node.

If you are concerned of the traffic and/or the data to be copied from your Primary node is large, it is recommended to use it during low peak period or rebuild your replica node using backup created and taken using ClusterControl. This shall be discussed on how to rebuild from backup in the following section below.

Default configuration

By default, ClusterControl deploys MySQL/MariaDB replication with the following configurations:

ClusterControl uses netcat or nc to stream data using a binary backup (xtrabackup or mariabackup). It uses port 9999, or if used, it will search between 9990-9998 range and will use the next port available.
MySQL GTID with log_slave_updates enabled (Oracle MySQL and Percona only).
MariaDB GTID with log_slave_updates enabled (MariaDB only).
Replica node shall be configured with read_only=ON and super_read_only=ON (if supported) for protection against accidental write.
Semi-sync replication is enabled
performance_schema is enabled
The generated account credentials are stored inside secrets-backup.cnf under the MySQL configuration directory.
Since ClusterControl will copy its data from the Primary node which means users will be copied as well, ClusterControl will add one user with its IP address or hostname to grant necessary privileges for replication:

Database user Purpose

rpl_user MySQL/MariaDB replication.

If you would like to customize the above configurations, modify the template base file to suit your needs before proceeding to the deployment. See Configuration Template.

Attention

ClusterControl sets read_only=ON on all slaves but a privileged user (SUPER) can still write to a slave (except for MySQL versions that support super_read_only).

Deployment steps

ClusterControl GUIClusterControl CLI

Go to the Clusters dashboard. Choose your target source cluster and click the ellipsis button (...)
Choose Add new → Replica node
Choose Create a replica node
First step is Node configuration.
1. Specify the Port of your target replica node. Default is set to 3306.
2. Data directory is set to /var/lib/mysql. If you have prepared a separate path or on a different volume, replace the value with your custom path.
3. Configuration template is set to my.cnf.repl84. If you have your own custom template, ClusterControl will show the templates that has been added or configured in /etc/cmon/templates. For more details see Configuration template
4. Install software is set to On. You can disable this if you do not want to install the software if the package is already installed. If set to On, ClusterControl will uninstall any existing package found and install the right package and version of the database instead.
5. Security configuration to either disable firewall or SELinux/AppArmor. Leave check if you need to Disable firewall (default). Leave check if you need to Disable SELinux/AppArmor (default). Otherwise uncheck if any of the two needs not be disabled. If you have uncheck Disable SELinux/AppArmor, make sure that you have configured properly to allow creating files and opening sockets for the database to run properly.
Click Continue.
Second step is Advanced Settings.
1. Rebuild from a backup is disabled. If you desire to rebuild your data instead of streaming from the Primary node, toggle it to On. This shall reveal a drop-down list with all full-backup created using Clustercontrol.
2. Delay the replica node is disabled. If you desire to delay the replica especially for data recovery purposes, switch this to On.
3. Include in LoadBalancer set (if exists) is disabled. Switch it to *On if you want ClusterControl to discover any existence of load balancer that ClusterControl supports.
4. Semi-synchronous replication is enabled by default. Switch to Off if you only need the traditional asynchronous replication.
Click Continue.
Third step is Add node.
1. Choose and select your Primary node.
2. In the Node field, this is where you have to specify and input the IP address or hostname of your target replica node. If you intend to add another one, this will just overwrite what was previously inputted and added since it only allows one node at a time.
Click Continue to proceed the last step Preview.
Fourth and the last step is Preview, this is the last step where it shows all the values and choosen options for the ClusterControl to use for setting up the node.

To be added soon

Recommended Next Steps

PostgreSQL

The replica will be staged with data from the primary. The primary's configuration will be altered to allow the replica to join the primary. The replica server must be reachable by SSH (key-based auth.) from the controller.

The following vendors and versions are supported for a new deployment:

PostgreSQL from postgresql.org repository - 12, 13, 14, 15, and 16.
PostgreSQL from EDB repository - 12, 13, 14, 15, and 16

Adding a replication requires a running and healthy primary node in the course of action in order to complete the job successfully. When the job is in progress, ClusterControl will copy the data through streaming from the primary node by using pg_basebackup. pg_basebackup is a binary backup which does not lock the primary node. If data is huge and streaming the data to the target replica node can take time, the process can create a significant amount of I/O load and impact performance. If you are concerned of such impact and overhead from your primary, it is recommended to use it during low peak period when adding a replica.

Default configuration

By default, ClusterControl deploys PostgreSQL instances with the following configurations:

Configure and load the pg_stat_statements module.
The WAL level is set to replica.
All replica will be set with hot_standby.
Configure the cluster_name value in postgresql.conf given if Instance name is provided n the GUI under Advanced settngs.
ClusterControl will configure the PostgreSQL instance with SSL encryption for clients and primary-replica connections.
The Disable firewall and Disable SELinux/AppArmor is hidden in the background but functionality is set to enabled.

ClusterControl will create and grant necessary privileges for the following database users:

Database user	Purpose
`cmon`	Management and automation.
`cmonexporter`	Prometheus exporter for database monitoring.
`cmonagent`	ClusterControl query monitoring agent.
`cmon_replication`	PostgreSQL streaming replication.

Deployment steps

ClusterControl GUIClusterControl CLI

Go to the Clusters dashboard. Choose your target source cluster and click the ellipsis button (...)
Choose Add new → Replica node
Choose Create a replica node
First step is Node configuration.
1. Specify the Port of your target replica node. Default is set to 3306.
2. Data directory is set to /var/lib/postgresql/<version>/main in Ubuntu/Debian, whereas /var/lib/pgsql/<version>/data in RHEL-variant Linux OS. If you have prepared a separate path or on a different volume, set Use package default for datadir to Off and specify the custom full path.
3. Install software is set to On. You can disable this if you do not want to install the software if the package is already installed. If set to On, ClusterControl will uninstall any existing package found and install the right package and version of the database instead.
Click Continue.
Second step is Advanced Settings.
1. Include in LoadBalancer set (if exists) is disabled. Switch it to *On if you want ClusterControl to discover any existence of load balancer that ClusterControl supports.
2. Synchronous replication is Off by default. Switch to On to enable synchronous replication.
3. Specify the Instance name which shall be the value of cluster_name in the postgresql.conf. Otherwise, you can leave it empty.
Click Continue.
Third step is Add node.
1. Choose and select your Primary node.
2. In the Node field, this is where you have to specify and input the IP address or hostname of your target replica node. If you intend to add another one, this will just overwrite what was previously inputted and added since it only allows one node at a time.
Click Continue to proceed the last step Preview.
Fourth and the last step is Preview, this is the last step where it shows all the values and choosen options for the ClusterControl to use for setting up the node.
Click Finish button to deploy the job.

To be added soon

Recommended Next Steps

TimescaleDB

The replica will be staged with data from the primary. The primary's configuration will be altered to allow the replica to join the primary. The replica server must be reachable by SSH (key-based auth.) from the controller.

The following vendors and versions are supported for a new deployment:

PostgreSQL from postgresql.org repository - 12, 13, 14, 15, and 16.
PostgreSQL from EDB repository - 12, 13, 14, 15, and 16

Adding a replication requires a running and healthy primary node in the course of action in order to complete the job successfully. When the job is in progress, ClusterControl will copy the data through streaming from the primary node by using pg_basebackup. pg_basebackup is a binary backup which does not lock the primary node. If data is huge and streaming the data to the target replica node can take time, the process can create a significant amount of I/O load and impact performance. If you are concerned of such impact and overhead from your primary, it is recommended to use it during low peak period when adding a replica.

Default configuration

By default, ClusterControl deploys PostgreSQL instances with the following configurations:

Configure and load the pg_stat_statements module.
The WAL level is set to replica.
All replica will be set with hot_standby.
Configure the cluster_name value in postgresql.conf given if Instance name is provided n the GUI under Advanced settngs.
ClusterControl will configure the PostgreSQL instance with SSL encryption for clients and primary-replica connections.
The Disable firewall and Disable SELinux/AppArmor is hidden in the background but functionality is set to enabled.

ClusterControl will create and grant necessary privileges for the following database users:

Database user	Purpose
`cmon`	Management and automation.
`cmonexporter`	Prometheus exporter for database monitoring.
`cmonagent`	ClusterControl query monitoring agent.
`cmon_replication`	PostgreSQL streaming replication.

Deployment steps

ClusterControl GUIClusterControl CLI

Go to the Clusters dashboard. Choose your target source cluster and click the ellipsis button (...)
Choose Add new → Replica node
Choose Create a replica node
First step is Node configuration.
1. Specify the Port of your target replica node. Default is set to 3306.
2. Data directory is set to /var/lib/postgresql/<version>/main in Ubuntu/Debian, whereas /var/lib/pgsql/<version>/data in RHEL-variant Linux OS. If you have prepared a separate path or on a different volume, set Use package default for datadir to Off and specify the custom full path.
3. Install software is set to On. You can disable this if you do not want to install the software if the package is already installed. If set to On, ClusterControl will uninstall any existing package found and install the right package and version of the database instead.
Click Continue.
Second step is Advanced Settings.
1. Include in LoadBalancer set (if exists) is disabled. Switch it to *On if you want ClusterControl to discover any existence of load balancer that ClusterControl supports.
2. Synchronous replication is Off by default. Switch to On to enable synchronous replication.
3. Specify the Instance name which shall be the value of cluster_name in the postgresql.conf. Otherwise, you can leave it empty.
Click Continue.
Third step is Add node.
1. Choose and select your Primary node.
2. In the Node field, this is where you have to specify and input the IP address or hostname of your target replica node. If you intend to add another one, this will just overwrite what was previously inputted and added since it only allows one node at a time.
Click Continue to proceed the last step Preview.
Fourth and the last step is Preview, this is the last step where it shows all the values and choosen options for the ClusterControl to use for setting up the node.
Click Finish button to deploy the job.

To be added soon

Recommended Next Steps

Redis/Valkey Cluster

The following vendors and versions are supported for adding replica node:

Redis Cluster - 6 and 7
Valkey Cluster - 7 and 8 (only RHEL 8/9 are supported)

Adding a replication requires a running and healthy primary node in the course of action in order to complete the job successfully. When the job is in progress, ClusterControl will setup the replica running full sync or incremental resynchronization running and executing CLUSTER MEET and CLUSTER REPLICATE commands. These commands update cluster metadata and replication roles without causing the master to become unresponsive.

Default Configuration

By default, ClusterControl deploys Redis cluster instances with the following configurations:

ClusterControl will configure the Redis/Valkey instances with appendonly enabled.
ClusterControl will secure the instance with AUTH enabled and configure the requirepass and masterauth options.
ClusterControl will enable TLS encryption. To access the cluster, one must use the --tls flag to connect.
The configuration maxMemory (70% of node's RAM, rounded to the nearest power of 2) and maxMemoryPolicy=allkeys-lru will be set, to reduce the risk of Redis being killed by OOM.

Deployment steps

ClusterControl GUIClusterControl CLI

Go to the Clusters dashboard. Choose your target source cluster and click the ellipsis button (...)
Choose Add new → Replica node
First step is Node configuration.
1. Specify the Port of your target replica node. Default is set to 6379.
2. Install software is set to On. You can disable this if you do not want to install the software if the package is already installed. If set to On, ClusterControl will uninstall any existing package found and install the right package and version of the database instead.
3. Security configuration to either disable firewall or SELinux/AppArmor. Leave check if you need to Disable firewall (default). Leave check if you need to Disable SELinux/AppArmor (default). Otherwise uncheck if any of the two needs not be disabled. If you have uncheck Disable SELinux/AppArmor, make sure that you have configured properly to allow creating files and opening sockets for the database to run properly.
Click Continue.
Third step is Add node.
1. Choose and select your Primary node.
2. In the Node field, this is where you have to specify and input the IP address or hostname of your target replica node. If you intend to add another one, this will just overwrite what was previously inputted and added since it only allows one node at a time.
Click Continue to proceed the last step Preview.
Fourth and the last step is Preview, this is the last step where it shows all the values and choosen options for the ClusterControl to use for setting up the node.
Click Finish button to deploy the job.

To be added soon

Recommended Next Steps

Post Deployment

Once a replica is added in the cluster, ClusterControl will immediately install Prometheus exporters on the database node and add it into the Prometheus monitoring list.