ClickHouse Replicated Cluster

This guide will take you through the process of setting up your first replicated ClickHouse cluster using ClusterControl. By following this tutorial, you’ll have a fully functional, multi-node ClickHouse cluster that keeps your data in sync across every node, so the loss of a single node doesn’t cost you data or availability.

ClickHouse is a column-oriented database management system (DBMS) for online analytical processing (OLAP). In a replicated cluster, two or more ClickHouse servers hold the same copy of your data. Replication is asynchronous and multi-master, so you can write to any node and the change is propagated to the others. Keeping track of which parts each replica already has and which it still needs to fetch is the job of ClickHouse Keeper, a Raft-based coordination service that replaces ZooKeeper.

Compared with a single-node deployment, a replicated cluster gives you data redundancy and lets you take a node down for maintenance without an outage. It is the right starting point whenever downtime or data loss is not acceptable.

Note

ClusterControl deploys a single-shard, multi-replica topology: every data node holds a full copy of the data. By default, ClickHouse Keeper is embedded on each data node (a co-located Raft participant), so you do not need to provide separate hosts to get a working replicated cluster. If you prefer to isolate coordination from your data nodes, you can add dedicated Keeper nodes during deployment instead. For a production Raft quorum, use an odd number of Keepers (3 or 5); a 2-node cluster works but has no fault tolerance for the quorum. ClusterControl also deploys ClickHouse in TLS-only mode, so the plaintext native (9000) and HTTP (8123) ports are disabled and clients connect over the secure ports.

See also

• ClickHouse: Data Replication
• ClickHouse Keeper

Prerequisites

Before proceeding, ensure you have:

• ClusterControl installed and running. If not, follow the instructions in Quickstart or use the Installer Script.
• At least four hosts (bare-metal or virtual machines):
  • One for the ClusterControl server.
  • Three for the ClickHouse data nodes. A three-node cluster gives you both data redundancy and a healthy Keeper quorum. See Operating System for supported distributions.
• SSH access to all servers.
• Internet access on the database hosts to install the required packages from the ClickHouse repositories.
• Network Time Protocol (NTP) configured and running on all hosts to keep their clocks synchronized. This matters more in a replicated cluster, where the nodes coordinate with each other.

Note

ClusterControl only supports managing x86-64 (64-bit) hosts for ClickHouse.

Architecture

Below is a simplified diagram of the final architecture for a three-node replicated deployment:

%%{init: {"themeVariables": {"fontSize": "11px"}, "flowchart": {"rankSpacing": 52, "nodeSpacing": 31}}}%%
flowchart TD
    A{{**Users/<br>clients/<br>apps**}} --> |SQL over TLS<br>native 9440 / HTTPS 8443| B1
    A --> B2
    A --> B3
    E[/**ClusterControl<br>192.168.40.5**/] -.- |monitors/manages| B1
    E -.- B2
    E -.- B3
    subgraph shard[" "]
        subgraph n1["192.168.40.20"]
            B1[**ClickHouse<br>Server**] --- K1[(Embedded<br>Keeper)]
        end
        subgraph n2["192.168.40.21"]
            B2[**ClickHouse<br>Server**] --- K2[(Embedded<br>Keeper)]
        end
        subgraph n3["192.168.40.22"]
            B3[**ClickHouse<br>Server**] --- K3[(Embedded<br>Keeper)]
        end
        K1 <-.->|Raft| K2
        K2 <-.->|Raft| K3
        K1 <-.->|Raft| K3
    end

ClusterControl deploys ClickHouse without any external supporting load balancers. All three data nodes belong to one shard with internal_replication enabled, so each node stores a complete copy of the data. The embedded Keepers form a Raft ensemble and coordinate replication between the nodes.

On the diagram above, the three ClickHouse nodes will use the following addresses for this example deployment:

• Node 1: 192.168.40.20 (data + embedded Keeper)
• Node 2: 192.168.40.21 (data + embedded Keeper)
• Node 3: 192.168.40.22 (data + embedded Keeper)

Step 1: Set up SSH key-based authentication

1. On the ClusterControl server, generate a new SSH key as the root user:

   ssh-keygen -t rsa
   

   Copy the public key to your first ClickHouse node (replace 192.168.40.20 with your node’s IP/hostname):

   ssh-copy-id -i /root/.ssh/id_rsa [email protected]
   

   If the target node uses a custom SSH key or port, you can add options:

   ssh-copy-id -i /root/.ssh/id_rsa -p 22 -o 'IdentityFile /root/myprivatekey.pem' [email protected]
   

   For some advanced setups where the user (for example mymainacc) is only allowed non-root access with public keys, password challenge is disabled, but the user has sudo privileges, you can copy the root public key as the root OS user like this:

   [root@ccnode ~]# ssh -i /home/mymainacc/.ssh/id_rsa [email protected] "sudo bash -c '
   umask 077
   mkdir -p   /root/.ssh
   cat >>     /root/.ssh/authorized_keys
   '" < ~/.ssh/id_rsa.pub
   

   This copies your root public key to the target node you will be setting up for the replicated ClickHouse deployment.

2. Test passwordless SSH from the ClusterControl server:

   ssh [email protected] "stat \$PWD/"
   

   Ensure there is no password prompt. If the command returns the directory status, you're set.

3. Repeat steps 1 and 2 for the remaining nodes, 192.168.40.21 and 192.168.40.22. ClusterControl needs passwordless SSH to every node it manages.

Step 2: Deploy a new cluster (replicated)

1. Open a web browser and go to the ClusterControl server’s IP or hostname.

2. On the ClusterControl dashboard, click Deploy a cluster (top-right) → Create a database cluster. This opens the Deploy cluster wizard.

3. Select ClickHouse from the Database dropdown. Specify the version you want to deploy by clicking the Version dropdown (for example, 24.8 or 25.3). Click Continue.

4. In the Deploy ClickHouse cluster wizard, configure the database cluster as below:

   (1) Cluster details(2) SSH configuration(3) Node configuration(4) Add nodes(5) Preview

   • Name: For example, ClickHouse-replicated. This name also becomes the cluster identifier used by ON CLUSTER statements later on, so pick something without spaces if you can.
   • Tags: (Optional) e.g., replicated, production, dc1.

   • SSH user: root
   • SSH user key path: /root/.ssh/id_rsa (ClusterControl will also autofill this field)
   • SSH port: 22 (default port)
   • SSH sudo password: (leave blank if you rely on key-based auth)
   • SSH sudo / OS elevation command: Choose either sudo (default), doas, or pbrun
   • Install software: On (default)
   • Disable firewall: Checked (default)
   • Disable SELinux/AppArmor: Checked (default)

   • Server port: 9440 (default). This is the ClickHouse native (secure TCP) port.
   • Keeper port: 9281 (default). This is the ClickHouse Keeper client port.
   • Admin user: admin (default)
   • Admin password: Password to be assigned to the database admin user (minimum 6 characters). The admin user is created on all nodes via CREATE USER ... ON CLUSTER.
   • Data directory: /var/lib/clickhouse (default)

   • ClickHouse node: Fill in the IP/hostname or FQDN of each data node — 192.168.40.20, 192.168.40.21, and 192.168.40.22 — pressing Enter after each one. Specifying two or more data nodes tells ClusterControl to deploy a replicated cluster with embedded Keeper on each node.
   • Keeper node: (Optional) Leave this empty to use embedded Keeper on the data nodes. Only fill it in if you want dedicated Keeper hosts separate from your data nodes; if so, add an odd number (3 or 5) for a proper quorum.
   • Wait until every node turns green. If a red warning appears, inspect the error and fix it before continuing. You can only proceed once all nodes are reachable.

   • Review your configuration. You can go back and adjust any previous section if necessary. The deployment settings are kept until you exit the wizard.

5. Click Finish to start deployment.

6. ClusterControl will now install and configure the three ClickHouse nodes, wire up the embedded Keepers into a Raft ensemble, and configure replication between the nodes. You can track progress in the Activity Center → Jobs. After a few minutes, your new replicated cluster will appear on the Home page.

Step 3: Monitor your cluster

Once deployed, you’ll see:

• Cluster health: The Home page provides the overall cluster state across all three nodes.
• Node health: Hover over the honeycomb diagram or check the Nodes tab. You can also see more detailed histograms under ClusterControl GUI → Clusters → choose the cluster → Dashboards.
• Recent alarms: Any triggered alarms will appear if there are configuration or resource issues.
• Automatic recovery status: If enabled, ClusterControl can attempt to restart a crashed ClickHouse server automatically.
• Topology viewer: You’ll see all three replicas and their Keeper participants.

To confirm the replicated topology from the database side, connect to any node and inspect the system.clusters table. It lists every replica in your cluster along with the cluster identifier you set during deployment:

clickhouse-client --host 192.168.40.20 --port 9440 --secure \
    --user admin --password 'myPassw0rd' \
    --query "SELECT cluster, shard_num, replica_num, host_name FROM system.clusters"

Step 4: Import data

In a replicated cluster you don’t load data into each node separately. Instead, you create a table using the ReplicatedMergeTree engine (typically with ON CLUSTER so the DDL runs on every node at once), then insert into any single node — ClickHouse and Keeper take care of copying the data to the other replicas. All examples connect over the secure native port (9440) using TLS, since ClusterControl deploys ClickHouse TLS-only.

Create a replicated tableInsert and verify replicationBulk import from a CSV file

Create the table on all nodes in one statement. The {shard} and {replica} placeholders are macros that ClusterControl configures per node, so the same command works everywhere. Replace ClickHouse-replicated with the cluster name you used during deployment:

clickhouse-client --host 192.168.40.20 --port 9440 --secure \
    --user admin --password 'myPassw0rd' \
    --query "CREATE TABLE default.trips ON CLUSTER 'ClickHouse-replicated'
             (id UInt64, city String, fare Float64)
             ENGINE = ReplicatedMergeTree('/clickhouse/tables/{shard}/{database}/{table}', '{replica}')
             ORDER BY id"

If you deployed with a self-signed certificate, add --accept-invalid-certificate (or point --config-file at a client config that trusts the ClusterControl CA).

Insert a few rows into one node, then read them back from a different node to confirm replication is working:

# Write to node 1
clickhouse-client --host 192.168.40.20 --port 9440 --secure \
    --user admin --password 'myPassw0rd' \
    --query "INSERT INTO default.trips VALUES (1,'Berlin',12.5),(2,'Oslo',9.0)"

# Read from node 2
clickhouse-client --host 192.168.40.21 --port 9440 --secure \
    --user admin --password 'myPassw0rd' \
    --query "SELECT count() FROM default.trips"

The count on node 2 should match what you inserted on node 1.

Stream a local CSV file into the replicated table using the native client. You only need to load it once, into any node:

clickhouse-client --host 192.168.40.20 --port 9440 --secure \
    --user admin --password 'myPassw0rd' \
    --query "INSERT INTO default.trips FORMAT CSV" < trips.csv

For very large files, ClickHouse also reads compressed input directly. For example, pipe a gzip-compressed file through zcat into the same command.

Tip

Because every node holds a full copy of the data, this cluster does not need a Distributed table for reads — any node can answer a query on its own.

Step 5: Connect to the database

Your application or client can connect to any node in the cluster over TLS, since each one holds a complete copy of the data. In this example we connect to node 1:

• Host: 192.168.40.20 (or any other node, 192.168.40.21 / 192.168.40.22)
• Native (secure TCP) port: 9440
• HTTPS port: 8443
• User/Password: The admin credentials specified during deployment. You can also inspect the file /etc/cmon.d/cmon_$CID.cnf, where $CID is the cluster ID of your ClickHouse deployment.

For high availability, point your application at more than one node (most ClickHouse drivers accept a list of hosts and will fail over automatically), or place a load balancer in front of the cluster. A quick connectivity test with the native client:

clickhouse-client --host 192.168.40.20 --port 9440 --secure \
    --user admin --password 'myPassw0rd' \
    --query "SELECT version()"

Step 6: Enable automatic backups

A replicated cluster protects you against losing a node, but it is not a substitute for backups — a bad DROP, a schema mistake, or corruption replicates just as fast as good data. ClusterControl makes it easy to schedule backups automatically. ClickHouse backups in ClusterControl use the native BACKUP SQL command, which writes a consistent copy to the local backups disk (typically /var/lib/clickhouse/backups/) or to S3-compatible object storage.

1. Go to ClusterControl GUI → choose the cluster → Backups.

2. Click Create Backup → Schedule a Backup. The Create a backup schedule wizard will open. Configure your backup as below:

   (1) Configuration(2) Advanced settings(3) Schedule(4) Preview

   • Schedule name: Daily ClickHouse replicated backup
   • Cluster: (defaults to your ClickHouse replicated cluster)
   • Backup method: Choose between Clickhouse native (full) and Clickhouse native (incr) (incremental)
   • Backup node: Select which node the backup should run on. Since each replica holds a full copy, backing up one node captures all your data.

   • Upload backup to cloud: Off (turn On to store the backup on S3-compatible object storage)
   • Retention: On (default)
   • Retention [textfield]: 4 (set to your desired number of days to retain backups)

   • Set backup schedule: Simple
   • Every: day at 02:00
   • Timezone: select your local timezone

   • Verify all settings. You can go back to adjust if needed.

3. Click Create to schedule it.

ClusterControl will now automatically perform your backups. All successful backups are listed on the Backups → All Backups page, where you can review the backup logs, size, and restore from them in the future if necessary for disaster recovery.

Note

ClickHouse native backups are written to the local filesystem of the node that runs them, so they are node-local. Keep your full and incremental backups on the same node: an incremental backup can only build on a parent that lives on the same node it runs on. Controller storage is not supported for ClickHouse native backups; use the local backups disk (default) or enable Upload backup to cloud for S3-compatible object storage.

Step 7: Configure alerts

To keep track of any issues or incidents in your cluster, it's important to set up alerting. ClusterControl supports sending alarms and alerts to email, web hooks, and third-party notification services like Slack or Telegram. In this example, we are going to use email.

Firstly, configure the mail server. Go to ClusterControl GUI → Settings → You don't have a mail server configured. Configure now → SMTP Server. Fill in all the necessary information about the SMTP server. You can also opt for Sendmail; however, a mail transfer agent (sendmail, postfix, or exim) must be installed on the ClusterControl server.

Once configured, we can configure the alert and recipients as below:

1. Go to ClusterControl GUI → choose the cluster → Settings → Email Notifications.
2. Select a user group (your group) from the User group dropdown.
3. Select your email address from the Users in the selected group dropdown.
4. Click Enable.
5. Set the Digest delivery time for when you want a digested (summarized events) email sent to you every day.
6. Set all Critical events to "Deliver" (default), all Warning events to "Digest", and you may ignore the Info events.

This ensures timely notifications when something goes wrong.

Tip

You can also configure alarms to be sent to third-party notification systems (Slack, Telegram), incident management systems (PagerDuty, ServiceNow, OpsGenie), or web hooks. See Integration → Notification Services.

Step 8: Manage your nodes

ClusterControl provides monitoring for your cluster and a system overview that displays workloads based on metrics. Once your ClickHouse cluster is deployed, agents using Prometheus exporters are deployed to gather metrics and provide more granular monitoring of your cluster and system workload across all nodes.

Apart from the monitoring, you can manage your nodes with the available options for this cluster:

• Database node management: Perform start/stop/restart node, reboot host. These features are available at ClusterControl GUI → choose the cluster → Nodes → Actions.
• Configuration management: Perform database configuration changes. This feature is available at ClusterControl GUI → choose the cluster → Manage → Configuration.
• Backup management: Create, schedule, and restore backups, store them in an off-cluster storage location such as AWS S3 or any S3-compatible storage, and set a retention period for your backups. These features are available at ClusterControl GUI → choose the cluster → Backups → Actions and ClusterControl GUI → choose the cluster → Backups → More.
• Maintenance management: Activate, deactivate, remark, and schedule maintenance mode for all nodes. Because your data is replicated, you can put one node into maintenance for patching or upgrades while the others keep serving traffic. This feature is available at ClusterControl GUI → choose the cluster → Nodes → Actions → Schedule maintenance.
• SSH console: Access your nodes directly from the ClusterControl GUI via web SSH console. This feature is available at ClusterControl GUI → choose the cluster → Nodes → Actions → SSH Console.

Step 9: Scale your cluster

Scaling out a replicated ClickHouse cluster means adding another replica. The new ClickHouse node joins the existing shard, pulls a full copy of the data from its peers, and starts serving queries, which improves both your redundancy and your read capacity.

ClusterControl assumes that the new node meets the requirements described under Prerequisites and is configured with proper SSH key-based authentication as shown in Step 1: Set up SSH key-based authentication.

To add a replica:

1. Go to ClusterControl GUI → choose the cluster → Actions... → Add new → Data node.

2. In the Create a data node wizard, configure the following:

   (1) Node configuration(2) Add node(3) Preview

   • Server port: 9440 (default)
   • Install software: On (default)
   • Disable firewall: Checked (default)
   • Disable SELinux/AppArmor: Checked (default)

   • Node: Specify the IP address, hostname, or FQDN of the node that you want to add (for example, 192.168.40.23) and press Enter.

   • Review the summary of the deployment. You can always go back to any previous section to modify your configuration. The deployment settings are kept until you exit the wizard.

3. Click Finish to trigger the deployment job.

4. ClusterControl will provision the new node, add it as a replica to the existing shard, and regenerate the cluster topology so the new node is included in remote_servers. You can monitor the progress from Activity Center → Jobs. Once complete, the node appears on the Nodes page and begins syncing data from the other replicas.

Tip

Adding another replica also adds a Keeper participant when embedded Keeper is used. Keep the Keeper ensemble at an odd size (3 or 5) for the healthiest quorum.

Conclusion

You have now deployed, monitored, and managed a replicated ClickHouse cluster using ClusterControl. Unlike a single node, a replicated cluster keeps a full copy of your data on every node and coordinates them with ClickHouse Keeper, so you get data redundancy and can take a node offline without an outage. As you operate it, keep the following in mind:

• Keep your backups current, and remember that ClickHouse native backups are node-local.
• Monitor performance and resource usage across all nodes, not just one.
• Secure your nodes by restricting access and using strong passwords.
• Scale out by adding replicas as your read traffic grows, keeping the Keeper quorum at an odd number.

From here, ClusterControl handles the day-to-day operations such as backups, alerts, maintenance, and scaling, so you can focus on your data.