Skip to content

Scalable Controllers Pool

The Scalable Controllers Pool removes the operational limits and failure risk of relying on a single CMON controller in larger ClusterControl environments. As the number of managed clusters grows, one controller can become a capacity bottleneck and a single failure domain for cluster monitoring, orchestration, and job handling.

How it works

The pool lets multiple CMON controllers coordinate cluster ownership: controller capacity scales horizontally, and ownership moves automatically when controllers are added, removed, restarted, or become unavailable.

The pool works identically regardless of the underlying infrastructure. Controllers can run on virtual machines, bare-metal servers, or cloud-hosted instances using NFS or Vault as the shared configuration backend, or on Kubernetes using Kubernetes Secrets. The only requirement is that every controller in the pool connects to the same CMON database and shares the same configuration backend.

All pool members share a common source of truth:

  • All pool members connect to the same CMON database.
  • All pool members must use a shared configuration backend.
  • Each controller publishes its presence and status.
  • Clusters are owned by one controller at a time, and ownership can move when controllers are added, removed, restarted, or become stale.

Here's a diagram on how the controller pool works:

graph TD
    subgraph "<b>User Interface & Entry</b>"
        User((User/<br>Admin))
        CLI[ClusterControl CLI]
        GUI[ClusterControl GUI]
    end

    subgraph "<b>Main Controller (Role: Main)</b>"
        MC[CMON Main Instance]
        Proxy[cmon-proxy]
        JobScheduler[Job Scheduler]
    end

    subgraph "<b>Controller Pool (Role: Members)</b>"
        C1[CMON Controller 1]
        C2[CMON Controller 2]
        C3[CMON Controller <i>N</i>]
    end

    subgraph "<b>Shared Infrastructure</b>"
        DB[(Shared CMON DB)]
        Storage[(Shared Config<br/>NFS or Kubernetes Secrets)]
    end

    subgraph "<b>Managed Clusters</b>"
        CL1[Cluster A]
        CL2[Cluster B]
        CL3[Cluster C]
        CL4[Cluster D]
    end

    %% Connections
    User --> CLI & GUI --> Proxy


    Proxy --> JobScheduler
    JobScheduler -- "assigns jobs" --> C1
    JobScheduler -- "assigns jobs" --> C2
    JobScheduler -- "assigns jobs" --> C3

    %% Shared Access
    MC & C1 & C2 & C3 -.- DB & Storage

    %% Ownership
    C1 -- "owns" --> CL1
    C1 -- "owns" --> CL2
    C2 -- "owns" --> CL3
    C3 -- "owns" --> CL4

Key runtime concepts

  • Controllers table: stores controller identity, endpoint, status, and heartbeat information.
  • Cluster assignment table: tracks which controller owns which cluster.
  • Refresh cycle: each controller refreshes status and adjusts ownership.
  • Staleness threshold: if a controller stops refreshing in time, peers can reclaim its clusters.
  • Fair-share ownership: cluster ownership is distributed across active controllers, subject to controller capacity limits.

Operationally, a controller only needs to manage the subset of clusters that it currently owns. When ownership changes, the controller loads or unloads cluster state accordingly.

Shared configuration requirement

Shared configuration is a hard requirement in pool mode. ClusterControl supports the following backends:

  • Kubernetes Secrets for Kubernetes-native environments.
  • NFS or shared filesystem for VM and non-Kubernetes deployments.
  • Vault as a secure backend direction for non-Kubernetes/shared deployments.

Deployment and configuration model

Minimum prerequisites:

  • All controllers must connect to the same CMON database.
  • Each controller must have a unique hostname:port identity.
  • All controllers must use the same shared configuration backend.
  • For Kubernetes shared-config deployments, controllers are started with --pool --k8s.

Example startup pattern for non-Kubernetes environment shared config

# /etc/default/cmon
EXTRA_OPTS=--pool

Example startup pattern for Kubernetes-backed shared config

# /etc/default/cmon
EXTRA_OPTS=--pool --k8s

In multi-VM or external-controller setups using Kubernetes Secrets as the shared configuration store, the documented pattern also includes Kubernetes API endpoint and certificate environment variables so that each controller can access the same secret store.

KUBERNETES_SERVICE_HOST=hostA
KUBERNETES_SERVICE_PORT=16443
KUBERNETES_SERVICE_CA_CERT_PATH=/etc/certs/ca.crt
KUBERNETES_CLIENT_CERT_PATH=/etc/certs/client.crt
KUBERNETES_CLIENT_KEY_PATH=/etc/certs/client.key

Non-Kubernetes deployment model

The feature is also intended for non-Kubernetes environments. In those cases, the shared configuration backend is typically a shared filesystem such as NFS, while controllers still point to the same CMON database. This makes the feature applicable to:

  • Traditional VM deployments
  • Physical server or bare-metal deployments
  • Cloud VMs or instances outside Kubernetes

Capacity and cold-start hardening

The following settings control how many clusters a controller may own and how quickly a new or restarted controller claims them.

Configuration key Meaning Default behavior described
controllers_pool_start_intervals_to_be_stable Refresh cycles before a starting controller becomes active 2 cycles
controllers_pool_max_clusters Per-controller ownership cap -2 auto, -1 unlimited, 0 inactive, >0 explicit cap
controllers_pool_clusters_per_gb RAM-based multiplier for auto capacity 10 clusters per GB
controllers_pool_cluster_claim_step Clusters claimed per refresh cycle during slow start Auto, about 20% of max capacity
controllers_pool_soft_alarm_secs Warning threshold for unassigned clusters 75 seconds
controllers_pool_hard_alarm_secs Critical threshold for unassigned clusters 150 seconds

These controls help avoid a cold-start controller claiming too many clusters at once, starving heartbeats, and triggering reassignment storms.

Managing the Controller Pool

Enabling the controller pool

Enabling the controller pool can be done from either the GUI or the s9s CLI.

  1. In the left-side navigation bar, click Controller pool. If pool mode is not yet enabled, the UI notifies you and shows a Restart CMON in pool mode button.

  2. Click Restart CMON in pool mode.

  3. In the Allowed controllers network field, specify the CIDR block or blocks controllers are allowed to connect from (for example, 192.168.40.0/24, 10.120.0.0/24, 10.8.0.0/24). Click Enable.

    Enabling pool mode adds a granted_controllers_network_mask entry to /etc/cmon.cnf (for example, granted_controllers_network_mask=192.168.40.0/24,192.168.121.0/24). The controller you enabled pool mode on keeps its existing clusters and is assigned the main role.

The s9s CLI adds a pool-controllers mode to manage the controller pool. To enable pool mode:

s9s pool-controllers --set-pool-mode --conf-storage=nfs --granted-network-mask=192.168.40.0/24
  • --set-pool-mode enables the controller pool.
  • --conf-storage=nfs sets the shared configuration backend. Use k8s for Kubernetes Secrets instead. Only nfs and k8s are allowed.
  • --granted-network-mask=192.168.40.0/24 sets the CIDR block controllers are allowed to connect from.

Adjusting controller capacity at runtime

Set the maximum number of clusters a controller may own with --set-max-clusters-capacity (CLI) or through the GUI. This is only available once the controller pool is enabled. See Capacity and cold-start hardening for what the -2/-1/0/>0 values mean.

  1. In the left-side navigation bar, click Controller pool.

  2. Choose a controller and click its ellipsis button (...).

  3. Choose Set capacity.

  4. In the Max clusters field, enter the maximum number of clusters this controller may own. Setting the capacity lower than the number of clusters currently owned causes the controller to abandon the excess clusters. To move clusters back onto this controller later, raise the capacity again.

  5. Click Apply.

s9s pool-controllers \
  --set-max-clusters-capacity=40 \
  --controller="https://HOST:9501" \
  --cmon-user=USER \
  --password=PASS
  • --set-max-clusters-capacity=40 sets this controller's capacity to 40 clusters.
  • --controller="https://HOST:9501" targets the controller to change.
  • --cmon-user and --password are your ClusterControl credentials. Omit them if they are already set in your S9S_USER_CONFIG file.
  • Add --force if lowering the capacity would require the controller to abandon clusters it already owns.

Starting or stopping a controller

Starting and stopping a controller works the same way as starting or stopping the cmon daemon via systemd. You can only start a stopped controller, and only stop a running one.

  1. In the left-side navigation bar, click Controller pool.

  2. Choose a controller and click its ellipsis button (...).

  3. Choose Stop or Start.

  4. Confirm the action in the prompt that appears.

s9s pool-controllers \
  --start \
  --controller-id SID
  • --start starts the controller's cmon daemon. Use --stop to stop a running controller instead.
  • --controller-id SID is the controller's SID, not its runtime ID. See View the controller pool for the difference.

Note

A controller with the main role cannot be stopped, only controllers with the member role.

Removing a controller from the pool

Removing a controller applies only to a controller that is already stopped.

  1. In the left-side navigation bar, click Controller pool.

  2. Choose a controller and click its ellipsis button (...).

  3. Choose Remove.

  4. In the prompt, choose one of two options:

    • Unregister controller only: Unregisters the controller from the pool but leaves the cmon daemon and its installation untouched.
    • Unregister and uninstall controller: Unregisters the controller and uninstalls its packages, ensuring the daemon is fully stopped.
s9s pool-controllers \
  --remove-controller \
  --controller-id SID
  • --remove-controller removes the controller from the pool.
  • --controller-id SID is the controller's SID, not its runtime ID. See View the controller pool for the difference.

Add --uninstall to also remove the cmon packages from that host:

s9s pool-controllers --remove-controller --controller-id SID --uninstall

Upgrading a controller's cmon version

  1. In the left-side navigation bar, click Controller pool.

  2. Choose a controller and click its ellipsis button (...).

  3. Choose Upgrade.

  4. Confirm the action in the prompt that appears.

The controller must be stopped before it can be upgraded.

s9s pool-controllers \
  --update-cmon \
  --controller-id SID
  • --update-cmon upgrades the controller to the latest cmon version available in the Severalnines public repository.
  • --controller-id SID is the controller's SID, not its runtime ID. See View the controller pool for the difference.

This also works against a specific pool member when run from the main controller.

Inspecting the controller pool with the CLI

Beyond the management operations above, the s9s CLI can also inspect pool status and ownership through related cluster and node commands.

View the controller pool

$ s9s pool-controllers --list

This shows controller SID, ID, hostname, port, status, role, and assigned clusters.

$ s9s pool-controllers --list
SID ID HOSTNAME      PORT STATUS ROLE   COUNT/MAX CLUSTERS
13  14 192.168.40.10 9500 active member      2/36 [2, 9]
7   24 192.168.40.5  9500 active main        2/36 [5, 10]
11  31 192.168.40.3  9500 active member      1/38 [1]
9   32 192.168.40.4  9500 active member      1/38 [8]

The ROLE field identifies which controller is your main controller, host 192.168.40.5 in this example.

The columns show:

  • SID: Static controller ID (persistent config identifier). Use this for operations like --start, --stop, --remove-controller.
  • ID: Runtime/dynamic controller instance ID (can change with runtime/re-registration).
  • HOSTNAME: Controller host/IP.
  • PORT: Controller RPC/API port (typically 9500/9501 depending setup).
  • STATUS: Current state (e.g., active, starting, stopped, restarting, etc.).
  • ROLE: Pool role (main = primary config/db holder, member = regular pool member).
  • COUNT/MAX: owned_clusters / max_cluster_capacity for that controller (e.g., 2/36 means owns 2 clusters, cap is 36).
  • CLUSTERS: Cluster IDs currently assigned to that controller (e.g., [2, 9]).

In the example above, the controller at 192.168.40.10 owns clusters 2 and 9, out of a capacity of 36.

Identifying ownership from the cluster perspective

Combine the result of s9s pool-controllers --list with s9s cluster --list to see which cluster names and details belong to a given controller:

$ s9s cluster --list --long --controller="https://192.168.40.10:9501" --cmon-user=admin2 --password="userP@55" --rpc-tls
Result displays same as the following example:

ID STATE   TYPE              OWNER  GROUP  NAME           COMMENT
 2 STARTED postgresql_single admin2 admins pgsql-18       All nodes are operational.
 9 STARTED galera            admin2 admins mariadb-galera All nodes are operational.
Total: 2

For security purposes, and you do not want to expose the password, you can use the following: 

$ export S9S_USER_CONFIG=/root/.s9s/s9s-controller10.conf;  s9s cluster --list --long --controller="https://192.168.40.10:9501" --rpc-tls
ID STATE   TYPE              OWNER  GROUP  NAME           COMMENT
 2 STARTED postgresql_single admin2 admins pgsql-18       All nodes are operational.
 9 STARTED galera            admin2 admins mariadb-galera All nodes are operational.
Total: 2

Where the file /root/.s9s/s9s-controller10.conf contains the following contents:

[global]
controller    = https://192.168.40.10:9501
cmon_user     = "admin"
cmon_password = "7201231b-8f0a-4d5e-a755-991e32ce3f66"

Note that querying controller 192.168.40.10 with s9s cluster --list --long only returns clusters 2 and 9, the clusters it owns. A cluster can only be viewed, listed, or identified through the controller that currently owns it, whether that controller is specified with --controller or through the controller parameter in S9S_USER_CONFIG.

Listing the nodes of the cluster

To list the nodes owned by a specific controller, run s9s node --list --long against that controller's address.

For example, from the main controller 192.168.40.5, you can check what nodes each pool member has:

$ s9s node --list --long --controller="https://192.168.40.3:9501"

STAT VERSION     CID CLUSTER        HOST           PORT COMMENT
ho-- 1.8.27        1  mariadb-10.5  192.168.121.22 9600 Process 'haproxy' is running.
coC- 2.5.0.22113   1  mariadb-10.5  192.168.40.3   9500 Up and running.
Po-- 2.53.5        1  mariadb-10.5  192.168.40.4   9090 Process 'prometheus' is running.
soM- 10.5.29       1  mariadb-10.5  192.168.40.60  3306 Up and running (read-write).
koM- 2.1           1  mariadb-10.5  192.168.40.60   112 Process 'keepalived' is running.
soS- 10.5.29       1  mariadb-10.5  192.168.40.61  3306 Up and running (read-only).
ho-- 1.8.27        1  mariadb-10.5  192.168.40.61  9600 Process 'haproxy' is running.
ko-- 2.1           1  mariadb-10.5  192.168.40.61   112 Process 'keepalived' is running.
Total: 8

Or check what nodes and clusters (by CID) controller host 192.168.40.4 has:

$ s9s node --list --long --controller="https://192.168.40.4:9501"
STAT VERSION     CID CLUSTER  HOST          PORT COMMENT
coC- 2.5.0.22113   8 pgsql-14 192.168.40.4  9500 Up and running.
poM- 14.23         8 pgsql-14 192.168.40.62 5432 Up and running (read-write).
poS- 14.23         8 pgsql-14 192.168.40.66 5432 Up and running (read-only).
Total: 3

Or check what nodes and clusters (by CID) controller host 192.168.40.10 has:

$ s9s node --list --long --controller="https://192.168.40.10:9501"
STAT VERSION     CID CLUSTER        HOST          PORT COMMENT
coC- 2.5.0.22113   2 pgsql-18       192.168.40.10 9500 Up and running.
Po-- 2.53.5        2 pgsql-18       192.168.40.3  9090 Process 'prometheus' is running.
p-S- 18.4          2 pgsql-18       192.168.40.63 5432 Shut down (read-only).
ho-- 1.8.27        2 pgsql-18       192.168.40.63 9600 Process 'haproxy' is running.
koM- 2.1           2 pgsql-18       192.168.40.63  112 Process 'keepalived' is running.
poM- 18.4          2 pgsql-18       192.168.40.64 5432 Up and running (read-write).
ho-- 1.8.27        2 pgsql-18       192.168.40.64 9600 Process 'haproxy' is running.
ko-- 2.1           2 pgsql-18       192.168.40.64  112 Process 'keepalived' is running.
coC- 2.5.0.22113   9 mariadb-galera 192.168.40.10 9500 Up and running.
Po-- 2.53.5        9 mariadb-galera 192.168.40.10 9090 Process 'prometheus' is running.
goM- 11.8.8        9 mariadb-galera 192.168.40.33 3306 Up and running (read-write).
goM- 11.8.8        9 mariadb-galera 192.168.40.34 3306 Up and running (read-write).
goM- 11.8.8        9 mariadb-galera 192.168.40.35 3306 Up and running (read-write).
Total: 13

Additionally, to see hostname/IP, node type, and database version for every node across all controllers in a single table, run the following from the main controller:

$ s9s cluster --list --all-pool --long --print-json | jq -r '
  ["cluster","cdt_path","ip","hostname","nodetype","version","version_comment","addresses"] as $cols
  | ($cols | "| " + join(" | ") + " |"),
    ($cols | map("---") | "| " + join(" | ") + " |"),
    ( [.. | objects | select(.class_name? == "CmonClusterInfo")][]
      | (.cluster_name // .cdt_path) as $cluster
      | .hosts[]?
      | [ $cluster,
          ((.cdt_path // "-") | gsub("^\\s+|\\s+$"; "")),
          (.ip // "-"),
          (.hostname // "-"),
          (.nodetype // .class_name // "-"),
          (.version // "-"),
          (.version_comment // "-"),
          ([.configuration[]?.address] | join("<br>") | if . == "" then "-" else . end)
        ]
      | map(tostring | gsub("\\|"; "\\|"))
      | "| " + join(" | ") + " |"
    )'
| cluster | cdt_path | ip | hostname | nodetype | version | version_comment | addresses |
| --- | --- | --- | --- | --- | --- | --- | --- |
|  mariadb-10.5  | / mariadb-10.5 | 192.168.40.60 | 192.168.40.60 | mysql | 10.5.29-MariaDB-log | MariaDB Server | - |
|  mariadb-10.5  | / mariadb-10.5 | 192.168.40.3 | 192.168.40.3 | controller | 2.5.0.22113 | - | - |
|  mariadb-10.5  | / mariadb-10.5 | 192.168.40.4 | 192.168.40.4 | prometheus | 2.53.5 | - | <br><br><br><br> |
|  mariadb-10.5  | / mariadb-10.5 | 192.168.40.61 | 192.168.40.61 | mysql | 10.5.29-MariaDB-log | MariaDB Server | - |
|  mariadb-10.5  | / mariadb-10.5 | 192.168.40.61 | 192.168.40.61 | haproxy | 1.8.27 | - | - |
|  mariadb-10.5  | / mariadb-10.5 | 192.168.40.60 | 192.168.40.60 | keepalived | 2.1 | - | - |
|  mariadb-10.5  | / mariadb-10.5 | 192.168.40.61 | 192.168.40.61 | keepalived | 2.1 | - | - |
|  mariadb-10.5  | / mariadb-10.5 | 192.168.121.22 | 192.168.121.22 | haproxy | 1.8.27 | - | - |
| pgsql-18 | /pgsql-18 | 192.168.40.63 | 192.168.40.63 | postgres | 18.4 | - | - |
| pgsql-18 | /pgsql-18 | 192.168.40.64 | 192.168.40.64 | postgres | 18.4 | - | - |
| pgsql-18 | /pgsql-18 | 192.168.40.10 | 192.168.40.10 | controller | 2.5.0.22113 | - | - |
| pgsql-18 | /pgsql-18 | 192.168.40.3 | 192.168.40.3 | prometheus | 2.53.5 | - | <br><br><br><br> |
| pgsql-18 | /pgsql-18 | 192.168.40.63 | 192.168.40.63 | haproxy | 1.8.27 | - | - |
| pgsql-18 | /pgsql-18 | 192.168.40.64 | 192.168.40.64 | haproxy | 1.8.27 | - | - |
| pgsql-18 | /pgsql-18 | 192.168.40.63 | 192.168.40.63 | keepalived | 2.1 | - | - |
| pgsql-18 | /pgsql-18 | 192.168.40.64 | 192.168.40.64 | keepalived | 2.1 | - | - |
| clickhouse | /clickhouse | 192.168.40.67 | 192.168.40.67 | clickhouse | 25.3 | - | - |
| clickhouse | /clickhouse | 192.168.40.68 | 192.168.40.68 | clickhouse | 25.3 | - | - |
| clickhouse | /clickhouse | 192.168.40.69 | 192.168.40.69 | clickhouse | 25.3 | - | - |
| clickhouse | /clickhouse | 192.168.40.30 | 192.168.40.30 | clickhouse_keeper | 25.3 | - | - |
| clickhouse | /clickhouse | 192.168.40.31 | 192.168.40.31 | clickhouse_keeper | 25.3 | - | - |
| clickhouse | /clickhouse | 192.168.40.32 | 192.168.40.32 | clickhouse_keeper | 25.3 | - | - |
| clickhouse | /clickhouse | 192.168.40.5 | 192.168.40.5 | controller | 2.5.0.22113 | - | - |
| clickhouse | /clickhouse | 192.168.40.10 | 192.168.40.10 | prometheus | 2.53.5 | - | <br><br><br><br> |
| pgsql-14 | /pgsql-14. | 192.168.40.66 | 192.168.40.66 | postgres | 14.23 | - | - |
| pgsql-14 | /pgsql-14. | 192.168.40.62 | 192.168.40.62 | postgres | 14.23 | - | - |
| pgsql-14 | /pgsql-14. | 192.168.40.4 | 192.168.40.4 | controller | 2.5.0.22113 | - | - |
| mariadb-galera | /mariadb-galera | 192.168.40.33 | 192.168.40.33 | galera | 11.8.8-MariaDB-log | MariaDB Server | - |
| mariadb-galera | /mariadb-galera | 192.168.40.10 | 192.168.40.10 | controller | 2.5.0.22113 | - | - |
| mariadb-galera | /mariadb-galera | 192.168.40.34 | 192.168.40.34 | galera | 11.8.8-MariaDB-log | MariaDB Server | - |
| mariadb-galera | /mariadb-galera | 192.168.40.35 | 192.168.40.35 | galera | 11.8.8-MariaDB-log | MariaDB Server | - |
| mariadb-galera | /mariadb-galera | 192.168.40.10 | 192.168.40.10 | prometheus | 2.53.5 | - | <br><br><br> |

Operational guidance

  1. Set up a shared CMON database.
  2. Choose a shared configuration backend appropriate to the environment.
  3. Start the initial controller with pool mode enabled.
  4. Add additional controllers that point to the same CMON database and shared config backend.
  5. Use s9s pool-controllers --list and s9s cluster --list --all-pool to verify ownership distribution.
  6. For larger fleets, tune capacity and claim-step settings to avoid cold-start overload.

Common gotchas

  • Assuming Kubernetes is required: it is not. Kubernetes is one supported model, but VMs, bare metal, and cloud-hosted servers are also valid if shared DB and shared config are in place.
  • Missing shared config: pool mode requires all controllers to see the same configuration state.
  • Wrong HA expectations: pool mode gives you horizontal scaling of controller capacity, but full HA/failover readiness has not shipped in the same release as pool mode for every version. Check your version's release notes before assuming both are available.
  • Privilege errors on pool-wide listing: use a superuser/admin account.
  • Runtime capacity changes disappearing after restart: only configuration-file values are persistent.
Copyright © 2014-2026 Severalnines AB. All rights reserved.

Severalnines and ClusterControl are registered trademarks in the US, UK, and EU. The 3rd-party trademarks on this site are property of their respective owners and are used for referential purposes only.

Made with Material for MkDocs