Skip to content

Scalable Controllers Pool with HA

The Scalable Controllers Pool with HA was created to remove the operational limits and failure risk of relying on a single CMON controller in larger ClusterControl environments. As the number of managed clusters grows, one controller can become a capacity bottleneck and a single failure domain for cluster monitoring, orchestration, and job handling.

Its purpose is to let multiple CMON controllers work as a coordinated pool, enabling horizontal scaling of controller capacity and supporting high availability/failover behavior for ClusterControl/CMON-managed environments.

The controller pool supports both Kubernetes and non-Kubernetes environments to provide scalability and high availability. Controllers can be deployed on virtual machines, bare-metal servers, or cloud-hosted instances — using NFS or Vault as the shared configuration backend — as well as on Kubernetes using K8s Secrets. As long as all controllers connect to the same CMON database and share a common configuration backend, the pool operates identically regardless of the underlying infrastructure.

Scale controller capacity by distributing cluster ownership across multiple CMON controllers.

Reduce single-controller risk by allowing ownership to move when controllers are added, removed, restarted, or become unavailable.

What the feature is for

A single CMON controller can become a scaling bottleneck in larger environments. The Scalable Controllers Pool feature addresses this by allowing multiple CMON controllers to coordinate and share cluster ownership.

Importantly, this is not a Kubernetes-only feature. The design and release material describe support for:

  • Kubernetes-based deployments, typically using shared configuration through Kubernetes Secrets with --pool --k8s.
  • VM-based deployments, including multi-VM setups using a shared filesystem such as NFS.
  • Bare-metal or cloud-hosted controller nodes, as long as all controllers use the same CMON database and the same shared configuration source.

The core idea is horizontal scaling of CMON controllers through shared state and coordinated cluster ownership, regardless of whether the controllers run on Kubernetes, virtual machines, bare metal, or cloud instances.

How it works

The controller pool works by having multiple CMON controllers share a common source of truth.

  • All pool members connect to the same CMON database.
  • All pool members must use a shared configuration backend.
  • Each controller publishes its presence and status.
  • Clusters are owned by one controller at a time, and ownership can move when controllers are added, removed, restarted, or become stale.

Here's a diagram on how the Controller Pool works: 

graph TD
    subgraph "User Interface & Entry"
        User((User / Admin))
        CLI[s9s CLI]
        GUI[ClusterControl UI]
    end

    subgraph "Main Controller (Role: Main)"
        MC[CMON Main Instance]
        Proxy[cmon-proxy]
        JobScheduler[Job Scheduler]
    end

    subgraph "Controller Pool (Role: Members)"
        C1[CMON Controller 1]
        C2[CMON Controller 2]
        C3[CMON Controller N]
    end

    subgraph "Shared Infrastructure"
        DB[(Shared CMON DB)]
        Storage{Shared Config<br/>NFS / K8s Secrets}
    end

    subgraph "Managed Clusters"
        CL1[Cluster A]
        CL2[Cluster B]
        CL3[Cluster C]
        CL4[Cluster D]
    end

    %% Connections
    User --> CLI
    User --> GUI
    CLI --> Proxy
    GUI --> Proxy

    Proxy --> JobScheduler
    JobScheduler -- "Assigns Jobs" --> C1
    JobScheduler -- "Assigns Jobs" --> C2

    %% Shared Access
    MC --> DB
    C1 --> DB
    C2 --> DB
    C3 --> DB

    MC --> Storage
    C1 --> Storage
    C2 --> Storage
    C3 --> Storage

    %% Ownership
    C1 -. "Owns" .-> CL1
    C1 -. "Owns" .-> CL2
    C2 -. "Owns" .-> CL3
    C3 -. "Owns" .-> CL4

    %% Logic Flow
    classDef main fill:#f9f,stroke:#333,stroke-width:2px;
    classDef member fill:#bbf,stroke:#333,stroke-width:1px;
    class MC main;
    class C1,C2,C3 member;

Key runtime concepts

  • Controllers table: stores controller identity, endpoint, status, and heartbeat information.
  • Cluster assignment table: tracks which controller owns which cluster.
  • Refresh cycle: each controller refreshes status and adjusts ownership.
  • Staleness threshold: if a controller stops refreshing in time, peers can reclaim its clusters.
  • Fair-share ownership: cluster ownership is distributed across active controllers, subject to controller capacity limits.

Operationally, a controller only needs to manage the subset of clusters that it currently owns. When ownership changes, the controller loads or unloads cluster state accordingly.

Shared configuration requirement

Shared configuration is a hard requirement in pool mode. The source material describes several supported or designed approaches:

  • Kubernetes Secrets for Kubernetes-native environments.
  • NFS or shared filesystem for VM and non-Kubernetes deployments.
  • Vault as a secure backend direction for non-Kubernetes/shared deployments.

Deployment and configuration model

Minimum prerequisites

  • All controllers must connect to the same CMON database.
  • Each controller must have a unique hostname:port identity.
  • All controllers must use the same shared configuration backend.
  • For Kubernetes shared-config deployments, controllers are started with --pool --k8s.

Example startup pattern for non-Kubernetes environment shared config

# /etc/default/cmon
EXTRA_OPTS=--pool

Example startup pattern for Kubernetes-backed shared config

# /etc/default/cmon
EXTRA_OPTS=--pool --k8s

In multi-VM or external-controller setups using Kubernetes Secrets as the shared configuration store, the documented pattern also includes Kubernetes API endpoint and certificate environment variables so that each controller can access the same secret store.

KUBERNETES_SERVICE_HOST=hostA
KUBERNETES_SERVICE_PORT=16443
KUBERNETES_SERVICE_CA_CERT_PATH=/etc/certs/ca.crt
KUBERNETES_CLIENT_CERT_PATH=/etc/certs/client.crt
KUBERNETES_CLIENT_KEY_PATH=/etc/certs/client.key

Non-Kubernetes deployment model

The feature is also intended for non-Kubernetes environments. In those cases, the shared configuration backend is typically a shared filesystem such as NFS, while controllers still point to the same CMON database. This makes the feature applicable to:

  • Traditional VM deployments
  • Physical server or bare-metal deployments
  • Cloud VMs or instances outside Kubernetes

Capacity and cold-start hardening

Later hardening work introduced settings that control how many clusters a controller may own and how quickly a new or restarted controller claims them.

Configuration key Meaning Default behavior described
controllers_pool_start_intervals_to_be_stable Refresh cycles before a starting controller becomes active 2 cycles
controllers_pool_max_clusters Per-controller ownership cap -2 auto, -1 unlimited, 0 inactive, >0 explicit cap
controllers_pool_clusters_per_gb RAM-based multiplier for auto capacity 10 clusters per GB
controllers_pool_cluster_claim_step Clusters claimed per refresh cycle during slow start Auto, about 20% of max capacity
controllers_pool_soft_alarm_secs Warning threshold for unassigned clusters 75 seconds
controllers_pool_hard_alarm_secs Critical threshold for unassigned clusters 150 seconds

These controls help avoid a cold-start controller claiming too many clusters at once, starving heartbeats, and triggering reassignment storms.

Managing the Controller Pool

Enabling the Controller or CMON Pool

Enabling the controller or cmon pool can be done both via GUI or s9s CLI

  1. In the left-side navigation bar, find the Controller pool and click. By defauult, the UI will notify you that you are not running in pool mode. Clicking the button Restart CMON in pool mode will bring you to the pool mode. Once you are in pool mode, this shall list all the available controllers you have added or registered in the controller pool.

  2. Click button Restart CMON in pool mode.

  3. In the text field Allowed controllers network, specify the CIDR notation or CIDR block (e.g. 192.168.40.0/24, 10.120.0/24, 10.8.0/24). Then click button Enable. Once enabled, this shall update the /etc/cmon.cnf adding a new entry with parameter granted_controllers_network_mask (e.g. granted_controllers_network_mask=192.168.40.0/24,192.168.121.0/24). After it has been enabled, it shall display your default controller where you operated the list of clusters assigned, plus it will be assigned as your main controller role.

  • s9s CLI tools added mode pool-controllers to manage cmon pool. To enable cmon-pool, simply run

    s9s pool-controllers --set-pool-mode --conf-storage=nfs --granted-network-mask=192.168.40.0/24
    • where --set-pol-mode to enable controller or cmon pool
    • where --conf-storage=nfs defines to use nfs as the storage. Use k8s if you want to store it using kubernetes. Only nfs and k8s are allowed.
    • where --granted-network-mask=192.168.40.0/24 sets the CIDR block you want controllers to be allowed

Adjusting the controller capacity at runtime

You can set the number of clusters to be assigned to the controller. By setting the maximum cluster capacity through parameter --set-max-clusters-capacity if using s9s CLI or through the GUI. This can only be used or found if the cmon or controller pool is enabled.

  1. In the left-side navigation bar, find the Controller pool and click.

  2. Choose one of the controllers and click the ellipsis button ...

  3. Chooose Set capacity

  4. In the text field *Max clusters, input the number you would like to set as the maximum capacity of clusters to be assigned to this controller. Take note, setting the capacity lower than the currently assigned clusters will abandone the cluster(s) from the controller. If you need the cluster get back again to that controller, assigned higher than the current number set into the controller.

  5. Click the Apply button to apply the value you desire to set.

  • Use the parameter --set-max-clusters-capacity to set the capacity of the clusters to the controller.
s9s pool-controllers \
  --set-max-clusters-capacity=40 \
  --controller="https://HOST:9501" \
  --cmon-user=USER \
  --password=PASS

- where `--set-max-clusters-capacity=40` sets the total clusters of 40 to be assigned to the controller
- where `--controller="https://HOST:9501"` defines to use nfs as the storage. Use `k8s` if you want to store it using kubernetes. Only `nfs` and `k8s` are allowed. 
- where `--cmon-user` and `--password` are the username/password combination for your ClusterControl user credentials. However, you can just ignore these options if the username and password is set alread in your S9S_USER_CONFIG file.

Start/Stopping the controller pool

Starting and stopping a controller is same as start/stopping the cmon daemon via systemd. Starting a controller pool applies only to a controller that is stopped. Stopping only applies as well to a running controller.

  1. In the left-side navigation bar, find the Controller pool and click.

  2. Choose one of the controllers and click the ellipsis button ...

  3. Chooose Stop or Start

  4. A prompt will show up. Make sure to click the Stop or Start button to apply your desired action.

  • Use the parameter --stop or --start. For example, to start the controller.
s9s pool-controllers \
  --start \
  --controller-id SID

- where `--start` the action to start the controller, i.e. start cmon daemon. Use `--stop` for a running controller that you intend to stop.
- where `--controller-id SID` where you specify the controller's `SID`, not the controller ID. See below for more info.

Note

Controller with a main role cannot be stopped. Only those controllers with a role of a member

Removing the controller pool

Removing a controller applies only to a stopped state controller.

  1. In the left-side navigation bar, find the Controller pool and click.

  2. Choose one of the controllers and click the ellipsis button ...

  3. Chooose Remove

  4. A prompt will show up. You will have two (2) options to take.

    1. Unregister controller only: This shall unregister the controller but let the cmon daemon running and untouched or installed.

    2. Unregister and uninstall controller: This shall unregister and uninstall the controller packages. This shall also ensure that the daemon is no longer running or is totally stopped or terminated.

  • Use the parameter --remove-controller to remove the controller.
s9s pool-controllers \
  --remove-controller \
  --controller-id SID

- where `--remove-controller` the action to remove the controller
- where `--controller-id SID` where you specify the controller's `SID`, not the controller ID. See below for more info.

Upgrading the controller cmon version

It is common that a cmon will have a new version. This section will guide you how to upgrade the version either from GUI or through s9s CLI

  1. In the left-side navigation bar, find the Controller pool and click.

  2. Choose one of the controllers and click the ellipsis button ...

  3. Chooose Upgrade

  4. A prompt will show up for confirmation. Make sure you hit the button Upgrade to apply the desired action.

  • Use the parameter --update-cmon to remove the controller. It requires that the controller or cmon is stop before you can proceed on executing or applying the action to upgrade the cmon to the latest version.
s9s pool-controllers \
  --update-cmon \
  --controller-id SID

- where `--update-cmon` the action to upgrade the cmon to the latest version available in the severalnines public repository
- where `--controller-id SID` where you specify the controller's `SID`, not the controller ID. See below for more info.

s9s CLI commands for cmon or controller pools

The controller pool is exposed through both pool-specific commands and related cluster/job commands in s9s.

View the controller pool

$ s9s pool-controllers --list
This shows controller SID, ID, hostname, port, status, role, and assigned clusters. 
$ s9s pool-controllers --list
SID ID HOSTNAME      PORT STATUS ROLE   COUNT/MAX CLUSTERS
13  14 192.168.40.10 9500 active member      2/36 [2, 9]
7   24 192.168.40.5  9500 active main        2/36 [5, 10]
11  31 192.168.40.3  9500 active member      1/38 [1]
9   32 192.168.40.4  9500 active member      1/38 [8]

Based on the above result, make sure of the ROLE field which reveals what is your main controller, i.e. host 192.168.40.5 based on this sample result.

The following field/column entails the following:

  • SID: Static controller ID (persistent config identifier). Use this for operations like --start, --stop, --remove-controller.
  • ID: Runtime/dynamic controller instance ID (can change with runtime/re-registration).
  • HOSTNAME: Controller host/IP.
  • PORT: Controller RPC/API port (typically 9500/9501 depending setup).
  • STATUS: Current state (e.g., active, starting, stopped, restarting, etc.).
  • ROLE: Pool role (main = primary config/db holder, member = regular pool member).
  • COUNT/MAX: owned_clusters / max_cluster_capacity for that controller (e.g., 2/36 means owns 2 clusters, cap is 36).
  • CLUSTERS: Cluster IDs currently assigned to that controller (e.g., [2, 9]).

In example above, consider the controller under hostname/IP 192.168.40.10, it owns clusters 2 and 9, with capacity 36.

Identifying ownership from the cluster perspective

Using the combination of s9s pool-controllers --list command result, you can verify what cluster names and its informaton using s9s cluster command. See how to do this: 

$ s9s cluster --list --long --controller="https://192.168.40.10:9501" --cmon-user=admin2 --password="userP@55" --rpc-tls
Result displays same as the following example: 
ID STATE   TYPE              OWNER  GROUP  NAME           COMMENT
 2 STARTED postgresql_single admin2 admins pgsql-18       All nodes are operational.
 9 STARTED galera            admin2 admins mariadb-galera All nodes are operational.
Total: 2

For security purposes, and you do not want to expose the password, you can use the following: 

$ export S9S_USER_CONFIG=/root/.s9s/s9s-controller10.conf;  s9s cluster --list --long --controller="https://192.168.40.10:9501" --rpc-tls
ID STATE   TYPE              OWNER  GROUP  NAME           COMMENT
 2 STARTED postgresql_single admin2 admins pgsql-18       All nodes are operational.
 9 STARTED galera            admin2 admins mariadb-galera All nodes are operational.
Total: 2

where the file /root/.s9s/s9s-controller10.conf contains the following contents: 

$ cat /root/.s9s/s9s-controller10.conf
[global]
controller    = https://192.168.40.10:9501
cmon_user     = "admin"
cmon_password = "7201231b-8f0a-4d5e-a755-991e32ce3f66"

From the result above, you notice that controller 192.168.40.10 contains only clusters with ID 2, 9. The result of querying to controller host 192.168.40.10 with s9s cluster --list --long only list clusters with id 2 and 9. This means that clusters can only be viewed, listed, or identified if the cluster belongs to that controller specified in --controller or in the S9S_USER_CONFIG config file parameter controller.

Listing the nodes of the cluster

If you want to check the list of nodes with s9s node --list --long, then you have to replace the controller host/IP to gather some information.

For example, given the results we have above, I am checking from my main controller 192.168.40.5. You can check the following controllers on what nodes do they have: 

[[email protected] ~]#  s9s node --list --long --controller="https://192.168.40.3:9501"

STAT VERSION     CID CLUSTER        HOST           PORT COMMENT
ho-- 1.8.27        1  mariadb-10.5  192.168.121.22 9600 Process 'haproxy' is running.
coC- 2.5.0.22113   1  mariadb-10.5  192.168.40.3   9500 Up and running.
Po-- 2.53.5        1  mariadb-10.5  192.168.40.4   9090 Process 'prometheus' is running.
soM- 10.5.29       1  mariadb-10.5  192.168.40.60  3306 Up and running (read-write).
koM- 2.1           1  mariadb-10.5  192.168.40.60   112 Process 'keepalived' is running.
soS- 10.5.29       1  mariadb-10.5  192.168.40.61  3306 Up and running (read-only).
ho-- 1.8.27        1  mariadb-10.5  192.168.40.61  9600 Process 'haproxy' is running.
ko-- 2.1           1  mariadb-10.5  192.168.40.61   112 Process 'keepalived' is running.
Total: 8
or check in controller host 192.168.40.4 which nodes and cluster (based on CID) does it have: 
[[email protected] ~]#  s9s node --list --long --controller="https://192.168.40.4:9501"
STAT VERSION     CID CLUSTER  HOST          PORT COMMENT
coC- 2.5.0.22113   8 pgsql-14 192.168.40.4  9500 Up and running.
poM- 14.23         8 pgsql-14 192.168.40.62 5432 Up and running (read-write).
poS- 14.23         8 pgsql-14 192.168.40.66 5432 Up and running (read-only).
Total: 3

or check in controller host 192.168.40.10 which nodes and cluster (based on CID) does it have: 

[[email protected] ~]#  s9s node --list --long --controller="https://192.168.40.10:9501"
STAT VERSION     CID CLUSTER        HOST          PORT COMMENT
coC- 2.5.0.22113   2 pgsql-18       192.168.40.10 9500 Up and running.
Po-- 2.53.5        2 pgsql-18       192.168.40.3  9090 Process 'prometheus' is running.
p-S- 18.4          2 pgsql-18       192.168.40.63 5432 Shut down (read-only).
ho-- 1.8.27        2 pgsql-18       192.168.40.63 9600 Process 'haproxy' is running.
koM- 2.1           2 pgsql-18       192.168.40.63  112 Process 'keepalived' is running.
poM- 18.4          2 pgsql-18       192.168.40.64 5432 Up and running (read-write).
ho-- 1.8.27        2 pgsql-18       192.168.40.64 9600 Process 'haproxy' is running.
ko-- 2.1           2 pgsql-18       192.168.40.64  112 Process 'keepalived' is running.
coC- 2.5.0.22113   9 mariadb-galera 192.168.40.10 9500 Up and running.
Po-- 2.53.5        9 mariadb-galera 192.168.40.10 9090 Process 'prometheus' is running.
goM- 11.8.8        9 mariadb-galera 192.168.40.33 3306 Up and running (read-write).
goM- 11.8.8        9 mariadb-galera 192.168.40.34 3306 Up and running (read-write).
goM- 11.8.8        9 mariadb-galera 192.168.40.35 3306 Up and running (read-write).
Total: 13

Additonally, if you want to verify what hostname/IP, node type, and database version, a helpful command below you can use to view what nodes are in the controller. For example, the command below is executed or ran in the main controller: 

[root@pupnode5 s9s]#
[root@pupnode5 s9s]# s9s cluster --list --all-pool --long --print-json | jq -r '
  ["cluster","cdt_path","ip","hostname","nodetype","version","version_comment","addresses"] as $cols
  | ($cols | "| " + join(" | ") + " |"),
    ($cols | map("---") | "| " + join(" | ") + " |"),
    ( [.. | objects | select(.class_name? == "CmonClusterInfo")][]
      | (.cluster_name // .cdt_path) as $cluster
      | .hosts[]?
      | [ $cluster,
          ((.cdt_path // "-") | gsub("^\\s+|\\s+$"; "")),
          (.ip // "-"),
          (.hostname // "-"),
          (.nodetype // .class_name // "-"),
          (.version // "-"),
          (.version_comment // "-"),
          ([.configuration[]?.address] | join("<br>") | if . == "" then "-" else . end)
        ]
      | map(tostring | gsub("\\|"; "\\|"))
      | "| " + join(" | ") + " |"
    )'
| cluster | cdt_path | ip | hostname | nodetype | version | version_comment | addresses |
| --- | --- | --- | --- | --- | --- | --- | --- |
|  mariadb-10.5  | / mariadb-10.5 | 192.168.40.60 | 192.168.40.60 | mysql | 10.5.29-MariaDB-log | MariaDB Server | - |
|  mariadb-10.5  | / mariadb-10.5 | 192.168.40.3 | 192.168.40.3 | controller | 2.5.0.22113 | - | - |
|  mariadb-10.5  | / mariadb-10.5 | 192.168.40.4 | 192.168.40.4 | prometheus | 2.53.5 | - | <br><br><br><br> |
|  mariadb-10.5  | / mariadb-10.5 | 192.168.40.61 | 192.168.40.61 | mysql | 10.5.29-MariaDB-log | MariaDB Server | - |
|  mariadb-10.5  | / mariadb-10.5 | 192.168.40.61 | 192.168.40.61 | haproxy | 1.8.27 | - | - |
|  mariadb-10.5  | / mariadb-10.5 | 192.168.40.60 | 192.168.40.60 | keepalived | 2.1 | - | - |
|  mariadb-10.5  | / mariadb-10.5 | 192.168.40.61 | 192.168.40.61 | keepalived | 2.1 | - | - |
|  mariadb-10.5  | / mariadb-10.5 | 192.168.121.22 | 192.168.121.22 | haproxy | 1.8.27 | - | - |
| pgsql-18 | /pgsql-18 | 192.168.40.63 | 192.168.40.63 | postgres | 18.4 | - | - |
| pgsql-18 | /pgsql-18 | 192.168.40.64 | 192.168.40.64 | postgres | 18.4 | - | - |
| pgsql-18 | /pgsql-18 | 192.168.40.10 | 192.168.40.10 | controller | 2.5.0.22113 | - | - |
| pgsql-18 | /pgsql-18 | 192.168.40.3 | 192.168.40.3 | prometheus | 2.53.5 | - | <br><br><br><br> |
| pgsql-18 | /pgsql-18 | 192.168.40.63 | 192.168.40.63 | haproxy | 1.8.27 | - | - |
| pgsql-18 | /pgsql-18 | 192.168.40.64 | 192.168.40.64 | haproxy | 1.8.27 | - | - |
| pgsql-18 | /pgsql-18 | 192.168.40.63 | 192.168.40.63 | keepalived | 2.1 | - | - |
| pgsql-18 | /pgsql-18 | 192.168.40.64 | 192.168.40.64 | keepalived | 2.1 | - | - |
| clickhouse | /clickhouse | 192.168.40.67 | 192.168.40.67 | clickhouse | 25.3 | - | - |
| clickhouse | /clickhouse | 192.168.40.68 | 192.168.40.68 | clickhouse | 25.3 | - | - |
| clickhouse | /clickhouse | 192.168.40.69 | 192.168.40.69 | clickhouse | 25.3 | - | - |
| clickhouse | /clickhouse | 192.168.40.30 | 192.168.40.30 | clickhouse_keeper | 25.3 | - | - |
| clickhouse | /clickhouse | 192.168.40.31 | 192.168.40.31 | clickhouse_keeper | 25.3 | - | - |
| clickhouse | /clickhouse | 192.168.40.32 | 192.168.40.32 | clickhouse_keeper | 25.3 | - | - |
| clickhouse | /clickhouse | 192.168.40.5 | 192.168.40.5 | controller | 2.5.0.22113 | - | - |
| clickhouse | /clickhouse | 192.168.40.10 | 192.168.40.10 | prometheus | 2.53.5 | - | <br><br><br><br> |
| pgsql-14 | /pgsql-14. | 192.168.40.66 | 192.168.40.66 | postgres | 14.23 | - | - |
| pgsql-14 | /pgsql-14. | 192.168.40.62 | 192.168.40.62 | postgres | 14.23 | - | - |
| pgsql-14 | /pgsql-14. | 192.168.40.4 | 192.168.40.4 | controller | 2.5.0.22113 | - | - |
| mariadb-galera | /mariadb-galera | 192.168.40.33 | 192.168.40.33 | galera | 11.8.8-MariaDB-log | MariaDB Server | - |
| mariadb-galera | /mariadb-galera | 192.168.40.10 | 192.168.40.10 | controller | 2.5.0.22113 | - | - |
| mariadb-galera | /mariadb-galera | 192.168.40.34 | 192.168.40.34 | galera | 11.8.8-MariaDB-log | MariaDB Server | - |
| mariadb-galera | /mariadb-galera | 192.168.40.35 | 192.168.40.35 | galera | 11.8.8-MariaDB-log | MariaDB Server | - |
| mariadb-galera | /mariadb-galera | 192.168.40.10 | 192.168.40.10 | prometheus | 2.53.5 | - | <br><br><br> |

Remove a controller from the pool

s9s pool-controllers --remove-controller --controller-id=<id>

Optionally uninstall the package during removal: 

s9s pool-controllers --remove-controller --controller-id=<id> --uninstall

Make sure that, after running the --uninstall, verify if the behavior does remove the cmon packages correctly.

Setting the maximum capacity of clusters for a controller

You can also set the value for --set-max-clusters-capacity with the following values with its definition:

For example, setting the maximum capacity to 10 clusters for a controller host 192.168.40.3:

      s9s pool-controllers  \
      --set-max-clusters-capacity=10 \
      --controller="https://192.168.40.3:9501"
Aside from setting your desired values, there are available values you can also set: - -2 = auto - -1 = unlimited - 0 = inactive - >0 = explicit cap

Use --force if reducing the cap would require the controller to abandon clusters it already owns.

Upgrade a pool member from the main controller

s9s pool-controllers --update-cmon --controller-id=679

where 679 is the SID value you can find in s9s pool-controllers --list. This operation is intended to be run against the main controller, and the target pool member must satisfy the documented validation rules.

Operational guidance

  1. Set up a shared CMON database.
  2. Choose a shared configuration backend appropriate to the environment.
  3. Start the initial controller with pool mode enabled.
  4. Add additional controllers that point to the same CMON database and shared config backend.
  5. Use s9s pool-controllers --list and s9s cluster --pool to verify ownership distribution.
  6. For larger fleets, tune capacity and claim-step settings to avoid cold-start overload.

Common gotchas

  • Assuming Kubernetes is required: it is not. Kubernetes is one supported model, but VMs, bare metal, and cloud-hosted servers are also valid if shared DB and shared config are in place.
  • Missing shared config: pool mode requires all controllers to see the same configuration state.
  • Wrong HA expectations: scalable pooling and full HA readiness were not introduced at exactly the same time in all releases.
  • Privilege errors on pool-wide listing: use a superuser/admin account.
  • Runtime capacity changes disappearing after restart: only configuration-file values are persistent.
Copyright © 2014-2026 Severalnines AB. All rights reserved.

Severalnines and ClusterControl are registered trademarks in the US, UK, and EU. The 3rd-party trademarks on this site are property of their respective owners and are used for referential purposes only.

Made with Material for MkDocs