Scalable Controllers Pool

The Scalable Controllers Pool removes the operational limits and failure risk of relying on a single CMON controller in larger ClusterControl environments. As the number of managed clusters grows, one controller can become a capacity bottleneck and a single failure domain for cluster monitoring, orchestration, and job handling.

How it works

The pool lets multiple CMON controllers coordinate cluster ownership: controller capacity scales horizontally, and ownership moves automatically when controllers are added, removed, restarted, or become unavailable.

The pool works identically regardless of the underlying infrastructure. Controllers can run on virtual machines, bare-metal servers, or cloud-hosted instances using NFS or Vault as the shared configuration backend, or on Kubernetes using Kubernetes Secrets. The only requirement is that every controller in the pool connects to the same CMON database and shares the same configuration backend.

All pool members share a common source of truth:

All pool members connect to the same CMON database.
All pool members must use a shared configuration backend.
Each controller publishes its presence and status.
Clusters are owned by one controller at a time, and ownership can move when controllers are added, removed, restarted, or become stale.

Here's a diagram on how the controller pool works:

graph TD
    subgraph "<b>User Interface & Entry</b>"
        User((User/<br>Admin))
        CLI[ClusterControl CLI]
        GUI[ClusterControl GUI]
    end

    subgraph "<b>Main Controller (Role: Main)</b>"
        MC[CMON Main Instance]
        Proxy[cmon-proxy]
        JobScheduler[Job Scheduler]
    end

    subgraph "<b>Controller Pool (Role: Members)</b>"
        C1[CMON Controller 1]
        C2[CMON Controller 2]
        C3[CMON Controller <i>N</i>]
    end

    subgraph "<b>Shared Infrastructure</b>"
        DB[(Shared CMON DB)]
        Storage[(Shared Config<br/>NFS or Kubernetes Secrets)]
    end

    subgraph "<b>Managed Clusters</b>"
        CL1[Cluster A]
        CL2[Cluster B]
        CL3[Cluster C]
        CL4[Cluster D]
    end

    %% Connections
    User --> CLI & GUI --> Proxy


    Proxy --> JobScheduler
    JobScheduler -- "assigns jobs" --> C1
    JobScheduler -- "assigns jobs" --> C2
    JobScheduler -- "assigns jobs" --> C3

    %% Shared Access
    MC & C1 & C2 & C3 -.- DB & Storage

    %% Ownership
    C1 -- "owns" --> CL1
    C1 -- "owns" --> CL2
    C2 -- "owns" --> CL3
    C3 -- "owns" --> CL4

Key runtime concepts

Controllers table: stores controller identity, endpoint, status, and heartbeat information.
Cluster assignment table: tracks which controller owns which cluster.
Refresh cycle: each controller refreshes status and adjusts ownership.
Staleness threshold: if a controller stops refreshing in time, peers can reclaim its clusters.
Fair-share ownership: cluster ownership is distributed across active controllers, subject to controller capacity limits.

Operationally, a controller only needs to manage the subset of clusters that it currently owns. When ownership changes, the controller loads or unloads cluster state accordingly.

Shared configuration requirement

Shared configuration is a hard requirement in pool mode. ClusterControl supports the following backends:

Kubernetes Secrets for Kubernetes-native environments.
NFS or shared filesystem for VM and non-Kubernetes deployments.
Vault as a secure backend direction for non-Kubernetes/shared deployments.

Deployment and configuration model

Minimum prerequisites:

All controllers must connect to the same CMON database.
Each controller must have a unique hostname:port identity.
All controllers must use the same shared configuration backend.
For Kubernetes shared-config deployments, controllers are started with --pool --k8s.

Example startup pattern for non-Kubernetes environment shared config

# /etc/default/cmon
EXTRA_OPTS=--pool

Example startup pattern for Kubernetes-backed shared config

# /etc/default/cmon
EXTRA_OPTS=--pool --k8s

In multi-VM or external-controller setups using Kubernetes Secrets as the shared configuration store, the documented pattern also includes Kubernetes API endpoint and certificate environment variables so that each controller can access the same secret store.

KUBERNETES_SERVICE_HOST=hostA
KUBERNETES_SERVICE_PORT=16443
KUBERNETES_SERVICE_CA_CERT_PATH=/etc/certs/ca.crt
KUBERNETES_CLIENT_CERT_PATH=/etc/certs/client.crt
KUBERNETES_CLIENT_KEY_PATH=/etc/certs/client.key

Non-Kubernetes deployment model

The feature is also intended for non-Kubernetes environments. In those cases, the shared configuration backend is typically a shared filesystem such as NFS, while controllers still point to the same CMON database. This makes the feature applicable to:

Traditional VM deployments
Physical server or bare-metal deployments
Cloud VMs or instances outside Kubernetes

Capacity and cold-start hardening

The following settings control how many clusters a controller may own and how quickly a new or restarted controller claims them.

Configuration key	Meaning	Default behavior described
`controllers_pool_start_intervals_to_be_stable`	Refresh cycles before a starting controller becomes active	2 cycles
`controllers_pool_max_clusters`	Per-controller ownership cap	`-2` auto, `-1` unlimited, `0` inactive, `>0` explicit cap
`controllers_pool_clusters_per_gb`	RAM-based multiplier for auto capacity	10 clusters per GB
`controllers_pool_cluster_claim_step`	Clusters claimed per refresh cycle during slow start	Auto, about 20% of max capacity
`controllers_pool_soft_alarm_secs`	Warning threshold for unassigned clusters	75 seconds
`controllers_pool_hard_alarm_secs`	Critical threshold for unassigned clusters	150 seconds

These controls help avoid a cold-start controller claiming too many clusters at once, starving heartbeats, and triggering reassignment storms.

Managing the Controller Pool

Enabling the controller pool

Enabling the controller pool can be done from either the GUI or the s9s CLI.

ClusterControl GUIClusterControl CLI

In the left-side navigation bar, click Controller pool. If pool mode is not yet enabled, the UI notifies you and shows a Restart CMON in pool mode button.
Click Restart CMON in pool mode.
In the Allowed controllers network field, specify the CIDR block or blocks controllers are allowed to connect from (for example, 192.168.40.0/24, 10.120.0.0/24, 10.8.0.0/24). Click Enable.

Enabling pool mode adds a granted_controllers_network_mask entry to /etc/cmon.cnf (for example, granted_controllers_network_mask=192.168.40.0/24,192.168.121.0/24). The controller you enabled pool mode on keeps its existing clusters and is assigned the main role.

The s9s CLI adds a pool-controllers mode to manage the controller pool. To enable pool mode:

s9s pool-controllers --set-pool-mode --conf-storage=nfs --granted-network-mask=192.168.40.0/24

--set-pool-mode enables the controller pool.
--conf-storage=nfs sets the shared configuration backend. Use k8s for Kubernetes Secrets instead. Only nfs and k8s are allowed.
--granted-network-mask=192.168.40.0/24 sets the CIDR block controllers are allowed to connect from.

Prerequisites (ClusterControl 2.5.0+)

From ClusterControl 2.5.0, the first controller of a pool needs two prerequisites before pool mode can be enabled on it:

CC DB cluster: cmon's own database made highly available, as a MySQL InnoDB Cluster behind a local MySQL Router. cmon connects through the Router, and other controllers and additional database instances join the cluster later. The target version is Oracle MySQL 8.4 LTS. An existing MySQL 8.0 cmon database is accepted, but the bootstrap logs a warning that MySQL 8.0 has reached end of life and that you should move to 8.4 LTS.
CC configuration storage: an OpenBao secret store that all controllers of the pool share.

Enabling pool mode no longer sets these up for you. Each one is created by its own job, so you can verify every stage before you restart cmon in pool mode. Until both are in place, enabling pool mode on the first controller fails, and the missing prerequisites are listed. There is nothing to set up when the controller runs in Kubernetes mode or when other controllers are already in the pool.

ClusterControl GUIClusterControl CLI

In the left-side navigation bar, click Controller pool. Every missing prerequisite is shown with an explanation of what it is for and a button that starts the job to set it up. Once both prerequisites are ready, the page continues with the enable wizard described in Enabling the controller pool.

Check which prerequisites are still missing. This command is read-only and works while pool mode is off. It prints the commands to run, in order:
```
s9s pool-controllers --pool-readiness
```
If cmon's database is MariaDB, migrate it to Oracle MySQL first, because the CC DB cluster needs Oracle MySQL:
```
s9s pool-controllers --migrate-db
```
The automatic migration is supported only on RHEL-family hosts (EL8, EL9 and EL10). On Debian and Ubuntu it is refused, and --pool-readiness shows the reason. Follow Migrating the CMON database to MySQL on Debian and Ubuntu instead.
Bootstrap the CC DB cluster. This job does not restart cmon, and it does nothing if the cluster already exists:
```
s9s pool-controllers --bootstrap-db --log
```

Install and bootstrap OpenBao on a host of your choice:

s9s pool-controllers --add-openbao --nodes=HOST --log

Run --pool-readiness again. When it reports ready, enable pool mode:
```
s9s pool-controllers --set-pool-mode
```

If you run cmon's database HA or the configuration storage yourself, skip the matching check with --no-require-db-cluster or --no-require-config-storage, added to --set-pool-mode.

The migration stops cmon

--migrate-db runs in the background, and cmon is stopped for several minutes. While it is stopped, managed clusters get no monitoring, no auto-recovery and no scheduled backups. Plan a maintenance window. The migration is refused while cmon has running jobs, so wait for them to finish first. Until cmon is back, --pool-readiness cannot connect. If the migration fails before cmon is running on MySQL, it rolls back to MariaDB and restarts cmon. Check journalctl -u cmon-db-migration before you retry.

Migrating the CMON database to MySQL on Debian and Ubuntu

On Debian and Ubuntu, --migrate-db is not available, so you have to move cmon's database from MariaDB to Oracle MySQL 8.4 LTS by hand. Run every step below as root on the controller host.

Check that the procedure applies. s9s pool-controllers --pool-readiness reports the DB backend as mariadb and says the migration is not supported on this host. cmon, and with it auto-recovery, stays down for the whole procedure, so plan a maintenance window. Take a backup of cmon's database before you start.

Record the current setup and back up the configuration. Note the database user cmon logs in as (cmon_user, or its alias cmondb_user, in /etc/cmon.cnf; cmon if neither is set), the mysql_* keys (hostname, port, socket and password), the hosts of that database user, and the installed MariaDB packages. The following steps use the CMON_USER shell variable, so run them in the same shell:

grep -E '^(cmon_user|cmondb_user|mysql_)' /etc/cmon.cnf
CMON_USER=$(sed -n -E 's/^(cmon_user|cmondb_user)[[:space:]]*=[[:space:]]*//p' /etc/cmon.cnf | tr -d "\"' " | head -n 1)
CMON_USER=${CMON_USER:-cmon}
mariadb -u"$CMON_USER" -p -e "SELECT user, host FROM mysql.user WHERE user = '$CMON_USER'"
dpkg -l | grep -i mariadb
cp -a /etc/cmon.cnf /root/cmon.cnf.mariadb
cp -a /etc/cmon.d /root/cmon.d.mariadb

Stop the ClusterControl services:

systemctl stop cmon cmon-cloud cmon-events cmon-ssh

Dump the cmon databases and save the row counts. Leave dcps out if that database does not exist:

mariadb-dump -u"$CMON_USER" -p --single-transaction --routines --triggers \
  --databases cmon dcps > /root/cmon-mariadb.sql

Recent MariaDB versions add a first line containing enable the sandbox mode, which MySQL cannot parse. Delete it. Then check that the dump uses no MariaDB-only uca1400 collations. The count must be 0. If it is not, stop here, because this procedure does not cover that case:

sed -i '1{/enable the sandbox mode/d}' /root/cmon-mariadb.sql
grep -c uca1400 /root/cmon-mariadb.sql

Save the per-table row counts to compare after the import:

mariadb -u"$CMON_USER" -p -N -e "SELECT CONCAT('SELECT ''', table_schema, '.', table_name, ''', COUNT(*) FROM \`', table_schema, '\`.\`', table_name, '\`;') FROM information_schema.tables WHERE table_schema IN ('cmon', 'dcps') AND table_type = 'BASE TABLE' ORDER BY 1" > /root/cmon-count.sql
mariadb -u"$CMON_USER" -p -N < /root/cmon-count.sql > /root/cmon-counts.mariadb.txt

Stop MariaDB and archive its data and configuration:

systemctl stop mariadb
systemctl disable mariadb
tar -C /var/lib -czpf /root/mariadb-datadir.tgz mysql
cp -a /etc/mysql /root/etc-mysql.mariadb

Remove the MariaDB server. Use remove, not purge, so that a rollback is possible. Simulate the removal first with apt-get -s, and do not continue if it would also remove ClusterControl packages. Then move the old data directory aside:
```
apt-get -s remove mariadb-server mariadb-client
apt-get remove -y mariadb-server mariadb-client
mv /var/lib/mysql /var/lib/mysql.mariadb
```
Install Oracle MySQL 8.4 LTS from repo.mysql.com. This uses the same keyring as the repository ClusterControl sets up for MySQL Shell and MySQL Router. Check that repo.mysql.com has a suite for your release. On Ubuntu 26.04 you can skip the repository and install the distribution's own MySQL 8.4 packages:
```
curl -fsSL https://repo.mysql.com/RPM-GPG-KEY-mysql-2025 | \
  gpg --batch --yes --dearmor -o /usr/share/keyrings/mysql.gpg
. /etc/os-release
echo "deb [signed-by=/usr/share/keyrings/mysql.gpg] https://repo.mysql.com/apt/${ID}/ ${VERSION_CODENAME} mysql-8.4-lts" \
  > /etc/apt/sources.list.d/mysql-8.4-lts-community.list
apt-get update
apt-cache policy mysql-server
apt-get -s install 'mysql-server=8.4.*'
DEBIAN_FRONTEND=noninteractive apt-get install -y 'mysql-server=8.4.*'
mysql -uroot -e "SELECT VERSION()"
```
Before you install, check that apt-cache policy lists an 8.4 version from repo.mysql.com. Also check that the apt-get -s dry run installs 8.4 packages only and removes no ClusterControl packages. After the install, SELECT VERSION() must return 8.4.x.

Note

ClusterControl writes /etc/apt/preferences.d/s9s-repo-mysql-com-pin, which gives repo.mysql.com priority 1 so that only MySQL Shell and MySQL Router come from it. If that file exists, you must keep the explicit =8.4.* version, or apt installs the distribution's MySQL instead. If the dry run still resolves a dependency to a distribution package, add that package to the command with =8.4.* as well.
Recreate the cmon database users with the password from mysql_password in /etc/cmon.cnf. Create one user for every host recorded in step 2, at least localhost and 127.0.0.1. Keep MySQL's default authentication plugin. Replace <cmon_user> with the value of $CMON_USER and <cmon_password> with that password:
```
CREATE USER '<cmon_user>'@'localhost' IDENTIFIED BY '<cmon_password>';
GRANT ALL ON *.* TO '<cmon_user>'@'localhost' WITH GRANT OPTION;
CREATE USER '<cmon_user>'@'127.0.0.1' IDENTIFIED BY '<cmon_password>';
GRANT ALL ON *.* TO '<cmon_user>'@'127.0.0.1' WITH GRANT OPTION;
```
Run these statements with mysql -uroot.

Import the dump and compare the row counts. The diff must be empty:

mysql -uroot < /root/cmon-mariadb.sql
mysql -uroot -N < /root/cmon-count.sql > /root/cmon-counts.mysql.txt
diff /root/cmon-counts.mariadb.txt /root/cmon-counts.mysql.txt

Point cmon at MySQL. In /etc/cmon.cnf, and in every /etc/cmon.d/cmon_*.cnf that sets these keys, set:
```
mysql_hostname=127.0.0.1
mysql_port=3306
mysql_socket=/var/run/mysqld/mysqld.sock
```

Start the services and verify:

systemctl start cmon cmon-cloud cmon-events cmon-ssh
s9s pool-controllers --pool-readiness

The DB backend must now be mysql, with no migration required.

Continue with the remaining prerequisites in the GUI or the CLI: bootstrap the CC DB cluster (--bootstrap-db), set up OpenBao (--add-openbao), then enable pool mode (--set-pool-mode). See Prerequisites (ClusterControl 2.5.0+).

Rollback

If a step fails before cmon is running on MySQL, go back to MariaDB:

systemctl stop cmon cmon-cloud cmon-events cmon-ssh
systemctl stop mysql
systemctl disable mysql
mv /var/lib/mysql /var/lib/mysql.oracle
mv /etc/mysql /etc/mysql.oracle
mv /var/lib/mysql.mariadb /var/lib/mysql
cp -a /root/etc-mysql.mariadb /etc/mysql
rm /etc/apt/sources.list.d/mysql-8.4-lts-community.list
apt-get update
apt-get remove -y mysql-server
DEBIAN_FRONTEND=noninteractive apt-get install -y -o Dpkg::Options::=--force-confold \
  mariadb-server mariadb-client
cp -a /root/cmon.cnf.mariadb /etc/cmon.cnf
rm -rf /etc/cmon.d && cp -a /root/cmon.d.mariadb /etc/cmon.d
systemctl enable --now mariadb
systemctl start cmon cmon-cloud cmon-events cmon-ssh

Reinstall the MariaDB packages and versions you recorded in step 2 (for example mariadb-server=VERSION). If they came from the MariaDB repository and not from the distribution, install them from that repository again. The MariaDB data and configuration are back in place before the packages are reinstalled, so the installation reuses them. If /var/lib/mysql.mariadb is missing or damaged, restore the data directory from the archive instead, with tar -C /var/lib -xzpf /root/mariadb-datadir.tgz.

Adjusting controller capacity at runtime

Set the maximum number of clusters a controller may own with --set-max-clusters-capacity (CLI) or through the GUI. This is only available once the controller pool is enabled. See Capacity and cold-start hardening for what the -2/-1/0/>0 values mean.

ClusterControl GUIClusterControl CLI

In the left-side navigation bar, click Controller pool.
Choose a controller and click its ellipsis button (...).
Choose Set capacity.
In the Max clusters field, enter the maximum number of clusters this controller may own. Setting the capacity lower than the number of clusters currently owned causes the controller to abandon the excess clusters. To move clusters back onto this controller later, raise the capacity again.
Click Apply.

s9s pool-controllers \
  --set-max-clusters-capacity=40 \
  --controller="https://HOST:9501" \
  --cmon-user=USER \
  --password=PASS

--set-max-clusters-capacity=40 sets this controller's capacity to 40 clusters.
--controller="https://HOST:9501" targets the controller to change.
--cmon-user and --password are your ClusterControl credentials. Omit them if they are already set in your S9S_USER_CONFIG file.
Add --force if lowering the capacity would require the controller to abandon clusters it already owns.

Starting or stopping a controller

Starting and stopping a controller works the same way as starting or stopping the cmon daemon via systemd. You can only start a stopped controller, and only stop a running one.

ClusterControl GUIClusterControl CLI

In the left-side navigation bar, click Controller pool.
Choose a controller and click its ellipsis button (...).
Choose Stop or Start.
Confirm the action in the prompt that appears.

s9s pool-controllers \
  --start \
  --controller-id SID

--start starts the controller's cmon daemon. Use --stop to stop a running controller instead.
--controller-id SID is the controller's SID, not its runtime ID. See View the controller pool for the difference.

Note

A controller with the main role cannot be stopped, only controllers with the member role.

Removing a controller from the pool

Removing a controller applies only to a controller that is already stopped.

ClusterControl GUIClusterControl CLI

In the left-side navigation bar, click Controller pool.
Choose a controller and click its ellipsis button (...).
Choose Remove.
In the prompt, choose one of two options:
- Unregister controller only: Unregisters the controller from the pool but leaves the cmon daemon and its installation untouched.
- Unregister and uninstall controller: Unregisters the controller and uninstalls its packages, ensuring the daemon is fully stopped.

s9s pool-controllers \
  --remove-controller \
  --controller-id SID

--remove-controller removes the controller from the pool.
--controller-id SID is the controller's SID, not its runtime ID. See View the controller pool for the difference.

Add --uninstall to also remove the cmon packages from that host:

s9s pool-controllers --remove-controller --controller-id SID --uninstall

Upgrading a controller's cmon version

ClusterControl GUIClusterControl CLI

In the left-side navigation bar, click Controller pool.
Choose a controller and click its ellipsis button (...).
Choose Upgrade.
Confirm the action in the prompt that appears.

The controller must be stopped before it can be upgraded.

s9s pool-controllers \
  --update-cmon \
  --controller-id SID

--update-cmon upgrades the controller to the latest cmon version available in the Severalnines public repository.
--controller-id SID is the controller's SID, not its runtime ID. See View the controller pool for the difference.

This also works against a specific pool member when run from the main controller.

Inspecting the controller pool with the CLI

Beyond the management operations above, the s9s CLI can also inspect pool status and ownership through related cluster and node commands.

View the controller pool

$ s9s pool-controllers --list

This shows controller SID, ID, hostname, port, status, role, and assigned clusters.

$ s9s pool-controllers --list
SID ID HOSTNAME      PORT STATUS ROLE   COUNT/MAX CLUSTERS
13  14 192.168.40.10 9500 active member      2/36 [2, 9]
7   24 192.168.40.5  9500 active main        2/36 [5, 10]
11  31 192.168.40.3  9500 active member      1/38 [1]
9   32 192.168.40.4  9500 active member      1/38 [8]

The ROLE field identifies which controller is your main controller, host 192.168.40.5 in this example.

The columns show:

SID: Static controller ID (persistent config identifier). Use this for operations like --start, --stop, --remove-controller.
ID: Runtime/dynamic controller instance ID (can change with runtime/re-registration).
HOSTNAME: Controller host/IP.
PORT: Controller RPC/API port (typically 9500/9501 depending setup).
STATUS: Current state (e.g., active, starting, stopped, restarting, etc.).
ROLE: Pool role (main = primary config/db holder, member = regular pool member).
COUNT/MAX: owned_clusters / max_cluster_capacity for that controller (e.g., 2/36 means owns 2 clusters, cap is 36).
CLUSTERS: Cluster IDs currently assigned to that controller (e.g., [2, 9]).

In the example above, the controller at 192.168.40.10 owns clusters 2 and 9, out of a capacity of 36.

Identifying ownership from the cluster perspective

Combine the result of s9s pool-controllers --list with s9s cluster --list to see which cluster names and details belong to a given controller:

$ s9s cluster --list --long --controller="https://192.168.40.10:9501" --cmon-user=admin2 --password="userP@55" --rpc-tls

Result displays same as the following example:

ID STATE   TYPE              OWNER  GROUP  NAME           COMMENT
 2 STARTED postgresql_single admin2 admins pgsql-18       All nodes are operational.
 9 STARTED galera            admin2 admins mariadb-galera All nodes are operational.
Total: 2

For security purposes, and you do not want to expose the password, you can use the following:

$ export S9S_USER_CONFIG=/root/.s9s/s9s-controller10.conf;  s9s cluster --list --long --controller="https://192.168.40.10:9501" --rpc-tls
ID STATE   TYPE              OWNER  GROUP  NAME           COMMENT
 2 STARTED postgresql_single admin2 admins pgsql-18       All nodes are operational.
 9 STARTED galera            admin2 admins mariadb-galera All nodes are operational.
Total: 2

Where the file /root/.s9s/s9s-controller10.conf contains the following contents:

[global]
controller    = https://192.168.40.10:9501
cmon_user     = "admin"
cmon_password = "7201231b-8f0a-4d5e-a755-991e32ce3f66"

Note that querying controller 192.168.40.10 with s9s cluster --list --long only returns clusters 2 and 9, the clusters it owns. A cluster can only be viewed, listed, or identified through the controller that currently owns it, whether that controller is specified with --controller or through the controller parameter in S9S_USER_CONFIG.

Listing the nodes of the cluster

To list the nodes owned by a specific controller, run s9s node --list --long against that controller's address.

For example, from the main controller 192.168.40.5, you can check what nodes each pool member has:

$ s9s node --list --long --controller="https://192.168.40.3:9501"

STAT VERSION     CID CLUSTER        HOST           PORT COMMENT
ho-- 1.8.27        1  mariadb-10.5  192.168.121.22 9600 Process 'haproxy' is running.
coC- 2.5.0.22113   1  mariadb-10.5  192.168.40.3   9500 Up and running.
Po-- 2.53.5        1  mariadb-10.5  192.168.40.4   9090 Process 'prometheus' is running.
soM- 10.5.29       1  mariadb-10.5  192.168.40.60  3306 Up and running (read-write).
koM- 2.1           1  mariadb-10.5  192.168.40.60   112 Process 'keepalived' is running.
soS- 10.5.29       1  mariadb-10.5  192.168.40.61  3306 Up and running (read-only).
ho-- 1.8.27        1  mariadb-10.5  192.168.40.61  9600 Process 'haproxy' is running.
ko-- 2.1           1  mariadb-10.5  192.168.40.61   112 Process 'keepalived' is running.
Total: 8

Or check what nodes and clusters (by CID) controller host 192.168.40.4 has:

$ s9s node --list --long --controller="https://192.168.40.4:9501"
STAT VERSION     CID CLUSTER  HOST          PORT COMMENT
coC- 2.5.0.22113   8 pgsql-14 192.168.40.4  9500 Up and running.
poM- 14.23         8 pgsql-14 192.168.40.62 5432 Up and running (read-write).
poS- 14.23         8 pgsql-14 192.168.40.66 5432 Up and running (read-only).
Total: 3

Or check what nodes and clusters (by CID) controller host 192.168.40.10 has:

$ s9s node --list --long --controller="https://192.168.40.10:9501"
STAT VERSION     CID CLUSTER        HOST          PORT COMMENT
coC- 2.5.0.22113   2 pgsql-18       192.168.40.10 9500 Up and running.
Po-- 2.53.5        2 pgsql-18       192.168.40.3  9090 Process 'prometheus' is running.
p-S- 18.4          2 pgsql-18       192.168.40.63 5432 Shut down (read-only).
ho-- 1.8.27        2 pgsql-18       192.168.40.63 9600 Process 'haproxy' is running.
koM- 2.1           2 pgsql-18       192.168.40.63  112 Process 'keepalived' is running.
poM- 18.4          2 pgsql-18       192.168.40.64 5432 Up and running (read-write).
ho-- 1.8.27        2 pgsql-18       192.168.40.64 9600 Process 'haproxy' is running.
ko-- 2.1           2 pgsql-18       192.168.40.64  112 Process 'keepalived' is running.
coC- 2.5.0.22113   9 mariadb-galera 192.168.40.10 9500 Up and running.
Po-- 2.53.5        9 mariadb-galera 192.168.40.10 9090 Process 'prometheus' is running.
goM- 11.8.8        9 mariadb-galera 192.168.40.33 3306 Up and running (read-write).
goM- 11.8.8        9 mariadb-galera 192.168.40.34 3306 Up and running (read-write).
goM- 11.8.8        9 mariadb-galera 192.168.40.35 3306 Up and running (read-write).
Total: 13

Additionally, to see hostname/IP, node type, and database version for every node across all controllers in a single table, run the following from the main controller:

$ s9s cluster --list --all-pool --long --print-json | jq -r '
  ["cluster","cdt_path","ip","hostname","nodetype","version","version_comment","addresses"] as $cols
  | ($cols | "| " + join(" | ") + " |"),
    ($cols | map("---") | "| " + join(" | ") + " |"),
    ( [.. | objects | select(.class_name? == "CmonClusterInfo")][]
      | (.cluster_name // .cdt_path) as $cluster
      | .hosts[]?
      | [ $cluster,
          ((.cdt_path // "-") | gsub("^\\s+|\\s+$"; "")),
          (.ip // "-"),
          (.hostname // "-"),
          (.nodetype // .class_name // "-"),
          (.version // "-"),
          (.version_comment // "-"),
          ([.configuration[]?.address] | join("<br>") | if . == "" then "-" else . end)
        ]
      | map(tostring | gsub("\\|"; "\\|"))
      | "| " + join(" | ") + " |"
    )'
| cluster | cdt_path | ip | hostname | nodetype | version | version_comment | addresses |
| --- | --- | --- | --- | --- | --- | --- | --- |
|  mariadb-10.5  | / mariadb-10.5 | 192.168.40.60 | 192.168.40.60 | mysql | 10.5.29-MariaDB-log | MariaDB Server | - |
|  mariadb-10.5  | / mariadb-10.5 | 192.168.40.3 | 192.168.40.3 | controller | 2.5.0.22113 | - | - |
|  mariadb-10.5  | / mariadb-10.5 | 192.168.40.4 | 192.168.40.4 | prometheus | 2.53.5 | - | <br><br><br><br> |
|  mariadb-10.5  | / mariadb-10.5 | 192.168.40.61 | 192.168.40.61 | mysql | 10.5.29-MariaDB-log | MariaDB Server | - |
|  mariadb-10.5  | / mariadb-10.5 | 192.168.40.61 | 192.168.40.61 | haproxy | 1.8.27 | - | - |
|  mariadb-10.5  | / mariadb-10.5 | 192.168.40.60 | 192.168.40.60 | keepalived | 2.1 | - | - |
|  mariadb-10.5  | / mariadb-10.5 | 192.168.40.61 | 192.168.40.61 | keepalived | 2.1 | - | - |
|  mariadb-10.5  | / mariadb-10.5 | 192.168.121.22 | 192.168.121.22 | haproxy | 1.8.27 | - | - |
| pgsql-18 | /pgsql-18 | 192.168.40.63 | 192.168.40.63 | postgres | 18.4 | - | - |
| pgsql-18 | /pgsql-18 | 192.168.40.64 | 192.168.40.64 | postgres | 18.4 | - | - |
| pgsql-18 | /pgsql-18 | 192.168.40.10 | 192.168.40.10 | controller | 2.5.0.22113 | - | - |
| pgsql-18 | /pgsql-18 | 192.168.40.3 | 192.168.40.3 | prometheus | 2.53.5 | - | <br><br><br><br> |
| pgsql-18 | /pgsql-18 | 192.168.40.63 | 192.168.40.63 | haproxy | 1.8.27 | - | - |
| pgsql-18 | /pgsql-18 | 192.168.40.64 | 192.168.40.64 | haproxy | 1.8.27 | - | - |
| pgsql-18 | /pgsql-18 | 192.168.40.63 | 192.168.40.63 | keepalived | 2.1 | - | - |
| pgsql-18 | /pgsql-18 | 192.168.40.64 | 192.168.40.64 | keepalived | 2.1 | - | - |
| clickhouse | /clickhouse | 192.168.40.67 | 192.168.40.67 | clickhouse | 25.3 | - | - |
| clickhouse | /clickhouse | 192.168.40.68 | 192.168.40.68 | clickhouse | 25.3 | - | - |
| clickhouse | /clickhouse | 192.168.40.69 | 192.168.40.69 | clickhouse | 25.3 | - | - |
| clickhouse | /clickhouse | 192.168.40.30 | 192.168.40.30 | clickhouse_keeper | 25.3 | - | - |
| clickhouse | /clickhouse | 192.168.40.31 | 192.168.40.31 | clickhouse_keeper | 25.3 | - | - |
| clickhouse | /clickhouse | 192.168.40.32 | 192.168.40.32 | clickhouse_keeper | 25.3 | - | - |
| clickhouse | /clickhouse | 192.168.40.5 | 192.168.40.5 | controller | 2.5.0.22113 | - | - |
| clickhouse | /clickhouse | 192.168.40.10 | 192.168.40.10 | prometheus | 2.53.5 | - | <br><br><br><br> |
| pgsql-14 | /pgsql-14. | 192.168.40.66 | 192.168.40.66 | postgres | 14.23 | - | - |
| pgsql-14 | /pgsql-14. | 192.168.40.62 | 192.168.40.62 | postgres | 14.23 | - | - |
| pgsql-14 | /pgsql-14. | 192.168.40.4 | 192.168.40.4 | controller | 2.5.0.22113 | - | - |
| mariadb-galera | /mariadb-galera | 192.168.40.33 | 192.168.40.33 | galera | 11.8.8-MariaDB-log | MariaDB Server | - |
| mariadb-galera | /mariadb-galera | 192.168.40.10 | 192.168.40.10 | controller | 2.5.0.22113 | - | - |
| mariadb-galera | /mariadb-galera | 192.168.40.34 | 192.168.40.34 | galera | 11.8.8-MariaDB-log | MariaDB Server | - |
| mariadb-galera | /mariadb-galera | 192.168.40.35 | 192.168.40.35 | galera | 11.8.8-MariaDB-log | MariaDB Server | - |
| mariadb-galera | /mariadb-galera | 192.168.40.10 | 192.168.40.10 | prometheus | 2.53.5 | - | <br><br><br> |

Operational guidance

Recommended workflow

Set up a shared CMON database.
Choose a shared configuration backend appropriate to the environment.
Start the initial controller with pool mode enabled.
Add additional controllers that point to the same CMON database and shared config backend.
Use s9s pool-controllers --list and s9s cluster --list --all-pool to verify ownership distribution.
For larger fleets, tune capacity and claim-step settings to avoid cold-start overload.

Common gotchas

Assuming Kubernetes is required: it is not. Kubernetes is one supported model, but VMs, bare metal, and cloud-hosted servers are also valid if shared DB and shared config are in place.
Missing shared config: pool mode requires all controllers to see the same configuration state.
Wrong HA expectations: pool mode gives you horizontal scaling of controller capacity, but full HA/failover readiness has not shipped in the same release as pool mode for every version. Check your version's release notes before assuming both are available.
Privilege errors on pool-wide listing: use a superuser/admin account.
Runtime capacity changes disappearing after restart: only configuration-file values are persistent.