Tuning ClusterControl for Large Deployment

Managing hundreds of cluster inside ClusterControl need some special adjustment in the controller node, database, and cmon service. Below is some configuration you might want to adjust for large scale deployment in ClusterControl:

1. Tune the database pool size in /etc/cmon.conf​, the default connection pool size for cmon is 64. You might want to increase the pool size depending on how much the cluster you have. You can increase 1.5x – 2x from the cluster size.

cmon_pool_size=128

2. Tune the file descriptors in /etc/sysctl.conf​, as we see in the logs. There are some errors related to the file descriptors with the message too many open files on the Prometheus. You can add/increase fs.file-max parameter:

fs.file-max=1048576

3. Tune soft/hard limits in /etc/security/limits.conf​, please set the soft/hard limits too in limits.conf related to open files:

* soft nproc 1048576
* hard nproc 1048576
* soft nofile 1048576
* hard nofile 1048576

4. Tune MySQL/MariaDB database parameters in my.cnf on the ClusterControl server:

innodb_buffer_pool_size      = 30G  # 50 - 70% of total RAM
innodb_buffer_pool_instances = 30   # innodb_buffer_pool_size divide by 1G, take the nearest integer
innodb_log_file_size         = 1G   # bigger is better, but tradeoff with crash recovery time
innodb_io_capacity           = 2000 # if you use SSD
innodb_io_capacity_max       = 4000 # if you use SSD
innodb_read_io_threads       = 16   # to engage more CPU cores 
innodb_write_io_threads      = 16   # to engage more CPU cores
max_connections              = 500  # value of max_used_connections status, multiply by 2 or 3, don't set too high

Also, use the UNIX socket instead of TCP/IP to reduce network overhead when CMON accessing the CMON database inside /etc/cmon.d/cmon_*.cnf, and /var/www/html/clustercontrol/bootstrap.php, all MySQL hostname variables are changed from 127.0.0.1 to localhost (to make use of UNIX socket, instead of TCP/IP). See the example shown in this blog post, How to Optimize Performance of ClusterControl and Its Components.