Tuning for Large Deployment
Managing hundreds of cluster inside ClusterControl need some special adjustment in the controller node, database, and cmon service.
Below are some configuration you might want to adjust for large scale deployment in ClusterControl.
Tuning InnoDB buffer pool size
ClusterControl components rely on a MySQL or MariaDB database as the persistent store for monitoring data collected from the managed nodes and all ClusterControl metadata (e.g., what jobs there are in the queue, backup schedules, backup statuses, etc.). By default, the installer script will install whatever version comes by the standard repository of the OS.
The installer script would do some basic tuning like configuring the datadir location, MySQL port, user, and InnoDB buffer pool size at the beginning of the installation stage. However, the tuning might not be suitable once you have imported or created more clusters/nodes. With an increased number of nodes to be monitored and managed, ClusterControl would use more resources, and the database layer is commonly the first bottleneck that users encounter. Some further tuning might be needed to contain the incoming load.
Buffer pool size is an important component and has to be configured upfront to improve the MySQL performance. It allows MySQL processing to be happening inside memory instead of hitting the disk. A simple rule of thumb is to check the InnoDB hit ratio and look for the following line under BUFFER POOL AND MEMORY section:
The hit rate of 986 / 1000 indicates that out of 1000 row reads, it was able to read the row in RAM 986 times. The remaining 14 times, it had to read the row of data from disk. Simply said, 1000 / 1000 is the best value that we are trying to achieve here, which means the frequently-accessed data fits fully in RAM.
Increasing the innodb_buffer_pool_size value will help a lot to accomodate more room for MySQL to work on. However, ensure you have sufficient RAM resources beforehand. By default, ClusterControl allocates 50% of the RAM to the buffer pool. If the host is dedicated to ClusterControl, you can even push it to a higher value like 70%, provided you spare at least 2GB of RAM to the OS processes and caches. If you can’t allocate that much, increasing the RAM is the only solution.
Changing this value requires a MySQL restart (older than MySQL 5.7.5), thus the correct service restart ordering will be:
- Modify
innodb_buffer_pool_sizevalue insidemy.cnf. - Stop the CMON service.
- Restart MySQL/MariaDB service.
- Start the CMON service.
Or simply reboot the host if you can afford a longer downtime.
Using MySQL socket file
By default, the installer script will automatically configure the following host value inside every ClusterControl database configuration files:
| Configuration File | Value |
|---|---|
/etc/cmon.cnf |
mysql_hostname=127.0.0.1 |
/etc/cmon.d/cmon_X.cnf (X is the cluster ID) |
mysql_hostname=127.0.0.1 |
The above will force the MySQL client to connect via TCP networking, just like connecting to a remote MySQL host although ClusterControl is running on the same server as the MySQL server. We purposely configured it this way to simplify the installation process since almost every OS platform pre-configures MySQL socket file differently, which greatly reduces the installation failure rate.
Changing the value to "localhost" will force the client to use the MySQL Unix socket file instead:
| Configuration File | Value |
|---|---|
/etc/cmon.cnf |
mysql_hostname=localhost |
/etc/cmon.d/cmon_X.cnf (X is the cluster ID) |
mysql_hostname=localhost |
On Unix based systems, MySQL programs treat the host name localhost specially, in a way that is likely different from what you expect compared to other network-based programs. For connections to localhost, MySQL programs attempt to connect to the local server by using a Unix socket file. This occurs even if a –port or -P option is given to specify a port number.
Using MySQL UNIX socket file is much more secure and will cut off the network overhead. It is always recommended over TCP. However, you need to make sure the socket file is configured correctly. It must exist on the following directives inside my.cnf and every MySQL option files on ClusterControl node, for example:
[mysqld]
socket=/var/lib/mysql/mysql.sock
[client]
socket=/var/lib/mysql/mysql.sock
[mysql]
socket=/var/lib/mysql/mysql.sock
[mysqldump]
socket=/var/lib/mysql/mysql.sock
Changing the socket file path in MySQL will also require a MySQL and CMON restart.
CMON pool size
Tune the pool size in /etc/cmon.conf, the default connection pool size for cmon is 64. You might want to increase the pool size in cmon.conf configurations depend on how much the cluster you have. You can increase 1,5x - 2x from the cluster size.
Restart ClusterControl controller (cmon) service to load the change:
File descriptors
In some huge database farm environment, there are some errors related to the file descriptors with message "too many open files" on the Prometheus. Tune the file descriptors in /etc/sysctl.conf by adding or increasing the fs.file-max parameter:
/etc/security/limits.conf, and set the soft/hard limits too in limits.conf related to open files:
Save the file and re-login the terminal session. Verify with:
OpenSSH with SSSD
ClusterControl uses SSH protocol as its main communication channel to manage and monitor remote nodes. The default OpenSSH configurations are pretty decent and should work in most cases. However, in some environments where SSH is integrated with other security enhancement tools like SElinux or System Security Services Daemon (SSSD), it could bring significant impact to the SSH performance.
We have seen many cases where an ever increasing amount of SSH connections established to each of the managed nodes and eventually, both the ClusterControl server and all managed nodes max out their system memory with SSH connections. In some cases, only a normal full system reboot nightly on the ClusterControl server could solve the problem.
If you running your infrastructure with System Security Services Daemon (SSSD), it’s advised for you to comment the following line inside OpenSSH client configuration at /etc/ssh/ssh_config on ClusterControl node:
The above will skip using SSSD to manage the host key, which will be handled by OpenSSH client instead.
File System and Partitioning
ClusterControl controller writes new entry in its log file almost every second for every cluster. For those who wants to take advantage of this sequential writes on disk and would like to save cost, you can use a spindle disk for this purpose. Modify the following line inside all cmon_X.cnf (where X is the cluster ID):
Restart CMON service to apply the change:
If you are using ClusterControl as the backup repository, it’s recommended for you to allocate sufficient disk space on a separate partition other than the root partition. It gets better if the partition resides on a networked or clustered file system for easy mounting with the targeted nodes when performing restoration operation. We have seen cases where the created backups ate up all disk space of the main partition, eventually impacting ClusterControl and its components.