Housekeeping

Prometheus storage retention

ClusterControl uses Prometheus to store time-series monitoring data. Prometheus is designed to store high-resolution time-series data, which can accumulate quickly. By default, the storage retention is set to 15 days with 10 seconds scrape interval. To put into perspective, with 5 database nodes (each has at least 3 exporters' endpoints), this can accumulate up to 5 GB total disk size (check the disk usage of /var/lib/prometheus).

You can adjust the Prometheus storage retention and scrape interval by going to ClusterControl GUI → Clusters → choose a cluster → Dashboards → More... → Configuration → Edit. Click Save and ClusterControl will reconfigure the agent-based monitoring (together with Prometheus reconfiguration) for the chosen cluster.

CMON data history

ClusterControl monitoring data will be purged based on the value set at ClusterControl GUI → Clusters → choose a cluster → Settings → System Settings → System → save_history_days (default is 7 days). Some users might find this value to be too low for auditing purposes. You can increase the value accordingly however, the longer collected data exist in the CMON database, the bigger space it needs. It is recommended to lower the disk space threshold under ClusterControl GUI → Clusters → choose a cluster → Settings → System Settings → Thresholds → diskspace so you will get an early warning in case the CMON database grows significantly.

Alternatively, you can modify the cmon configuration file (/etc/cmon.d/cmon_X.cnf, where X is the cluster id) and set save_history_days and pass how many days of the history you would like to keep. Please keep in mind you have to restart the cmon service to apply the change.

If you have configured a very short interval recurring jobs like backup jobs running every hour, it would produce lots of job activities. We suggest lowering the controller job history retention period, to like 1 or 2 days. It can be done by setting save_history_days=1 which mean the controller only keeps the last two days of job history.

Log rotation

The cmon process has internal log rotation scheduling where it will log up to 5 MB in size before archiving /var/log/cmon.log and /var/log/cmon_{cluster ID}.log. The archived log will be named as cmon.log.1 (or cmon_{cluster ID}.log.1) sequentially, with up to 9 archived log files (total of 10 log file rotation).