ClusterControl

This section provides information is your go-to resource for diagnosing and resolving issues that may arise within the ClusterControl platform.

Logging

ClusterControl consists of a number of components that write to their own log file. These files reside on the ClusterControl node. By default, cmon runs without the debug option enabled. See Debugging on how to get the run in debug mode.

If you encounter any problems with ClusterControl, it is highly recommended to examine the related log files.

Log Type	Default location (RedHat/CentOS)	Default location (Debian/Ubuntu)
CMON process log	/var/log/cmon.log or /var/log/cmon_<cluster_id>.log	/var/log/cmon.log or /var/log/cmon_<cluster_id>.log
CMON cloud log	/var/log/cmon-cloud.log	/var/log/cmon-cloud.log
CMON events log	/var/log/cmon-events.log	/var/log/cmon-events.log
CMON audit log	/var/log/cmon_audit.log	/var/log/cmon_audit.log
CMON upgrade log	/var/log/cmon_upgrade.log	/var/log/cmon_upgrade.log
CMON discovery log	/var/log/cmon_discovery.log	/var/log/cmon_discovery.log
Apache error log	- /var/log/httpd/ssl_error_log - /var/log/httpd/error_log	/var/log/apache2/error.log
Apache access log	- /var/log/httpd/access_log - /var/log/httpd/ssl_access_log	- /var/log/apache2/access.log -/var/log/apache2/ssl_access.log

Debugging

The debugging section describes more advanced troubleshooting in ClusterControl. It's usually used for cases related to cmon crashes and requires in-depth investigation.

Debugging ClusterControl Controller (CMON)

In case if you encounter a CMON crash, please install the debuginfo package and the necessary packages as shown below.

Install debugging components

Red Hat/CentOSDebian/Ubuntu

1. Enable the debug repo under /etc/yum.repos.d/CentOS-Debuginfo.repo and set enabled=1.

2. Install yum utilities:

   yum -y install yum-utils
   

3. Install ClusterControl debuginfo and gdb:

   yum -y install clustercontrol-controller-debuginfo gdb
   

4. Then, run:

   debuginfo-install clustercontrol-controller
   

1. Install ClusterControl debuginfo package and gdb:

   apt-get install clustercontrol-controller-dbg gdb
   

   Optionally, you can

2. Install the debugging components’ library:

   apt-get install libstdc++6-4.8-dbg libc6-dbg
   

   However, this totally depends on the libstdc++6 version installed. Print the shared object dependencies using ldd:

   ldd /usr/sbin/cmon | grep libstdc
           libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007ff508001000)
   

   Based on the library path, locate the package name that provides this library:

   dpkg -S /usr/lib/x86_64-linux-gnu/libstdc++.so.6
   libstdc++6:amd64: /usr/lib/x86_64-linux-gnu/libstdc++.so.6
   

   Then, find the package’s version:

   dpkg -l | grep libstdc++6
   ii libstdc++6:amd64 4.9.2-10 amd64 GNU Standard C++ Library v3
   

   In this case, we have version “4.9” installed for libstc++6. Finally, install the corresponding debug packages:

   apt-get install gdb libc6-dbg libstdc++-6-4.9-dbg
   

Debugging Steps

Debugging is a program that produces a core dump. It consists of the recorded state of the working memory of a computer program at a specific time, generally when the program has crashed or otherwise terminated abnormally. ClusterControl Controller (CMON) package comes with a crown file installed under /etc/cron.d/ which will auto-restart if the cmon process is terminated abnormally. Typically, you may notice if the cmon process has crashed by looking at the dmesg output.

In such cases, generating a core dump is the only way to backtrace the issue. Make sure you have the debugging components installed as described in the previous section beforehand. On ClusterControl node as the root user, increase the CPU limit, adjust the kernel’s core pattern value and run CMON on the foreground:

systemctl stop cmon
service cmon stop # for systemd only, to disable /etc/cron.d/cmon
ulimit -c unlimited
echo "/tmp/core.%e.%p.%h.%t" > /proc/sys/kernel/core_pattern
cmon -d

Or, if you want to run CMON as the background process, we need to create a unit override file for the cmon service:

systemctl edit cmon

And add the following lines:

[Service]
LimitCORE=infinity  # only when debugging

Then, you may start/stop the service using the systemctl command:

systemctl daemon-reload
systemctl start cmon

Warning

Only set LimitCORE=infinity for debugging purposes. To revert back to the standard setting, remove /etc/systemd/system/cmon.service.d/override.conf and reload systemd using the systemctl daemon-reload command.

When cmon crashes, there will now be a core file under /tmp. Compress the core dump (gzip is recommended) and attach it to a support ticket so we can take a look and perform the necessary fix. Alternatively, you can send only the backtrace in a support ticket by using the following command:

gdb /usr/sbin/cmon /tmp/<corefile>
thread apply all bt full

Attach the full output and potentially replace sensitive information with “XXXXXXXXX”. Traces may contain password information.

CMON on Foreground

If you would like to run cmon as a foreground process, you can do that by invoking -d option:

$ service cmon stop
$ CMON_DEBUG=1 cmon -d

CMON will enable LOG_DEBUG messages and print detailed information on the screen (stdout) as well as /var/log/cmon.log or /var/log/cmon_{cluster ID}.log. Press Ctrl + C to terminate the process. In some cases, this type of CMON output might be needed to get insight into the problem.

ClusterControl GUI diagnostic

For ClusterControl GUI-related issue, you can use Diagnostic feature available at ClusterControl GUI → Settings → Diagnostics and toggle ON UI audit data collection. This allows user to record and capture the necessary data while reproducing an issue in the user interface. Once the issue has been replicated, click on "Download audit data" for analysis or to share with our Support team for troubleshooting and debugging.

Warning

Remember to switch off the setting afterward to stop audit data collection.

Note

Note that we are not collecting any personal data during this process.