2. Home
  4. Docs
  6. ClusterControl
  8. Troubleshooting
  10. Reporting and Debugging

Reporting and Debugging

If the above does not help identify the root cause of the issues, please refer to our knowledge base or contact us via our Support Portal by creating a support request. We encourage you to make use of our error reporting tool as described in the next section.

Error Report

ClusterControl provides an error reporting tool called s9s_error_reporter. This can greatly facilitate the troubleshooting process as it collects the necessary information on the entire database cluster setup and archives it in a compressed package. Use this tool to generate an error report and then attach the generated tarball package when creating a Severalnines Support Ticket.

To generate an error report for one particular cluster, simply go to ClusterControl → pick a cluster → Logs → Error Reports → Create an Error Report. Once the job completes, the error report package will be listed in this page. You can then download it into your workstation and use it for debugging and troubleshooting purposes or attach it to alongside your Severalnines Support Ticket.

You can also use the helper script to generate an error report via the command-line interface. Run the following command on ClusterControl node:

# as root
s9s_error_reporter -i 1
# or as a non-root user
sudo s9s_error_reporter -i 1

Where 1 is the cluster ID of the corresponding cluster. For other clusters, just change the cluster ID accordingly. The cluster ID can be retrieved from Database Cluster List or by looking into the CMON configuration file for the respective cluster inside /etc/cmon.d. If you have no cluster managed by ClusterControl, you might want to specify 0 instead, where 0 has a special meaning in ClusterControl – collect information about ClusterControl installation, configurations, and logs on the local node only.

# as root
s9s_error_reporter -i 0

At the end of the execution, it will print out something like this:

Executing tar -C /var/tmp/cmon-000809-42b7c3b595d5282a -czf '/var/www/html/clustercontrol/app/tmp/logs/error-report-2018-11-19_065637-cluster0.tar.gz' error-report-2018-11-19_065637-cluster0
Please attach /var/www/html/clustercontrol/app/tmp/logs/error-report-2018-11-19_065637-cluster0.tar.gz to the support issue.

Attach the generated tarball to your support issue. Sometimes, the generated tarball might get too big to be uploaded into our support system, where the attachment limit is 20MB in size. You probably want to use cloud storage services like Google Drive, Dropbox or WeTransfer to upload the error report there and share the download link in the support ticket. Please note that the error report might contain sensitive and confidential information. Restrict the file from public access and only share with us the HTTPS download URL.

Note

We also recommend you to take screenshots showing the area of the problem, e.g, the Overview page from the UI allows us to understand the current state of nodes and clusters.

Debugging ClusterControl Controller (CMON)

Starting from ClusterControl v1.3.0, ClusterControl comes with a debuginfo package. In case if you encounter a CMON crash, please install the debuginfo package and the necessary packages as shown below.

Install Debugging Components

Red Hat/CentOSDebian/Ubuntu

1. Enable the debug repo under /etc/yum.repos.d/CentOS-Debuginfo.repo and set enabled=1.

2. Install yum utilities:

yum -y install yum-utils

3. Install ClusterControl debuginfo and gdb:

yum -y install clustercontrol-controller-debuginfo gdb

4. Then, run:

debuginfo-install clustercontrol-controller

1. Install ClusterControl debuginfo package and gdb:

apt-get install clustercontrol-controller-dbg gdb

Optionally, you can

2. Install the debugging components’ library:

apt-get install libstdc++6-4.8-dbg libc6-dbg

However, this totally depends on the libstdc++6 version installed. Print the shared object dependencies using ldd:

ldd /usr/sbin/cmon | grep libstdc
        libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007ff508001000)

Based on the library path, locate the package name that provides this library:

dpkg -S /usr/lib/x86_64-linux-gnu/libstdc++.so.6
libstdc++6:amd64: /usr/lib/x86_64-linux-gnu/libstdc++.so.6

Then, find the package’s version:

dpkg -l | grep libstdc++6
ii libstdc++6:amd64 4.9.2-10 amd64 GNU Standard C++ Library v3

In this case, we have version “4.9” installed for libstc++6. Finally, install the corresponding debug packages:

apt-get install gdb libc6-dbg libstdc++-6-4.9-dbg

Debugging Steps

Debugging is a program that produces a core dump. It consists of the recorded state of the working memory of a computer program at a specific time, generally when the program has crashed or otherwise terminated abnormally. ClusterControl Controller (CMON) package comes with a crown file installed under /etc/cron.d/ which will auto-restart if the cmon process is terminated abnormally. Typically, you may notice if the cmon process has crashed by looking at the dmesg output.

In such cases, generating a core dump is the only way to backtrace the issue. Make sure you have the debugging components installed as described in the previous section beforehand. On ClusterControl node as the root user, increase the CPU limit, adjust the kernel’s core pattern value and run CMON on the foreground:

systemctl stop cmon
service cmon stop # for systemd only, to disable /etc/cron.d/cmon
ulimit -c unlimited
echo "/tmp/core.%e.%p.%h.%t" > /proc/sys/kernel/core_pattern
cmon -d

Or, if you want to run CMON as the background process, we need to create a unit override file for the cmon service:

systemctl edit cmon

And add the following lines:

[Service]
LimitCORE=infinity  # only when debugging

Then, you may start/stop the service using the systemctl command:

systemctl daemon-reload
systemctl start cmon
Warning

Only set LimitCORE=infinity for debugging purposes. To revert back to the standard setting, remove /etc/systemd/system/cmon.service.d/override.conf and reload systemd using the systemctl daemon-reload command.

When cmon crashes, there will now be a core file under /tmp. Compress the core dump (gzip is recommended) and attach it to a support ticket so we can take a look and perform the necessary fix. Alternatively, you can send only the backtrace in a support ticket by using the following command:

gdb /usr/sbin/cmon /tmp/<corefile>
thread apply all bt full

Attach the full output and potentially replace sensitive information with “XXXXXXXXX”. Traces may contain password information.

CMON on Foreground

If you would like to run cmon as a foreground process, you can do that by invoking -d option:

$ service cmon stop
$ CMON_DEBUG=1 cmon -d

CMON will enable LOG_DEBUG messages and print detailed information on the screen (stdout) as well as /var/log/cmon.log or /var/log/cmon_{cluster ID}.log. Press Ctrl + C to terminate the process. In some cases, this type of CMON output might be needed to get insight into the problem.

Debugging ClusterControl UI

To enable ClusterControl UI for debugging, SSH into the ClusterControl node and modify the following values inside /var/www/html/clustercontrol/app/Config/core.php:

Configure::write('debug', 0);

Where,

  • 0: Production mode. All errors and warnings are suppressed.
  • 1: Errors and warnings shown, model caches refreshed, flash messages halted.
  • 2: As in 1, but also with full debug messages and SQL output.

Make sure /var/www/html/clustercontrol/app/tmp has the write permission and is owned by Apache user for the debug and error log to be generated.

Was this article helpful to you? Yes No