ClusterControl Controller
ClusterControl Controller (cmon) is the core backend process that performs all automation and management procedures. It is installed as /usr/sbin/cmon and comes with a collection of helper scripts in /usr/bin the directory (prefixed with s9s_) to complete specific tasks. However, some of the scripts have been deprecated due to the corresponding tasks are now being handled by the CMON core process.
ClusterControl Controller packages are available at Severalnines Repository page. The packages are also available at the Severalnines download site. RedHat-based systems should download and install the RPM package while Debian-based systems should download and install the DEB package. The package name is formatted as:
- RPM package (RedHat-based systems):
clustercontrol-controller-{version}-{build number}-{architecture}.rpm - DEB package (Debian-based systems):
clustercontrol-controller-{version}-{build number}-{architecture}.deb
A configuration file /etc/cmon.cnf is required to initially run a cmon process. It is possible to have several configuration files for multiple clusters as described in the Configuration Files.
Command Line Arguments
By default, if you just run cmon (without any arguments), cmon defaults to run in the background. The binary supports several command-line options as shown below:
| Shorthand, Option | Description |
|---|---|
-h, --help |
Print help. |
--help-config |
Print the manual for configuration parameters. See Configuration Options. |
--help-init |
Shows the special options for --init. |
-v, --version |
Prints out the version number and build info. |
--logfile=[filepath] |
The path of the log file to be used. |
-s, --syslog |
Also, log to Syslog. |
-b, --bind-addr='ip1,ip2..' |
Bind Remote Procedure Call (RPC) to IP addresses (default is 127.0.0.1,::1). By default cmon binds to 127.0.0.1 and ::1. If another bind-address is needed, then it is possible to define the bind addresses in the file /etc/default/cmon. See Startup File. |
-c, --cloud-service=URL |
A custom clustercontrol-cloud service URL. |
-d, --nodaemon |
Run in the foreground. Ctrl + C to exit. |
-e, --events-client=URL |
Additional RPC URL where backend sends events. |
-g, --grant |
Create grants. |
-i, --init |
Creates configuration file and database. |
--log-rpc |
Log every RPC call (very verbose). |
--no-safety-checks |
Do not check if another cmon is connected. |
-p, --rpc-port=[integer] |
Listen on RPC port. The default is 9500. |
-r, --directory=[directory] |
Running directory. |
-u, --upgrade-schema |
Try to upgrade the CMON schema (Supported from CMON version 1.2.12 and later). |
Startup File
To customize the cmon startup process, you can define the command line arguments in a custom file, instead of hacking up the init script directly. The cmon init script (or systemd) will append all configuration options defined inside /etc/default/cmon when starting up the cmon process and translates those options into command line arguments. For example:
$ cat /etc/default/cmon
RPC_PORT=9500
RPC_BIND_ADDRESSES="10.10.10.13,192.168.33.1,127.0.0.1"
EVENTS_CLIENT=http://127.0.0.1:9510
CLOUD_SERVICE=http://127.0.0.1:9518
In the example above, cmon will bind into those IP addresses and listen to port 9500 once started. If you filter out the ps output from the server, you should get the following:
/usr/sbin/cmon --rpc-port=9500 --bind-addr='10.10.10.13,192.168.33.1,127.0.0.1' --events-client='http://127.0.0.1:9510' --cloud-service='http://127.0.0.1:9518
Configuration Options
Values that consist of special characters must be enclosed with a single-quote. Any value change to the configuration options that shall be applied to the cmon configuration file or indirectly via GUI or through CLI command, requires a service restart before they are applied.
Following is the list of configuration options inside the CMON configuration file. You can also see them by using --help-config the parameter in the terminal:
General Configurations
| Option | Description |
|---|---|
cluster_id=<integer> |
|
name=<string> |
|
type=<string> |
|
tags=<string> |
|
created_by_job=<integer> |
|
vendor=<string> |
|
use_internal_repos=<boolean integer> |
|
enable_html_emails=<boolean integer> |
|
frontend_url=<url> |
|
error_report_dir=<path> |
|
ops_report_retention=<integer> |
|
CMON
| Option | Description |
|---|---|
hostname=<string> |
|
controller_id=<integer> |
|
mode=<string> |
|
agentless=<boolean integer> |
|
logfile=<path> |
|
pidfile=<path> |
|
mysql_hostname=<string> |
|
mysql_password=<string> |
|
mysql_port=<integer> |
|
mysql_socket=<boolean integer> |
|
mysql_basedir=<path> |
|
mysql_bindir=<path> |
|
mysql_scriptdir=<path> |
|
config_file_path=<path> |
|
cmon_db=<string> |
|
cmon_user=<string> |
|
cmondb_ssl_key=<path> |
|
cmondb_ssl_cert=<path> |
|
cmondb_ssl_ca=<path> |
|
cluster_ssl_key=<path> |
|
cluster_ssl_cert=<path> |
|
cluster_ssl_ca=<path> |
|
cluster_certs_store=<path> |
|
cluster_ssl_enforce=<boolean integer> |
|
rpc_key=<string> |
|
rpc_version=<string> |
|
net_read_timeout=<integer> |
|
net_write_timeout=<integer> |
|
connect_timeout=<integer> |
|
owner=<integer> |
|
group_owner=<integer> |
|
cdt_path=<string> |
|
acl=<string> |
|
staging_dir=<path> |
|
plugin_dir=<path> |
|
cmon_use_mail=<boolean integer> |
|
cmon_mail_sender=<email> |
|
cmon_pool_size=<integer> |
|
cmon_ha_heartbeat_interval_millis=<integer> |
|
cmon_ha_heartbeat_millis_rand_max=<integer> |
|
cmon_ha_heartbeat_network_timeout=<integer> |
|
cmon_ha_heartbeat_timeout_millis=<integer> |
|
cmon_ha_max_send_retry=<integer> |
|
cmon_ha_send_retry_delay=<integer> |
|
vault_addr=<URI> |
|
vault_auto_migrate=<boolean integer> |
|
vault_path=<path> |
|
vault_token=<string> |
|
Operating System
| Option | Description |
|---|---|
os=<string> |
|
osuser=<string> |
|
sudo=<command> |
|
osuser_home=<path> |
|
software_packagedir=<path> |
|
local_repo_name=<string> |
|
init_name=<string> |
|
disable_numa=<boolean integer> |
|
SSH
| Option | Description |
|---|---|
ssh_identify=<path> |
|
ssh_port=<integer> |
|
ssh_acquire_tty=<boolean integer> |
|
ssh_password=<string> |
|
ssh_timeout=<integer> |
|
libssh_loglevel=<integer> |
|
slow_ssh_warning=<integer> |
|
slow_ssh_critical=<integer> |
|
Automatic Recovery
| Option | Description |
|---|---|
enable_cluster_autorecovery=<boolean integer> |
|
enable_node_autorecovery=<boolean integer> |
|
enable_autorecovery=<boolean integer> |
|
node_recovery_lock_file=<path> |
|
node_recovery_timeout=<integer> |
|
add_cluster_timeout=<integer> |
|
send_clear_cluster_failure=<boolean integer> |
|
Monitoring and Thresholds
| Option | Description |
|---|---|
monitored_mountpoints=<paths> |
|
monitored_nics=<string> |
|
db_stats_collection_interval=<integer> |
|
host_stats_collection_interval=<integer> |
|
host_stats_window_size=<integer> |
|
lb_stats_collection_interval=<integer> |
|
db_schema_stats_collection_interval=<integer> |
|
db_proc_stats_collection_interval=<integer> |
|
db_log_collection_interval=<integer> |
|
db_deadlock_check_interval=<integer> |
|
db_schema_max_objects=<integer> |
|
db_exporter_user=<string> |
|
db_exporter_password=<string> |
|
db_exporter_use_nonlocal_address=<boolean integer> |
|
db_hourly_stats_collection_interval=<integer> |
|
enable_mysql_timemachine=<boolean integer> |
|
swap_warning=<integer> |
|
swap_critical=<integer> |
|
swap_inout_period=<integer> |
|
swap_inout_warning=<integer> |
|
swap_inout_critical=<integer> |
|
save_history_days=<integer> |
|
mysqlmemory_warning=<integer> |
|
mysqlmemory_critical=<integer> |
|
ram_warning=<integer> |
|
ram_critical=<integer> |
|
diskspace_warning=<integer> |
|
diskspace_critical=<integer> |
|
cpu_warning=<integer> |
|
cpu_critical=<integer> |
|
cpu_steal_warning=<integer> |
|
cpu_steal_critical=<integer> |
|
cpu_iowait_warning=<integer> |
|
cpu_iowait_critical=<integer> |
|
monitor_cpu_temperature=<boolean integer> |
|
redobuffer_warning=<integer> |
|
redobuffer_critical=<integer> |
|
indexmemory_warning=<integer> |
|
indexmemory_critical=<integer> |
|
datamemory_warning=<integer> |
|
datamemory_critical=<integer> |
|
tablespace_warning=<integer> |
|
tablespace_critical=<integer> |
|
redolog_warning=<integer> |
|
redolog_critical=<integer> |
|
enable_is_queries=<boolean integer> |
|
max_replication_lag=<integer> |
|
enable_icmp_ping=<boolean integer> |
|
Query Monitor
| Option | Description |
|---|---|
long_query_time=<float> |
|
log_queries_not_using_indexes=<boolean integer> |
|
query_monitor_use_local_settings=<boolean integer> |
|
db_long_query_time_alarm=<integer> |
|
enable_query_monitor=<integer> |
|
enable_query_monitor_auto_purge_ps=<boolean integer> |
|
query_monitor_long_running_query_ms=<integer> |
|
query_monitor_alert_long_running_query=<boolean integer> |
|
query_monitor_kill_long_running_query=<boolean integer> |
|
query_monitor_long_running_query_matching_info=<regex> |
|
query_monitor_long_running_query_matching_info_negate=<regex> |
|
query_monitor_long_running_query_matching_host=<regex> |
|
query_monitor_long_running_query_matching_db=<regex> |
|
query_monitor_long_running_query_matching_user=<regex> |
|
query_monitor_long_running_query_matching_user_negate=<regex> |
|
query_monitor_long_running_query_matching_command=<regex> |
|
Backup
| Option | Description |
|---|---|
netcat_port=<string> |
|
backup_user=<string> |
|
backup_user_password=<string> |
|
backup_encryption_key=<string> |
|
backupdir=<path> |
|
backup_subdir=<string> |
|
backup_retention=<integer> |
|
backup_cloud_retention=<integer> |
|
backup_n_safety_copies=<integer> |
|
datadir_backup_path=<path> |
|
disable_backup_email=<boolean integer> |
|
backup_create_hash=<boolean integer> |
|
MySQL, MariaDB Nodes
| Option | Description |
|---|---|
mysql_server_addresses=<string> |
|
monitored_mysql_port=<integer> |
|
monitored_mysql_root_user=<string> |
|
monitored_mysql_root_password=<string> |
|
skip_name_resolve=<boolean integer> |
|
mysql_version=<string> |
|
galera_port=<integer> |
|
galera_version=<string> |
|
galera_vendor=<string> |
|
repl_user=<string> |
|
repl_password=<string> |
|
replication_failover_whitelist=<string> |
|
replication_failover_blacklist=<string> |
|
replication_skip_apply_missing_txs=<boolean integer> |
|
replication_stop_on_error=<boolean integer> |
|
replication_failover_wait_to_apply_timeout=<integer> |
|
replication_stop_on_error=<boolean integer> |
|
replication_auto_rebuild_slave=<boolean integer> |
|
replication_onfail_failover_script=<path> |
|
replication_pre_failover_script=<path> |
|
replication_post_failover_script=<path> |
|
replication_post_unsuccessful_failover_script=<path> |
|
replication_failed_reslave_failover_script=<path> |
|
replication_pre_switchover_script=<path> |
|
replication_post_switchover_script=<path> |
|
replication_check_external_bf_failover=<boolean integer> |
|
replication_check_binlog_filtration_bf_failover=<boolean integer> |
|
replication_failover_events=<boolean integer> |
|
auto_manage_readonly=<boolean integer> |
|
schema_change_detection_address=<string> |
|
schema_change_detection_pause_time_ms=<integer> |
|
schema_change_detection_databases=<string> |
|
datanode_addresses=<string> |
|
mgmnode_addresses=<string> |
|
ndb_connectstring=<string> |
|
ndb_binary=<string> |
|
ndbd_datadir=<path> |
|
mgmd_datadir=<path> |
|
db_configdir=<string> |
|
MongoDB Nodes
| Option | Description |
|---|---|
mongodb_server_addresses=<string> |
|
mongoarbiter_server_addresses=<string> |
|
mongocfg_server_addresses=<string> |
|
mongos_server_addresses=<string> |
|
mongodb_basedir=<path> |
|
mongodb_user=<string> |
|
mongodb_password=<string> |
|
mongodb_authdb=<string> |
|
mongodb_cluster_key=<path> |
|
server_selection_timeout_ms=<integer> |
|
server_selection_try_once=<boolean integer> |
|
PostgreSQL, TimescaleDB Nodes
| Option | Description |
|---|---|
postgresql_server_addresses=<string> |
|
postgresql_user=<string> |
|
postgresql_password=<string> |
|
wal_retention_hours=<boolean integer> |
|
replication_auto_rebuild_slave=<boolean integer> |
|
repl_user=<string> |
|
repl_password=<string> |
|
replication_failover_whitelist=<string> |
|
replication_failover_blacklist=<string> |
|
replication_pre_failover_script=<path> |
|
replication_post_failover_script=<path> |
|
replication_post_unsuccessful_failover_script=<path> |
|
LDAP
ClusterControl has a built-in support for LDAP authentication and requires another configuration file called cmon-ldap.cnf. The configuration file resides under the /etc/cmon.d directory and comes by default with the package. When an LDAP configuration change event happens in ClusterControl GUI → User management → LDAP → LDAP Settings, ClusterControl will commit the change into the controller's runtime and save the configuration directly into cmon-ldap.cnf.
LDAP integration is only available in the Enterprise edition.
Example output
The following is an example configuration if one has an Active Directory server integrated with ClusterControl:
#
# Enables LDAP authentication. The native authentication in ClusterControl
# will also work.
#
enabled = true
#
# Enter the LDAP or LDAPS Uniform Resource Identifier (URI), with the port
# number (if applicable)
#
ldapServerUri = "ldap://support-ad.severalnines.com"
#
# The “Distinguished Name” is used to bind the LDAP server. This user
# requires read access to all LDAP user and group entries to work
# correctly. ClusterControl must perform a LDAP search using the DN before
# any user can log in. This field is case-sensitive.
#
ldapAdminUser = "CN=Administrator,CN=Users,DC=severalnines,DC=com"
#
# Enter the password for the “Distinguished Name” login.
#
ldapAdminPassword = "<redacted>"
#
# Enter the user’s “Relative Distinguished Name” (RDN) to locate this
# user’s information. This field is case-sensitive.
#
ldapUserSearchRoot = "CN=Users,DC=severalnines,DC=com"
#
# The group’s “Relative Distinguished Name” (RDN) is used to locate the
# group information. ClusterControl does not support LDAP users that do
# not belong to at least one LDAP group.
#
ldapGroupSearchRoot = "OU=Groups,DC=severalnines,DC=com"
[ldap_settings]
#
# This is used to filter the object class of the LDAP users. If empty,
# all object classes will be returned.
#
# ldapUserClassName = "objectclass=*"
#
# The LDAP attribute which should contain the username, separated by a
# comma (whitespace value is not allowed). For Active Directory, this is
# commonly “sAMAccountName” and “uid” for OpenLDAP. If more than one
# attribute is specified ClusterControl will attempt to look up all of them
# with the first non-empty reply (in the particular order) used for the
# login username.
#
ldapUsernameAttributes = "sAMAccountName"
#
# The LDAP attribute which should contain the full name of the user,
# separated by a comma. For Active Directory, this is commonly
# “displayName” and “cn” for OpenLDAP. If more than one attribute is
# specified ClusterControl will attempt to look up all of them with the
# first non-empty reply (in the particular order) used for the user’s full
# name.
#
ldapRealnameAttributes = "displayName"
#
# The LDAP attribute which should contain the email address of the user,
# separated by a comma. For Active Directory, this is commonly
# “userPrincipalName” and “mail” for OpenLDAP. If more than one attribute
# is specified, ClusterControl will attempt to look up all of them with the
# first non-empty reply (in the particular order) used for the user’s email
# address.
#
ldapEmailAttributes = "userPrincipalName"
#
# Filter the object class for the LDAP groups. If empty, all object classes
# will be returned.
#
# ldapGroupClassName = ""
#
# The LDAP attribute which represents the group’s members, separated by
# a comma. For Active Directory, this is commonly “member” and “memberUid”
# for OpenLDAP. These attributes will be used to create combinations with
# the value of Group mapping attributes when querying the LDAP server to
# retrieve the group’s information.
#
ldapMemberAttributes = "member"
#
# The LDAP attribute of which name holds the group membership, separated
# by a comma. For Active Directory, this is commonly “dn” and “uid” for
# OpenLDAP. The value of these attributes will be used to create
# combinations with Static member attributes to query the LDAP server to
# retrieve the group.
#
ldapGroupIdAttributes = "dn"
#
# The LDAP attribute where the name represents the group’s name. This is
# commonly “cn”. The value of this attribute will be used in the group
# mapping with “ClusterControl’s Team” for authorization purposes.
#
ldapGroupNameAttribute = "cn"
#
# This variable specifies the connection timeout in seconds if the LDAP
# server is unreachable or takes too long to respond.
#
ldapNetworkTimeout = 5
#
# This is the current LDAP protocol version being used.
#
ldapProtocolVersion = 3
#
# This variable specifies the limit in seconds for an LDAP query to finish,
# any query which takes longer will be aborted.
#
ldapQueryTimeLimit = 5
[security]
#
# This is the location of the CA certification file. (Only for LDAPS).
#
# caCertFile = "/etc/ssl/certs/ldap-ca.crt"
#
# This is the location of the certification file. (Only for LDAPS).
#
# certFile = "/etc/ssl/certs/ldap.crt"
#
# This is the location of the certificate key file. (Only for LDAPS).
#
# keyFile = "/etc/ssl/certs/ldap.pem"
[DBAadmins]
ldapGroupId = "DBA"
cmonGroupName = "admins"
CMON Database
The CMON database is the persistent store for all monitoring data collected from the managed nodes, as well as all ClusterControl metadata (e.g. what jobs there are in the queue, backup schedules, backup statuses, etc.). The ClusterControl Controller requires a MySQL database running on mysql_hostname as defined in the configuration file. The database name and the user is "cmon" and is immutable. The CMON database dump files are shipped together with the ClusterControl Controller package and can be found under /usr/share/cmon once it is installed.
MySQL user "cmon" needs to have proper access to the CMON database by performing the following grant:
Grant all privileges to "cmon" at hostname value (as defined in CMON configuration file) on ClusterControl host:
GRANT ALL PRIVILEGES ON *.* TO 'cmon'@'<hostname>' IDENTIFIED BY '<mysql_password>' WITH GRANT OPTION;
Grant all privileges for "cmon" at 127.0.0.1 on ClusterControl host:
GRANT ALL PRIVILEGES ON *.* TO 'cmon'@'127.0.0.1' IDENTIFIED BY '<mysql_password>' WITH GRANT OPTION;
On every managed database server, grant all privileges to cmon at the controller's hostname value (as defined in CMON configuration file) on each of the managed database host:
GRANT ALL PRIVILEGES ON *.* TO 'cmon'@'<hostname>' IDENTIFIED BY '<mysql_password>' WITH GRANT OPTION;
If one deploys a cluster using ClusterControl deployment wizard, the above privileges will be configured automatically.