1. Home
  2. Docs
  3. ClusterControl
  4. User Guide (CLI)
  5. s9s-node

s9s-node

View and handle nodes.

Usage

s9s node {command} {options}

Command

Name, shorthand Description
−−change-config Changes configuration values for the given node.
−−enable-binary-logging Creates a job to enable binary logging on a specific node. Not all clusters support this feature (MySQL does). One needs to enable binary logging in order to set up a cluster to cluster replication.
−−list, -L List nodes. Default to all clusters.
−−list-config Lists the configuration values for the given node.
−−pull-config Copy the configuration file(s) from the node to the local computer. Use the --output-dir to control where the files will be created.
--push-config Copy the configuration file(s) to a node.
--register Registers a node that already is working.
−−restart Restarts the node. This means the process that provides the main functionality on the node (e.g. the MySQL daemon on a Galera node) will be stopped then start again.
−−set Sets various properties of the specified node/host.
−−set-config Changes configuration values for the given node.
−−set-read-only Creates a job that sets the node to read-only mode. Please note that not all cluster types support read-only mode.
−−set-read-write Creates a job that sets the node to read-write mode if it was previously set to read-only mode. Please note that not all cluster types support read-only mode.
−−start Starts the node. This means the process that provides the main functionality on the node (e.g. the MySQL daemon on a Galera node) will be started.
−−stat Prints detailed node information. It can be used in conjunction with --graph to produce statistical data. See Graph Options.
−−stop Stops the node. This means the process that provides the main functionality on the node (e.g. the MySQL daemon on a Galera node) will be stopped.
−−unregister Unregisters the node from ClusterControl.

Options

Name, shorthand Description
−−cluster-id=ID, -i The ID of the cluster in which the node is.
−−cluster-name=NAME, -n Name of the cluster to list.
−−nodes=NODELIST The nodes to list or manipulate. See Node List.
−−node-format=FORMAT The format string used to print nodes.
−−opt-group=GROUP The configuration option group.
−−opt-name=NAME The name of the configuration option.
−−opt-value=VALUE The value of the configuration option.
−−output-dir=DIR The directory where the files are created.
−−−−graph=GRAPH_NAME The name of the graph to show. See Graph Options.
−−begin=TIMESTAMP The start of the graph interval.
−−end=TIMESTAMP The end of the graph interval.
−−force Force the execution of potentially dangerous operations like restarting a master node in a MySQL Replication cluster.
−−begin=TIMESTAMP The start time of the graph (the X-axis).
−−density If this option is provided will be a probability density function (or histogram) instead of a timeline. The X-axis shows the measured values (e.g. MByte/s) while the Y-axis shows how many percent of the measurements contain the value. If for example, the CPU usage is between 0% and 1% at 90% of the time the graph will show a 90% bump at the lower end.
−−end=TIMESTAMP The end of the graph.
−−node-format[=FORMATSTRING] The string that controls the format of the printed information about the nodes. See Node Format.
−−properties=ASSIGNMENT One or more assignments specifying property names and values. The assignment operator is the = character (e.g. --properties='alias="newname"'), multiple assignments are separated by the semicolon ;.
−−output-dir=DIRECTORY The directory where the output files will be created on the local computer.

Graph Options

When providing a valid graph name together with the --stat option a graph will be printed with statistical data. Currently, the following graphs are available:

Option Description
cpughz The graph will show the CPU clock frequency measured in GHz.
cpuload Shows the average CPU load of the host computer.
cpusys Percent of time the CPU spent in kernel mode.
cpuidle Percent of time the CPU is idle on the host.
cpuiowait Percent of time the CPU is waiting for IO operations.
cputemp The temperature of the CPU measured in degree Celsius. Please note that to measure the CPU temperature some kernel module might be needed (e.g. it might be necessary to run sudo modprobe coretemp). On multiprocessor systems, the graph might show only the first processor.
cpuuser Percent of time the CPU is running user space programs.
diskfree The amount of free disk space measured in GBytes.
diskreadspeed Disk read speed measured in MBytes/sec.
diskreadwritespeed Disk read and write speed measured in MBytes/sec.
diskwritespeed Disk write speed measured in MBytes/sec.
diskutilization The bandwidth utilization for the device in percent.
memfree The amount of the free memory measure in GBytes.
memutil The memory utilization of the host measured in percent.
neterrors The number of receive and transmit errors on the network interface.
netreceivedspeed Network read speed in MByte/sec.
netreceiveerrors The number of packets received with error on the given network interface.
nettransmiterrors The number of packets failed to transmit.
netsentspeed Network write speed in MByte/sec.
netspeed Network read and write speed in MByte/sec.
sqlcommands Shows the number of SQL commands executed measured in 1/s.
sqlcommits The number of commits measured in 1/s.
sqlconnections Shows the number of SQL connections.
sqlopentables The number of open tables at any given moment.
sqlqueries The number of SQL queries in 1/s.
sqlreplicationlag Replication lag on the SQL server.
sqlslowqueries The number of slow queries in 1/s.
swapfree The size of the free swap space measured in GBytes.

Node List

Using the --list and --long command-line options a detailed list of the nodes can be printed. Here is an example of such a list:

$ s9s node --list --long '192.168.1.1*'
STAT VERSION CID CLUSTER HOST PORT COMMENT
poM- 9.6.2 1 ft_postgresql_11794 192.168.1.117 8089 Up and running
coC- 1.4.2 1 ft_postgresql_11794 192.168.1.127 9555 Up and running
Total: 3
Note

The list in the example is created using a filter (that is ‘192.168.1.1*’ in the example). The last line shows a 3 as the total, the number of nodes maintained by the controller, but only two of the nodes are printed in the list because of the filter.

The list contains the following fields:

Field Description
STAT Nodetype
This is the type of node. It can be c for controller, g for Galera node, x for MaxScale node, k for Keepalived node, p for PostgreSQL, m for Mongo, e for MemCached, y for ProxySql, h for HaProxy, b for PgBouncer, B for PgBackRest, t for PBMAgent, a for Garbd, r for group replication host, A for cmon agent, P for Prometheus, s for generic MySQL nodes, S for Redis sentinel, R for Redis, E for Elasticsearch, and ? for unknown nodes.
Hoststatus
The status of the node. It can be o for online, l for off-line, f for failed nodes, r for nodes performing recovery, - for nodes that are shut down and ? for nodes in an unknown state.
Role This field shows the role of the node in the cluster. This can be M for master, S for Slave, U for multi (master and slave), C for controller, V for backup verification node, A for arbiter, R for backup repository host D for Elasticsearch data host c for Elasticsearch coordinator_only host and - for everything else.
Maintenance
This field shows if the node is in maintenance mode. The character is M for nodes in maintenance mode and - for nodes that are not in maintenance mode.
VERSION This field shows the version string of the software that provides the service represented in the given line.
CID The cluster ID of the cluster that holds the node as a member. Every node belongs to exactly one cluster.
CLUSTER The name of the cluster that holds the node as a member.
HOST The hostname of the host. This can be a real DNS hostname, the IP address, or the Cmon alias name of the node depending on the configuration and the command line options. The cluster is usually configured to use IP addresses (the Cmon configuration file contains IP addresses) so this field usually shows IP addresses.
PORT The IP port on which the node accepts requests. The same DNS hostname or IP address can be added multiple times to the same or to multiple clusters, but the host:port pair must be unique. In other words, the same host with the same port can not be added to the same Cmon controller twice. Since the hostname:port pair is unique the nodes are identified by this and every line of the node list is representing a hostname:port node. There is one exception to this rule: the Cmon Controller can manage multiple clusters and so be a part of more than one cluster with the same hostname and port.

Node Format

The string that controls the format of the printed information about the nodes. When this command-line option is used the specified information will be printed instead of the default columns. The format string uses the % character to mark variable fields and flag characters as they are specified in the standard printf() C library functions. The % specifiers are ended by field name letters to refer to various properties of the nodes.

The %+12i format string for example has the +12 flag characters in it with the standard meaning the field will be 12 character-wide and the + or - sign will always be printed with the number. The properties of the node are encoded by letters. In the %16D for example, the letter D encodes the “data directory” field, so the full path of the data directory on the node will be substituted. Standard \ notation is also available, \n for example encodes a new-line character.

The s9s-tools support the following fields:

Field Description
A The IP address of the node.
a Maintenance mode flag. If the node is in maintenance mode a letter M, otherwise -.
b The master host for this slave if it is applicable.
C The configuration file for the most important process on the node (e.g. the configuration file of the MySQL daemon on a Galera node).
c The total number of CPU cores in the host. Please note that this number may be affected by hyper-threading. When a computer has 2 identical CPUs, with four cores each and uses 2x hyper-threading it will count as 2x4x2 = 16.
D The data directory of the node. This is usually the data directory of the SQL server.
d The PID file on the node.
G The name of the group owner of the node.
g The log file on the node.
h The CDT path of the node.
I The numerical ID of the node.
i The total number of monitored disk devices (partitions) in the cluster.
k The total number of disk bytes found on the monitored devices in the node. This is a double-precision floating-point number measured in Terabytes.
L The replay location. This field currently only has valid value in PostgreSQL clusters.
l The received location. This field currently only has valid value in PostgreSQL clusters.
M A message, describing the node’s status in a human-readable format.
m The total memory size found in the host, measured in GBytes. This value is represented by a double-precision floating-point number, so formatting it with precision (e.g. %6.2m) is possible. When used with the f modifier (e.g. %6.2fm) this reports the free memory, the memory that available for allocation, used for cache or used for buffers.
N The name of the node. If the node has an alias that is used, otherwise the name of the node is used. If the node is registered using the IP address, the IP address is the name.
n The total number of monitored network interfaces in the host.
O The user name of the owner of the cluster that holds the node.
o The name and version of the operating system together with the codename.
P The port on which the most important service is waiting for requests.
p The PID (process ID) on the node that presents the service (e.g. the PID of the MySQL daemon on a Galera node).
R The role of the node (e.g. “controller”, “master”, “slave” or “none”).
r The work read-only or read-write indicating if the server is in read-only mode or not.
S The status of the host (e.g. CmonHostUnknown, CmonHostOnline, CmonHostOffLine, CmonHostFailed, CmonHostRecovery, CmonHostShutDown).
s The list of slaves of the given host in one string.
T The type of the node, e.g. “controller”, “galera”, “postgres”.
t The total network traffic (both received and transmitted) measured in MBytes/seconds.
U The number of physical CPUs on the host.
u The CPU usage percent found on the host.
V The version string of the most important software (e.g. the version of the PostgreSQL installed on a PostgreSQL node).
v The ID of the container/VM in “CLOUD/ID” format. The – string if no container ID is set for the node.
w The total swap space found in the host measured in GigaBytes. With the f modifier (e.g. %6.2fw) this reports the free swap space in GigaBytes.
Z The name of the CPU model. Should the host have multiple CPUs, this will return the model name of the first CPU.
% The % character itself.

Examples

List all nodes:

$ s9s node --list --long
ST  VERSION                  CID CLUSTER        HOST       PORT COMMENT
go- 10.1.22-MariaDB-1~xenial   1 MariaDB Galera 10.0.0.185 3306 Up and running
co- 1.4.1.1856                 1 MariaDB Galera 10.0.0.205 9500 Up and running
go- 10.1.22-MariaDB-1~xenial   1 MariaDB Galera 10.0.0.209 3306 Up and running
go- 10.1.22-MariaDB-1~xenial   1 MariaDB Galera 10.0.0.82  3306 Up and running
Total: 4

Print the configuration for a node:

$ s9s node --list-config --nodes=10.0.0.3
...
mysqldump   max_allowed_packet                     512M
mysqldump   user                                   backupuser
mysqldump   password                               nWC6NSm7PnnF8zQ9
xtrabackup  user                                   backupuser
xtrabackup  password                               nWC6NSm7PnnF8zQ9
MYSQLD_SAFE pid-file                               /var/lib/mysql/mysql.pid
MYSQLD_SAFE basedir                                /usr/
Total: 71

The following example shows how a node in a given cluster can be restarted. When this command executed a new job will be created to restart a node. The command-line tool will stop and show the job messages until the job is finished:

$ s9s node \
        --restart \
        --cluster-id=1 \
        --nodes=192.168.1.117 \
        --log

Change a configuration value for a PostgreSQL server:

$ s9s node \
        --change-config \
        --nodes=192.168.1.115 \
        --opt-name=log_line_prefix \
        --opt-value='%m '

Push a configuration option inside my.cnf (max_connections=500) on node 10.0.0.3:

$ s9s node \
        --change-config \
        --nodes=10.0.0.3 \
        --opt-group=mysqld \
        --opt-name=max_connections \
        --opt-value=500

Import two existing Keepalived nodes (192.168.20.56 and 192.168.20.57) where the virtual IP is 192.168.20.59 into cluster ID 13:

# import primary keepalived
$ s9s node \
        --register \
        --nodes="keepalived://192.168.20.56" \
        --virtual-ip=192.168.20.59 \
        --cluster-id=13 \
        --wait

# import secondary keepalived
$ s9s node \
        --register \
        --nodes="keepalived://192.168.20.57" \
        --virtual-ip=192.168.20.59 \
        --cluster-id=13 \
        --wait

Listing the Galera hosts. This can be done by filtering the list of nodes by their properties:

$ s9s node \
        --list \
        --long \
        --properties="class_name=CmonGaleraHost"

Create a set of graphs, one for each node shown in the terminal about the load on the hosts. If the terminal is wide enough the graphs will be shown side by side for a compact view:

$ s9s node \
        --stat \
        --cluster-id=1 \
        --begin="08:00" \
        --end="14:00" \
        --graph=load

The density function can also be printed to show what were the typical values for the given statistical data. The following example shows what was the typical values for the user mode CPU usage percent:

$ s9s node \
        --stat \
        --cluster-id=2 \
        --begin=00:00 \
        --end=16:00 \
        --density \
        --graph=cpuuser

The following example shows how a custom list can be created to show some information about the CPU(s) in some specific hosts:

$ s9s node \
        --list \
        --node-format="%N %U CPU %c Cores %6.2u%% %Z\n" 192.168.1.191 192.168.1.195
192.168.1.191 2 CPU 16 Cores 22.54% Intel(R) Xeon(R) CPU L5520 @ 2.27GHz
192.168.1.195 2 CPU 16 Cores 23.12% Intel(R) Xeon(R) CPU L5520 @ 2.27GHz

The following list shows some information about the memory, the total memory and the memory available for the applications to allocate (including cache and buffer with the free memory):

$ s9s node \
        --list \
        --node-format="%4.2m GBytes %4.2fm GBytes %N\n"
16.00 GBytes 15.53 GBytes 192.168.1.191
47.16 GBytes 38.83 GBytes 192.168.1.127

Set a node to read-write mode if it was previously set to read-only mode:

$ s9s node \
        --set-read-write \
        --cluster-id=1 \
        --nodes=192.168.0.78 \
        --log

Copy configuration file(s) from a PostgreSQL server 192.168.0.232 into the localhost:

$ s9s node \
        --pull-config \
        --nodes="192.168.0.232" \
        --output-dir="tmp"
Was this article helpful to you? Yes No