4. Managing Nodes

4.1. Resource Discovery

As part of the Robin Platform installation process, resource discovery is run on the node wherein which details about the physical configuration, hardware limits and resource availability are discovered. The purpose of this is two fold. First this process allows Robin to gain a better understanding of the machine in terms of both the compute and storage resources it can provide for application deployment as well as allow Robin to better optimize the nodes usage within the cluster. Moreover it allows Robin to reserve resources that were already in-use pre-Robin installation so as to make sure resources aren’t diverted inappropriately.

The following properties of the node are discovered:

  • Disks

  • CPUs

  • Random Access Memory (RAM)

  • Network Devices

  • NUMA Configuration

Details on what is captured with regards to each one of the above aspects alongside how they are captured are described below.

4.1.1. Disk Discovery

Robin leverages a multitude of sources to discover the disks that are available to a node. Some of the commands and directories used to attain the below details are: lsblk, partprobe, pvs, blkid and /proc/mounts. The following details are captured for each disk (if present):

  • Devpath

  • Capacity

  • Physical Sector size

  • WWN (along with make and model)

  • Media type

This discovery is primarily performed in order determine the storage capacity of the node and consequently the amount of storage resources the host can provide to its assigned resource pool. However not all discovered disks are considered to be viable to be used as storage. Those that are excluded are primarily considered to be root devices. Moreover we do not consider devices that have parititions, file mounts or are LVMs as devices to be used for storage. This is due to the fact that when a storage role is assigned to a host Robin formats devices marked for storage.

4.1.2. Disk Partitions (LVM) Discovery

Environments where resources are constrained, such as Edge servers, may not have dedicated data disks for Robin to consume and instead may only contain disks which are partitioned. By default, partitioned disks are discovered and marked as Reserved to avoid any user data being overwritten. As a result, in order for the partition(s) to serve as data disks they will have to be setup manually by the process described below.

First ensure the target partitions are discovered appropriately by Robin by running the following commands:

# lsblk
sdb              8:16   0   50G  0 disk
├─sdb1           8:17   0   10G  0 part
└─sdb2           8:18   0   40G  0 part
  └─vg-robinds   253:0    0   39G  0 lvm

# robin drive list --role=all
ID | WWN                                                  | Host         | Path /dev/disk/by-id                                                         | Size(GB) | Movable | Type | Free/Max(GB) | Vols | Role     | Status  | LastOpr
---+------------------------------------------------------+--------------+------------------------------------------------------------------------------+----------+---------+------+--------------+------+----------+---------+---------
-  | 0xQEMU_QEMU_HARDDISK_561637eb-07d0-4a0d-8            | vnode-89-142 | scsi-0QEMU_QEMU_HARDDISK_561637eb-07d0-4a0d-8                                | 50       | N       | HDD  | 38/38 (100%) | 0/10 | RootDisk | UNKNOWN | INIT
-  | 0xQEMU_QEMU_HARDDISK_561637eb-07d0-4a0d-8-vg-robinds | vnode-89-142 | dm-uuid-LVM-DOy7w9WSdGi2PcSERuceOHkfM7dotzr7nm5EuoAWsyAHkvbYGT02MaeWDro05F3R | 39       | N       | HDD  | 30/30 (100%) | 0/10 | Reserved | UNKNOWN | INIT

Note

Details on the robin drive list command can be found here.

In order to mark the partition as ready to be used as storage and confirm that its role has been updated in accordance, run the following command:

# robin drive update 0xQEMU_QEMU_HARDDISK_561637eb-07d0-4a0d-8-vg-robinds --role storage --wait
Job: 5922 Name: DiskModify           State: PROCESSED       Error: 0
Job: 5922 Name: DiskModify           State: COMPLETED       Error: 0

# robin drive list
ID | WWN                                                  | Host         | Path /dev/disk/by-id                                                         | Size(GB) | Movable | Type | Free/Max(GB) | Vols | Role     | Status  | LastOpr
---+------------------------------------------------------+--------------+------------------------------------------------------------------------------+----------+---------+------+--------------+------+----------+---------+---------
-  | 0xQEMU_QEMU_HARDDISK_561637eb-07d0-4a0d-8-vg-robinds | vnode-89-142 | dm-uuid-LVM-DOy7w9WSdGi2PcSERuceOHkfM7dotzr7nm5EuoAWsyAHkvbYGT02MaeWDro05F3R | 39       | N       | HDD  | 30/30 (100%) | 0/10 | Storage  | UNKNOWN | INIT

Note

Details on the robin drive update command can be found here.

Lastly to intialize the paritition and confirm that is ready for use, retrieve the name of the host it is currently associated with and run the following commands:

# robin host add-role vnode-89-142 storage --wait
Job: 5923 Name: HostAddRoles         State: PROCESSED       Error: 0
Job: 5923 Name: HostAddRoles         State: WAITING         Error: 0
Job: 5923 Name: HostAddRoles         State: COMPLETED       Error: 0

# robin drive list
ID | WWN                                                  | Host         | Path /dev/disk/by-id                                                         | Size(GB) | Movable | Type | Free/Max(GB) | Vols | Role     | Status  | LastOpr
---+------------------------------------------------------+--------------+------------------------------------------------------------------------------+----------+---------+------+--------------+------+----------+---------+---------
4  | 0xQEMU_QEMU_HARDDISK_561637eb-07d0-4a0d-8-vg-robinds | vnode-89-142 | dm-uuid-LVM-DOy7w9WSdGi2PcSERuceOHkfM7dotzr7nm5EuoAWsyAHkvbYGT02MaeWDro05F3R | 39       | N       | HDD  | 30/30 (100%) | 0/10 | Storage  | ONLINE  | READY

Note

Details on the robin host add-role command can be found here.

4.1.3. CPU Discovery

Robin leverages the information stored within the /proc/cpuinfo file in order to discover the CPUs present on a machine. The following details are captured for each CPU (if they are present):

  • CPU Index

  • Physical ID

  • Vendor ID

  • CPU Clock Speed

  • Model Name

  • Cache Size

  • Core ID

In addition to the above, Robin discovers if a processor is online, and the min/max frequencies it can operate at by utilizing the values stored in the cpufrequency directory. The reason behind this extensive detail gathering is to enable resource tracking via identification for chargeback purposes and resource accounting whilst planning application deployment.

Moreover if there are any isolated CPUs present on the system, Robin will distinguish them from the general CPUs. This is because Robin will only utilize isolated cores, if present, for application deployment due to the increased performance that can be achieved with isolation and CPU pinning. As a result, from Robin’s perspective the host will only have these isolated cores to contribute to the resource pool.

4.1.4. Memory Discovery

Robin leverages the information stored within the /proc/meminfo file in order to discover the overall memory statistics of the host as well as certain breakdowns. The following details are captured (if they are present):

  • Total memory available on the host

  • Total number of 2M (2048 KB) Hugepages

  • Total size of 2M (2048 KB) Hugepages

  • Total number of 1G (1024 MB) Hugepages

  • Total size of 1G (1024 MB) Hugepages

  • Breakdown of memory banks including their size, speed, manufacturer and type

Once again the reason behind this extensive detail gathering is to enable resource tracking via identification for chargeback purposes and resource accounting whilst planning application deployment. Moreover these details are necessary to determine how much memory to reserve for use by Robin services and to account for the memory that is already being used by non-Robin services. The latter is especially important as memory is not an overprovisionable resource.

4.1.5. Network Device Discovery

Robin leverages the information stored within the /sys/class/net directory in order to discover properties about the physical interfaces present on the node. The following details are captured for each NIC (if they are present):

  • Name of the NIC

  • Maximum Transmission Unit

  • MAC address

  • Product ID

  • Vendor ID

  • Attached CPUs

  • Domain, bus and slot configuration

  • The NUMA node to which it is attached if applicable

  • The number of Virtual Functions present (for SR-IOV NICs)

  • IP Addresses

  • IPv4/v6 Gateways

The reason behind this extensive detail gathering is to enable appropriate IP allocation/tracking when deploying pods and ensuring network connectivity between instantiated containers. In addition the details help to validate, plan and orchestrate certain application deployment configurations.

4.1.6. NUMA Configuration Discovery

Robin leverages the information stored within the /sys/devices/system/node* directories in order to discover properties and resource limits of each NUMA node present on the host. The following details are captured for each NUMA node (if they are present):

  • NUMA Node ID

  • Total memory available

  • Total size of 1G (1024 MB) Hugepages

  • Total size of 2M (2048 KB) Hugepages

  • Associated General CPU IDs

  • Associated Isolated CPU IDs

  • Number of allocations present on each associated CPU ID

The reason behind this extensive detail gathering is to ensure that whilst application deployment planning we do not over allocate resources or incorrectly track the used/available resources. This is especially true given the fact that Robin attempts to make every application as NUMA aware as possible in order to boost performance and optimize workloads. As with CPU discovery, if any isolated CPUs are attached to the NUMA node they will be the only processors tracked for the node.

Moreover given all the above is captured alongside details of any SR-IOV Nics attached to particular NUMA node, Robin has the capability to deploy applications completely enclosed within the NUMA boundary if specified as condition during application deployment.

Note

The aforementioned isolated CPUs can be achieved by manually editing the isolcpus within the GRUB configuration for the host which results in the kernel scheduler never utilizing the CPU’s specified.

4.2. Robin Node Roles

There are three Robin roles that can be assigned to hosts: Manager, Compute and Storage. Depending on the resources present on a node you can initialize it to have one or more of these roles. This allows for a more granular control over what resources a node can provide for allocation. Below is a description for each role:

  • The Manager role should be designated to a node which is intended to be part of the Robin Control Plane. The first node which is added as a Manager will be considered the master and have essential Robin services, including the RCM server, file server and event server, running on it. As a result, it will essentially control and manage (hence the name of the role) the agent nodes. This entails handling all external communication with Robin via APIs, maintaining the most current replica of the PostgreSQL database etc. However if a failover occurs, these services are moved and run on the newly elected master. For every Robin cluster there is a maximum of 3 Manager nodes (this is necessary for HA installations).

  • The Compute role should be designated to a node which is intended to host pods and their respective containers. As a result, these nodes will have resources such as memory and CPU utilized by application deployments. When orchestrating applications the Robin server will try to provision pods on nodes with this roles, and track the aforementioned resource allocations/availability.

  • The Storage role should be designated to a node which is intended to provide storage, as indicated by its name, for applications deployed on Robin. As a result, any volumes needed for deployed applications will be created and mounted on devices on nodes with this role set.

The aforementioned roles are not mutually exclusive to one another. As a result, a node could have any combination of the three roles (including all of them at one time). However a host within Robin cluster must always have role assigned to it unless it is to be decommissioned.

The following commands are described in this section:

robin host add-role

Add one or more role(s) to a host

robin host enable-role

Move one or more role(s) out of maintenance mode for a host

robin host disable-role

Move one or more role(s) into of maintenance mode for a host

robin host remove-role

Remove one or more role(s) from a host

4.2.1. Adding role(s)

To add a role to a host that is already registered within the Robin cluster, issue the following command:

# robin host add-role [<hosts>] [<roles>]
                                 --rpool <rpool>
                                 --disks <disks>

hosts

Comma separated list of hosts to add role to.

roles

Comma separated list of roles. Valid values are ‘storage’, ‘compute’, and ‘manager’

--rpool

Assign a resource pool prior to adding a role.

--disks

Comma separated list of disk WWNs to add as part of storage role addition.

Example:

# robin host add-role centos-60-212,centos-60-214 Compute,Storage --wait
Job: 18  Name: HostAddRoles     State: VALIDATED       Error: 0
Job: 18  Name: HostAddRoles     State: WAITING         Error: 0
Job: 18  Name: HostAddRoles     State: COMPLETED       Error: 0

Adds a role to a host that is already registered within the Robin cluster.

End Point: /api/v3/robin_server/hosts/<hostname>

Method: PUT

URL Parameters: None

Data Parameters:

  • action: add_roles - This mandatory field within the payload specifies that the add role operation is to be performed.

  • roles: <list_of_roles> - This mandatory field within the payload is a list of roles that should be added to the specified host. Valid values include ‘storage’, ‘compute’, and ‘manager’.

  • drives: <list_of_wwns> - Utilizing this parameter by specifiying a list of WWNs of drives results in the disks associated with the aforementioned WWNs being added alongside the addition of the storage role.

  • rpool: <rpool_name> - Utilizing this parameter by specifying a resource pool name results in a resource pool being assigned to the host prior to the addition of roles.

Port: RCM Port (default value is 29442)

Headers:

  • Authorization: <auth_token> : Authorization token to identify which user is sending the request. The token can be acquired from the login API.

Success Response Code: 202

Error Response Code: 500 (Internal Server Error), 404 (Not Found Error), 401 (Unauthorized Error), 400 (Invalid API Usage Error)

Example Response:

Output
{
   "jobid":20
}

4.2.2. Enabling role(s)

To move a role, which is already added to a host, out of maintenance mode and thus enable it for use again, issue the following command:

# robin host enable-role [<host>] [<roles>]

hosts

Fully qualified hostname

roles

Comma separated list of roles to enable. Valid values are ‘storage’ and ‘compute’

Example:

# robin host enable-role centos-60-212.robinsystems.com Compute
Role(s) 'Compute' enabled on host centos-60-212.robinsystems.com

Enables a role that was previously disabled on a host.

End Point: /api/v3/robin_server/hosts/<hostname>

Method: PUT

URL Parameters: None

Data Parameters:

  • action: enable-role - This mandatory field within the payload specifies that the enable role operation is to be performed.

  • roles: <list_of_roles> - This mandatory field within the payload is a list of roles that should be enabled on the specified host. Valid values include ‘compute’ and ‘storage’.

Port: RCM Port (default value is 29442)

Headers:

  • Authorization: <auth_token> : Authorization token to identify which user is sending the request. The token can be acquired from the login API.

Success Response Code: 200

Error Response Code: 500 (Internal Server Error), 404 (Not Found Error), 401 (Unauthorized Error), 400 (Invalid API Usage Error)

Example Response: On success the reponse is empty.

4.2.3. Disabling role(s)

Within Robin when one disables a role, the role is said to be put into maintenance mode. This in turn means that for all intents and purposes the host does not have access to this role. This is useful for debugging purposes and to temporarily reserve the hosts resources. To move a role into maintenance mode and thus disable it for use, issue the following command:

# robin host disable-role [<host>] [<roles>]

hosts

Fully qualified hostname

roles

Comma separated list of roles to enable. Valid values are ‘storage’ and ‘compute’

Example:

# robin host disable-role centos-60-212.robinsystems.com Compute
Role(s) 'Compute' disabled on host centos-60-212.robinsystems.com

Disables a role for a host such that the host temporarily does not have access to it.

End Point: /api/v3/robin_server/hosts/<hostname>

Method: PUT

URL Parameters: None

Data Parameters:

  • action: disable-role - This mandatory field within the payload specifies that the disable role operation is to be performed.

  • roles: <list_of_roles> - This mandatory field within the payload is a list of roles that should be disabled on the specified host. Valid values include ‘compute’ and ‘storage’.

Port: RCM Port (default value is 29442)

Headers:

  • Authorization: <auth_token> : Authorization token to identify which user is sending the request. The token can be acquired from the login API.

Success Response Code: 200

Error Response Code: 500 (Internal Server Error), 404 (Not Found Error), 401 (Unauthorized Error), 400 (Invalid API Usage Error)

Example Response: On success the reponse is empty.

4.2.4. Removing role(s)

In order to a remove a role that had previously been assigned to a node, issue the following command:

# robin host remove-role [<hosts>] [<roles>]
                                    --force
                                    --yes

hosts

Comma separated list of hosts to remove role from.

roles

Comma separated list of roles. Valid values are ‘storage’, ‘compute’, and ‘manager’

--force

Required if retrying a remove-role operation.

--yes

Do not prompt the user for confirmation of removal

Example:

# robin host remove-role centos-60-212 Compute,Storage --wait
Job: 192  Name: HostRemoveRoles     State: VALIDATED       Error: 0
Job: 192  Name: HostRemoveRoles     State: WAITING         Error: 0
Job: 192  Name: HostRemoveRoles     State: COMPLETED       Error: 0

Removes a role that had previously been assigned to a host.

End Point: /api/v3/robin_server/hosts/<hostname>

Method: PUT

URL Parameters: None

Data Parameters:

  • action: remove_roles - This mandatory field within the payload specifies that the remove role operation is to be performed.

  • roles: <list_of_roles> - This mandatory field within the payload is a list of roles that should be removed from the specified host. Valid values include ‘storage’, ‘compute’, and ‘manager’.

  • force: true - This field is mandatory when retrying the role removal operation, otherwise it can be excluded.

Port: RCM Port (default value is 29442)

Headers:

  • Authorization: <auth_token> : Authorization token to identify which user is sending the request. The token can be acquired from the login API.

Success Response Code: 202

Error Response Code: 500 (Internal Server Error), 404 (Not Found Error), 401 (Unauthorized Error), 400 (Invalid API Usage Error)

Example Response:

Output
{
   "jobid":24
}

4.2.5. Shutting Down, Rebooting or Power cycling a manager node

In a Robin HA cluster, there are three manager nodes. One node will have the master role whilst the others will have slave roles. Before a manager node is brought down via a shutdown, reboot or power cycle, make sure that all three manager nodes are ONLINE and Ready. This will ensure there are two healthy managers remaining when the target node goes offline.

Note

A minimum of two Manager nodes are required to address split-brain failure scenarios.

If the manager node that is to be taken offline is the current master, or a transition of roles is desired, then the master lock must be released before the operation can proceed. The release of the lock results in one of the slave manager nodes acquiring it and consequently transitioning into being the node with the master role. Run the following command to force the manager node with the master role to release the master lock:

# robin host release-master-lock <master_node_hostname> –-force

When the above command is run, the current master manager node stops all control plane services whilst also releasing the lock. Once this is done, the current master manager node then puts itself to sleep and waits for one of the slaves to acquire the master lock in order to transition into become the new master manager node. Once the transition is complete, the previous master manager node takes on the slave role and follows the new master manager node.

Note

After releasing the master lock, make sure all three Managers are ONLINE and Ready state, before you shut down, reboot, or power cycle any of the nodes.

4.3. Managing Resource Pools

Resource pools are a construct in Robin which allow you to group nodes in the cluster together for allocation purposes and consequently provide resource isolation. In turn this isolation has a multi-layered impact; primarily it determines what resources a tenant (and its associated users) has access to, which in turn affects the type of applications that can be deployed. As a result, resource pools can provide isolation at a user level as well as an application level since workloads can be manipulated such that they are deployed on a particular set of nodes. This is useful when one needs performant applications to run on more powerful or specifically configured nodes for example. The flip side of this is also true as nodes can be cordoned off from application deployments. The aforementioned isolation is maintained by the fact each node can be only be part of a single resource pool.

Upon installation, Robin creates a resource pool called default, which has no nodes associated with it and is attached to the Adminstrator’s tenant, for convienence sake.

Note

Nodes need to be added to resource pools before they can be initialized via roles and once they are added to a resource pool they cannot be removed.

The following commands are described in this section:

robin rpool add

Add a resource pool

robin rpool list

List all resource pools

robin rpool remove

Delete a resource pool

robin host assign-rpool

Assign a resource pool to one or more hosts

4.3.1. Adding a resource pool

In order to create an empty resource pool, issue the following the command:

# robin rpool add [<name>]
                   --zone <zone>
                   --tenant <tenant>
                   --desc <description>

name

Resource pool name

--zone <zone>

Name of zone the resource pool belongs to

--tenant <tenant>

Name of tenant the resource pool should be assigned to

--desc <description>

Description for resource pool

Example:

# robin rpool add demo --desc "Resource pool for demo" --tenant Administrators
Added resource pool demo.

Creates an empty resource pool.

End Point: /api/v3/robin_server/rpools

Method: POST

URL Parameters: None

Data Parameters:

  • name: <rpool_name> - This mandatory field within the payload specifies the name of the resource pool to be created.

  • description: <description> - This mandatory field within the payload specifies the description of the resource pool to be created.

  • zone: <zone_name> - This mandatory field within the payload specifies the name of the zone in which the resource pool should be created.

  • tenant: <tenant> - Utilizing this parameter within the payload, by specifying a tenant name, results in the consequent resource pool being associated with the aforementioned tenant.

Port: RCM Port (default value is 29442)

Headers:

  • Authorization: <auth_token> : Authorization token to identify which user is sending the request. The token can be acquired from the login API.

Success Response Code: 200

Error Response Code: 500 (Internal Server Error), 404 (Not Found Error), 401 (Unauthorized Error), 400 (Invalid API Usage Error), 409 (Duplicate Resource Error)

Example Response:

Output
{
   "message":"Added resource pool demo.\n"
}

4.3.2. Listing all resource pools

In order to list all resource pools currently present on the cluster along with details off their states, number of applications etc, issue the following command:

# robin rpool list [<name>]
                    --full
                    --json

name

Resource pool name to filter by. Note this is an optional argument

--full

Display additional information about each resource pool

--json

Output in JSON

Example:

# robin rpool list
ID | Name    | %cores | %mem   | Nodes-ON/OF/FA | Apps-ON/OF/FA/AW/OT | Description
---+---------+--------+--------+----------------+---------------------+------------------------
1  | default | 20/100 | 20/100 |     3- 3/ 0/ 0 |    4- 4/ 0/ 0/ 0/ 0 | Default Resource Pool
2  | demo    | 20/100 | 20/100 |     0- 0/ 0/ 0 |    0- 0/ 0/ 0/ 0/ 0 | Resource pool for demo

(ON: Online, OF: Offline, FA: Faulted, AW: Admin Wait, OT: Other states

Returns information on all resource pools currently present on the cluster along with details off their states, number of applications etc.

End Point: /api/v3/robin_server/rpools

Method: GET

URL Parameters: None

Data Parameters: None

Port: RCM Port (default value is 29442)

Headers:

  • Authorization: <auth_token> : Authorization token to identify which user is sending the request. The token can be acquired from the login API.

Success Response Code: 200

Error Response Code: 500 (Internal Server Error)

Example Response:

Output
{
   "items":[
      {
         "zoneid":1597147518,
         "hdd_size":429496729600,
         "apps":[
            {
               "name":"demo",
               "state":6
            }
         ],
         "id":1,
         "name":"default",
         "ssd_size":0,
         "num_cores":80,
         "nodes":[
            {
               "zoneid":1597147518,
               "k8s_node_status":"Ready",
               "name":"cscale-82-140",
               "remove_taint":true,
               "state":"ONLINE",
               "status":"Ready",
               "hostname":"cscale-82-140.robinsystems.com",
               "nodeid":1,
               "roles":[
                  [
                     "MANAGER",
                     "ONLINE",
                     "READY"
                  ],
                  [
                     "STORAGE",
                     "ONLINE",
                     "READY"
                  ],
                  [
                     "COMPUTE",
                     "ONLINE",
                     "READY"
                  ]
               ]
            },
            {
               "zoneid":1597147518,
               "k8s_node_status":"Notready",
               "name":"cscale-82-139",
               "remove_taint":false,
               "state":"UNREACHABLE",
               "status":"Notready",
               "hostname":"cscale-82-139.robinsystems.com",
               "nodeid":2,
               "roles":[
                  [
                     "STORAGE",
                     "OFFLINE",
                     "NOTREADY"
                  ],
                  [
                     "COMPUTE",
                     "OFFLINE",
                     "NOTREADY"
                  ]
               ]
            }
         ],
         "hdd_used_size":73000000000.0,
         "num_nodes":2,
         "ssd_used_size":0,
         "mem_size":67111419904,
         "description":"Default Resource Pool"
      }
   ]
}

4.3.3. Removing a resource pool

In order to remove a resource pool from Robin, issue the following command:

Note

Only empty resource pools which are not attached to any tenants can be removed.

# robin rpool remove [<name>]
                     --yes

name

Resource pool name to remove

--yes

Do not prompt the user for confirmation of removal

Example:

# robin rpool remove demo
Deleted resource pool 'demo'.

Removes a resource pool from Robin.

End Point: /api/v3/robin_server/rpools/<rpool_name>

Method: DELETE

URL Parameters: None

Data Parameters: None

Port: RCM Port (default value is 29442)

Headers:

  • Authorization: <auth_token> : Authorization token to identify which user is sending the request. The token can be acquired from the login API.

Success Response Code: 200

Error Response Code: 500 (Internal Server Error), 404 (Not Found Error), 401 (Unauthorized Error), 400 (Invalid API Usage Error)

Example Response:

Output
{
   "message":"Deleted resource pool demo.\n"
}

4.3.4. Assigning a resource pool to a host

When one assigns a resource pool to a host, it results in the resources attached to the host contributing to the resource pool. These resources include storage drives, CPUs and memory. In turn any application created by a user in a tenant that is associated with the resource pool will have the ability to utilize the aforementioned resources. In order to assign a resource pool to a host, issue the following command:

# robin host assign-rpool [<hosts>] [<rpool>]

hosts

A single host or a comma separated list of hosts to assign a resource pool too

rpool

Name of resource pool to assign

Example:

# robin host assign-rpool centos-60-212.robinsystems.com demo --wait
Job: 22  Name: HostAddResourcePool     State: VALIDATED       Error: 0
Job: 22  Name: HostAddResourcePool     State: WAITING         Error: 0
Job: 22  Name: HostAddResourcePool     State: COMPLETED       Error: 0

Assigns a resource pool to a host, such that the resources such as storage drives, CPU etc. attached to the host contribute to the resource pool and are available for use by applications to be deployed within that resource pool.

End Point: /api/v3/robin_server/hosts/<hostname>

Method: PUT

URL Parameters: None

Data Parameters:

  • action: add_rpool - This mandatory field within the payload specifies the add resource pool action is to be performed.

  • host_rpool: <dict_of_key_value_pairs>

    • hostname: <hostname> - This mandatory key value pairing within the host_rpool dictionary indicates which host the operation is to be performed on.

    • rpool: <rpool_name> - This mandatory key value pairing within the host_rpool dictionary indicates which resource pool should be assigned to the aformentioned host.

Port: RCM Port (default value is 29442)

Headers:

  • Authorization: <auth_token> : Authorization token to identify which user is sending the request. The token can be acquired from the login API.

Success Response Code: 202

Error Response Code: 500 (Internal Server Error), 404 (Not Found Error), 401 (Unauthorized Error), 400 (Invalid API Usage Error)

Example Response:

Output
{
   "jobid":109
}

4.4. Managing File Collections

File collections are repositories in a Robin cluster where application bundles and images required for the provisioning of applications are stored. Robin File Server, one of the core Robin management services, is responsible for managing file collections and the files they contain. Storage for each file collection is allocated from Robin managed storage. The volumes allocated for each file collection can be configured with or without replication, however it is highly recommended that replicated storage be used. This will ensure that there will not be any data loss should one of the hosts providing the storage go down or one of storage disks fail.

The following commands are described in this section:

robin collection create

Create a file collection

robin collection delete

Delete a file collection

robin collection list

List all file collections

robin collection offline

Disable a file collection

robin collection online

Enable a file collection

robin collection move

Move the contents of a file collection

4.4.1. Creating a file collection

In order to create a file collection in order to store logs, images, bundles etc, issue the following the command:

# robin collection create <media_type> <rpool>
                                       --collection_type <collection_type>
                                       --storage_type <storage_type>
                                       --replicas <replicas>
                                       --size <size>
                                       --force

media_type

The media type of storage drives to allocate from. Options include ‘HDD’ or ‘SSD’

rpool

Name of the resource pool to allocate from.

--collection_type <collection_type>

Type of collection to create. Options include ‘LOG_COLLECTION’ or ‘FILE_COLLECTION’

--storage_type <storage_type>

Type of storage to use. Options include ‘STORMGR_VOLUME’, ‘LOCAL’, or ‘STORAGE_ARRAY’

--replicas <replicas>

Replication factor of the collection volume

--size <size>

Size of allocated storage volume

--force

Override the requirement that the number of replicas must be set to 3

Example:

# robin collection create HDD default --replicas 1 --size 5G --force --wait
Job:  216 Name: CollectionAdd        State: PROCESSED       Error: 0
Job:  216 Name: CollectionAdd        State: WAITING         Error: 0
Job:  216 Name: CollectionAdd        State: COMPLETED       Error: 0

Creates a file collection in order to store logs, images, bundles etc.

End Point: /api/v3/robin_server/collections

Method: POST

URL Parameters: None

Data Parameters:

  • media: <media_type> - This mandatory field within the payload specifies the media type of storage drives to allocate from. Options include ‘HDD’ or ‘SSD’.

  • rpool: <resource_pool> - This mandatory field within the payload specifies the resource pool name from which to allocate from.

  • collection_type: <collection_type> - This mandatory field within the payload specifies the type of collection create. Options include ‘LOG_COLLECTION’ or ‘FILE_COLLECTION’.

  • storage_type: <storage_type> - This mandatory field within the payload specifies the type of storage to use. Options include ‘STORMGR_VOLUME’, ‘LOCAL’, or ‘STORAGE_ARRAY’.

  • replicas: <replicas> - This mandatory field within the payload specifies the replication factor of the collection volume. Valid values include: 1, 2, and 3.

  • size: <size> - Utilizing this parameter, by specifying the size of the collection volume in bytes, results in a volume of the aforementioned size being created. The default size is 50GB.

  • force: true - Utilizing this parameter allows one to override the requirment that the number of replicas for a collection value must be 3.

Port: RCM Port (default value is 29442)

Headers:

  • Authorization: <auth_token> : Authorization token to identify which user is sending the request. The token can be acquired from the login API.

Success Response Code: 202

Error Response Code: 500 (Internal Server Error), 404 (Not Found Error), 401 (Unauthorized Error), 400 (Invalid API Usage Error)

Example Response:

Output
{
   "plan":{
      "collection_type":"FILE_COLLECTION",
      "storage":[
         {
            "size":5368709120,
            "rpool":"default",
            "faultdomain":"host",
            "replication":1,
            "media":"HDD"
         }
      ],
      "storage_type":"STORMGR_VOLUME",
      "authorization_token":"eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJ0ZW5hbnRfaWQiOjEsInVzZXJfaWQiOjMsImV4cCI6MTYwMDIyMjY5NH0.qsv3pBCXAfGnS1JrhvncQiubEqQcyPhV2fzGse8aA-A",
      "hostname":"vnode-95-42.robinsystems.com",
      "collection_id":1600138723311,
      "force":true,
      "collection_name":"file-collection-1600138723311"
   },
   "jobid":48
}

4.4.2. Deleting a file collection

In order to delete a file collection, issue the following the command:

# robin collection delete <collection_id>
                          --force
                          --yes

collection_id

The media type of storage drives to allocate from. Options include HDD or SSD

--force

Delete a collection forcibly if the backing storage is down

--yes

Do not prompt the user for confirmation of deletion

Example:

# robin collection delete 1583820760654 --yes --wait
Job:  221 Name: CollectionDelete     State: VALIDATED       Error: 0
Job:  221 Name: CollectionDelete     State: COMPLETED       Error: 0

Deletes a file collection.

End Point: /api/v3/robin_server/collections

Method: DELETE

URL Parameters: None

Data Parameters:

  • collection_id: <collection_id> - This mandatory field within the payload specifies the ID of the collection that should be deleted.

  • force: true - Utilizing this parameter results in the collection being forcibly removed if the backing storage is down.

Port: RCM Port (default value is 29442)

Headers:

  • Authorization: <auth_token> : Authorization token to identify which user is sending the request. The token can be acquired from the login API.

Success Response Code: 202

Error Response Code: 500 (Internal Server Error), 404 (Not Found Error), 401 (Unauthorized Error), 400 (Invalid API Usage Error)

Example Response:

Output
{
   "jobid":81,
   "plan":{
      "collection_id":1597897988633,
      "authorization_token":"eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJ1c2VyX2lkIjozLCJ0ZW5hbnRfaWQiOjEsImV4cCI6MTU5Nzk2NzM1M30.Tcv6pAb66lC2k0VJpLj4oRT1elK99Qi7WBv_7Iid_zo"
   }
}

4.4.3. Listing all file collections

In order to view all file collections available on the cluster alongside details such the host it is mounted on, its size, replication factor etc, issue the following command:

# robin collection list --name <name>
                        --full
                        --json

--name <name>

File collection name to filter by

--full

Display additional information about each file collection

--json

Output in JSON

Example:

# robin collection list
Collection Id | Collection Type | Storage Type   | Name                          | Hostname                 | Status | LastOpr |  Used | Size (GB) | Replication Factor
--------------+-----------------+----------------+-------------------------------+--------------------------+--------+---------+-------+-----------+--------------------
1583356117884 | FILE_COLLECTION | STORMGR_VOLUME | file-collection-1583356117884 | vnode36.robinsystems.com |  READY |  Online | 1.3M  | 10        | 1
1583820760654 | FILE_COLLECTION | STORMGR_VOLUME | file-collection-1583820760654 | vnode36.robinsystems.com |  READY |  Online | 2.2M  | 5         | 1

Returns information on all file collections available on the cluster alongside details such the host it is mounted on, its size, replication factor etc.

End Point: /api/v3/robin_server/collections

Method: GET

URL Parameters:

  • name=<collection_name> : Utilizing this parameter results in only information for the file collection with the specified name being returned.

Data Parameters: None

Port: RCM Port (default value is 29442)

Headers:

  • Authorization: <auth_token> : Authorization token to identify which user is sending the request. The token can be acquired from the login API.

Success Response Code: 200

Error Response Code: 500 (Internal Server Error)

Example Response:

Output
{
   "collections":[
      {
         "notifications":{

         },
         "size":5368709120,
         "id":1,
         "used":"1.3M",
         "name":"file-collection-1597122699552",
         "pathname":"\/usr\/local\/robin\/collections\/file-collection-1597122699552",
         "state":"Online",
         "status":"READY",
         "timestamp":"August 10, 2020 22:11:39",
         "collection_type":"FILE_COLLECTION",
         "replication":1,
         "storage_type":"STORMGR_VOLUME",
         "hostname":"cscale-82-140.robinsystems.com",
         "collection_id":1597122699552
      }
   ]
}

4.4.4. Disabling a file collection

In order to disable a file collection such that it temporarily cannot be used/accessed, issue the following command:

# robin collection offline <collection_id>

collection_id

File collection to disable

Example:

# robin collection offline 1583820760654 --wait
Job:  218 Name: CollectionOffline    State: PROCESSED       Error: 0
Job:  218 Name: CollectionOffline    State: COMPLETED       Error: 0

# robin collection list
Collection Id | Collection Type | Storage Type   | Name                          | Hostname                 |    Status | LastOpr | Size (GB) | Replication Factor
--------------+-----------------+----------------+-------------------------------+--------------------------+-----------+---------+-----------+--------------------
1583356117884 | FILE_COLLECTION | STORMGR_VOLUME | file-collection-1583356117884 | vnode36.robinsystems.com |     READY |  Online | 10        | 1
1583820760654 | FILE_COLLECTION | STORMGR_VOLUME | file-collection-1583820760654 | vnode36.robinsystems.com | NOT_READY | Offline | 5         | 1

Disable a file collection such that it temporarily cannot be used/accessed.

End Point: /api/v3/robin_server/collections

Method: PUT

URL Parameters: None

Data Parameters:

  • action: offline - This mandatory field within the payload specifies that the offline operation is to be performed.

  • collection_id: <collection_id> - This mandatory field within the payload specifies which collection should be disabled.

Port: RCM Port (default value is 29442)

Headers:

  • Authorization: <auth_token> : Authorization token to identify which user is sending the request. The token can be acquired from the login API.

Success Response Code: 202

Error Response Code: 500 (Internal Server Error), 404 (Not Found Error), 401 (Unauthorized Error), 400 (Invalid API Usage Error)

Example Response:

Output
{
   "plan":{
      "authorization_token":"eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJ0ZW5hbnRfaWQiOjEsInVzZXJfaWQiOjMsImV4cCI6MTU5Nzc2NjE1NH0._D_s9sj_6HoOQkqytCGqlgtL6aIsp1oym6BOwntM8iU",
      "origin":1,
      "collection_id":1597122699552
   },
   "jobid":2073
}

4.4.5. Enabling a file collection

In order to enable a file collection such that it is usable again by remounting it on the master node of the cluster, issue the following command:

# robin collection online <collection_id>

collection_id

File collection to enable

Example:

# robin collection online 1583820760654 --wait
Job:  219 Name: CollectionOnline    State: PROCESSED       Error: 0
Job:  219 Name: CollectionOnline    State: COMPLETED       Error: 0

# robin collection list
Collection Id | Collection Type | Storage Type   | Name                          | Hostname                 | Status | LastOpr | Size (GB) | Replication Factor
--------------+-----------------+----------------+-------------------------------+--------------------------+--------+---------+-----------+--------------------
1583356117884 | FILE_COLLECTION | STORMGR_VOLUME | file-collection-1583356117884 | vnode36.robinsystems.com |  READY |  Online | 10        | 1
1583820760654 | FILE_COLLECTION | STORMGR_VOLUME | file-collection-1583820760654 | vnode36.robinsystems.com |  READY |  Online | 5         | 1

Enables a file collection such that it can be used again by remounting it on the master node of the cluster.

End Point: /api/v3/robin_server/collections

Method: PUT

URL Parameters: None

Data Parameters:

  • action: online - This mandatory field within the payload specifies that the online operation is to be performed.

  • collection_id: <collection_id> - This mandatory field within the payload specifies which collection should be enabled.

Port: RCM Port (default value is 29442)

Headers:

  • Authorization: <auth_token> : Authorization token to identify which user is sending the request. The token can be acquired from the login API.

Success Response Code: 202

Error Response Code: 500 (Internal Server Error), 404 (Not Found Error), 401 (Unauthorized Error), 400 (Invalid API Usage Error)

Example Response:

Output
{
   "jobid":2108,
   "plan":{
      "origin":1,
      "collection_id":1597122699552,
      "authorization_token":"eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJ0ZW5hbnRfaWQiOjEsInVzZXJfaWQiOjMsImV4cCI6MTU5Nzc2NjE1NH0._D_s9sj_6HoOQkqytCGqlgtL6aIsp1oym6BOwntM8iU"
   }
}

4.4.6. Moving a file collection

In order to move the resources contained within one file collection into another, issue the following command:

# robin collection move <collection_id> <target_collection_id>

collection_id

ID of source file collection from which to move resources from

target_collection_id

ID of target file collection to move resource to

Example:

# robin collection move 1583820760654 1583356117884 --wait
Job:  225 Name: CollectionMove    State: PROCESSED       Error: 0
Job:  225 Name: CollectionMove    State: COMPLETED       Error: 0

Moves the resources contained within one file collection into another.

End Point: /api/v3/robin_server/collections

Method: PUT

URL Parameters: None

Data Parameters:

  • action: moveresources - This mandatory field within the payload specifies that the resources within the source collection should be moved.

  • collection_id: <collection_id> - This mandatory field within the payload specifies the source collection the resources should be moved from.

  • target_collection_id: <target_collection_id> - This mandatory field within the payload specifies the destination collection the resources should be moved to.

Port: RCM Port (default value is 29442)

Headers:

  • Authorization: <auth_token> : Authorization token to identify which user is sending the request. The token can be acquired from the login API.

Success Response Code: 202

Error Response Code: 500 (Internal Server Error), 404 (Not Found Error), 401 (Unauthorized Error), 400 (Invalid API Usage Error)

Example Response:

Output
{
   "jobid":47,
   "plan":{
      "authorization_token":"eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJ0ZW5hbnRfaWQiOjEsInVzZXJfaWQiOjMsImV4cCI6MTYwNTI0NjA3OH0.wSV2mDr7y-kn_1MgafG7HxvjEdgu-cbTg3bSP4tQoZo",
      "target_collection_id":1605164780025,
      "origin":1,
      "collection_id":1605164771262
   }
}

4.5. Gathering information on nodes

Robin exposes multiple endpoints that provides a user the means by which to attain information about the hosts registered with a Robin cluster. The information that is returned is a combination of the physical attributes (obtained by the resource discovery described here), resource utilization and status of services for the host. This gives a user insight into the state of the cluster alongside granular details for each individual host and enables application deployment planning.

The following commands are described in this section:

robin host list

View all hosts in a cluster

robin host info

Display detailed information about a host

4.5.1. List all hosts

In order to view all hosts within a cluster alongside information on their statuses (from Robin’s perspective), resource consumption, and roles within the cluster, issue the following command:

# robin host list --services
                  --resources
                  --network
                  --devices
                  --tags
                  --json

--services

Show status information for each host

--resources

Show resource utilization for each host

--network

Show network resource utilization info for each host

--devices

Show devices resource utilization info for each host

--tags

Show tag information for each host

--json

Output in JSON

Example 1 (Listing all hosts):

# robin host list
Id           | Hostname                      | Version   | Status | RPool   | LastOpr | Roles  | Isol Cores(SHR/DED/Total) | Non-Isol Cores | GPUs | Mem(Free/Alloc/Total) | HDD(#/Alloc/Total) | SSD(#/Alloc/Total) | Pod Usage | Joined Time
-------------+-------------------------------+-----------+--------+---------+---------+--------+---------------------------+----------------+------+-----------------------+--------------------+--------------------+-----------+----------------------
1596566663:1 | cscale-82-81.robinsystems.com | 5.3.0-172 | Ready  | default | ONLINE  | M*,S,C | 0/0/0                     | 1/40           | 0/0  | 24G/6G/31G            | 1/5G/100G          | -/-/-              | 11/110    | 04 Aug 2020 04:44:47
1596566663:2 | cscale-82-82.robinsystems.com | 5.3.0-172 | Ready  | default | ONLINE  | M,S,C  | 0/0/0                     | 2/40           | 0/0  | 24G/7G/31G            | 2/40G/200G         | -/-/-              | 13/110    | 04 Aug 2020 04:51:52
1596566663:3 | cscale-82-83.robinsystems.com | 5.3.0-172 | Ready  | default | ONLINE  | M,S,C  | 0/0/0                     | 1/40           | 0/0  | 24G/6G/31G            | 2/-/200G           | -/-/-              | 11/110    | 04 Aug 2020 04:58:25
1596566663:4 | qct-07.robinsystems.com       | 5.3.0-172 | Ready  | workers | ONLINE  | S,C    | 8/68/76                   | 1/4            | 0/0  | 280G/95G/376G         | 1/554G/893G        | -/-/-              | 83/110    | 04 Aug 2020 05:05:24
1596566663:5 | qct-08.robinsystems.com       | 5.3.0-172 | Ready  | workers | ONLINE  | S,C    | 0/76/76                   | 1/4            | 0/0  | 284G/91G/376G         | 1/547G/893G        | -/-/-              | 88/110    | 04 Aug 2020 05:12:06
1596566663:6 | qct-11.robinsystems.com       | 5.3.0-172 | Ready  | workers | ONLINE  | S,C    | 0/27/76                   | 1/4            | 0/0  | 147G/40G/187G         | 1/581G/893G        | -/-/-              | 34/110    | 04 Aug 2020 05:18:47
1596566663:7 | cscale-82-80.robinsystems.com | 5.3.0-172 | Ready  | workers | ONLINE  | C,S    | 0/21/36                   | 1/40           | 0/0  | 0.7G/30G/31G          | 1/-/100G           | -/-/-              | 28/110    | 04 Aug 2020 20:21:40

Example 2 (Retrieving status information):

# robin host list --services
+-------------------------------+-------+-------+------+-------+-------+------+-------+------+------+------+-------+------+------+-------+---------+---------+
|              Host             | ConCl | ConSr | GCli | Httpd | Iomgr | RMon | Pgsql | RAgt | RAer | REvt | RFile | NMon | RSer | RWdog | Metrics | Stormgr |
+-------------------------------+-------+-------+------+-------+-------+------+-------+------+------+------+-------+------+------+-------+---------+---------+
| cscale-82-81.robinsystems.com |  DOWN |   UP  |  UP  |   UP  |   UP  |  UP  |   UP  |  UP  |  UP  |  UP  |   UP  |  UP  |  UP  |   UP  |   DOWN  |    UP   |
| cscale-82-82.robinsystems.com |  DOWN |   UP  |  UP  |   UP  |   UP  |  UP  |   UP  |  UP  | DOWN | DOWN |  DOWN | DOWN | DOWN |   UP  |   DOWN  |   DOWN  |
| cscale-82-83.robinsystems.com |  DOWN |   UP  |  UP  |   UP  |   UP  |  UP  |   UP  |  UP  | DOWN | DOWN |  DOWN | DOWN | DOWN |   UP  |   DOWN  |   DOWN  |
|    qct-07.robinsystems.com    |   UP  |  DOWN |  UP  |   UP  |   UP  |  UP  |  DOWN |  UP  | DOWN | DOWN |  DOWN | DOWN | DOWN |  DOWN |   DOWN  |   DOWN  |
|    qct-08.robinsystems.com    |   UP  |  DOWN |  UP  |   UP  |   UP  |  UP  |  DOWN |  UP  | DOWN | DOWN |  DOWN | DOWN | DOWN |  DOWN |   DOWN  |   DOWN  |
|    qct-11.robinsystems.com    |   UP  |  DOWN |  UP  |   UP  |   UP  |  UP  |  DOWN |  UP  | DOWN | DOWN |  DOWN | DOWN | DOWN |  DOWN |   DOWN  |   DOWN  |
| cscale-82-80.robinsystems.com |   UP  |  DOWN |  UP  |   UP  |   UP  |  UP  |  DOWN |  UP  | DOWN | DOWN |  DOWN | DOWN | DOWN |  DOWN |   DOWN  |   DOWN  |
+-------------------------------+-------+-------+------+-------+-------+------+-------+------+------+------+-------+------+------+-------+---------+---------+

        UP: Running
        CRIT: Critical and Down
        DOWN: Not Running

Returns information on all hosts within a cluster including details on their statuses (from Robin’s perspective), resource consumption, and roles within the cluster.

End Point: /api/v5/robin_server/hosts

Method: GET

URL Parameters:

  • details=tags : Utilizing this parameter results in tag information for each host being present in the response payload.

Data Parameters: None

Port: RCM Port (default value is 29442)

Headers:

  • Authorization: <auth_token> : Authorization token to identify which user is sending the request. The token can be acquired from the login API.

Success Response Code: 200

Error Response Code: 500 (Internal Server Error)

Example Response:

Output
{
   "page_num":1,
   "page_size":3,
   "total":3,
   "num_items":3,
   "nodes_count":3,
   "items":[
      {
         "id":3,
         "zoneid":1664284022,
         "state":"ONLINE",
         "gpu_cores":0,
         "cpu_cores":40,
         "maintenance_mode":"DISABLED",
         "memory":270079143936,
         "name":"poch03",
         "sysinfo":{
            "distribution":"rocky",
            "system":"Linux",
            "release":"4.18.0-372.9.1.el8.x86_64",
            "version":"#1 SMP Tue May 10 14:48:47 UTC 2022",
            "processor":"x86_64",
            "boot_time":1664310097,
            "install_date":"Tue May 10 14:48:47 UTC 2022",
            "iqn":"iqn.1994-05.com.redhat:6c454716a37",
            "wwpns":[

            ],
            "robin_software":[
               {
                  "version":"5.4.1",
                  "release":"25",
                  "patch":"",
                  "patch_date":"",
                  "full_version":"5.4.1-25",
                  "build_info":"robin-defd5ab9adce8fdfb7df57d6cbb50c28742deeb2:robin-ui:c45019c5e0e0dc0a6df2e28f5fa9d8b59ab5abb4:robingraph:6cc6485a7ffa817926962c683730d007b33636cc",
                  "install_date":"2022-09-26"
               }
            ],
            "uuid":"",
            "current_version":"5.4.1-25",
            "join_time":1664259055
         },
         "hostname":"poch03.robinsystems.com",
         "public_hostname":"poch03.robinsystems.com",
         "mem_for_storage":21143632852,
         "rpool":"default",
         "services":{
            "iomgr-server":{
               "ActiveState":"active"
            },
            "consul-client":{
               "ActiveState":"active"
            },
            "robin-agent":{
               "ActiveState":"active"
            },
            "monitor-server":{
               "ActiveState":"active"
            },
            "consul_dns":true
         },
         "memory_allocated":0,
         "cpu_cores_allocated":0,
         "gpu_cores_allocated":0,
         "memory_reserved":25438600148,
         "ninstances":11,
         "disks":[
            {
               "node_ref":3,
               "wwn":"0x5002538c4006a2f8",
               "role":"Storage",
               "type":"SSD",
               "state":"READY",
               "capacity":960197124096,
               "pfree":936873295872,
               "maintenance_mode":"DISABLED",
               "make":"Samsung based SSDs",
               "zoneid":1664284022,
               "devpath":"\/dev\/disk\/by-id\/ata-SAMSUNG_MZ7KM960HAHP-00005_S2HTNXAG905537",
               "model":"SAMSUNG MZ7KM960HAHP-00005",
               "alloc_slices":0,
               "spf":0.8,
               "dev":"\/dev\/sdi",
               "protected":0,
               "reattachable":0,
               "max_volumes_per_disk":40,
               "max_latency_sensitive_vols_per_disk":2,
               "max_throughput_intensive_vols_per_disk":1,
               "pused":0,
               "allocated":0,
               "max_alloc_slices":693,
               "free_alloc_slices":693,
               "status":"ONLINE"
            },
            {
               "node_ref":3,
               "wwn":"0x5002538c4006a9d0",
               "role":"RootDisk",
               "type":"SSD",
               "state":"INIT",
               "capacity":960197124096,
               "pfree":0,
               "maintenance_mode":"DISABLED",
               "make":"Samsung based SSDs",
               "zoneid":1664284022,
               "devpath":"\/dev\/disk\/by-id\/ata-SAMSUNG_MZ7KM960HAHP-00005_S2HTNXAG905999",
               "model":"SAMSUNG MZ7KM960HAHP-00005",
               "alloc_slices":0,
               "spf":0.8,
               "dev":"\/dev\/sdc",
               "protected":0,
               "reattachable":0,
               "max_volumes_per_disk":40,
               "max_latency_sensitive_vols_per_disk":2,
               "max_throughput_intensive_vols_per_disk":1,
               "pused":0,
               "allocated":0,
               "max_alloc_slices":693,
               "free_alloc_slices":693,
               "status":"UNKNOWN"
            },
            {
               "node_ref":3,
               "wwn":"0x5002538c4006a9d1",
               "role":"Storage",
               "type":"SSD",
               "state":"READY",
               "capacity":960197124096,
               "pfree":936873295872,
               "maintenance_mode":"DISABLED",
               "make":"Samsung based SSDs",
               "zoneid":1664284022,
               "devpath":"\/dev\/disk\/by-id\/ata-SAMSUNG_MZ7KM960HAHP-00005_S2HTNXAG906000",
               "model":"SAMSUNG MZ7KM960HAHP-00005",
               "alloc_slices":0,
               "spf":0.8,
               "dev":"\/dev\/sdd",
               "protected":0,
               "reattachable":0,
               "max_volumes_per_disk":40,
               "max_latency_sensitive_vols_per_disk":2,
               "max_throughput_intensive_vols_per_disk":1,
               "pused":0,
               "allocated":0,
               "max_alloc_slices":693,
               "free_alloc_slices":693,
               "status":"ONLINE"
            },
            {
               "node_ref":3,
               "wwn":"0x500a0751094903bd",
               "role":"Storage",
               "type":"SSD",
               "state":"READY",
               "capacity":960197124096,
               "pfree":748230279168,
               "maintenance_mode":"DISABLED",
               "make":"Crucial\/Micron BX\/MX1\/2\/3\/500",
               "zoneid":1664284022,
               "devpath":"\/dev\/disk\/by-id\/ata-Micron_M500_MTFDDAK960MAV_1332094903BD",
               "model":"Micron_M500_MTFDDAK960MAV",
               "alloc_slices":374,
               "spf":0.8,
               "dev":"\/dev\/sdf",
               "protected":0,
               "reattachable":0,
               "max_volumes_per_disk":40,
               "max_latency_sensitive_vols_per_disk":2,
               "max_throughput_intensive_vols_per_disk":1,
               "pused":188273917952,
               "allocated":343,
               "max_alloc_slices":693,
               "free_alloc_slices":319,
               "status":"ONLINE"
            },
            {
               "node_ref":3,
               "wwn":"0x5002538c400656de",
               "role":"RootDisk",
               "type":"SSD",
               "state":"INIT",
               "capacity":960197124096,
               "pfree":0,
               "maintenance_mode":"DISABLED",
               "make":"Samsung based SSDs",
               "zoneid":1664284022,
               "devpath":"\/dev\/disk\/by-id\/ata-SAMSUNG_MZ7KM960HAHP-00005_S2HTNXAG904734",
               "model":"SAMSUNG MZ7KM960HAHP-00005",
               "alloc_slices":0,
               "spf":0.8,
               "dev":"\/dev\/sdb",
               "protected":0,
               "reattachable":0,
               "max_volumes_per_disk":40,
               "max_latency_sensitive_vols_per_disk":2,
               "max_throughput_intensive_vols_per_disk":1,
               "pused":0,
               "allocated":0,
               "max_alloc_slices":693,
               "free_alloc_slices":693,
               "status":"UNKNOWN"
            },
            {
               "node_ref":3,
               "wwn":"0x500a0751094904f1",
               "role":"Storage",
               "type":"SSD",
               "state":"READY",
               "capacity":960197124096,
               "pfree":936504197120,
               "maintenance_mode":"DISABLED",
               "make":"Crucial\/Micron BX\/MX1\/2\/3\/500",
               "zoneid":1664284022,
               "devpath":"\/dev\/disk\/by-id\/ata-Micron_M500_MTFDDAK960MAV_1332094904F1",
               "model":"Micron_M500_MTFDDAK960MAV",
               "alloc_slices":0,
               "spf":0.8,
               "dev":"\/dev\/sde",
               "protected":0,
               "reattachable":0,
               "max_volumes_per_disk":40,
               "max_latency_sensitive_vols_per_disk":2,
               "max_throughput_intensive_vols_per_disk":1,
               "pused":0,
               "allocated":0,
               "max_alloc_slices":693,
               "free_alloc_slices":693,
               "status":"ONLINE"
            },
            {
               "node_ref":3,
               "wwn":"0x500a075109604e78",
               "role":"Storage",
               "type":"SSD",
               "state":"READY",
               "capacity":960197124096,
               "pfree":936504197120,
               "maintenance_mode":"DISABLED",
               "make":"Crucial\/Micron BX\/MX1\/2\/3\/500",
               "zoneid":1664284022,
               "devpath":"\/dev\/disk\/by-id\/ata-Micron_M500_MTFDDAK960MAV_140109604E78",
               "model":"Micron_M500_MTFDDAK960MAV",
               "alloc_slices":0,
               "spf":0.8,
               "dev":"\/dev\/sdg",
               "protected":0,
               "reattachable":0,
               "max_volumes_per_disk":40,
               "max_latency_sensitive_vols_per_disk":2,
               "max_throughput_intensive_vols_per_disk":1,
               "pused":0,
               "allocated":0,
               "max_alloc_slices":693,
               "free_alloc_slices":693,
               "status":"ONLINE"
            },
            {
               "node_ref":3,
               "wwn":"0x5002538c4006570e",
               "role":"RootDisk",
               "type":"SSD",
               "state":"INIT",
               "capacity":960197124096,
               "pfree":0,
               "maintenance_mode":"DISABLED",
               "make":"Samsung based SSDs",
               "zoneid":1664284022,
               "devpath":"\/dev\/disk\/by-id\/ata-SAMSUNG_MZ7KM960HAHP-00005_S2HTNXAG904752",
               "model":"SAMSUNG MZ7KM960HAHP-00005",
               "alloc_slices":0,
               "spf":0.8,
               "dev":"\/dev\/sda",
               "protected":0,
               "reattachable":0,
               "max_volumes_per_disk":40,
               "max_latency_sensitive_vols_per_disk":2,
               "max_throughput_intensive_vols_per_disk":1,
               "pused":0,
               "allocated":0,
               "max_alloc_slices":693,
               "free_alloc_slices":693,
               "status":"UNKNOWN"
            },
            {
               "node_ref":3,
               "wwn":"0x500a075109604e6c",
               "role":"Storage",
               "type":"SSD",
               "state":"READY",
               "capacity":960197124096,
               "pfree":469661384704,
               "maintenance_mode":"DISABLED",
               "make":"Crucial\/Micron BX\/MX1\/2\/3\/500",
               "zoneid":1664284022,
               "devpath":"\/dev\/disk\/by-id\/ata-Micron_M500_MTFDDAK960MAV_140109604E6C",
               "model":"Micron_M500_MTFDDAK960MAV",
               "alloc_slices":692,
               "spf":0.8,
               "dev":"\/dev\/sdj",
               "protected":0,
               "reattachable":0,
               "max_volumes_per_disk":40,
               "max_latency_sensitive_vols_per_disk":2,
               "max_throughput_intensive_vols_per_disk":1,
               "pused":466842812416,
               "allocated":650,
               "max_alloc_slices":693,
               "free_alloc_slices":1,
               "status":"ONLINE"
            },
            {
               "node_ref":3,
               "wwn":"0x500a075109604a0b",
               "role":"Storage",
               "type":"SSD",
               "state":"READY",
               "capacity":960197124096,
               "pfree":936504197120,
               "maintenance_mode":"DISABLED",
               "make":"Crucial\/Micron BX\/MX1\/2\/3\/500",
               "zoneid":1664284022,
               "devpath":"\/dev\/disk\/by-id\/ata-Micron_M500_MTFDDAK960MAV_140109604A0B",
               "model":"Micron_M500_MTFDDAK960MAV",
               "alloc_slices":0,
               "spf":0.8,
               "dev":"\/dev\/sdh",
               "protected":0,
               "reattachable":0,
               "max_volumes_per_disk":40,
               "max_latency_sensitive_vols_per_disk":2,
               "max_throughput_intensive_vols_per_disk":1,
               "pused":0,
               "allocated":0,
               "max_alloc_slices":693,
               "free_alloc_slices":693,
               "status":"ONLINE"
            }
         ],
         "roles":[
            [
               "STORAGE",
               "ONLINE",
               "READY"
            ],
            [
               "COMPUTE",
               "ONLINE",
               "READY"
            ]
         ],
         "napps":11,
         "ipaddresses":[
            {
               "mac_address":"00:25:90:fa:49:c8",
               "ip_address":"10.10.2.33",
               "netmask":"255.255.255.0"
            }
         ],
         "nics":[
            {
               "mac_address":"00:25:90:fa:49:c8",
               "name":"br0",
               "physical_nic":"ens18f0",
               "numa_node":null,
               "native_vlan":null,
               "all_vlans_allowed":false,
               "untagged":false,
               "num_vfs":0,
               "used_vfs":0,
               "native_vfdriver":null,
               "mtu":1500,
               "vendor":null,
               "vendor_desc":null,
               "domain":null,
               "bus":null,
               "slot":null,
               "function":null,
               "maintenance_mode":null,
               "allowed_vlans":[

               ],
               "vfdrivers":[

               ],
               "state":"up",
               "max_bandwidth":0,
               "used_bandwidth":0,
               "vlan_interfaces":{

               },
               "linkstate":"up"
            },
            {
               "mac_address":"00:25:90:fa:49:c9",
               "name":"ens18f1",
               "physical_nic":"ens18f1",
               "numa_node":0,
               "native_vlan":null,
               "all_vlans_allowed":false,
               "untagged":false,
               "num_vfs":0,
               "used_vfs":0,
               "native_vfdriver":null,
               "mtu":1500,
               "vendor":"8086",
               "vendor_desc":"Intel Corporation",
               "domain":"0000",
               "bus":"01",
               "slot":"00",
               "function":"1",
               "maintenance_mode":null,
               "allowed_vlans":[

               ],
               "vfdrivers":[

               ],
               "state":"up",
               "max_bandwidth":0,
               "used_bandwidth":0,
               "vlan_interfaces":{

               },
               "linkstate":"up"
            }
         ],
         "pci":[

         ],
         "hdd_faulted":0,
         "ssd_faulted":0,
         "status":"Ready",
         "memory_used":10337910784,
         "sysmem":[
            270079143936,
            257787629568,
            9442762752,
            0,
            0,
            0,
            4200140800,
            126976
         ],
         "cpu_cores_used":0,
         "cpu_prov_factor":1,
         "hugepages_2m":0,
         "hugepages_1g":0,
         "hugepages_2m_allocated":0,
         "hugepages_1g_allocated":0,
         "k8s_node_name":"poch03",
         "k8s_node_status":"Ready",
         "remove_taint":true,
         "pods":100,
         "pods_used":21,
         "cpu_cores_present":40,
         "pods_reserved":10,
         "availability_zone":"N\/A",
         "gpus":[

         ],
         "public_ip":"10.10.2.33",
         "reserved_cpus":[
            0
         ],
         "non_isol_total":40,
         "isol_total":0,
         "kvm_emulatorpin_cpuset":"",
         "shared_cores_used":3,
         "guaranteed_cores_used":1,
         "robin_ephemeral_pods":0,
         "available_guaranteed_cores":36,
         "available_shared_cores":36,
         "prelim_total_cpus":39,
         "topology_manager":true,
         "rcm_ha_role":"NONE"
      },
      {
         "id":1,
         "zoneid":1664284022,
         "state":"ONLINE",
         "gpu_cores":0,
         "cpu_cores":40,
         "maintenance_mode":"DISABLED",
         "memory":270079152128,
         "name":"poch01",
         "sysinfo":{
            "distribution":"rocky",
            "system":"Linux",
            "release":"4.18.0-372.9.1.el8.x86_64",
            "version":"#1 SMP Tue May 10 14:48:47 UTC 2022",
            "processor":"x86_64",
            "boot_time":1664306878,
            "install_date":"Tue May 10 14:48:47 UTC 2022",
            "iqn":"iqn.1994-05.com.redhat:280cd28a85d",
            "wwpns":[

            ],
            "robin_software":[
               {
                  "version":"5.4.1",
                  "release":"25",
                  "patch":"",
                  "patch_date":"",
                  "full_version":"5.4.1-25",
                  "build_info":"robin-defd5ab9adce8fdfb7df57d6cbb50c28742deeb2:robin-ui:c45019c5e0e0dc0a6df2e28f5fa9d8b59ab5abb4:robingraph:6cc6485a7ffa817926962c683730d007b33636cc",
                  "install_date":"2022-09-26"
               }
            ],
            "uuid":"",
            "current_version":"5.4.1-25",
            "join_time":1664258953
         },
         "hostname":"poch01.robinsystems.com",
         "public_hostname":"poch01.robinsystems.com",
         "mem_for_storage":21143632852,
         "rpool":"default",
         "services":{
            "iomgr-server":{
               "ActiveState":"active"
            },
            "consul-client":{
               "ActiveState":"active"
            },
            "robin-agent":{
               "ActiveState":"active"
            },
            "monitor-server":{
               "ActiveState":"active"
            },
            "consul_dns":true
         },
         "memory_allocated":0,
         "cpu_cores_allocated":0,
         "gpu_cores_allocated":0,
         "memory_reserved":25438600148,
         "ninstances":27,
         "disks":[
            {
               "node_ref":1,
               "wwn":"0x500a075109604a16",
               "role":"Storage",
               "type":"SSD",
               "state":"READY",
               "capacity":960197124096,
               "pfree":841410936832,
               "maintenance_mode":"DISABLED",
               "make":"Crucial\/Micron BX\/MX1\/2\/3\/500",
               "zoneid":1664284022,
               "devpath":"\/dev\/disk\/by-id\/ata-Micron_M500_MTFDDAK960MAV_140109604A16",
               "model":"Micron_M500_MTFDDAK960MAV",
               "alloc_slices":356,
               "spf":0.8,
               "dev":"\/dev\/sdg",
               "protected":0,
               "reattachable":0,
               "max_volumes_per_disk":40,
               "max_latency_sensitive_vols_per_disk":2,
               "max_throughput_intensive_vols_per_disk":1,
               "pused":95093260288,
               "allocated":312,
               "max_alloc_slices":693,
               "free_alloc_slices":337,
               "status":"ONLINE"
            },
            {
               "node_ref":1,
               "wwn":"0x5002538c4006a30d",
               "role":"Storage",
               "type":"SSD",
               "state":"READY",
               "capacity":960197124096,
               "pfree":936873295872,
               "maintenance_mode":"DISABLED",
               "make":"Samsung based SSDs",
               "zoneid":1664284022,
               "devpath":"\/dev\/disk\/by-id\/ata-SAMSUNG_MZ7KM960HAHP-00005_S2HTNXAG905558",
               "model":"SAMSUNG MZ7KM960HAHP-00005",
               "alloc_slices":0,
               "spf":0.8,
               "dev":"\/dev\/sde",
               "protected":0,
               "reattachable":0,
               "max_volumes_per_disk":40,
               "max_latency_sensitive_vols_per_disk":2,
               "max_throughput_intensive_vols_per_disk":1,
               "pused":0,
               "allocated":0,
               "max_alloc_slices":693,
               "free_alloc_slices":693,
               "status":"ONLINE"
            },
            {
               "node_ref":1,
               "wwn":"0x5002538c4006a30e",
               "role":"Storage",
               "type":"SSD",
               "state":"READY",
               "capacity":960197124096,
               "pfree":936873295872,
               "maintenance_mode":"DISABLED",
               "make":"Samsung based SSDs",
               "zoneid":1664284022,
               "devpath":"\/dev\/disk\/by-id\/ata-SAMSUNG_MZ7KM960HAHP-00005_S2HTNXAG905559",
               "model":"SAMSUNG MZ7KM960HAHP-00005",
               "alloc_slices":0,
               "spf":0.8,
               "dev":"\/dev\/sdf",
               "protected":0,
               "reattachable":0,
               "max_volumes_per_disk":40,
               "max_latency_sensitive_vols_per_disk":2,
               "max_throughput_intensive_vols_per_disk":1,
               "pused":0,
               "allocated":0,
               "max_alloc_slices":693,
               "free_alloc_slices":693,
               "status":"ONLINE"
            },
            {
               "node_ref":1,
               "wwn":"0x5002538c4006a30c",
               "role":"RootDisk",
               "type":"SSD",
               "state":"INIT",
               "capacity":960197124096,
               "pfree":0,
               "maintenance_mode":"DISABLED",
               "make":"Samsung based SSDs",
               "zoneid":1664284022,
               "devpath":"\/dev\/disk\/by-id\/ata-SAMSUNG_MZ7KM960HAHP-00005_S2HTNXAG905557",
               "model":"SAMSUNG MZ7KM960HAHP-00005",
               "alloc_slices":0,
               "spf":0.8,
               "dev":"\/dev\/sdi",
               "protected":0,
               "reattachable":0,
               "max_volumes_per_disk":40,
               "max_latency_sensitive_vols_per_disk":2,
               "max_throughput_intensive_vols_per_disk":1,
               "pused":0,
               "allocated":0,
               "max_alloc_slices":693,
               "free_alloc_slices":693,
               "status":"UNKNOWN"
            },
            {
               "node_ref":1,
               "wwn":"0x5002538c400656e8",
               "role":"Storage",
               "type":"SSD",
               "state":"READY",
               "capacity":960197124096,
               "pfree":936873295872,
               "maintenance_mode":"DISABLED",
               "make":"Samsung based SSDs",
               "zoneid":1664284022,
               "devpath":"\/dev\/disk\/by-id\/ata-SAMSUNG_MZ7KM960HAHP-00005_S2HTNXAG904739",
               "model":"SAMSUNG MZ7KM960HAHP-00005",
               "alloc_slices":0,
               "spf":0.8,
               "dev":"\/dev\/sdc",
               "protected":0,
               "reattachable":0,
               "max_volumes_per_disk":40,
               "max_latency_sensitive_vols_per_disk":2,
               "max_throughput_intensive_vols_per_disk":1,
               "pused":0,
               "allocated":0,
               "max_alloc_slices":693,
               "free_alloc_slices":693,
               "status":"ONLINE"
            },
            {
               "node_ref":1,
               "wwn":"0x5002538c4006567c",
               "role":"Storage",
               "type":"SSD",
               "state":"READY",
               "capacity":960197124096,
               "pfree":936873295872,
               "maintenance_mode":"DISABLED",
               "make":"Samsung based SSDs",
               "zoneid":1664284022,
               "devpath":"\/dev\/disk\/by-id\/ata-SAMSUNG_MZ7KM960HAHP-00005_S2HTNXAG904686",
               "model":"SAMSUNG MZ7KM960HAHP-00005",
               "alloc_slices":0,
               "spf":0.8,
               "dev":"\/dev\/sdb",
               "protected":0,
               "reattachable":0,
               "max_volumes_per_disk":40,
               "max_latency_sensitive_vols_per_disk":2,
               "max_throughput_intensive_vols_per_disk":1,
               "pused":0,
               "allocated":0,
               "max_alloc_slices":693,
               "free_alloc_slices":693,
               "status":"ONLINE"
            },
            {
               "node_ref":1,
               "wwn":"0x500a07510966e793",
               "role":"Storage",
               "type":"SSD",
               "state":"READY",
               "capacity":960197124096,
               "pfree":936504197120,
               "maintenance_mode":"DISABLED",
               "make":"Crucial\/Micron BX\/MX1\/2\/3\/500",
               "zoneid":1664284022,
               "devpath":"\/dev\/disk\/by-id\/ata-Micron_M500_MTFDDAK960MAV_13270966E793",
               "model":"Micron_M500_MTFDDAK960MAV",
               "alloc_slices":0,
               "spf":0.8,
               "dev":"\/dev\/sdj",
               "protected":0,
               "reattachable":0,
               "max_volumes_per_disk":40,
               "max_latency_sensitive_vols_per_disk":2,
               "max_throughput_intensive_vols_per_disk":1,
               "pused":0,
               "allocated":0,
               "max_alloc_slices":693,
               "free_alloc_slices":693,
               "status":"ONLINE"
            },
            {
               "node_ref":1,
               "wwn":"0x5002538c40065688",
               "role":"RootDisk",
               "type":"SSD",
               "state":"INIT",
               "capacity":960197124096,
               "pfree":0,
               "maintenance_mode":"DISABLED",
               "make":"Samsung based SSDs",
               "zoneid":1664284022,
               "devpath":"\/dev\/disk\/by-id\/ata-SAMSUNG_MZ7KM960HAHP-00005_S2HTNXAG904693",
               "model":"SAMSUNG MZ7KM960HAHP-00005",
               "alloc_slices":0,
               "spf":0.8,
               "dev":"\/dev\/sda",
               "protected":0,
               "reattachable":0,
               "max_volumes_per_disk":40,
               "max_latency_sensitive_vols_per_disk":2,
               "max_throughput_intensive_vols_per_disk":1,
               "pused":0,
               "allocated":0,
               "max_alloc_slices":693,
               "free_alloc_slices":693,
               "status":"UNKNOWN"
            },
            {
               "node_ref":1,
               "wwn":"0x5002538c400656e7",
               "role":"RootDisk",
               "type":"SSD",
               "state":"INIT",
               "capacity":960197124096,
               "pfree":0,
               "maintenance_mode":"DISABLED",
               "make":"Samsung based SSDs",
               "zoneid":1664284022,
               "devpath":"\/dev\/disk\/by-id\/ata-SAMSUNG_MZ7KM960HAHP-00005_S2HTNXAG904738",
               "model":"SAMSUNG MZ7KM960HAHP-00005",
               "alloc_slices":0,
               "spf":0.8,
               "dev":"\/dev\/sdd",
               "protected":0,
               "reattachable":0,
               "max_volumes_per_disk":40,
               "max_latency_sensitive_vols_per_disk":2,
               "max_throughput_intensive_vols_per_disk":1,
               "pused":0,
               "allocated":0,
               "max_alloc_slices":693,
               "free_alloc_slices":693,
               "status":"UNKNOWN"
            },
            {
               "node_ref":1,
               "wwn":"0x500a0751096049b7",
               "role":"Storage",
               "type":"SSD",
               "state":"READY",
               "capacity":960197124096,
               "pfree":454796771328,
               "maintenance_mode":"DISABLED",
               "make":"Crucial\/Micron BX\/MX1\/2\/3\/500",
               "zoneid":1664284022,
               "devpath":"\/dev\/disk\/by-id\/ata-Micron_M500_MTFDDAK960MAV_1401096049B7",
               "model":"Micron_M500_MTFDDAK960MAV",
               "alloc_slices":693,
               "spf":0.8,
               "dev":"\/dev\/sdh",
               "protected":0,
               "reattachable":0,
               "max_volumes_per_disk":40,
               "max_latency_sensitive_vols_per_disk":2,
               "max_throughput_intensive_vols_per_disk":1,
               "pused":481707425792,
               "allocated":651,
               "max_alloc_slices":693,
               "free_alloc_slices":0,
               "status":"ONLINE"
            }
         ],
         "roles":[
            [
               "STORAGE",
               "ONLINE",
               "READY"
            ],
            [
               "COMPUTE",
               "ONLINE",
               "READY"
            ]
         ],
         "napps":23,
         "ipaddresses":[
            {
               "mac_address":"00:25:90:fa:4a:5c",
               "ip_address":"10.10.2.31",
               "netmask":"255.255.255.0"
            }
         ],
         "nics":[
            {
               "mac_address":"00:25:90:fa:4a:5c",
               "name":"br0",
               "physical_nic":"ens18f0",
               "numa_node":null,
               "native_vlan":null,
               "all_vlans_allowed":false,
               "untagged":false,
               "num_vfs":0,
               "used_vfs":0,
               "native_vfdriver":null,
               "mtu":1500,
               "vendor":null,
               "vendor_desc":null,
               "domain":null,
               "bus":null,
               "slot":null,
               "function":null,
               "maintenance_mode":null,
               "allowed_vlans":[

               ],
               "vfdrivers":[

               ],
               "state":"up",
               "max_bandwidth":0,
               "used_bandwidth":0,
               "vlan_interfaces":{

               },
               "linkstate":"up"
            },
            {
               "mac_address":"00:25:90:fa:4a:5d",
               "name":"ens18f1",
               "physical_nic":"ens18f1",
               "numa_node":0,
               "native_vlan":null,
               "all_vlans_allowed":false,
               "untagged":false,
               "num_vfs":0,
               "used_vfs":0,
               "native_vfdriver":null,
               "mtu":1500,
               "vendor":"8086",
               "vendor_desc":"Intel Corporation",
               "domain":"0000",
               "bus":"01",
               "slot":"00",
               "function":"1",
               "maintenance_mode":null,
               "allowed_vlans":[

               ],
               "vfdrivers":[

               ],
               "state":"up",
               "max_bandwidth":0,
               "used_bandwidth":0,
               "vlan_interfaces":{

               },
               "linkstate":"up"
            }
         ],
         "pci":[

         ],
         "hdd_faulted":0,
         "ssd_faulted":0,
         "status":"Ready",
         "memory_used":25689063424,
         "sysmem":[
            270079152128,
            227238158336,
            20731609088,
            0,
            0,
            0,
            23560785920,
            98304
         ],
         "cpu_cores_used":0,
         "cpu_prov_factor":1,
         "hugepages_2m":0,
         "hugepages_1g":0,
         "hugepages_2m_allocated":0,
         "hugepages_1g_allocated":0,
         "k8s_node_name":"poch01",
         "k8s_node_status":"Ready",
         "remove_taint":true,
         "pods":100,
         "pods_used":43,
         "cpu_cores_present":40,
         "pods_reserved":10,
         "availability_zone":"N\/A",
         "gpus":[

         ],
         "public_ip":"10.10.2.31",
         "reserved_cpus":[
            0
         ],
         "non_isol_total":40,
         "isol_total":0,
         "kvm_emulatorpin_cpuset":"",
         "shared_cores_used":6,
         "guaranteed_cores_used":1,
         "robin_ephemeral_pods":1,
         "available_guaranteed_cores":33,
         "available_shared_cores":33,
         "prelim_total_cpus":39,
         "topology_manager":true,
         "rcm_ha_role":"NONE"
      },
      {
         "id":2,
         "zoneid":1664284022,
         "state":"ONLINE",
         "gpu_cores":0,
         "cpu_cores":40,
         "maintenance_mode":"DISABLED",
         "memory":270079143936,
         "name":"poch02",
         "sysinfo":{
            "distribution":"rocky",
            "system":"Linux",
            "release":"4.18.0-372.9.1.el8.x86_64",
            "version":"#1 SMP Tue May 10 14:48:47 UTC 2022",
            "processor":"x86_64",
            "boot_time":1664305461,
            "install_date":"Tue May 10 14:48:47 UTC 2022",
            "iqn":"iqn.1994-05.com.redhat:9634b931eb9b",
            "wwpns":[

            ],
            "robin_software":[
               {
                  "version":"5.4.1",
                  "release":"25",
                  "patch":"",
                  "patch_date":"",
                  "full_version":"5.4.1-25",
                  "build_info":"robin-defd5ab9adce8fdfb7df57d6cbb50c28742deeb2:robin-ui:c45019c5e0e0dc0a6df2e28f5fa9d8b59ab5abb4:robingraph:6cc6485a7ffa817926962c683730d007b33636cc",
                  "install_date":"2022-09-26"
               }
            ],
            "uuid":"",
            "current_version":"5.4.1-25",
            "join_time":1664259004
         },
         "hostname":"poch02.robinsystems.com",
         "public_hostname":"poch02.robinsystems.com",
         "mem_for_storage":3020518979,
         "rpool":"default",
         "services":{
            "iomgr-server":{
               "ActiveState":"active"
            },
            "consul-client":{
               "ActiveState":"active"
            },
            "robin-agent":{
               "ActiveState":"active"
            },
            "monitor-server":{
               "ActiveState":"active"
            },
            "consul_dns":true
         },
         "memory_allocated":0,
         "cpu_cores_allocated":0,
         "gpu_cores_allocated":0,
         "memory_reserved":7315486275,
         "ninstances":28,
         "disks":[
            {
               "node_ref":2,
               "wwn":"0x5002538c4006a9d2",
               "role":"RootDisk",
               "type":"SSD",
               "state":"INIT",
               "capacity":960197124096,
               "pfree":0,
               "maintenance_mode":"DISABLED",
               "make":"Samsung based SSDs",
               "zoneid":1664284022,
               "devpath":"\/dev\/disk\/by-id\/ata-SAMSUNG_MZ7KM960HAHP-00005_S2HTNXAG906001",
               "model":"SAMSUNG MZ7KM960HAHP-00005",
               "alloc_slices":0,
               "spf":0.8,
               "dev":"\/dev\/sda",
               "protected":0,
               "reattachable":0,
               "max_volumes_per_disk":40,
               "max_latency_sensitive_vols_per_disk":2,
               "max_throughput_intensive_vols_per_disk":1,
               "pused":0,
               "allocated":0,
               "max_alloc_slices":693,
               "free_alloc_slices":693,
               "status":"UNKNOWN"
            },
            {
               "node_ref":2,
               "wwn":"0x500a0751096049c0",
               "role":"RootDisk",
               "type":"SSD",
               "state":"INIT",
               "capacity":960197124096,
               "pfree":0,
               "maintenance_mode":"DISABLED",
               "make":"Crucial\/Micron BX\/MX1\/2\/3\/500",
               "zoneid":1664284022,
               "devpath":"\/dev\/disk\/by-id\/ata-Micron_M500_MTFDDAK960MAV_1401096049C0",
               "model":"Micron_M500_MTFDDAK960MAV",
               "alloc_slices":0,
               "spf":0.8,
               "dev":"\/dev\/sdh",
               "protected":0,
               "reattachable":0,
               "max_volumes_per_disk":40,
               "max_latency_sensitive_vols_per_disk":2,
               "max_throughput_intensive_vols_per_disk":1,
               "pused":0,
               "allocated":0,
               "max_alloc_slices":693,
               "free_alloc_slices":693,
               "status":"UNKNOWN"
            },
            {
               "node_ref":2,
               "wwn":"0x5002538c4006a308",
               "role":"RootDisk",
               "type":"SSD",
               "state":"INIT",
               "capacity":960197124096,
               "pfree":0,
               "maintenance_mode":"DISABLED",
               "make":"Samsung based SSDs",
               "zoneid":1664284022,
               "devpath":"\/dev\/disk\/by-id\/ata-SAMSUNG_MZ7KM960HAHP-00005_S2HTNXAG905553",
               "model":"SAMSUNG MZ7KM960HAHP-00005",
               "alloc_slices":0,
               "spf":0.8,
               "dev":"\/dev\/sde",
               "protected":0,
               "reattachable":0,
               "max_volumes_per_disk":40,
               "max_latency_sensitive_vols_per_disk":2,
               "max_throughput_intensive_vols_per_disk":1,
               "pused":0,
               "allocated":0,
               "max_alloc_slices":693,
               "free_alloc_slices":693,
               "status":"UNKNOWN"
            },
            {
               "node_ref":2,
               "wwn":"0x5002538c4006a2b0",
               "role":"RootDisk",
               "type":"SSD",
               "state":"INIT",
               "capacity":960197124096,
               "pfree":0,
               "maintenance_mode":"DISABLED",
               "make":"Samsung based SSDs",
               "zoneid":1664284022,
               "devpath":"\/dev\/disk\/by-id\/ata-SAMSUNG_MZ7KM960HAHP-00005_S2HTNXAG905465",
               "model":"SAMSUNG MZ7KM960HAHP-00005",
               "alloc_slices":0,
               "spf":0.8,
               "dev":"\/dev\/sdf",
               "protected":0,
               "reattachable":0,
               "max_volumes_per_disk":40,
               "max_latency_sensitive_vols_per_disk":2,
               "max_throughput_intensive_vols_per_disk":1,
               "pused":0,
               "allocated":0,
               "max_alloc_slices":693,
               "free_alloc_slices":693,
               "status":"UNKNOWN"
            },
            {
               "node_ref":2,
               "wwn":"0x5002538c4006a9d3",
               "role":"RootDisk",
               "type":"SSD",
               "state":"INIT",
               "capacity":960197124096,
               "pfree":0,
               "maintenance_mode":"DISABLED",
               "make":"Samsung based SSDs",
               "zoneid":1664284022,
               "devpath":"\/dev\/disk\/by-id\/ata-SAMSUNG_MZ7KM960HAHP-00005_S2HTNXAG906002",
               "model":"SAMSUNG MZ7KM960HAHP-00005",
               "alloc_slices":0,
               "spf":0.8,
               "dev":"\/dev\/sdb",
               "protected":0,
               "reattachable":0,
               "max_volumes_per_disk":40,
               "max_latency_sensitive_vols_per_disk":2,
               "max_throughput_intensive_vols_per_disk":1,
               "pused":0,
               "allocated":0,
               "max_alloc_slices":693,
               "free_alloc_slices":693,
               "status":"UNKNOWN"
            },
            {
               "node_ref":2,
               "wwn":"0x5002538c400656e6",
               "role":"RootDisk",
               "type":"SSD",
               "state":"INIT",
               "capacity":960197124096,
               "pfree":0,
               "maintenance_mode":"DISABLED",
               "make":"Samsung based SSDs",
               "zoneid":1664284022,
               "devpath":"\/dev\/disk\/by-id\/ata-SAMSUNG_MZ7KM960HAHP-00005_S2HTNXAG904737",
               "model":"SAMSUNG MZ7KM960HAHP-00005",
               "alloc_slices":0,
               "spf":0.8,
               "dev":"\/dev\/sdd",
               "protected":0,
               "reattachable":0,
               "max_volumes_per_disk":40,
               "max_latency_sensitive_vols_per_disk":2,
               "max_throughput_intensive_vols_per_disk":1,
               "pused":0,
               "allocated":0,
               "max_alloc_slices":693,
               "free_alloc_slices":693,
               "status":"UNKNOWN"
            },
            {
               "node_ref":2,
               "wwn":"0x5002538c400656e9",
               "role":"RootDisk",
               "type":"SSD",
               "state":"INIT",
               "capacity":960197124096,
               "pfree":0,
               "maintenance_mode":"DISABLED",
               "make":"Samsung based SSDs",
               "zoneid":1664284022,
               "devpath":"\/dev\/disk\/by-id\/ata-SAMSUNG_MZ7KM960HAHP-00005_S2HTNXAG904740",
               "model":"SAMSUNG MZ7KM960HAHP-00005",
               "alloc_slices":0,
               "spf":0.8,
               "dev":"\/dev\/sdc",
               "protected":0,
               "reattachable":0,
               "max_volumes_per_disk":40,
               "max_latency_sensitive_vols_per_disk":2,
               "max_throughput_intensive_vols_per_disk":1,
               "pused":0,
               "allocated":0,
               "max_alloc_slices":693,
               "free_alloc_slices":693,
               "status":"UNKNOWN"
            },
            {
               "node_ref":2,
               "wwn":"0x500a075109604a12",
               "role":"RootDisk",
               "type":"SSD",
               "state":"INIT",
               "capacity":960197124096,
               "pfree":0,
               "maintenance_mode":"DISABLED",
               "make":"Crucial\/Micron BX\/MX1\/2\/3\/500",
               "zoneid":1664284022,
               "devpath":"\/dev\/disk\/by-id\/ata-Micron_M500_MTFDDAK960MAV_140109604A12",
               "model":"Micron_M500_MTFDDAK960MAV",
               "alloc_slices":0,
               "spf":0.8,
               "dev":"\/dev\/sdg",
               "protected":0,
               "reattachable":0,
               "max_volumes_per_disk":40,
               "max_latency_sensitive_vols_per_disk":2,
               "max_throughput_intensive_vols_per_disk":1,
               "pused":0,
               "allocated":0,
               "max_alloc_slices":693,
               "free_alloc_slices":693,
               "status":"UNKNOWN"
            },
            {
               "node_ref":2,
               "wwn":"0x5002538c4006a2fb",
               "role":"RootDisk",
               "type":"SSD",
               "state":"INIT",
               "capacity":960197124096,
               "pfree":0,
               "maintenance_mode":"DISABLED",
               "make":"Samsung based SSDs",
               "zoneid":1664284022,
               "devpath":"\/dev\/disk\/by-id\/ata-SAMSUNG_MZ7KM960HAHP-00005_S2HTNXAG905540",
               "model":"SAMSUNG MZ7KM960HAHP-00005",
               "alloc_slices":0,
               "spf":0.8,
               "dev":"\/dev\/sdi",
               "protected":0,
               "reattachable":0,
               "max_volumes_per_disk":40,
               "max_latency_sensitive_vols_per_disk":2,
               "max_throughput_intensive_vols_per_disk":1,
               "pused":0,
               "allocated":0,
               "max_alloc_slices":693,
               "free_alloc_slices":693,
               "status":"UNKNOWN"
            },
            {
               "node_ref":2,
               "wwn":"0x500a0751096049dc",
               "role":"Storage",
               "type":"SSD",
               "state":"READY",
               "capacity":960197124096,
               "pfree":855872897024,
               "maintenance_mode":"DISABLED",
               "make":"Crucial\/Micron BX\/MX1\/2\/3\/500",
               "zoneid":1664284022,
               "devpath":"\/dev\/disk\/by-id\/ata-Micron_M500_MTFDDAK960MAV_1401096049DC",
               "model":"Micron_M500_MTFDDAK960MAV",
               "alloc_slices":254,
               "spf":0.8,
               "dev":"\/dev\/sdj",
               "protected":0,
               "reattachable":0,
               "max_volumes_per_disk":40,
               "max_latency_sensitive_vols_per_disk":2,
               "max_throughput_intensive_vols_per_disk":1,
               "pused":80631300096,
               "allocated":229,
               "max_alloc_slices":693,
               "free_alloc_slices":439,
               "status":"ONLINE"
            }
         ],
         "roles":[
            [
               "STORAGE",
               "ONLINE",
               "READY"
            ],
            [
               "COMPUTE",
               "ONLINE",
               "READY"
            ]
         ],
         "napps":28,
         "ipaddresses":[
            {
               "mac_address":"00:25:90:fa:48:88",
               "ip_address":"10.10.2.32",
               "netmask":"255.255.255.0"
            }
         ],
         "nics":[
            {
               "mac_address":"00:25:90:fa:48:88",
               "name":"br0",
               "physical_nic":"ens18f0",
               "numa_node":null,
               "native_vlan":null,
               "all_vlans_allowed":false,
               "untagged":false,
               "num_vfs":0,
               "used_vfs":0,
               "native_vfdriver":null,
               "mtu":1500,
               "vendor":null,
               "vendor_desc":null,
               "domain":null,
               "bus":null,
               "slot":null,
               "function":null,
               "maintenance_mode":null,
               "allowed_vlans":[

               ],
               "vfdrivers":[

               ],
               "state":"up",
               "max_bandwidth":0,
               "used_bandwidth":0,
               "vlan_interfaces":{

               },
               "linkstate":"up"
            },
            {
               "mac_address":"00:25:90:fa:48:89",
               "name":"ens18f1",
               "physical_nic":"ens18f1",
               "numa_node":0,
               "native_vlan":null,
               "all_vlans_allowed":false,
               "untagged":false,
               "num_vfs":0,
               "used_vfs":0,
               "native_vfdriver":null,
               "mtu":1500,
               "vendor":"8086",
               "vendor_desc":"Intel Corporation",
               "domain":"0000",
               "bus":"01",
               "slot":"00",
               "function":"1",
               "maintenance_mode":null,
               "allowed_vlans":[

               ],
               "vfdrivers":[

               ],
               "state":"up",
               "max_bandwidth":0,
               "used_bandwidth":0,
               "vlan_interfaces":{

               },
               "linkstate":"up"
            }
         ],
         "pci":[

         ],
         "hdd_faulted":0,
         "ssd_faulted":0,
         "status":"Ready",
         "memory_used":35581329408,
         "sysmem":[
            270079143936,
            254536040448,
            11728973824,
            0,
            0,
            0,
            5252657152,
            3022848
         ],
         "cpu_cores_used":0,
         "cpu_prov_factor":1,
         "hugepages_2m":0,
         "hugepages_1g":0,
         "hugepages_2m_allocated":0,
         "hugepages_1g_allocated":0,
         "k8s_node_name":"poch02",
         "k8s_node_status":"Ready",
         "remove_taint":true,
         "pods":100,
         "pods_used":57,
         "cpu_cores_present":40,
         "pods_reserved":10,
         "availability_zone":"N\/A",
         "gpus":[

         ],
         "public_ip":"10.10.2.32",
         "reserved_cpus":[
            0
         ],
         "non_isol_total":40,
         "isol_total":0,
         "kvm_emulatorpin_cpuset":"",
         "shared_cores_used":7,
         "guaranteed_cores_used":1,
         "robin_ephemeral_pods":0,
         "available_guaranteed_cores":32,
         "available_shared_cores":32,
         "prelim_total_cpus":39,
         "topology_manager":true,
         "rcm_ha_role":"NONE"
      }
   ]
}

4.5.2. Show information about a specific host

In order to display detailed information for a host such as the storage allocation breakdown, discovered physical attributes with their utilization (NUMA configuration, network topology etc.) and service details, issue the following command:

# robin host info <hostname>
                  --services
                  --resources
                  --config
                  --consul
                  --json

hostname

FQDN of host

--services

Show status information for the host

--resources

Show resource utilization for the host

--config

Show config info

--consul

Show consul cluster info

--json

Output in JSON

Example:

# robin host info poch01.robin.io
Output
Host: qct-07.robinsystems.com
  Zone Id: 1596566663
  Host Id: 4
  Type: physical
  Version: 5.3.0-172
  Kernel Version: 3.10.0-1062.el7.x86_64
  Boot Time: 04 Aug 2020 03:18:01
  Resource pool: workers
  CPU:
    Total Cores: 80
    Total Isolated Cores: 76
    Total Non-Isolated Cores: 4
    Non-Isolated CPUs allocated:        1
    Shared Isolated CPUs allocated:     8
    Dedicated Isolated CPUs allocated:  68
    Provisioning Factor: 1
  NUMA Topology:
    Node 0:
        Total Memory:                         187G
        Total Isolated CPUs:                  38
        Total Non-Isolated CPUs:              2
        Total Reserved CPUs:                  0
        Non-Isolated Pinned CPUs:             0
        Isolated Shared Pinned CPUs:          0
        Isolated Dedicated Pinned CPUs:       38
        Total HugePages_1G:                   -
        Total HugePages_2M:                   -
        CPU List:  1-19,41-59
        NIC List:            enp94s0f0,enp59s0f0,enp59s0f1,enp94s0f1
    Node 1:
        Total Memory:                         188G
        Total Isolated CPUs:                  38
        Total Non-Isolated CPUs:              2
        Total Reserved CPUs:                  0
        Non-Isolated Pinned CPUs:             0
        Isolated Shared Pinned CPUs:          8
        Isolated Dedicated Pinned CPUs:       30
        Total HugePages_1G:                   -
        Total HugePages_2M:                   -
        CPU List:  21-39,61-79
        NIC List:            enp175s0f1,enp175s0f0
  GPU:
    Total Cores: 0
  Memory:
    System Total: 376G
    Allocatable Total: 376G
    Reserved: 6G
      Robin Manager services: -
      Robin Compute services: 4G
      Robin Storage services: 2G
    Memory allocated to instances: 88G
    Free Total: 292G
    HugePages_2M:
      Total: -
      Allocated for Robin apps: -
    HugePages_1G:
      Total: -
      Allocated for Robin apps: -
  POD Utilization: 83/110
  Network:
    Bridge Interface: br0
      Physical Interface: enp94s0f1
      MTU: 1500
      Product Info: 158B - Ethernet Controller XXV710 for 25GbE SFP28 (Ethernet Network Adapter XXV710)
      Vendor Info: 8086 - Intel Corporation
      NUMA Node: 0
      H/W Info: 0000:5e:00.1
      IP Addresses: 10.9.20.15/16
    Interface: enp175s0f1
      MTU: 1500
      Product Info: 158B - Ethernet Controller XXV710 for 25GbE SFP28 (Ethernet Network Adapter XXV710)
      Vendor Info: 8086 - Intel Corporation
      NUMA Node: 1
      H/W Info: 0000:af:00.1
    Interface: enp175s0f0
      MTU: 1500
      Product Info: 158B - Ethernet Controller XXV710 for 25GbE SFP28 (Ethernet Network Adapter XXV710-2)
      Vendor Info: 8086 - Intel Corporation
      NUMA Node: 1
      H/W Info: 0000:af:00.0
    Interface: enp94s0f0
      MTU: 1500
      Product Info: 158B - Ethernet Controller XXV710 for 25GbE SFP28 (Ethernet Network Adapter OCP XXV710-2)
      Vendor Info: 8086 - Intel Corporation
      NUMA Node: 0
      H/W Info: 0000:5e:00.0
    Interface: enp59s0f0
      MTU: 1500
      Product Info: 158B - Ethernet Controller XXV710 for 25GbE SFP28 (Ethernet Network Adapter XXV710-2)
      Vendor Info: 8086 - Intel Corporation
      NUMA Node: 0
      H/W Info: 0000:3b:00.0
    Interface: enp59s0f1
      MTU: 1500
      Product Info: 158B - Ethernet Controller XXV710 for 25GbE SFP28 (Ethernet Network Adapter XXV710)
      Vendor Info: 8086 - Intel Corporation
      NUMA Node: 0
      H/W Info: 0000:3b:00.1
  Public IP Address: 10.9.20.15
  Public Hostname: qct-07.robinsystems.com
  Instances: 69
  State: ONLINE
  Status: Ready
  K8s_Node_Status: Ready
  Maintenance Mode: DISABLED
  Consul state: UP
  Roles:
    STORAGE: ONLINE Status:READY
    COMPUTE: ONLINE Status:READY

  Storage:
  Type | Used (GB) | Robin Allocated (GB) | K8s Allocated (GB) | Total (GB)
  -----+-----------+----------------------+--------------------+------------
  HDD  |        38 |                  538 |                 16 | 893
  SSD  |         - |                    - |                  - | -


  Services:
  Name               | State | RoleTags | PID  | Started
  -------------------+-------+----------+------+----------------------------
  consul-client      | UP    | A        | 351  | 2020-08-04 17:51:31.538657
  consul-server      | DOWN  | M        | 0    | 0
  gui-cli            | UP    | -        | 1132 | 2020-08-04 17:51:50.479143
  httpd              | UP    | *        | 1077 | 2020-08-04 17:51:50.116133
  iomgr-server       | UP    | C S      | 5391 | 2020-08-04 17:52:16.467809
  monitor-server     | UP    | M C S    | 539  | 2020-08-04 17:51:42.748944
  postgresql-9.6     | DOWN  | M        | 0    | 0
  robin-agent        | UP    | M C S    | 600  | 2020-08-04 17:51:45.394012
  robin-auth-server  | DOWN  | M*       | 0    | 0
  robin-event-server | DOWN  | M*       | 0    | 0
  robin-file-server  | DOWN  | M*       | 0    | 0
  robin-node-monitor | DOWN  | M*       | 0    | 0
  robin-server       | DOWN  | M*       | 0    | 0
  robin-watchdog     | DOWN  | M        | 0    | 0
  sherlock-server    | DOWN  | -        | 0    | 0
  stormgr-server     | DOWN  | M*       | 0    | 0

  Last updated (04 Aug 2020 18:40:14)

  UP: Running
  CRIT: Critical and Down
  DOWN: Not Running

  Root Disk storage info:
  Partition                 | Name            | Size (GB) | Available (GB)
  --------------------------+-----------------+-----------+----------------
  /var/log                  | RobinLog        |       299 |            268
  /var/lib/pgsql            | Pgsql           |       299 |            268
  /var/crash                | Crash           |       299 |            268
  /var/lib/robin            | RobinLib        |       299 |            268
  /var/lib/[appropriateCRI] | ContainerImages |        15 |              -

  Unused container images: 6G
  Image                                       | Size (GB)
  --------------------------------------------+-----------
  k8s.gcr.io/kube-controller-manager:v1.18.6  |      0.15
  k8s.gcr.io/kube-apiserver:v1.18.6           |      0.16
  k8s.gcr.io/kube-scheduler:v1.18.6           |      0.09
  robinsys/robinimg:5.2.7-18                  |         3
  quay.io/k8scsi/csi-provisioner:v1.6.0_robin |      0.04
  k8s.gcr.io/kube-proxy:v1.17.5               |      0.11
  k8s.gcr.io/kube-apiserver:v1.17.5           |      0.16
  k8s.gcr.io/kube-controller-manager:v1.17.5  |      0.15
  k8s.gcr.io/kube-scheduler:v1.17.5           |      0.09
  quay.io/k8scsi/snapshot-controller:v2.1.0   |      0.04
  k8s.gcr.io/pause:3.2                        |       0.0
  prom/prometheus:v2.16.0                     |      0.12
  quay.io/k8scsi/csi-attacher:v2.1.0          |      0.04
  calico/typha:v3.11.1                        |      0.05
  calico/pod2daemon-flexvol:v3.11.1           |       0.1
  calico/cni:v3.11.1                          |      0.18
  calico/kube-controllers:v3.11.1             |      0.05
  quay.io/k8scsi/csi-provisioner:v1.4.0_robin |      0.05
  k8s.gcr.io/kube-proxy:v1.16.3               |      0.08
  k8s.gcr.io/kube-apiserver:v1.16.3           |       0.2
  k8s.gcr.io/kube-controller-manager:v1.16.3  |      0.15
  k8s.gcr.io/kube-scheduler:v1.16.3           |      0.08
  k8s.gcr.io/coredns:1.6.5                    |      0.04
  metallb/controller:v0.8.2                   |      0.04
  metallb/speaker:v0.8.2                      |      0.04
  k8s.gcr.io/etcd:3.4.3-0                     |      0.27
  quay.io/k8scsi/csi-snapshotter:v1.2.2       |      0.04
  k8s.gcr.io/etcd:3.3.15-0                    |      0.23
  quay.io/k8scsi/csi-attacher:v1.2.1          |      0.04
  k8s.gcr.io/coredns:1.6.2                    |      0.04
  robinsys/genie-plugin:v3.0                  |      0.02
  quay.io/k8scsi/csi-provisioner:v1.0.0_robin |      0.04
  quay.io/k8scsi/csi-provisioner:v0.4.1_robin |      0.04
  robinsys/coredns:1.2.2                      |      0.03

Returns detailed information for a host such as the storage allocation breakdown, discovered physical attributes with their utilization (NUMA configuration, network topology etc.) and service details.

End Point: /api/v3/robin_server/hosts/<hostname>

Method: GET

URL Parameters:

  • diskinfo=true : Utilizing this parameter results in details of the disks attached to the specified host being returned.

Data Parameters: None

Port: RCM Port (default value is 29442)

Headers:

  • Authorization: <auth_token> : Authorization token to identify which user is sending the request. The token can be acquired from the login API.

Success Response Code: 200

Error Response Code: 500 (Internal Server Error), 404 (Not Found Error), 401 (Unauthorized Error)

Example Response:

Output
{
   "items":[
      {
         "memory_used":2692743168,
         "hdd_lalloc":35433480192.0,
         "memory":33555709952,
         "lab":"default",
         "saas_mode":false,
         "zoneid":1596601846,
         "non_isol_total_with_prov":400,
         "zone_name":"default",
         "rack":"default",
         "ipaddresses":[
            {
               "ip_address":"10.9.82.140",
               "mac_address":"00:15:5d:14:06:0e",
               "netmask":"255.255.0.0"
            }
         ],
         "public_ip":"10.9.82.140",
         "ssd_nonrobin_usage":0,
         "k8s_node_name":"cscale-82-140",
         "mem_for_storage":1073741824,
         "id":1,
         "ssd_for_storage":0,
         "rcm_ha_role":"MANAGER_MASTER",
         "ssd_robin_usage":0,
         "gpu_cores_allocated":0,
         "numa_map":{
            "0":{
               "memory_used":0,
               "hugepages_1g_used":0,
               "isol_total":0,
               "isol_shared_map":{

               },
               "cpu_reserved":0,
               "numa_id":0,
               "non_isol_cores_used":2,
               "cpu_ids":"",
               "cpu_used":0,
               "mem_used":1493172224,
               "non_isol_total":20,
               "hugepages_2m_used":0,
               "gpu_used":0,
               "isol_shared_cores_used":0,
               "hugepages_1g_total":0,
               "cpu_total":20,
               "hugepages_2m_total":0,
               "memory_total":16777626965,
               "isol_dedicated_cores_used":0
            },
            "1":{
               "memory_used":0,
               "hugepages_1g_used":0,
               "isol_total":0,
               "isol_shared_map":{

               },
               "cpu_reserved":0,
               "numa_id":1,
               "non_isol_cores_used":0,
               "cpu_ids":"",
               "cpu_used":0,
               "mem_used":0,
               "non_isol_total":20,
               "hugepages_2m_used":0,
               "gpu_used":0,
               "isol_shared_cores_used":0,
               "hugepages_1g_total":0,
               "cpu_total":20,
               "hugepages_2m_total":0,
               "memory_total":16778082988,
               "isol_dedicated_cores_used":0
            }
         },
         "roles":[
            [
               "MANAGER",
               "ONLINE",
               "READY"
            ],
            [
               "COMPUTE",
               "ONLINE",
               "READY"
            ],
            [
               "STORAGE",
               "ONLINE",
               "READY"
            ]
         ],
         "ssd_max_alloc_slices":0,
         "cpu_prov_factor":10,
         "services":{
            "update_time":1596761892.3700919151,
            "services":{
               "consul_dns":true,
               "stormgr-server":{
                  "MainPID":2299,
                  "Type":"simple",
                  "ExecMainStartTimestamp":"2020-08-04 14:31:26.794916",
                  "RoleTags":[
                     "M*"
                  ],
                  "Id":"stormgr-server",
                  "State":"UP",
                  "ActiveState":"active"
               },
               "gui-cli":{
                  "MainPID":2647,
                  "Type":"simple",
                  "ExecMainStartTimestamp":"2020-08-04 14:31:32.291459",
                  "RoleTags":[
                     "-"
                  ],
                  "Id":"gui-cli",
                  "State":"UP",
                  "ActiveState":"active"
               },
               "consul-client":{
                  "MainPID":0,
                  "Type":"simple",
                  "ExecMainStartTimestamp":0,
                  "RoleTags":[
                     "A"
                  ],
                  "Id":"consul-client",
                  "State":"DOWN",
                  "ActiveState":"inactive"
               },
               "robin-server":{
                  "MainPID":64400,
                  "Type":"simple",
                  "ExecMainStartTimestamp":"2020-08-06 17:57:55.208502",
                  "RoleTags":[
                     "M*"
                  ],
                  "Id":"robin-server",
                  "State":"UP",
                  "ActiveState":"active"
               },
               "robin-node-monitor":{
                  "MainPID":1278,
                  "Type":"simple",
                  "ExecMainStartTimestamp":"2020-08-04 14:31:13.459063",
                  "RoleTags":[
                     "M*"
                  ],
                  "Id":"robin-node-monitor",
                  "State":"UP",
                  "ActiveState":"active"
               },
               "iomgr-server":{
                  "MainPID":7384,
                  "Type":"simple",
                  "ExecMainStartTimestamp":"2020-08-04 14:55:13.851492",
                  "RoleTags":[
                     "C",
                     "S"
                  ],
                  "Id":"iomgr-server",
                  "State":"UP",
                  "ActiveState":"active"
               },
               "consul_members":[
                  {
                     "DelegateMax":5,
                     "ProtocolCur":2,
                     "Port":29460,
                     "Status":1,
                     "ProtocolMax":5,
                     "DelegateCur":4,
                     "Tags":{
                        "dc":"consul",
                        "role":"consul",
                        "vsn":"2",
                        "wan_join_port":"29461",
                        "segment":"",
                        "port":"29459",
                        "raft_vsn":"2",
                        "vsn_min":"2",
                        "vsn_max":"3",
                        "id":"9dbc13cd-bbb4-1bf1-9bcd-f3d7e0f0026f",
                        "bootstrap":"1",
                        "build":"0.9.4:40f243a+"
                     },
                     "ProtocolMin":1,
                     "Name":"cscale-82-140.robinsystems.com",
                     "Addr":"10.9.82.140",
                     "DelegateMin":2
                  }
               ],
               "robin-file-server":{
                  "MainPID":1071,
                  "Type":"simple",
                  "ExecMainStartTimestamp":"2020-08-04 14:30:55.801664",
                  "RoleTags":[
                     "M*"
                  ],
                  "Id":"robin-file-server",
                  "State":"UP",
                  "ActiveState":"active"
               },
               "robin-event-server":{
                  "MainPID":1039,
                  "Type":"simple",
                  "ExecMainStartTimestamp":"2020-08-04 14:30:55.322709",
                  "RoleTags":[
                     "M*"
                  ],
                  "Id":"robin-event-server",
                  "State":"UP",
                  "ActiveState":"active"
               },
               "sherlock-server":{
                  "MainPID":0,
                  "Type":"simple",
                  "ExecMainStartTimestamp":0,
                  "RoleTags":[
                     "-"
                  ],
                  "Id":"sherlock-server",
                  "State":"DOWN",
                  "ActiveState":"inactive"
               },
               "robin-agent":{
                  "MainPID":9186,
                  "Type":"simple",
                  "ExecMainStartTimestamp":"2020-08-04 14:59:23.165676",
                  "RoleTags":[
                     "M",
                     "C",
                     "S"
                  ],
                  "Id":"robin-agent",
                  "State":"UP",
                  "ActiveState":"active"
               },
               "postgresql-9.6":{
                  "MainPID":660,
                  "Type":"simple",
                  "ExecMainStartTimestamp":"2020-08-04 14:30:34.988682",
                  "RoleTags":[
                     "M"
                  ],
                  "Id":"postgresql-9.6",
                  "State":"UP",
                  "ActiveState":"active"
               },
               "consul-server":{
                  "MainPID":564,
                  "Type":"simple",
                  "ExecMainStartTimestamp":"2020-08-04 14:30:32.432940",
                  "RoleTags":[
                     "M"
                  ],
                  "Id":"consul-server",
                  "State":"UP",
                  "ActiveState":"active"
               },
               "httpd":{
                  "MainPID":2613,
                  "Type":"simple",
                  "ExecMainStartTimestamp":"2020-08-04 14:31:32.124472",
                  "RoleTags":[
                     "*"
                  ],
                  "Id":"httpd",
                  "State":"UP",
                  "ActiveState":"active"
               },
               "robin-auth-server":{
                  "MainPID":1010,
                  "Type":"simple",
                  "ExecMainStartTimestamp":"2020-08-04 14:30:50.844131",
                  "RoleTags":[
                     "M*"
                  ],
                  "Id":"robin-auth-server",
                  "State":"UP",
                  "ActiveState":"active"
               },
               "robin-watchdog":{
                  "MainPID":860,
                  "Type":"simple",
                  "ExecMainStartTimestamp":"2020-08-04 14:30:37.982382",
                  "RoleTags":[
                     "M"
                  ],
                  "Id":"robin-watchdog",
                  "State":"UP",
                  "ActiveState":"active"
               },
               "monitor-server":{
                  "MainPID":2687,
                  "Type":"simple",
                  "ExecMainStartTimestamp":"2020-08-04 14:31:32.455445",
                  "RoleTags":[
                     "M",
                     "C",
                     "S"
                  ],
                  "Id":"monitor-server",
                  "State":"UP",
                  "ActiveState":"active"
               }
            }
         },
         "non_isol_total":40,
         "gpu_cores":0,
         "hdd_robin_usage":35433480192,
         "visibledisks":[
            "0x600224801d3ac9b6650afd3280aa5898",
            "0x600224801d3ac9b6650afd3280aa5898-centos-root",
            "0x600224801d3ac9b6650afd3280aa5898-centos-swap",
            "0x600224801d3ac9b6650afd3280aa5898-centos-home",
            "0x600224804c48fd7e16c608dea0919064",
            "0x600224803bcdafde95b1f5cd27ceb5fb"
         ],
         "ssd_faulted":0,
         "isol_shared_cores_used":0,
         "nic_details":{
            "br0":{
               "vfdrivers":[

               ],
               "product_desc":null,
               "all_vlans_allowed":false,
               "mtu":1500,
               "bus":null,
               "vendor_id":null,
               "slot":null,
               "physical_nic":"eth0",
               "allowed_vlans":[

               ],
               "num_vfs":0,
               "function":null,
               "ips":[
                  "10.9.82.140\/16"
               ],
               "product_id":null,
               "local_cpulist":null,
               "native_vlan":null,
               "vendor_desc":null,
               "domain":null,
               "untagged":false,
               "used_vfs":0,
               "numa_node":null
            }
         },
         "ninstances":2,
         "non_isol_cores_used":3,
         "maintenance_mode":"DISABLED",
         "memory_allocated":0,
         "mem_for_management":1073741824.0,
         "rpool_id":1,
         "hdd_for_storage":214748364800,
         "datacenter":"default",
         "tags":{
            "kubernetes.io\/os":[
               "linux"
            ],
            "robin.io\/robinrpool":[
               "default"
            ],
            "kubernetes.io\/arch":[
               "amd64"
            ]
         },
         "hugepages_1g":0,
         "hugepages_2m_allocated":0,
         "rpool":"default",
         "hdd_total":428414599168,
         "ssd_free_alloc_slices":0,
         "pods":110,
         "state":"ONLINE",
         "status":"Ready",
         "instances":[
            {
               "state":"STARTED",
               "name":"rohan-app.nginx.03",
               "hostname":"rohan-app-nginx-03.t001-u000003.svc.cluster.local"
            },
            {
               "state":"STARTED",
               "name":"test-RIC-1.server.01",
               "hostname":"test-ric-1-server-01.t001-u000003.svc.cluster.local"
            }
         ],
         "isol_dedicated_cores_used":0,
         "host_type":"physical",
         "ssd_pused":0,
         "isol_total":0,
         "primary_ip":"10.9.82.140",
         "isol_shared_map":{

         },
         "hugepages_2m":0,
         "pods_used":26,
         "hdd_lused":0,
         "hdd_faulted":0,
         "napps":2,
         "hdd_nonrobin_usage":0,
         "cpu_cores_present":40,
         "ssd_total":0,
         "cpu_cores_allocated":0,
         "is_master":true,
         "memory_reserved":6442450944.0,
         "cpu_cores":400,
         "config":{
            "stormgr_rest_port":29454,
            "monitor_host_mem_lowmark":0.8,
            "monitor_host_root_volume_highmark":0.9,
            "rio_rest_port":29456,
            "stormgr_rest_listen_addr":"127.0.0.1",
            "kvm_enabled":true,
            "hard_reset_on_isolation":0,
            "monitor_host_cpu_lowmark":0.8,
            "monitor_host_var_crash_volume_highmark":0.9,
            "monitor_interval":1,
            "monitor_host_var_pgsql_volume_lowmark":0.5,
            "kubelet_restart_bursttime":25,
            "server_rest_port":29442,
            "kubelet_restart_burstlimit":2,
            "event_server_port":29449,
            "rio_rpc_port":29453,
            "rdvm_bmapcache_skip_all":0,
            "rdvm_mem_maxcap":25769803776,
            "rest_server":"cscale-82-140.robinsystems.com",
            "registration_timeout":10,
            "rdvm_rpc_port":29452,
            "node_exporter_port":29457,
            "stormgr_rpc_port":29451,
            "monitor_host_root_volume_lowmark":0.85,
            "database_port":29458,
            "rdvm_rest_listen_addr":"127.0.0.1",
            "https_port":29443,
            "metrics_grafana_details":"{\"url\": \"\", \"auth\": \":\"}",
            "monitor_host_var_volume_lowmark":0.85,
            "monitor_host_cpu_highmark":0.85,
            "rio_rest_listen_addr":"127.0.0.1",
            "monitor_host_var_robin_volume_highmark":0.9,
            "rdvm_mem_alloc":1073741824,
            "monitor_num_samples":3600,
            "monitor_host_swap_lowmark":0.75,
            "watchdog_loop_interval":3,
            "rdvm_rest_port":29455,
            "monitor_container_swap_highmark":0.8,
            "consul_serfwan_port":29461,
            "saas_mode":false,
            "file_object_cache":"\/var\/lib\/robin\/file_object_cache",
            "node_monitor_port":29467,
            "monitor_influx_details":"{\"url\": \"\", \"dbname\": \"robin\", \"auth\": \":\" }",
            "consul_http_port":29462,
            "monitor_container_swap_lowmark":0.75,
            "hostname":"cscale-82-140.robinsystems.com",
            "monitor_host_var_volume_highmark":0.9,
            "network_type":4,
            "suicide_threshold":50,
            "mem_for_compute":null,
            "mem_for_management":null,
            "sherlock_rest_port":29446,
            "nfs_mount_options":"nolock,rw,timeo=60",
            "rediscover_timeout":120,
            "kvm_emulatorpin_cpuset":"",
            "rdvm_bmapcache_invalidate_all":0,
            "consul_serflan_port":29460,
            "monitor_host_var_robin_volume_lowmark":0.85,
            "rest_port":29450,
            "monitor_report_interval":5,
            "host_type":"physical",
            "monitor_host_swap_highmark":0.8,
            "nodejs_port":29447,
            "monitor_push_interval":60,
            "ovs_enabled":true,
            "monitor_host_var_log_volume_highmark":0.9,
            "kubelet_restart_tolerance":15,
            "monitor_host_mem_highmark":0.85,
            "monitor_host_var_log_volume_lowmark":0.85,
            "log_level":10,
            "monitor_host_var_crash_volume_lowmark":0.85,
            "monitor_container_volume_highmark":0.9,
            "monitor_container_volume_lowmark":0.85,
            "consul_server_port":29459,
            "monitor_host_var_pgsql_volume_highmark":0.7
         },
         "ssd_lused":0,
         "hugepages_1g_allocated":0,
         "k8s_node_status":"Ready",
         "hdd_free_alloc_slices":293131517952.0,
         "ssd_lalloc":0,
         "hostname":"cscale-82-140.robinsystems.com",
         "public_hostname":"cscale-82-140.robinsystems.com",
         "hdd_max_alloc_slices":328564998144.0,
         "memory_total":33555709952,
         "mem_for_compute":4294967296,
         "sysinfo":{
            "join_time":1596576678,
            "current_version":"5.3.0-171",
            "iqn":"iqn.1994-05.com.redhat:329b8568de1",
            "install_date":"Tue Mar 17 23:49:17 UTC 2020",
            "wwpns":[

            ],
            "distribution":"CentOS Linux",
            "version":"#1 SMP Tue Mar 17 23:49:17 UTC 2020",
            "uuid":"",
            "boot_time":1596576222,
            "robin_software":[
               {
                  "version":"5.3.0",
                  "patch":"",
                  "full_version":"5.3.0-171",
                  "install_date":"2020-08-03",
                  "patch_date":"",
                  "release":"171",
                  "build_info":"robin-c2edf85eaa83a42ced9512e7de9c7c2f1e4fa962:robin-ui:9ee33fd00273ba19861d4dc3ef8c6169d822d3e0:robingraph:cf0ceefe696ccac2dbd2eeb1d28b859955452843"
               }
            ],
            "release":"3.10.0-1062.18.1.el7.x86_64",
            "system":"Linux",
            "processor":"x86_64"
         },
         "hdd_pused":1174405120,
         "disks":[
            {
               "spf":0.8,
               "zoneid":1596601846,
               "dev":"\/dev\/sda",
               "aslices":0,
               "nodeid":1,
               "maintenance_mode":"OFF",
               "role":"RootDisk",
               "protected":0,
               "status":"UNKNOWN",
               "make":null,
               "reattachable_nodes":[
                  [
                     "cscale-82-140.robinsystems.com",
                     "ONLINE"
                  ]
               ],
               "capacity":107374182400,
               "max_latency_sensitive_vols_per_disk":2,
               "pfree":0,
               "node_hostname":"cscale-82-140.robinsystems.com",
               "tags":{

               },
               "pused":0,
               "type":"HDD",
               "nvols":0,
               "state":"INIT",
               "reattachpolicy":{
                  "restarts_done":0,
                  "burst_count":0,
                  "burst_start_time":0,
                  "burst_interval":600,
                  "id":1,
                  "restart_limit":5
               },
               "max_alloc_slices":77,
               "stormgrid":0,
               "free_alloc_slices":77,
               "slices":0,
               "availability_zone":null,
               "max_throughput_intensive_vols_per_disk":1,
               "model":null,
               "lused_size":0,
               "devpath":"\/dev\/disk\/by-id\/scsi-3600224801d3ac9b6650afd3280aa5898",
               "alloc_slices":0,
               "reattachable":0,
               "max_volumes_per_disk":10,
               "wwn":"0x600224801d3ac9b6650afd3280aa5898",
               "allocations":[

               ],
               "alloc_score":0,
               "node_ref":1,
               "preserved":0
            },
            {
               "spf":0.8,
               "zoneid":1596601846,
               "dev":"\/dev\/dm-0",
               "aslices":0,
               "nodeid":1,
               "maintenance_mode":"OFF",
               "role":"RootDisk",
               "protected":0,
               "status":"UNKNOWN",
               "make":null,
               "reattachable_nodes":[
                  [
                     "cscale-82-140.robinsystems.com",
                     "ONLINE"
                  ]
               ],
               "capacity":53687091200,
               "max_latency_sensitive_vols_per_disk":2,
               "pfree":0,
               "node_hostname":"cscale-82-140.robinsystems.com",
               "tags":{

               },
               "pused":0,
               "type":"HDD",
               "nvols":0,
               "state":"INIT",
               "reattachpolicy":{
                  "restarts_done":0,
                  "burst_count":0,
                  "burst_start_time":0,
                  "burst_interval":600,
                  "id":4,
                  "restart_limit":5
               },
               "max_alloc_slices":38,
               "stormgrid":0,
               "free_alloc_slices":38,
               "slices":0,
               "availability_zone":null,
               "max_throughput_intensive_vols_per_disk":1,
               "model":null,
               "lused_size":0,
               "devpath":"\/dev\/disk\/by-id\/dm-uuid-LVM-vI83PDTxV3H0dWyAXfH5ef7rxTOuYyphgpZcvqGdfOKaXbEbOZzNthc6btsoSXDj",
               "alloc_slices":0,
               "reattachable":0,
               "max_volumes_per_disk":10,
               "wwn":"0x600224801d3ac9b6650afd3280aa5898-centos-root",
               "allocations":[

               ],
               "alloc_score":0,
               "node_ref":1,
               "preserved":0
            },
            {
               "spf":0.8,
               "zoneid":1596601846,
               "dev":"\/dev\/dm-1",
               "aslices":0,
               "nodeid":1,
               "maintenance_mode":"OFF",
               "role":"RootDisk",
               "protected":0,
               "status":"UNKNOWN",
               "make":null,
               "reattachable_nodes":[
                  [
                     "cscale-82-140.robinsystems.com",
                     "ONLINE"
                  ]
               ],
               "capacity":8254390272,
               "max_latency_sensitive_vols_per_disk":2,
               "pfree":0,
               "node_hostname":"cscale-82-140.robinsystems.com",
               "tags":{

               },
               "pused":0,
               "type":"HDD",
               "nvols":0,
               "state":"INIT",
               "reattachpolicy":{
                  "restarts_done":0,
                  "burst_count":0,
                  "burst_start_time":0,
                  "burst_interval":600,
                  "id":5,
                  "restart_limit":5
               },
               "max_alloc_slices":5,
               "stormgrid":0,
               "free_alloc_slices":5,
               "slices":0,
               "availability_zone":null,
               "max_throughput_intensive_vols_per_disk":1,
               "model":null,
               "lused_size":0,
               "devpath":"\/dev\/disk\/by-id\/dm-uuid-LVM-vI83PDTxV3H0dWyAXfH5ef7rxTOuYyphaFy4aq3EUo1yluonS8FG0LF16ycBrdEw",
               "alloc_slices":0,
               "reattachable":0,
               "max_volumes_per_disk":10,
               "wwn":"0x600224801d3ac9b6650afd3280aa5898-centos-swap",
               "allocations":[

               ],
               "alloc_score":0,
               "node_ref":1,
               "preserved":0
            },
            {
               "spf":0.8,
               "zoneid":1596601846,
               "dev":"\/dev\/dm-2",
               "aslices":0,
               "nodeid":1,
               "maintenance_mode":"OFF",
               "role":"RootDisk",
               "protected":0,
               "status":"UNKNOWN",
               "make":null,
               "reattachable_nodes":[
                  [
                     "cscale-82-140.robinsystems.com",
                     "ONLINE"
                  ]
               ],
               "capacity":44350570496,
               "max_latency_sensitive_vols_per_disk":2,
               "pfree":0,
               "node_hostname":"cscale-82-140.robinsystems.com",
               "tags":{

               },
               "pused":0,
               "type":"HDD",
               "nvols":0,
               "state":"INIT",
               "reattachpolicy":{
                  "restarts_done":0,
                  "burst_count":0,
                  "burst_start_time":0,
                  "burst_interval":600,
                  "id":6,
                  "restart_limit":5
               },
               "max_alloc_slices":32,
               "stormgrid":0,
               "free_alloc_slices":32,
               "slices":0,
               "availability_zone":null,
               "max_throughput_intensive_vols_per_disk":1,
               "model":null,
               "lused_size":0,
               "devpath":"\/dev\/disk\/by-id\/dm-uuid-LVM-vI83PDTxV3H0dWyAXfH5ef7rxTOuYyphQObDlS6eMUSpSxH5zsvyg9I5a0Gpuj5W",
               "alloc_slices":0,
               "reattachable":0,
               "max_volumes_per_disk":10,
               "wwn":"0x600224801d3ac9b6650afd3280aa5898-centos-home",
               "allocations":[

               ],
               "alloc_score":0,
               "node_ref":1,
               "preserved":0
            },
            {
               "spf":0.8,
               "zoneid":1596601846,
               "dev":"\/dev\/sdb",
               "aslices":7,
               "nodeid":1,
               "maintenance_mode":"OFF",
               "role":"Storage",
               "write_unit":4096,
               "status":"ONLINE",
               "make":null,
               "reattachable_nodes":[
                  [
                     "cscale-82-140.robinsystems.com",
                     "ONLINE"
                  ]
               ],
               "protected":0,
               "capacity":107374182400,
               "max_latency_sensitive_vols_per_disk":2,
               "pfree":104287174656,
               "node_hostname":"cscale-82-140.robinsystems.com",
               "tags":{

               },
               "pused":234881024,
               "type":"HDD",
               "nvols":3,
               "state":"READY",
               "reattachpolicy":{
                  "restarts_done":0,
                  "burst_count":0,
                  "burst_start_time":0,
                  "burst_interval":600,
                  "id":2,
                  "restart_limit":5
               },
               "max_alloc_slices":77,
               "stormgrid":1,
               "free_alloc_slices":68,
               "slices":6390,
               "availability_zone":null,
               "max_throughput_intensive_vols_per_disk":1,
               "model":null,
               "lused_size":0,
               "devpath":"\/dev\/disk\/by-id\/scsi-3600224804c48fd7e16c608dea0919064",
               "alloc_slices":9,
               "reattachable":0,
               "max_volumes_per_disk":10,
               "wwn":"0x600224804c48fd7e16c608dea0919064",
               "allocations":[
                  {
                     "vols":[
                        {
                           "media":"HDD",
                           "pused":167772160,
                           "id":"1",
                           "size":5368709120,
                           "state":"ONLINE",
                           "name":"file-collection-1596578146092.269f9b38-f828-48c2-a382-8921dd74ee53"
                        }
                     ],
                     "volume_group":"file-collection-1596578146092.269f9b38-f828-48c2-a382-8921dd74ee53.72.1.673abece-0975-4234-9fc2-56a06bf54031",
                     "name":"file-collection-1596578146092.269f9b38-f828-48c2-a382-8921dd74ee53.0.970a44c7-a15c-4612-ac57-9b4f15ae386e",
                     "volume":{
                        "media":"HDD",
                        "pused":167772160,
                        "id":"1",
                        "size":5368709120,
                        "state":"ONLINE",
                        "name":"file-collection-1596578146092.269f9b38-f828-48c2-a382-8921dd74ee53"
                     },
                     "slices":5
                  },
                  {
                     "vols":[
                        {
                           "media":"HDD",
                           "pused":67108864,
                           "id":"8",
                           "size":1073741824,
                           "state":"ONLINE",
                           "name":"test-RIC-1.server.01.data.1.382f1ad5-1294-4e24-8297-9c6025eacfe5"
                        }
                     ],
                     "volume_group":"test-RIC-1.server.01.72.1.ea297971-f931-4787-99cc-6782e026b77c",
                     "name":"test-RIC-1.server.01.72.1.ea297971-f931-4787-99cc-6782e026b77c.0.1e8feb41-fd42-4409-b8c1-751331febdc1",
                     "volume":{
                        "media":"HDD",
                        "pused":67108864,
                        "id":"8",
                        "size":1073741824,
                        "state":"ONLINE",
                        "name":"test-RIC-1.server.01.data.1.382f1ad5-1294-4e24-8297-9c6025eacfe5"
                     },
                     "slices":2
                  },
                  {
                     "vols":[
                        {
                           "media":"HDD",
                           "pused":0,
                           "id":"9",
                           "size":1073741824,
                           "state":"ONLINE",
                           "name":"test-RIC-1.server.01.block.1.1053eaeb-4542-42a5-a173-d69a76703ead"
                        }
                     ],
                     "volume_group":"test-RIC-1.server.01.72.1.1e483fd0-2d5c-434c-aef0-91a87796977a",
                     "name":"test-RIC-1.server.01.72.1.1e483fd0-2d5c-434c-aef0-91a87796977a.0.5bf728c4-ea26-4e0f-82d7-c584fcf0bd9a",
                     "volume":{
                        "media":"HDD",
                        "pused":0,
                        "id":"9",
                        "size":1073741824,
                        "state":"ONLINE",
                        "name":"test-RIC-1.server.01.block.1.1053eaeb-4542-42a5-a173-d69a76703ead"
                     },
                     "slices":2
                  }
               ],
               "alloc_score":95,
               "node_ref":1,
               "preserved":0
            },
            {
               "spf":0.8,
               "zoneid":1596601846,
               "dev":"\/dev\/sdc",
               "aslices":20,
               "nodeid":1,
               "maintenance_mode":"OFF",
               "role":"Storage",
               "write_unit":4096,
               "status":"ONLINE",
               "make":null,
               "reattachable_nodes":[
                  [
                     "cscale-82-140.robinsystems.com",
                     "ONLINE"
                  ]
               ],
               "protected":0,
               "capacity":107374182400,
               "max_latency_sensitive_vols_per_disk":2,
               "pfree":103582531584,
               "node_hostname":"cscale-82-140.robinsystems.com",
               "tags":{

               },
               "pused":939524096,
               "type":"HDD",
               "nvols":1,
               "state":"READY",
               "reattachpolicy":{
                  "restarts_done":0,
                  "burst_count":0,
                  "burst_start_time":0,
                  "burst_interval":600,
                  "id":3,
                  "restart_limit":5
               },
               "max_alloc_slices":77,
               "stormgrid":2,
               "free_alloc_slices":53,
               "slices":6390,
               "availability_zone":null,
               "max_throughput_intensive_vols_per_disk":1,
               "model":null,
               "lused_size":0,
               "devpath":"\/dev\/disk\/by-id\/scsi-3600224803bcdafde95b1f5cd27ceb5fb",
               "alloc_slices":24,
               "reattachable":0,
               "max_volumes_per_disk":10,
               "wwn":"0x600224803bcdafde95b1f5cd27ceb5fb",
               "allocations":[
                  {
                     "vols":[
                        {
                           "media":"HDD",
                           "pused":939524096,
                           "id":"16",
                           "size":21474836480,
                           "state":"ONLINE",
                           "name":"rohan-app.nginx.03.data.1.83d03fbf-3bfe-4723-8abe-5cbd51014e0c"
                        }
                     ],
                     "volume_group":"rohan-app.nginx.03.72.1.44251a23-0221-4fda-837e-db26bca3ccb8",
                     "name":"rohan-app.nginx.03.72.1.44251a23-0221-4fda-837e-db26bca3ccb8.0.6e2afb65-5c4c-42e4-972e-161de3fb3856",
                     "volume":{
                        "media":"HDD",
                        "pused":939524096,
                        "id":"16",
                        "size":21474836480,
                        "state":"ONLINE",
                        "name":"rohan-app.nginx.03.data.1.83d03fbf-3bfe-4723-8abe-5cbd51014e0c"
                     },
                     "slices":24
                  }
               ],
               "alloc_score":89,
               "node_ref":1,
               "preserved":0
            }
         ]
      }
   ]
}

4.6. Tagging a node

Whilst Robin provides host isolation/grouping at a higher level via a construct called resource pools (more information detailed here), a more granular control is sometimes needed. In order to achieve this Robin allows users to tag hosts with user defined tags which not only allows for separation within a resource pool but also enables the user to take advantage of Robin’s advanced application policies and enforce exactly which host(s) an application can be deployed on. This granular control is useful as it allows for on the fly alignment of resources and provide a temporary boundary for resources in comparison to resource pools which are more static in nature. Tags are defined by key-value pairs and need to be registered with Robin before they can be assigned to hosts.

The following commands are described in this section:

robin tag add-key

Register a tag

robin host add-tags

Assign a tag to a host

robin host del-tags

Unassign a tag from a host

robin tag del-key

Unregister a tag

4.6.1. Registering a tag

In order to add a tag with Robin such that it can be assigned to a host, issue the following command:

# robin tag add-key [<key>] [<values>]

Note

If a key already exists, the above command can be used to append values for the aforementioned key

key

Key to be used for tag

values

Space separated list of values to be associated with key

Example:

# robin tag add-key color blue red
Added key 'color' with values: ['red', 'blue']

Registers a tag with Robin such that it can be assigned to a host.

End Point: /api/v3/robin_server/tags/

Method: PUT

URL Parameters: None

Data Parameters:

  • action: addkey - This mandatory field within the payload specifies the add key action is to be performed.

  • key: <key_value> - This mandatory field within the payload specifies the key to be used for the tag.

  • values: <list_of_values> - This mandatory field within the payload specifies a list of comma seperated strings to be used as the values associated with the aforementioned key.

Port: RCM Port (default value is 29442)

Headers:

  • Authorization: <auth_token> : Authorization token to identify which user is sending the request. The token can be acquired from the login API.

Success Response Code: 200

Error Response Code: 500 (Internal Server Error), 401 (Unauthorized Error)

Example Response:

Output
{
   "message":"Added the following values: red, blue for key 'color'"
}

4.6.2. Assigning a tag to a node

In order to assign one or more tags to a host, issue the following command:

# robin host add-tags [<hostname>] [<tags>]

Note

Only one value for one key can be assigned to each host at any given moment.

hostname

FQDN of host

tags

Tags to assign in <key>=<value format

Example:

# robin host add-tags vnode36.robinsystems.com color=blue
Job:   80 Name: SetTag               State: VALIDATED       Error: 0
Job:   80 Name: SetTag               State: COMPLETED       Error: 0

Assigns one or more tags to a host.

End Point: /api/v3/robin_server/hosts/<hostname>

Method: PUT

URL Parameters: None

Data Parameters:

  • action: add_tag - This mandatory field within the payload specifies the add tag action is to be performed.

  • tags: <dict_of_key_value_pairs>

    • <tag_key>: <list_of_tag_values> - Each key value pair should be the tag’s key and a list of comma seperated values already associated with it. An example would be "color":["blue"]

Port: RCM Port (default value is 29442)

Headers:

  • Authorization: <auth_token> : Authorization token to identify which user is sending the request. The token can be acquired from the login API.

Success Response Code: 202

Error Response Code: 500 (Internal Server Error), 401 (Unauthorized Error), 404 (Not Found Error), 400 (Invalid API Usage Error)

Example Response:

Output
{
   "jobid": 27
}

4.6.3. Unassigning a tag from a node

In order to unassign one or more tags to a host, issue the following command:

# robin host del-tags [<hostname>] [<tags>]

hostname

FQDN of host

tags

Tags to unassign in <key>=<value> format

Example:

# robin host del-tags vnode36.robinsystems.com color=blue
Job:   81 Name: RemoveTag            State: PROCESSED       Error: 0
Job:   81 Name: RemoveTag            State: COMPLETED       Error: 0

Unassigns one or more tags from a host.

End Point: /api/v3/robin_server/hosts/<hostname>

Method: PUT

URL Parameters: None

Data Parameters:

  • action: del_tag - This mandatory field within the payload specifies the delete tag action is to be performed.

  • tags: <dict_of_key_value_pairs> -

    • <tag_key>: <list_of_tag_values> - Each key value pair should be the tag’s key and a list of comma seperated values already associated with it. An example would be "color":["blue"]

Port: RCM Port (default value is 29442)

Headers:

  • Authorization: <auth_token> : Authorization token to identify which user is sending the request. The token can be acquired from the login API.

Success Response Code: 202

Error Response Code: 500 (Internal Server Error), 401 (Unauthorized Error), 404 (Not Found Error), 400 (Invalid API Usage Error)

Example Response:

Output
{
   "jobid": 28
}

4.6.4. Unegistering a tag

In order to remove a tag from Robin, issue the following command:

# robin tag del-key [<key>] [<values>]

Note

In order to delete a key, all of its associated values must be specified.

key

Key of tag to be deleted

values

Space separated list of values to be unassociated with the key

Example:

# robin tag del-key color blue red
Deleted tags with key 'color' for the following values: 'red, blue'

Removes a tag registered with Robin or disassociates certain values from a key.

End Point: /api/v3/robin_server/tags/

Method: PUT

URL Parameters: None

Data Parameters:

  • action: delkey - This mandatory field within the payload specifies the delete key action is to be performed.

  • key: <key_value> - This mandatory field within the payload specifies the key of the tag to be removed/edited.

  • values: <list_of_values> - This mandatory field within the payload specifies a list of comma seperated strings to be disassociated with the aforementioned key. If all values associated with the key are specified, the key is removed and the tag is deleted.

Port: RCM Port (default value is 29442)

Headers:

  • Authorization: <auth_token> : Authorization token to identify which user is sending the request. The token can be acquired from the login API.

Success Response Code: 200

Error Response Code: 500 (Internal Server Error), 401 (Unauthorized Error), 404 (Not Found Error)

Example Response:

Output
{
   "message":"Deleted tags with key 'color' for the following values: 'blue'"
}

4.7. Disabling a node

In certain situations, a user might not want any resources (storage or compute) for an application to be allocated from a particular host due to a malfunction with the physical machine or simply because the host is temporarily undergoing maintenance. Instead of requiring the user to remove the node from an existing cluster, Robin allows one to place a host into maintenance mode. This effectively isolates the host with regards to resource availability as it entails that none of the host’s storage capacity or compute resources can be used for future application deployment regardless of the Robin roles assigned to the node. This mode can be toggled using the commands detailed below. For more granular control, please review the section on disabling/enabling particular roles here.

The following commands are described in this section:

robin host set-maintenance

Place a host into maintenance mode

robin host unset-maintenance

Place a host into non-maintenance (normal) mode

4.7.1. Placing a host into maintenance mode

In order to put a host into maintenance and thus temporarily suspend it from providing either storage or compute resources for future application deployments, issue the following command:

# robin host set-maintenance <hostname>

hostname

FQDN of host

Example:

# robin host set-maintenance vnode36.robinsystems.com
Host vnode36.robinsystems.com set in maintenance mode

Puts a host into maintenance mode, which in turn temporarily suspends it from providing storage and compute resources for application deployments.

End Point: /api/v3/robin_server/hosts/<hostname>

Method: PUT

URL Parameters: None

Data Parameters:

  • action: set_maintenance - This mandatory field within the payload specifies that the set maintenance mode operation is to be performed.

Port: RCM Port (default value is 29442)

Headers:

  • Authorization: <auth_token> : Authorization token to identify which user is sending the request. The token can be acquired from the login API.

Success Response Code: 200

Error Response Code: 500 (Internal Server Error), 404 (Not Found Error), 401 (Unauthorized Error), 400 (Invalid API Usage Error)

Example Response:

Output
{
   "message":"Maintenance mode set"
}

4.7.2. Placing a host into non-maintenance mode

In order to revert a host back into its normal setting and thus allow it to provide resources for future application deployments, issue the following command:

# robin host unset-maintenance <hostname>

hostname

FQDN of host

Example:

# robin host unset-maintenance vnode36.robinsystems.com
Host vnode36.robinsystems.com out of maintenance mode

Removes a host from maintenance mode, which in turn allows it to provide storage and compute resources for application deployments.

End Point: /api/v3/robin_server/hosts/<hostname>

Method: PUT

URL Parameters: None

Data Parameters:

  • action: unset_maintenance - This mandatory field within the payload specifies that the unset maintenance mode operation is to be performed.

Port: RCM Port (default value is 29442)

Headers:

  • Authorization: <auth_token> : Authorization token to identify which user is sending the request. The token can be acquired from the login API.

Success Response Code: 200

Error Response Code: 500 (Internal Server Error), 404 (Not Found Error), 401 (Unauthorized Error), 400 (Invalid API Usage Error)

Example Response:

Output
{
   "message":"Maintenance mode unset"
}

4.8. Remove a Worker or Master node

You might need to remove a worker node or Master node from your cluster if the node malfunctioned for any reason and replace it with a new one. You must ensure that a Robin HA cluster must have three master nodes at all times.

Prerequisites

The following are the prerequisites to remove a worker or master node:

  • Move all pods from the node you plan to remove to another healthy node on the cluster. You can use the robin instance relocate command to move. For more information, see here

  • Move all volumes from the node you plan to remove to drives residing on other healthy nodes. Use the robin drive evacuate command. For more information, see here

  • Move all Kubernetes pods from the node you plan to another node by running the kubectl drain <node name> command.

The following are the high-level steps for removing a Worker node or Master node:

4.8.1. Step 1 - Remove assigned node role

You must first remove any assigned node role from the required node before removing a node from the cluster. A node generally has storage and compute roles.

To remove an assigned role from the node, complete the following steps:

  1. Log in to one of the master nodes and run the following command to view the list of nodes in a cluster, and find the node name that you want to remove.

    robin host list
    

    Example

    # robin host list
    
    Id           | Hostname                    | Version  | Status | RPool   | Avail. Zone | LastOpr | Roles | Cores     | GPUs  | Mem            | HDD(#/Alloc/Total) | SSD(#/Alloc/Total) | Pod Usage | Joined Time
    -------------+-----------------------------+----------+--------+---------+-------------+---------+-------+-----------+-------+----------------+--------------------+--------------------+-----------+----------------------
    1663838470:1 | rakuten-12.robinsystems.com | 5.4.1-21 | Ready  | default | N/A         | ONLINE  | S,C   | 124/4/128 | 0/0/0 | 974G/33G/1007G | -/-/-              | 6/5G/17884G        | 76/24/100 | 21 Sep 2022 19:23:02
    1663838470:2 | rakuten-14.robinsystems.com | 5.4.1-21 | Ready  | default | N/A         | ONLINE  | S,C   | 123/5/128 | 0/0/0 | 972G/34G/1007G | -/-/-              | 7/5G/20865G        | 71/29/100 | 21 Sep 2022 19:23:53
    1663838470:3 | rakuten-15.robinsystems.com | 5.4.1-21 | Ready  | default | N/A         | ONLINE  | S,C   | 123/5/128 | 0/0/0 | 973G/33G/1007G | -/-/-              | 7/-/20865G         | 72/28/100 | 21 Sep 2022 19:24:43
    
    * Note: all values indicated above in the format XX/XX/XX represent the Free/Allocated/Total values of the respective resource unless otherwise specified. In addition allocated values for compute resource such as cpu, memory and pod usage includes reserved values for the corresponding resource.
    
  2. Run the following command to remove the role for the required node.

    robin host remove-role <node name> storage, compute –wait
    

    Example

    # robin host remove-role rakuten-15.robinsystems.com storage,compute --wait
    
    Are you sure you want to remove the role(s) 'storage,compute' from host 'rakuten-15.robinsystems.com' [y/n] ? y
    Job:  117 Name: HostRemoveRoles      State: PROCESSED       Error: 0
    Job:  117 Name: HostRemoveRoles      State: WAITING         Error: 0
    Job:  117 Name: HostRemoveRoles      State: COMPLETED       Error: 0
    
  3. Run the robin host list command to verify that roles are removed for the node.

    Example

    # robin host list
    Id           | Hostname                    | Version  | Status | RPool   | Avail. Zone | LastOpr | Roles | Cores     | GPUs  | Mem            | HDD(#/Alloc/Total) | SSD(#/Alloc/Total) | Pod Usage | Joined Time
    -------------+-----------------------------+----------+--------+---------+-------------+---------+-------+-----------+-------+----------------+--------------------+--------------------+-----------+----------------------
    1663838470:1 | rakuten-12.robinsystems.com | 5.4.1-21 | Ready  | default | N/A         | ONLINE  | S,C   | 124/4/128 | 0/0/0 | 974G/33G/1007G | -/-/-              | 6/5G/17884G        | 76/24/100 | 21 Sep 2022 19:23:02
    1663838470:2 | rakuten-14.robinsystems.com | 5.4.1-21 | Ready  | default | N/A         | ONLINE  | S,C   | 123/5/128 | 0/0/0 | 972G/34G/1007G | -/-/-              | 7/5G/20865G        | 71/29/100 | 21 Sep 2022 19:23:53
    1663838470:3 | rakuten-15.robinsystems.com | 5.4.1-21 | Ready  | default | N/A         | SYNCED  |       | 123/5/128 | 0/0/0 | 1001G/5G/1007G | -/-/-              | -/-/-              | 72/28/100 | 21 Sep 2022 19:24:43
    
    * Note: all values indicated above in the format XX/XX/XX represent the Free/Allocated/Total values of the respective resource unless otherwise specified. In addition allocated values for compute resource such as cpu, memory and pod usage includes reserved values for the corresponding resource
    

4.8.2. Step 2 - Remove host from Robin cluster

After you remove the assigned roles from the node, you must remove the host from the Robin cluster.

To remove the host, complete the following steps:

  1. Run the following command to remove the host.

    robin host remove <host name> –wait
    

    Example

    # robin host remove rakuten-15.robinsystems.com
    Are you sure you want to delete [y/n] ? y
    Submitted job '129'. Use 'robin job wait 129' to track the progress
    # robin job wait 129
    Job:  129 Name: HostRemove           State: COMPLETED       Error: 0
    
  2. Run the robin host list command to confirm the required host is removed from the Robin Cluster.

After removing the host, you must delete the node from the Kubernetes cluster.

4.8.3. Step 3 - Delete node from the Kubernetes cluster

After removing the host from the Robin cluster, you must delete the corresponding Kubernetes node from the Kubernetes cluster.

To delete a node from the Kubernetes, cluster complete the following steps:

  1. Run the following command to view the list of Kubernetes nodes. Make a note of the node name that you need to delete from the list.

    kubectl get nodes
    

    Example

    # kubectl get nodes
    NAME                          STATUS   ROLES                  AGE    VERSION
    rakuten-12.robinsystems.com   Ready    control-plane,master   115m   v1.23.8
    rakuten-14.robinsystems.com   Ready    control-plane,master   114m   v1.23.8
    rakuten-15.robinsystems.com   Ready    control-plane,master   114m   v1.23.8
    
  2. Run the following command to delete the required nodes from the Kubernetes cluster.

    Note

    You need to cordon the node if the node is online.

    kubectl delete node <node name>
    

    Example

    # kubectl delete node rakuten-15.robinsystems.com
    node "rakuten-15.robinsystems.com" deleted
    
  3. Run the kubectl get nodes command to confirm that node is deleted from the Kubernetes cluster. You should not see the deleted node in the list.

After deleting the Kubernetes node, you must remove the deleted node from the ETCD cluster

4.8.4. Step 4 - Remove deleted node from ETCD cluster

After you delete the Kubernetes node, you must remove the deleted node ID from ETCD members.

To remove deleted node from the ETCD, complete the following steps:

  1. Run the following command to view ETCD members and note the deleted node’s ID from the list.

    retcd get-members
    

    Example

    # retcd get-members
    +------------------+---------+-----------------------------+-----------------------------------------------+-----------------------------------------------+------------+
    |        ID        | STATUS  |            NAME             |                  PEER ADDRS                   |                 CLIENT ADDRS                  | IS LEARNER |
    +------------------+---------+-----------------------------+-----------------------------------------------+-----------------------------------------------+------------+
    | 40d4cb4906cb56b6 | started | rakuten-12.robinsystems.com | https://[fd74:ca9b:3a09:868c:10:9:20:92]:2380 | https://[fd74:ca9b:3a09:868c:10:9:20:92]:2379 |      false |
    | b839707ae1e0a0ab | started | rakuten-15.robinsystems.com | https://[fd74:ca9b:3a09:868c:10:9:20:95]:2380 | https://[fd74:ca9b:3a09:868c:10:9:20:95]:2379 |      false |
    | d2a42873a4133282 | started | rakuten-14.robinsystems.com | https://[fd74:ca9b:3a09:868c:10:9:20:94]:2380 | https://[fd74:ca9b:3a09:868c:10:9:20:94]:2379 |      false |
    +------------------+---------+-----------------------------+-----------------------------------------------+-----------------------------------------------+------------+
    
  2. Run the following command to remove deleted node from the ETCD cluster member.

    retcd remove-member <member ID>
    

    Example

    # retcd remove-member --member-id=b839707ae1e0a0ab
    Member b839707ae1e0a0ab removed from cluster a7b3089f00157a28
    
  3. Run the retcd get-members command to confirm the removal of the node from the ETCD cluster.

4.9. Add or replace a Worker or Master Node

You might need to add a new worker node to your Robin cluster or replace a failed node (master or worker) back to the cluster.

To add or replace the worker or master node, you must copy the Robin configuration from the Robin cluster to the node that you are adding or replacing.

The following are the high-level steps to add or replace a worker or master node:

Prerequisites

  • Kubernetes node is added (new or replacement)

  • Copy the GoRobin tar file to the new node and untar it

  • Login credentials of the new or replacing node

4.9.1. Step 1 - Copy config from the existing Master node

You need to copy the configuration file from any of the existing master nodes to add or replace worker or master nodes.

Complete the following steps to copy the configuration file:

  1. Log in to any of the existing master nodes.

  2. Run the following command to copy the configuration file.

    scp $HOME/.kube/config <new or replacing node name>:/tmp/kubeconfig
    

    Example

    # scp $HOME/.kube/config root@test-node:/tmp/kubeconfig
    

4.9.2. Step 2 - Install Robin Host

After copying the configuration file from one of the Master nodes to the new or replacing node, you must install the Robin host to make the host part of the cluster. You can install the Robin host using the GoRobin utility.

Complete the following steps to install Robin Host:

  1. Log in to the new or replacing node (node where you have copied the configuration file).

  2. Access the folder where you have copied the GoRobin tar file and untar the file using the following command.

    tar -xvf <gorobintar-x.x.x-xxx.tar>
    
  3. Run the ls command to view the list of files inside the gorobintar folder.

    Example

    # ls
    gorobintar-5.4.1-21.tar
    
    # tar -xvf gorobintar-5.4.1-21.tar
    gorobintar/
    gorobintar/host-script-el7.sh
    gorobintar/host-script-el8.sh
    gorobintar/k8s-script-el7.sh
    gorobintar/k8s-script-el8.sh
    gorobintar/k8splus-script.sh
    gorobintar/robin-script.sh
    gorobintar/k8simg.tar.gz
    gorobintar/robinimg.tar.gz
    gorobintar/bundles.tar.gz
    
  4. Inside the gorobintar folder, run the appropriate host script to install Robin host (host-script-el7 or host-script-el8).

    Note

    Use the host-script-el7.sh script for Red Hat Linux 7.x and use host-script-el8.sh for Rockey Linux 8.x.

    ./host-script-el8.sh install --robin-image-archive=<path for robinimg.tar.gz> --k8s-image-archive=<path for k8simg.tar.gz file> --host-type=<physical> --ignore-warnings --ip-protocol=<ipv4 or ipv6> --nics=<nic name>
    

    Example

    ./host-script-el8.sh install --robin-image-archive=/tmp/gorobintar/robinimg.tar.gz --k8s-image-archive=/tmp/gorobintar/k8simg.tar.gz --host-type=physical --ignore-warnings --ip-protocol=ipv6 --nics=ens1f1
    
                                      Robin Host Install
    
     Checking Tar...                                   : DONE
     Extracting Payload                                : DONE
     Validating Input Parameters                       : DONE
     Inspecting System Configuration                   : DONE
        Checking Python...                                                : PASSED
        Checking for curl ...                                             : PASSED
        Checking curl version >= 7.29.0 ...                               : PASSED
        Checking host type...                                             : PASSED
        Checking IPv6 forwarding...                                       : WARNING
        Checking kernel version...                                        : PASSED
        Checking if tcmu modules can be loaded...                         : PASSED
        Checking whether system has an FQDN...                            : PASSED
        Checking validity of system FQDN...                               : PASSED
        Checking localhost entry in /etc/hosts file...                    : PASSED
        Checking system FQDN format in /etc/hosts...                      : PASSED
        Checking if fqdn is resolvable...                                 : PASSED
        Checking / usage is less than 90%...                              : PASSED
        Checking space in /tmp...                                         : PASSED
        Checking for stale TCMU devices...                                : PASSED
        Checking /var/lib/docker is on xfs...                             : PASSED
        Checking /var/lib xfs dtype set...                                : PASSED
        Checking if /var/lib/docker is a partition...                     : WARNING
        Checking space in /var/lib/docker...                              : WARNING
        Checking space in /var/lib...                                     : PASSED
        Checking isolated cpu siblings...                                 : PASSED
        Checking Hugepages configuration...                               : PASSED
        Checking Multiple sized Hugepages ...                             : PASSED
        Checking Valid NIC NUMA nodes...                                  : PASSED
        Checking KVM Virtualization...                                    : PASSED
        Checking KVM vhost-net...                                         : PASSED
        Checking KVM networking...                                        : PASSED
        Checking KVM PCI device passthrough...                            : WARNING
        Checking if directory /var/run/udev exists...                     : PASSED
        Checking hostname resolution...                                   : PASSED
        Checking if /home/robinds/var/log is a partition...               : WARNING
        Checking space in /home/robinds/var/log...                        : WARNING
        Checking if /home/robinds/var/crash is a partition...             : WARNING
        Checking space in /home/robinds/var/crash...                      : WARNING
        Checking if /home/robinds is a partition...                       : WARNING
        Checking space in /home/robinds...                                : WARNING
        Checking if /home/robinds/var/lib/pgsql is a partition...         : WARNING
        Checking space in /home/robinds/var/lib/pgsql...                  : WARNING
        Checking if Swap is disabled...                                   : PASSED
        Checking if network ports are in use...                           : PASSED
        Checking firewalls...                                             : PASSED
        Checking /tmp folder for permissions...                           : PASSED
        Checking DNS Configuration...                                     : PASSED
        Checking Domain Configuration...                                  : PASSED
        Checking for incomplete YUM transactions...                       : PASSED
        Checking YUM repolist..                                           : PASSED
        Checking if EPEL repository...                                    : PASSED
        Checking restart of network service...                            : PASSED
        Checking if ntpd service is running...                            : WARNING
        Checking if kernel dump is enabled...                             : PASSED
        Checking if crash kernel is loaded...                             : WARNING
        Checking internet connection...                                   : WARNING
        Checking 8.8.8.8 DNS ...                                          : WARNING
        Checking CPU count                                                : PASSED
        Checking available CPUs                                           : PASSED
        Checking available memory...                                      : PASSED
        Checking if required PATHs are present...                         : PASSED
        Checking for NIC bonding...                                       : PASSED
        Checking for partitions ...                                       : WARNING
        Checking for LVM drives ...                                       : PASSED
        Checking for drive mounts ...                                     : PASSED
        Checking for drives to use as Robin storage...                    : WARNING
    
                                      Robin Precheck warnings/errors
    
           WARNING: IPv4/IPv6 forwarding is disabled, will be enabled as part of the installation
           WARNING: /var/lib/docker not a partition (Folder not present)
           WARNING: /var/lib/docker folder not present. Required space: 40G
           WARNING: KVM PCI device passthrough not enabled. Set iommu=pt and intel_iommu=on or amd_iommu=on in GRUB if planning to deploy KVM+SRIOV apps
           WARNING: /home/robinds/var/log not a partition (Folder not present)
           WARNING: /home/robinds/var/log folder not present. Required space: 60G
           WARNING: /home/robinds/var/crash not a partition (Folder not present)
           WARNING: /home/robinds/var/crash folder not present. Required space: 100G
           WARNING: /home/robinds not a partition (Folder not present)
           WARNING: /home/robinds folder not present. Required space: 40G
           WARNING: /home/robinds/var/lib/pgsql not a partition (Folder not present)
           WARNING: /home/robinds/var/lib/pgsql folder not present. Required space: 50G
           WARNING: Service ntpd is not running
           WARNING: Crash kernel is not loaded
           WARNING: Internet is not accessible
           WARNING: 8.8.8.8 is not accessible
           WARNING: Partitioned drives nvme0n1 nvme1n1 nvme2n1 nvme4n1 nvme3n1 nvme5n1 nvme7n1 nvme6n1  cannot be used for ROBIN storage
           WARNING: No drives available for storage (No storage role)
    
     Ignoring Precheck warnings
     Using --ignore-warnings may limit logging and core dumps by ROBIN processes.
     Installing Host Packages                          : DONE
     Validating Host Packages                          : DONE
     Setting up Binaries and Services                  : DONE
     Setting up Host                                   : DONE
     Setting up Runtimes                               : DONE
     Loading Kubernetes Images                         : DONE
     Checking K8s Images                               : DONE
     Setting up Default CRI                            : DONE
     Loading ROBIN Image                               : DONE
     Checking Robin Image                              : DONE
     Setting up ROBIN CRI                              : DONE
     Successfully prepared host for ROBIN CNP
    

4.9.3. Step 3 - Add or replace Worker node or Master node

After installing the necessary host packages on the node, you need to add a worker node or master node according to your requirements.

4.9.3.1. Add or replace Worker node

You can add a new worker node or replace a malfunctioned node. Run the following command to add a worker node:

Note

Use the k8s-script-el7.sh script for Red Hat Linux 7.x and use k8s-script-el8.sh for Rockey Linux 8.x.

./k8s-script-el8.sh worker --kubeconfig=/tmp/kubeconfig --ignore-warnings

Example

./k8s-script-el8.sh worker --kubeconfig=/tmp/kubeconfig --ignore-warnings

                                Robin K8s Worker

Extracting Payload                                : DONE
Installing Kubernetes Packages                    : DONE
Validating Kubernetes Packages                    : DONE
Fetching Global Configuration                     : DONE
Setting up Kubernetes Node                        : DONE
Joining Kubernetes Master                         : DONE
Setting up Helm                                   : DONE
Successfully setup Kubernetes node

4.9.3.2. Add or replace Master node

You can add a new master node or replace a malfunctioned Robin master node.

Run the following command to add or replace a Master node:

./k8s-script-el8.sh master --kubeconfig=/tmp/kubeconfig --ignore-warnings

Example

./k8s-script-el8.sh master --kubeconfig=/tmp/kubeconfig --ignore-warnings

                                 Robin K8s Master

Extracting Payload                                : DONE
Installing Kubernetes Packages                    : DONE
Validating Kubernetes Packages                    : DONE
Fetching Global Configuration                     : DONE
Setting up Kubernetes Node                        : DONE
Setting up Kubernetes HA                          : DONE
Setting up Helm                                   : DONE
Successfully setup kubernetes node

4.9.4. Step 4 - Verification after adding or replacing worker or master node

Once you add or replace a worker or master node, you must verify the status of the following to ensure that the addition or replacement of the node is successful:

  • Nodes

  • Pods

  • Patroni cluster

  • ETCD cluster

4.9.4.1. Verify nodes

After you add or replace a worker or master node, you must verify that all nodes are in the Ready status.

Run the following command to verify nodes:

kubectl get nodes

Example

# kubectl get nodes
NAME                          STATUS   ROLES                  AGE     VERSION
rakuten-12.robinsystems.com   Ready    control-plane,master   3h14m   v1.23.8
rakuten-14.robinsystems.com   Ready    control-plane,master   3h13m   v1.23.8
rakuten-15.robinsystems.com   Ready    control-plane,master   116s    v1.23.8

4.9.4.2. Verify pods

After you add or replace a worker or master node, you must verify that all pods are in the Running status.

Run the following command to verify pods:

kubectl get pods -A

You can grep with the newly added node name.

Example

# kubectl get pods -A | grep rakuten-15

kube-system   etcd-rakuten-15.robinsystems.com                      1/1     Running   0               2m7s
kube-system   kube-apiserver-rakuten-15.robinsystems.com            1/1     Running   0               82s
kube-system   kube-controller-manager-rakuten-15.robinsystems.com   1/1     Running   0               2m12s
kube-system   kube-scheduler-rakuten-15.robinsystems.com            1/1     Running   0               52s

4.9.4.3. Verify Patroni cluster

You must ensure that Patroni cluster pods are in the Running status after replacing or adding a new worker or master node.

Run the following command to verify Patroni cluster:

kubectl get pods -A -o wide | grep patroni

Example

# kubectl get pods -A -o wide | grep patroni

robinio       robin-patroni-0                                       1/1     Running   0               137m    fd74:ca9b:3a09:868c:172:18:0:7548   rakuten-12.robinsystems.com   <none>           <none>
robinio       robin-patroni-1                                       1/1     Running   2 (138m ago)    3h12m   fd74:ca9b:3a09:868c:172:18:0:6f5a   rakuten-14.robinsystems.com   <none>           <none>
robinio       robin-patroni-2                                       1/1     Running   0               2m21s   fd74:ca9b:3a09:868c:172:18:0:4c82   rakuten-15.robinsystems.com   <none>           <none>
robinio       robin-patroni-postgres-operator-75fd786c4b-x4mwg      1/1     Running   0               78m     fd74:ca9b:3a09:868c:172:18:0:6f5b   rakuten-14.robinsystems.com   <none>           <none>

# robin host list

Id           | Hostname                    | Version  | Status | RPool   | Avail. Zone | LastOpr | Roles | Cores     | GPUs  | Mem            | HDD(#/Alloc/Total) | SSD(#/Alloc/Total) | Pod Usage | Joined Time
-------------+-----------------------------+----------+--------+---------+-------------+---------+-------+-----------+-------+----------------+--------------------+--------------------+-----------+----------------------
1663838470:1 | rakuten-12.robinsystems.com | 5.4.1-21 | Ready  | default | N/A         | ONLINE  | S,C   | 123/5/128 | 0/0/0 | 973G/33G/1007G | -/-/-              | 6/5G/17884G        | 72/28/100 | 21 Sep 2022 19:23:02
1663838470:2 | rakuten-14.robinsystems.com | 5.4.1-21 | Ready  | default | N/A         | ONLINE  | S,C   | 123/5/128 | 0/0/0 | 972G/35G/1007G | -/-/-              | 7/5G/20865G        | 70/30/100 | 21 Sep 2022 19:23:53
1663838470:5 | rakuten-15.robinsystems.com | 5.4.1-21 | Ready  | None    | N/A         | SYNCED  |       | 124/4/128 | 0/0/0 | 1002G/5G/1007G | -/-/-              | -/-/-              | 77/23/100 | 21 Sep 2022 22:30:54

* Note: all values indicated above in the format XX/XX/XX represent the Free/Allocated/Total values of the respective resource unless otherwise specified. In addition allocated values for compute resource such as cpu, memory and pod usage includes reserved values for the corresponding resource.

4.9.4.4. Verify ETCD cluster

You must ensure that ETCD cluster nodes are in the started status after replacing or adding a new worker or master node.

Run the following command to verify ETCD cluster status:

retcd get-members

Example

# retcd get-members
+------------------+---------+-----------------------------+-----------------------------------------------+-----------------------------------------------+------------+
|        ID        | STATUS  |            NAME             |                  PEER ADDRS                   |                 CLIENT ADDRS                  | IS LEARNER |
+------------------+---------+-----------------------------+-----------------------------------------------+-----------------------------------------------+------------+
|  3dcbe3353219099 | started | rakuten-15.robinsystems.com | https://[fd74:ca9b:3a09:868c:10:9:20:95]:2380 | https://[fd74:ca9b:3a09:868c:10:9:20:95]:2379 |      false |
| 40d4cb4906cb56b6 | started | rakuten-12.robinsystems.com | https://[fd74:ca9b:3a09:868c:10:9:20:92]:2380 | https://[fd74:ca9b:3a09:868c:10:9:20:92]:2379 |      false |
| d2a42873a4133282 | started | rakuten-14.robinsystems.com | https://[fd74:ca9b:3a09:868c:10:9:20:94]:2380 | https://[fd74:ca9b:3a09:868c:10:9:20:94]:2379 |      false |
+------------------+---------+-----------------------------+-----------------------------------------------+-----------------------------------------------+------------+

4.10. Decommissioning a node

When a server physically malfunctions or needs to be replaced, one has to remove the node from its respective Robin and Kubernetes clusters. Detailed below is a step-by-step walkthrough which showcases how to achieve this, whilst also ensuring your Robin cluster continues to function normally.

Setting a host to maintenance mode:

Before a host can be removed from the Robin cluster it is good practice to set it to maintenance mode. This disables all roles on the host for the time being and is important as it ensures no other applications or volumes can be provisioned on the host to be removed. This in turn makes the following steps of the walkthrough much easier to perform and results in a smoother removal of the node. In order to set a host to maintenance mode use the robin host set-maintenance command. More details about this command can be found here.

# robin host set-maintenance intel-1.robinsystems.com
Host intel-1.robinsystems.com set in maintenance mode

Removing roles from a host:

Before a host can be removed from the Robin cluster all the roles currently assigned to it must be removed. Detailed below are the different preconditions that need to be met before removing a particular role.

If the host in question has the Compute role assigned to it, all pods from this host will have to be relocated to other hosts in the cluster. This can be achieved by running the robin instance relocate command for all the Vnodes that are currently residing on the host. More details about this command can be found here. The reason for this is that the Compute role denotes that resources such as CPU and memory can be allocated from this host. As a result in order to remove the role no compute resources should be taken from the host. This is the resulting consequence of moving the respective pods over.

# robin instance list --host intel-1.robinsystems.com
ID | Owner/Tenant         | ResPool | Container    | Hostname                                    | IP Addresses   | Status | Autopilot | LastOpr | Host    | Engine | Cores | GPUs | Mem (GB) | Ctime
---+----------------------+---------+--------------+---------------------------------------------+----------------+--------+-----------+---------+---------+--------+-------+------+----------+----------------------
1  | robin/Administrators | default | a1.server.01 | a1-server-01.t001-u000003.svc.cluster.local | 172.21.17.3/16 | ONLINE | Enabled   | STARTED | intel-1 | docker | 1     | 0    | 0.38     | 27 Feb 2020 13:40:13

# robin instance relocate a1.server.01 --wait --yes
Job:   63 Name: VnodeDeploy          State: PROCESSED       Error: 0
Job:   63 Name: VnodeDeploy          State: WAITING         Error: 0
Job:   63 Name: VnodeDeploy          State: AGENT_WAIT      Error: 0
Job:   63 Name: VnodeDeploy          State: FINALIZED       Error: 0
Job:   63 Name: VnodeDeploy          State: COMPLETED       Error: 0

# robin instance list
ID | Owner/Tenant         | ResPool | Container    | Hostname                                    | IP Addresses   | Status | Autopilot | LastOpr | Host         | Engine | Cores | GPUs | Mem (GB) | Ctime
---+----------------------+---------+--------------+---------------------------------------------+----------------+--------+-----------+---------+--------------+--------+-------+------+----------+----------------------
1  | robin/Administrators | default | a1.server.01 | a1-server-01.t001-u000003.svc.cluster.local | 172.21.17.3/16 | ONLINE | Enabled   | STARTED | cscale-82-83 | docker | 1     | 0    | 0.38     | 27 Feb 2020 13:40:13

If the host in question has the Storage role assigned to it, all volumes that are currently allocated on drives which belong to that host must be evacuated to drives residing on other hosts. This can be achieved by running the robin drive evacuate command. More details about this command can be found here. The reason for this is that the Storage role denotes that the host is available to provide storage capacity for volumes that are created alongisde applications. As a result, in order to remove the role there must be no volume allocations tied with the host. This is the resulting consequence of evacuating all the volumes from each drive on the node.

# robin drive list --host intel-1.robinsystems.com
ID | WWN                                       | Host    | Path /dev/disk/by-id                          | Size(GB) | Movable | Type | Free/Max(GB) | Vols | Role    | Status | LastOpr
---+-------------------------------------------+---------+-----------------------------------------------+----------+---------+------+--------------+------+---------+--------+---------
3  | 0xQEMU_QEMU_HARDDISK_e54d6149-0a4e-48ce-b | intel-1 | scsi-0QEMU_QEMU_HARDDISK_e54d6149-0a4e-48ce-b | 100      | N       | HDD  | 63/77 (82%)  | 2/10 | Storage | ONLINE | READY
4  | 0xQEMU_QEMU_HARDDISK_89fc0488-2050-4f44-a | intel-1 | scsi-0QEMU_QEMU_HARDDISK_89fc0488-2050-4f44-a | 100      | N       | HDD  | 77/77 (100%) | 0/10 | Storage | ONLINE | READY

Only storage disks are shown. Issue `robin disk list --role all` to view all disks

# robin drive evacuate 0xQEMU_QEMU_HARDDISK_e54d6149-0a4e-48ce-b  --exclude-disks 0xQEMU_QEMU_HARDDISK_e54d6149-0a4e-48ce-b, 0xQEMU_QEMU_HARDDISK_89fc0488-2050-4f44-a --wait --yes
Job:   65 Name: DiskEvacuate         State: VALIDATED       Error: 0
Job:   65 Name: DiskEvacuate         State: WAITING         Error: 0
Job:   65 Name: DiskEvacuate         State: COMPLETED       Error: 0

# robin drive list --host intel-1.robinsystems.com
ID | WWN                                       | Host    | Path /dev/disk/by-id                          | Size(GB) | Movable | Type | Free/Max(GB) | Vols | Role    | Status | LastOpr
---+-------------------------------------------+---------+-----------------------------------------------+----------+---------+------+--------------+------+---------+--------+---------
3  | 0xQEMU_QEMU_HARDDISK_e54d6149-0a4e-48ce-b | intel-1 | scsi-0QEMU_QEMU_HARDDISK_e54d6149-0a4e-48ce-b | 100      | N       | HDD  | 77/77 (100%) | 0/10 | Storage | ONLINE | READY
4  | 0xQEMU_QEMU_HARDDISK_89fc0488-2050-4f44-a | intel-1 | scsi-0QEMU_QEMU_HARDDISK_89fc0488-2050-4f44-a | 100      | N       | HDD  | 77/77 (100%) | 0/10 | Storage | ONLINE | READY

If the host in question has the Manager role assigned to it, there are no explicit preconditions that need to be met. However we recommend that for high availability clusters that there are at least 2 nodes with the Manager role assigned to them at all times. This is because if the master Manager node does fail or is rebooted, a cluster failover can occur and the Robin cluster is not affected.

Once all the necessary conditions have been met, the roles can be removed as shown in the following example:

# robin host list
Id           | Hostname                      | Version    | Status | LastOpr | Resource Pool | Roles | Cores | GPUs | Mem(Free/Alloc/Total) | HDD(#/Alloc/Total) | SSD(#/Alloc/Total) | Instances | Joined Time
-------------+-------------------------------+------------+--------+---------+---------------+-------+-------+------+-----------------------+--------------------+--------------------+-----------+----------------------
1582820722:1 | cscale-82-81.robinsystems.com | 5.2.1-9769 | Ready  | ONLINE  | default       | M*    | 2/400 | 0/0  | 30G/1G/31G            | -/-/-              | -/-/-              | 0         | 27 Feb 2020 00:25:46
1582820722:2 | cscale-82-82.robinsystems.com | 5.2.1-9769 | Ready  | ONLINE  | default       | M     | 1/400 | 0/0  | 30G/1G/31G            | -/-/-              | -/-/-              | 0         | 27 Feb 2020 00:32:47
1582820722:3 | cscale-82-83.robinsystems.com | 5.2.1-9769 | Ready  | ONLINE  | default       | M     | 1/400 | 0/0  | 30G/1G/31G            | -/-/-              | -/-/-              | 0         | 27 Feb 2020 00:41:56
1582820722:4 | intel-1.robinsystems.com      | 5.2.1-9769 | Ready  | ONLINE  | default       | S,C   | 1/720 | 0/0  | 175G/12G/187G         | 2/-/200G           | -/-/-              | 0         | 27 Feb 2020 00:53:35

# robin host remove-role intel-1.robinsystems.com storage,compute --yes --wait
Job: 208  Name: HostRemoveRoles     State: VALIDATED       Error: 0
Job: 208  Name: HostRemoveRoles     State: WAITING         Error: 0
Job: 208  Name: HostRemoveRoles     State: COMPLETED       Error: 0

# robin host list
Id           | Hostname                      | Version    | Status | LastOpr | Resource Pool | Roles | Cores | GPUs | Mem(Free/Alloc/Total) | HDD(#/Alloc/Total) | SSD(#/Alloc/Total) | Instances | Joined Time
-------------+-------------------------------+------------+--------+---------+---------------+-------+-------+------+-----------------------+--------------------+--------------------+-----------+----------------------
1582820722:1 | cscale-82-81.robinsystems.com | 5.2.1-9769 | Ready  | ONLINE  | default       | M*    | 2/400 | 0/0  | 30G/1G/31G            | -/-/-              | -/-/-              | 0         | 27 Feb 2020 00:25:46
1582820722:2 | cscale-82-82.robinsystems.com | 5.2.1-9769 | Ready  | ONLINE  | default       | M     | 1/400 | 0/0  | 30G/1G/31G            | -/-/-              | -/-/-              | 0         | 27 Feb 2020 00:32:47
1582820722:3 | cscale-82-83.robinsystems.com | 5.2.1-9769 | Ready  | ONLINE  | default       | M,C   | 1/400 | 0/0  | 30G/1G/31G            | -/-/-              | -/-/-              | 0         | 27 Feb 2020 00:41:56
1582820722:4 | intel-1.robinsystems.com      | 5.2.1-9769 | Ready  | SYNCED  | default       |       | 1/720 | 0/0  | 187G/0.05G/187G       | -/-/-              | -/-/-              | 0         | 27 Feb 2020 00:53:35

Removing the host:

Once all the the roles are removed from a host, issue the following command to remove the host from a Robin cluster:

# robin host remove [<host>]
                    --force
                    --yes

host

FQDN of host

--force

Forcibly remove a host from a cluster. Required if removing a node that is down

--yes

Do not prompt the user for confirmation of removal

Example:

# robin host list
Id           | Hostname                      | Version    | Status | LastOpr | Resource Pool | Roles | Cores | GPUs | Mem(Free/Alloc/Total) | HDD(#/Alloc/Total) | SSD(#/Alloc/Total) | Instances | Joined Time
-------------+-------------------------------+------------+--------+---------+---------------+-------+-------+------+-----------------------+--------------------+--------------------+-----------+----------------------
1582820722:1 | cscale-82-81.robinsystems.com | 5.2.1-9769 | Ready  | ONLINE  | default       | M*    | 2/400 | 0/0  | 30G/1G/31G            | -/-/-              | -/-/-              | 0         | 27 Feb 2020 00:25:46
1582820722:2 | cscale-82-82.robinsystems.com | 5.2.1-9769 | Ready  | ONLINE  | default       | M     | 1/400 | 0/0  | 30G/1G/31G            | -/-/-              | -/-/-              | 0         | 27 Feb 2020 00:32:47
1582820722:3 | cscale-82-83.robinsystems.com | 5.2.1-9769 | Ready  | ONLINE  | default       | M     | 1/400 | 0/0  | 30G/1G/31G            | -/-/-              | -/-/-              | 0         | 27 Feb 2020 00:41:56
1582820722:4 | intel-1.robinsystems.com      | 5.2.1-9769 | Ready  | ONLINE  | default       | S,C   | 1/720 | 0/0  | 175G/12G/187G         | 2/-/200G           | -/-/-              | 0         | 27 Feb 2020 00:53:35

# kubectl get nodes
NAME           STATUS   ROLES    AGE   VERSION
cscale-82-81   Ready    master   21h   v1.16.3
cscale-82-82   Ready    master   21h   v1.16.3
cscale-82-83   Ready    master   21h   v1.16.3
intel-1        Ready    worker   21h   v1.16.3

# robin host remove intel-1.robinsystems.com --yes --wait
Job: 215  Name: HostRemove     State: VALIDATED       Error: 0
Job: 215  Name: HostRemove     State: WAITING         Error: 0
Job: 215  Name: HostRemove     State: COMPLETED       Error: 0

# robin host list
Id           | Hostname                      | Version    | Status | LastOpr | Resource Pool | Roles | Cores | GPUs | Mem(Free/Alloc/Total) | HDD(#/Alloc/Total) | SSD(#/Alloc/Total) | Instances | Joined Time
-------------+-------------------------------+------------+--------+---------+---------------+-------+-------+------+-----------------------+--------------------+--------------------+-----------+----------------------
1582820722:1 | cscale-82-81.robinsystems.com | 5.2.1-9769 | Ready  | ONLINE  | default       | M*    | 2/400 | 0/0  | 30G/1G/31G            | -/-/-              | -/-/-              | 0         | 27 Feb 2020 00:25:46
1582820722:2 | cscale-82-82.robinsystems.com | 5.2.1-9769 | Ready  | ONLINE  | default       | M     | 1/400 | 0/0  | 30G/1G/31G            | -/-/-              | -/-/-              | 0         | 27 Feb 2020 00:32:47
1582820722:3 | cscale-82-83.robinsystems.com | 5.2.1-9769 | Ready  | ONLINE  | default       | M     | 1/400 | 0/0  | 30G/1G/31G            | -/-/-              | -/-/-              | 0         | 27 Feb 2020 00:41:56

# kubectl get nodes
NAME           STATUS   ROLES    AGE   VERSION
cscale-82-81   Ready    master   21h   v1.16.3
cscale-82-82   Ready    master   21h   v1.16.3
cscale-82-83   Ready    master   21h   v1.16.3

4.11. Managing a cluster via the remote client

In addition to the Robin CLI, which is available on all hosts where Robin is installed, a remote client is shipped with each cluster that is deployed. This client mirrors the functionality of the native CLI with regards to the commands available and hence it provides the management capabilities that are described throughout this document. One advantage of utilizing this client is that it can be used to manage a multitude of Robin clusters via the concept of contexts. A context in this scenario refers to a Robin cluster and is identified by the server name or IP Address. In addition to this primary key, the following attributes can also be set within a context: the port values for various Robin services (including the Robin Server, File Server, Event Server, Watchdog Server, and Metrics Server) along with the logging level. The attributes are discussed in more detail in the following sections. After creating the appropriate context for a Robin cluster, one can set it to be the current context and communicate with the respective cluster. The commands which can be used to achieve this are described below.

The following commands are described in this section:

robin client add-context

Add a Robin cluster context

robin client list-contexts

List all registered Robin cluster contexts

robin client set-current

Set a Robin cluster context as the current context

robin client update-context

Update attributes for the current Robin cluster context

robin client delete-context

Delete a Robin cluster context

4.11.1. Downloading the Robin client

In order to download the Robin client from an existing Robin cluster, issue the following command:

# curl -k https://<master_ip>:<port>/api/v3/robin_server/download?file=robincli&os=<os> -o robin

master_ip

IP Address of the Master Node or VIP

port

Port number for the Robin Server

os

The operating system to download the client for. Supported operating systems include: Linux, MacOS.

Example:

# curl -k 'https://vnode42:29442/api/v3/robin_server/download?file=robincli&os=linux' -o robin
% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                Dload  Upload   Total   Spent    Left  Speed
100 10.1M  100 10.1M    0     0  1421k      0  0:00:07  0:00:07 --:--:-- 1483k
# ls -lart
-rw-r--r--    1 demo  staff    10655536 Mar 26 14:12 robin

4.11.2. Adding a Context

A context is a construct that can be used to define a Robin cluster in a manner that the remote client can understand. In order to add a context, issue the following command:

Note

If a context already exists with the server specified, that context will be updated with the values supplied.

# robin client add-context <server>
                           --port <port>
                           --file-port <file_port>
                           --event-port <event_port>
                           --watchdog-port <watch_port>
                           --metrics-port <metrics_port>
                           --log-level <log_level>
                           --product <product_type>
                           --set-current

server

FQDN/IP Address of the Master Node or VIP

--port <port>

Port number for the Robin Server. Default value is 29442

--file-port <file_port>

Port number for the File Server. Default value is 29445

--event-port <event_port>

Port number for the Event Server. Default value is 29449

--watchdog-port <watch_port>

Port number for the Watchdog Server. Default value is 29444

--metrics-port <metrics_port>

Port number for the Metrics Server. Default is 29446

--log-level <log_level>

Number indicating the verbosity of logs. Valid values are 10 (DEBUG), 20 (INFO), 40 (ERROR). Default value is 40.

--product <product_type>

Type of ROBIN installation. Valid choices are ‘platform’ or ‘storage’. Default value is ‘platform’.

--set-current

Set context to be created as the current

Note

If the target Robin cluster was deployed as a highly available cluster, ensure that the robincp_mode config attribute is enabled and set the port value for the Robin, file and event server to 29465. In addition the server parameter should be set to the value of the VIP in this case. Otherwise the remote client will not be fully operational.

Example:

# robin client add-context centos-60-214 --port 29442
Context robin-cluster-centos-60-214 created successfully

4.11.3. Listing all available contexts

In order to list all contexts that have already been registered with the client alongside additional details such as the port values specified or the log level, issue the following command:

# robin client list-contexts --full

--full

Show additional details about all registered contexts

Example:

# robin client list-contexts --full

   | Server                            | Port  | Version    | Tenant         | Last Login           | Tenants        | FPort | WPort | MPort | LogLevel
---+-----------------------------------+-------+------------+----------------+----------------------+----------------+-------+-------+-------+----------
   | master.robin-server.service.robin | 29442 | -          | -              | -                    |                | 29445 | 29444 | 29446 | ERR
   | centos-60-214                     | 29443 | -          | Administrators | -                    |                | 29445 | 29444 | 29446 | ERR
 * | 172.19.174.194                    | 29442 | 5.2.3-9842 | Administrators | 26 Mar 2020 16:10:58 | Administrators | 29445 | 29444 | 29446 | ERR

Note

The asterisk displayed above indicates the current context.

4.11.4. Setting the current context

In order to access a particular Robin cluster, its respective context needs to be set as the current context. To achieve this, issue the following command:

# robin client set-current <context>

context

The server attribute of the context to be set as current

Example:

# robin client set-current centos-60-214
Current context set to robin-cluster-centos-60-214

4.11.5. Updating the current context

In certain situations, such as a reinstallation, the attributes of a context might be altered whilst retaining the same server IP Address or hostname. As a result, the context which refers to this cluster will have to be updated. In order to do so, issue the following command:

Note

The below command only updates the current context.

# robin client update-context --port <port>
                              --file-port <file_port>
                              --event-port <event_port>
                              --watchdog-port <watch_port>
                              --metrics-port <metrics_port>
                              --log-level <log_level>

--port <port>

Updated port number for the Robin Server

--file-port <file_port>

Updated port number for the File Server

--event-port <event_port>

Updated port number for the Event Server

--watchdog-port <watch_port>

Updated port number for the Watchdog Server

--metrics-port <metrics_port>

Updated port number for the Metrics Server

--log-level <log_level>

Updated number indicating the verbosity of logs. Valid values are 10 (DEBUG), 20 (INFO), 40 (ERROR)

Example:

# robin client update-context --port 29942 --file-port 29445 --watchdog-port 29444 --metrics-port 29446
Updating attributes for context robin-cluster-centos-60-214
Server: centos-60-214
Context config updated for robin-cluster-centos-60-214

4.11.6. Deleting a context

In order to remove a registered context, issue the following command:

# robin client delete-context <context>

context

The server attribute of the context to be deleted

Example:

# robin client delete-context centos-60-214
Context centos-60-214 deleted

4.12. Managing Unused Images

Unused docker images tend to bloat and consume system storage. Whilst they can be deleted, manually locating and removing them becomes cumbersome. Robin facilitates the deletion of unused images (which are not in use by any containers) on all hosts (or a given subset) within the cluster. In addition, users can schedule a job to prune unnecessary images so as to mimic garbage collection for these images.

The following commands are described in this section:

robin host pruneimage

Prune unused images from hosts within the cluster

4.12.1. Pruning Images

In order to remove unused images on a host, issue the following command:

# robin host pruneimage [<hostnames>]
                        --skip-images-file <skip_images_path>
                        --all

hostnames

Comma seperated list of host FQDNs

--skip-images-file <skip_images_path>

Path to file containing a list of images that should not be pruned. Each image should be specified on a seperate line and in the following format: <image>:<tag>

--all

Prune unused images from all hosts. Should be specified if no hostname is given

Example:

# robin host pruneimage vnode36.robinsystems.com --wait
Job:  226 Name: HostPruneImagesMulti           State: PROCESSED       Error: 0
Job:  226 Name: HostPruneImagesMulti           State: COMPLETED       Error: 0

Removes unused images from a specified host.

End Point: /api/v3/robin_server/hosts/<hostname>

Method: PUT

URL Parameters: None

Data Parameters:

  • action: pruneimage - This mandatory field within the payload specifies that the image prune operation should be performed.

  • skip_images: <list_of_imgs> - Utilizing this parameter within the payload, by specifying a list of images in the format <image_name>:<tag>, results in the given images not being removed even if they are unused at the moment.

Port: RCM Port (default value is 29442)

Headers:

  • Authorization: <auth_token> : Authorization token to identify which user is sending the request. The token can be acquired from the login API.

Success Response Code: 202

Error Response Code: 500 (Internal Server Error), 404 (Not Found Error), 401 (Unauthorized Error), 400 (Invalid API Usage Error)

Example Response:

Output
{
   "jobid":1039
}

4.12.2. Enabling Pruning Schedule

In order to enable a schedule that removes all unused images via a CRON string, issue the following command:

# robin config update server image_prune_cron <cron_string>

cron_string

CRON string representing the time at which the schedule shouild run

Note

In order to set the schedule via a CRON string, it has to be enabled first via the following command: robin config update server enable_image_prune_schedule true

Example:

# robin config update server enable_image_prune_schedule true
The 'server' attribute 'enable_image_prune_schedule' has been updated

# robin config update server image_prune_cron "0 2 * * *"
The 'server' attribute 'image_prune_cron' has been updated

4.12.3. Disabling Pruning Schedule

In order to disable the schedule that removes all unused images, issue the following command:

# robin config update server enable_image_prune_schedule false

Example:

# robin config update server enable_image_prune_schedule false
The 'server' attribute 'enable_image_prune_schedule' has been updated