******************************** Sherlock - Troubleshooting Tool ******************************** Sherlock is a troubleshooting and self-diagnostic command-line tool (CLI) in Robin. It is designed to assist Robin administrators to identify and analyze any problems with Robin clusters. Using Sherlock, you as an administrator can diagnose cluster-wide problems or any information regarding specific applications, nodes, containers, volumes, devices, and so on. It provides details of problems affecting from applications to spindle. .. Note:: By default, Sherlock displays only unhealthy objects. Sherlock tool generates details for troubleshooting by querying a range of Robin APIs and direct database calls to generate cluster health reports. .. Note:: You can access the Sherlock tool only from the active Robin master-pod. On CNS clusters, you can access using the primary ``robinds`` Pod. =================== Sherlock Use Cases =================== You can use the Sherlock tool when you notice any problem with your Robin cluser to know details of the problem, plan maintenance, etc. **Analyze problem details** When there is an issue with the Robin cluster, Sherlock detects it automatically and displays all the impacted objects when you use the Sherlock tool. For example, if a disk goes down on the cluster, Sherlock displays all the impacted objects because of the disk(device) issue, such as volumes, Pods, applications, users, etc. Similarly, Sherlock detects the problems with other objects and displays details. This information helps you to detect the issue faster and fix it. **Plan maintenance activities** Sherlock also helps you to plan maintenance activities. For example, if you want to bring down any node for maintenance, you can run the Sherlock tool and check the impacted objects (Pods, volumes, applications, and users) for planning maintenance. =================================== Details you can View Using Sherlock =================================== Sherlock is useful as it allows you to map resources both top-down and bottom-up through the resource hierarchy. The Sherlock tool provides details of the following: - Application to Pod mappings - Source of computing resources for Pods - Volumes attached to Pods - Source of storage resources for Volume - Number of snapshots, replicas - Source of replication storage resources - Disks attached to hosts - Disks capacity and their consumption - Volumes that are hosted and Pods that owns the volumes - Robin cluster critical services status **For example:** Node → Disk Node → Pod → Application Disk → Volumes → Pods → Application Volumes → Pods → Application Volumes → File-object → Bundle → App Pods → Application Critical node services → Node → Pods → Application Application → Users/Tenant ================ Sherlock Report ================ A report generated by Sherlock using the ``sherlock`` command has the following sections: ============================================================= =========================================================================================================================================================================================================================== **SHOWING APPLICATIONS THAT NEED ATTENTION:** This section of the report displays the unhealthy applications running on your cluster (with -H healthy apps), where the Pods comprising the application are located and volumes, devices on which data is saved. **SHOWING UNHEALTHY VOLUMES THAT NEED ATTENTION:** This section of the report displays details of the unhealth Pods (with -H healthy Pods) running on your cluster and the volumes, devices that are being used for storing data. **SHOWING UNHEALTHY NODES THAT NEED ATTENTION:** This section of the report displays details of unhealthy volumes (with -H health volumes) running on your cluster and the devices that these volumes are using for storing data. **SHOWING UNHEALTHY NODES THAT NEED ATTENTION:** This section of the report displays unhealthy nodes (with -H healthy) of the nodes and services running on the nodes, Pods in the nodes, volumes, and devices. **SHOWING UNHEALTHY DEVICES THAT NEED ATTENTION:** This section of the report displays the status of unhealthy (with -H healthy) devices and impacted objects. ============================================================= =========================================================================================================================================================================================================================== ================ Access Sherlock ================ After you log into your Robin cluster you can access the Sherlock troubleshooting tool. The Sherlock tool is part of Robin CNS and Robin CNP. You do not need to install anything manually to start using it. The tool displays details of the following components that might need your attention in the form of a report: - Applications - Pods - Volumes - Nodes - Devices - Unavailable file collection - Unavailable bundle details .. Note:: If all components are fine, the tool displays all healthy or available messages. **Prerequisites** - You must access the Sherlock tool only from the active master node. - On CNS clusters, you must run the Sherlock commands from the ``primary robinds`` Pod. To access Sherlock, run the following command: .. code-block:: text # sherlock .. Note:: You can run the ``sherlock --help`` after accessing Sherlock to view commands options. **Example showing all healthy and available** .. code-block:: text # sherlock SHOWING APPLICATIONS THAT NEED ATTENTION: All apps are healthy SHOWING PODS THAT NEED ATTENTION: All pods are healthy SHOWING UNHEALTHY VOLUMES THAT NEED ATTENTION All volumes are healthy SHOWING UNHEALTHY NODES THAT NEED ATTENTION: All nodes are healthy SHOWING UNHEALTHY DEVICES THAT NEED ATTENTION: All devices are healthy SHOWING UNAVAILABLE FILE COLLECTIONS THAT NEED ATTENTION: All file collection are available SHOWING UNAVAILABLE BUNDLES THAT NEED ATTENTION: All bundles are available Only unhealthy objects are shown. To see everything re-run with -H|--healthy opt ion To see more details rerun with -V|--verbose option sherlock produced results in 155 milliseconds (Sat Sep 18 06:14:59 PM 2021). |-- 3 nodes, 26 disks, 3 vols, 3 snapshots, 0 apps, 0 pods, 1 file-collections, 2 bundles, 1 users and 1 tenants were analyzed **Example showing unhealthy app and Pods** .. code-block:: text # sherlock SHOWING APPLICATIONS THAT NEED ATTENTION: |-- robinte STATE: PLANNED Robin Systems 2/2 pods unhealthy KIND: ROB IN SHOWING USERS WHO ARE AFFECTED: |-- Robin Systems (Firstname: Robin LastName: Systems Email: None) | |-- APPS 1: robinte SHOWING PODS THAT NEED ATTENTION: o-- POD/VNODE ID 121: robinte.R1.01 INSTALLING/ONLINE 1 CPU, 50 MB MEM NODE : UP, RIO: UP |-- POD/VNODE ID 122: robinte.R2.01 INSTALLING/ONLINE 1 CPU, 50 MB MEM NODE : UP, RIO: UP SHOWING UNHEALTHY VOLUMES THAT NEED ATTENTION All volumes are healthy SHOWING UNHEALTHY NODES THAT NEED ATTENTION: All nodes are healthy SHOWING UNHEALTHY DEVICES THAT NEED ATTENTION: All devices are healthy SHOWING UNAVAILABLE FILE COLLECTIONS THAT NEED ATTENTION: All file collection are available SHOWING UNAVAILABLE BUNDLES THAT NEED ATTENTION: All bundles are available =========================== View Health of All objects =========================== You can view the health of all objects using the -H option. The -H option displays healthy objects along with unhealthy objects in the report. Sherlock displays health of all objects(Pods, volumes, applications, nodes, devices, file collection, bundle). To view health of all objects, run the following command: .. code-block:: text # sherlock -H .. Note:: You can use the ``-V`` option to view the details of all healthy and unhealthy objects. Also, you can the ``-H`` (healthy) and ``-V`` (verbose) options together with other commands as well. **Example with -H** .. code-block:: text # sherlock -H No matching apps found No matching pods found SHOWING HEALTH OF 3/3 VOLUMES IN THE CLUSTER |-- VOLID 1: file-collection-1631971248912.0798c2d5-332f-4c6f-96e6-8283a431851b, usage: 448 MB / 20 GB, 1 snapshots, resync progress: SYNCED, using 1 devices |-- VOLID 132: pvc-94229d46-e381-4e3c-99a1-ddfe389d7839 , usage: 352 MB / 5 GB, 1 snapshots, resync progress: SYNCED, using 1 devices |-- VOLID 131: pvc-a140c841-0e2a-4d91-be7c-c7c75b5756b1 , usage: 576 MB / 11 GB, 1 snapshots, resync progress: SYNCED, using 1 devices All volumes are healthy SHOWING HEALTH OF 3/3 NODES RUNNING IN THE CLUSTER |-- eqx01-flash16 ONLINE 0 errors, 0 warnings NODE: UP, AGENT: UP, IOMGR: UP, K8S Service(s): READY |-- eqx04-flash05 ONLINE 0 errors, 0 warnings NODE: UP, AGENT: UP, IOMGR: UP, K8S Service(s): READY |-- eqx01-flash15 ONLINE 0 errors, 0 warnings NODE: UP, AGENT: UP, IOMGR: UP, K8S Service(s): READY SHOWING HEALTH OF 26/26 DEVICES IN THE CLUSTER |-- /dev/sdi@eqx01-flash16 | 1/100 vols | 11/119194 slices | 1.8 TB free of 1.8 TB, NODE: ONLINE, RDVM: UP, DEV: READY | (WWN: 0x5000c5008c0fe71f PATH: /dev/disk/by-id/ata-ST2000NX0253_S46059W1) | |-- /dev/sde@eqx01-flash16 | 0/100 vols | 0/119194 slices | 1.8 TB free of 1.8 TB, NODE: ONLINE, RDVM: UP, DEV: READY | (WWN: 0x5000c5008c0feea2 PATH: /dev/disk/by-id/ata-ST2000NX0253_S46059R8) | |-- /dev/sde@eqx01-flash15 | 1/100 vols | 5/119194 slices | 1.8 TB free of 1.8 TB, NODE: ONLINE, RDVM: UP, DEV: READY | (WWN: 0x5000c5008c0db2c7 PATH: /dev/disk/by-id/ata-ST2000NX0253_S4605EN0) | |-- /dev/sdf@eqx01-flash15 | 0/100 vols | 0/119194 slices | 1.8 TB free of 1.8 TB, NODE: ONLINE, RDVM: UP, DEV: READY | (WWN: 0x5000c5008c0db9be PATH: /dev/disk/by-id/ata-ST2000NX0253_S4605E90) | |-- /dev/sdi@eqx01-flash15 | 0/100 vols | 0/119194 slices | 1.8 TB free of 1.8 TB, NODE: ONLINE, RDVM: UP, DEV: READY | (WWN: 0x5000c5008c0dbae3 PATH: /dev/disk/by-id/ata-ST2000NX0253_S4605DY2) | |-- /dev/sdg@eqx01-flash15 | 0/100 vols | 0/119194 slices | 1.8 TB free of 1.8 TB, NODE: ONLINE, RDVM: UP, DEV: READY | (WWN: 0x5000c5008c0ddd62 PATH: /dev/disk/by-id/ata-ST2000NX0253_S4605D33) | |-- /dev/sdh@eqx01-flash15 | 0/100 vols | 0/119194 slices | 1.8 TB free of 1.8 TB, NODE: ONLINE, RDVM: UP, DEV: READY | (WWN: 0x5000c5008c0df3ba PATH: /dev/disk/by-id/ata-ST2000NX0253_S4605CZ7) | |-- /dev/sdd@eqx01-flash15 | 0/100 vols | 0/119194 slices | 1.8 TB free of 1.8 TB, NODE: ONLINE, RDVM: UP, DEV: READY | (WWN: 0x5000c5008c101de8 PATH: /dev/disk/by-id/ata-ST2000NX0253_S4605TKT) | |-- /dev/sdb@eqx01-flash15 | 1/100 vols | 20/57194 slices | 893.8 GB free of 894.3 GB, NODE: ONLINE, RDVM: UP, DEV: READY | (WWN: 0x500a075109604998 PATH: /dev/disk/by-id/ata-Micron_M500_MTFDDAK960MAV_140109604998) | |-- /dev/sdb@eqx04-flash05 | 0/100 vols | 0/57194 slices | 894.3 GB free of 894.3 GB, NODE: ONLINE, RDVM: UP, DEV: READY | (WWN: 0x500a07510ec79d1f PATH: /dev/disk/by-id/ata-Crucial_CT960M500SSD1_14330EC79D1F) | |-- /dev/sdh@eqx01-flash16 | 0/100 vols | 0/119194 slices | 1.8 TB free of 1.8 TB, NODE: ONLINE, RDVM: UP, DEV: READY | (WWN: 0x5000c5008c0d9e30 PATH: /dev/disk/by-id/ata-ST2000NX0253_S4605EZW) | |-- /dev/sdf@eqx01-flash16 | 0/100 vols | 0/119194 slices | 1.8 TB free of 1.8 TB, NODE: ONLINE, RDVM: UP, DEV: READY | (WWN: 0x5000c5008c0dc21f PATH: /dev/disk/by-id/ata-ST2000NX0253_S4605DSS) | |-- /dev/sdb@eqx01-flash16 | 0/100 vols | 0/119194 slices | 1.8 TB free of 1.8 TB, NODE: ONLINE, RDVM: UP, DEV: READY | (WWN: 0x5000c5008c0dd039 PATH: /dev/disk/by-id/ata-ST2000NX0253_S4605DQ3) | |-- /dev/sdg@eqx01-flash16 | 0/100 vols | 0/119194 slices | 1.8 TB free of 1.8 TB, NODE: ONLINE, RDVM: UP, DEV: READY | (WWN: 0x5000c5008c0dee42 PATH: /dev/disk/by-id/ata-ST2000NX0253_S4605DCS) | |-- /dev/sdd@eqx01-flash16 | 0/100 vols | 0/119194 slices | 1.8 TB free of 1.8 TB, NODE: ONLINE, RDVM: UP, DEV: READY | (WWN: 0x5000c5008c0df26c PATH: /dev/disk/by-id/ata-ST2000NX0253_S4605DB2) | |-- /dev/sdc@eqx04-flash05 | 0/100 vols | 0/57194 slices | 894.3 GB free of 894.3 GB, NODE: ONLINE, RDVM: UP, DEV: READY | (WWN: 0x500a07510ee9a052 PATH: /dev/disk/by-id/ata-Crucial_CT960M500SSD1_14280EE9A052) | SHOWING 1 FILE COLLECTIONS IN THE CLUSTER |-- file-collection-1631971248912 Online 0 errors 0 warnings SHOWING 1 BUNDLES IN THE CLUSTER |-- wordpress ONLINE 0 errors 0 warnings To see more details rerun with -V|--verbose option sherlock produced results in 200 milliseconds (Sat Sep 18 11:51:21 PM 2021). |-- 3 nodes, 26 disks, 3 vols, 3 snapshots, 0 apps, 0 pods, 1 file-collections, 1 bundles, 3 users and 3 tenants were analyzed **Example with -H and -V** .. code-block:: text # sherlock -H -V No matching apps found No matching pods found SHOWING HEALTH OF 3/3 VOLUMES IN THE CLUSTER |-- VOLID 1: file-collection-1631971248912.0798c2d5-332f-4c6f-96e6-8283a431851b, usage: 448 MB / 20 GB, 1 snapshots, resync progress: SYNCED, using 1 devices | |-- DEVID 1: /dev/sdb on eqx01-flash15 using 448 MB/894.3 GB capacity, 14/20 slices, 14 segs, segspernap=1 RDVM: UP, DEV: READY | | (WWN: 0x500a075109604998 PATH: /dev/disk/by-id/ata-Micron_M500_MTFDDAK960MAV_140109604998) | | | |-- SNAPSHOTS: 1 CREATED DEV OWN CLONES STATE SIZE | | |-- SNAPID 1: 1969/12/31 16:00:00 14 14 0 READY 448 MB | | | | |-- VOLID 132: pvc-94229d46-e381-4e3c-99a1-ddfe389d7839 , usage: 352 MB / 5 GB, 1 snapshots, resync progress: SYNCED, using 1 devices | |-- DEVID 2: /dev/sde on eqx01-flash15 using 352 MB/1.8 TB capacity, 11/5 slices, 11 segs, segspernap=3 RDVM: UP, DEV: READY | | (WWN: 0x5000c5008c0db2c7 PATH: /dev/disk/by-id/ata-ST2000NX0253_S4605EN0) | | | |-- SNAPSHOTS: 1 CREATED DEV OWN CLONES STATE SIZE | | |-- SNAPID 1: 1969/12/31 16:00:00 11 11 0 READY 352 MB | | | | |-- VOLID 131: pvc-a140c841-0e2a-4d91-be7c-c7c75b5756b1 , usage: 576 MB / 11 GB, 1 snapshots, resync progress: SYNCED, using 1 devices | |-- DEVID 11: /dev/sdi on eqx01-flash16 using 576 MB/1.8 TB capacity, 18/11 slices, 18 segs, segspernap=2 RDVM: UP, DEV: READY | | (WWN: 0x5000c5008c0fe71f PATH: /dev/disk/by-id/ata-ST2000NX0253_S46059W1) | | | |-- SNAPSHOTS: 1 CREATED DEV OWN CLONES STATE SIZE | | |-- SNAPID 1: 1969/12/31 16:00:00 18 18 0 READY 576 MB | | | | All volumes are healthy |-- eqx01-flash16 ONLINE 0 errors, 0 warnings NODE: UP, AGENT: UP, IOMGR: UP, K8S Service(s): READY ============================================================================================================================================================================================= 0 PODS ARE RUNNING ON THIS NODE 9 DEVICES ARE ATTACHED TO THIS NODE |-- DEVID 11: /dev/sdi READY 1.8 TB free=1.8 TB (100%) 1/100 vols, 11/119194 slices, 18 segs | |-- VOLID 131: pvc-a140c841-0e2a-4d91-be7c-c7c75b5756b1 576 MB nslices=11 nsnaps=1 nsegs=18 nsegs_per_snap=2 | |-- DEVID 12: /dev/sde READY 1.8 TB free=1.8 TB (100%) 0/100 vols, 0/119194 slices, 0 segs |-- DEVID 0: /dev/sdc INIT 14.9 GB free=14.9 GB (100%) 0/100 vols, 0/0 slices, 0 segs |-- DEVID 9: /dev/sdh READY 1.8 TB free=1.8 TB (100%) 0/100 vols, 0/119194 slices, 0 segs |-- DEVID 8: /dev/sdf READY 1.8 TB free=1.8 TB (100%) 0/100 vols, 0/119194 slices, 0 segs |-- DEVID 10: /dev/sdb READY 1.8 TB free=1.8 TB (100%) 0/100 vols, 0/119194 slices, 0 segs |-- DEVID 14: /dev/sdg READY 1.8 TB free=1.8 TB (100%) 0/100 vols, 0/119194 slices, 0 segs |-- DEVID 13: /dev/sdd READY 1.8 TB free=1.8 TB (100%) 0/100 vols, 0/119194 slices, 0 segs |-- DEVID 0: /dev/sda INIT 1.8 TB free=1.8 TB (100%) 0/100 vols, 0/0 slices, 0 segs |-- eqx04-flash05 ONLINE 0 errors, 0 warnings NODE: UP, AGENT: UP, IOMGR: UP, K8S Service(s): READY ============================================================================================================================================================================================= 0 PODS ARE RUNNING ON THIS NODE 8 DEVICES ARE ATTACHED TO THIS NODE |-- DEVID 16: /dev/sdb READY 894.3 GB free=894.3 GB (100%) 0/100 vols, 0/57194 slices, 0 segs |-- DEVID 0: /dev/sda INIT 894.3 GB free=894.3 GB (100%) 0/100 vols, 0/0 slices, 0 segs |-- DEVID 0: /dev/sdd INIT 894.3 GB free=894.3 GB (100%) 0/100 vols, 0/0 slices, 0 segs |-- DEVID 15: /dev/sdc READY 894.3 GB free=894.3 GB (100%) 0/100 vols, 0/57194 slices, 0 segs |-- DEVID 0: /dev/sde INIT 59.6 GB free=59.6 GB (100%) 0/100 vols, 0/0 slices, 0 segs |-- DEVID 0: /dev/dm-1 INIT 17.4 GB free=17.4 GB (100%) 0/100 vols, 0/0 slices, 0 segs |-- DEVID 0: /dev/dm-2 INIT 35.7 GB free=35.7 GB (100%) 0/100 vols, 0/0 slices, 0 segs |-- DEVID 0: /dev/dm-0 INIT 6.0 GB free=6.0 GB (100%) 0/100 vols, 0/0 slices, 0 segs |-- eqx01-flash15 ONLINE 0 errors, 0 warnings NODE: UP, AGENT: UP, IOMGR: UP, K8S Service(s): READY ============================================================================================================================================================================================= 0 PODS ARE RUNNING ON THIS NODE 9 DEVICES ARE ATTACHED TO THIS NODE |-- DEVID 0: /dev/sda INIT 894.3 GB free=894.3 GB (100%) 0/100 vols, 0/0 slices, 0 segs |-- DEVID 0: /dev/sdc INIT 14.9 GB free=14.9 GB (100%) 0/100 vols, 0/0 slices, 0 segs |-- DEVID 2: /dev/sde READY 1.8 TB free=1.8 TB (100%) 1/100 vols, 5/119194 slices, 11 segs | |-- VOLID 132: pvc-94229d46-e381-4e3c-99a1-ddfe389d7839 352 MB nslices=5 nsnaps=1 nsegs=11 nsegs_per_snap=3 | |-- DEVID 3: /dev/sdf READY 1.8 TB free=1.8 TB (100%) 0/100 vols, 0/119194 slices, 0 segs |-- DEVID 7: /dev/sdi READY 1.8 TB free=1.8 TB (100%) 0/100 vols, 0/119194 slices, 0 segs |-- DEVID 5: /dev/sdg READY 1.8 TB free=1.8 TB (100%) 0/100 vols, 0/119194 slices, 0 segs |-- DEVID 6: /dev/sdh READY 1.8 TB free=1.8 TB (100%) 0/100 vols, 0/119194 slices, 0 segs |-- DEVID 4: /dev/sdd READY 1.8 TB free=1.8 TB (100%) 0/100 vols, 0/119194 slices, 0 segs |-- DEVID 1: /dev/sdb READY 894.3 GB free=893.8 GB (100%) 1/100 vols, 20/57194 slices, 14 segs | |-- VOLID 1: file-collection-1631971248912.0798c2d5-332f-4c6f-96e6-8283a431851b 448 MB nslices=20 nsnaps=1 nsegs=14 nsegs_per_snap=1 | DEVICE /dev/sdi on eqx01-flash16 1/100 vols | 11/119194 slices | 1.8 TB free of 1.8 TB NODE: ONLINE, RDVM: UP, DEV: READY ============================================================================================================================================================================================= |==> WWN: 0x5000c5008c0fe71f | PATH: /dev/disk/by-id/ata-ST2000NX0253_S46059W1 |-- VOL: 131 pvc-a140c841-0e2a-4d91-be7c-c7c75b5756b1 576 MB nslices=11 nsegs=18 (2 ) nsnaps=1 DEVICE /dev/sde on eqx01-flash16 0/100 vols | 0/119194 slices | 1.8 TB free of 1.8 TB NODE: ONLINE, RDVM: UP, DEV: READY ============================================================================================================================================================================================= |==> WWN: 0x5000c5008c0feea2 | PATH: /dev/disk/by-id/ata-ST2000NX0253_S46059R8 DEVICE /dev/sde on eqx01-flash15 1/100 vols | 5/119194 slices | 1.8 TB free of 1.8 TB NODE: ONLINE, RDVM: UP, DEV: READY ============================================================================================================================================================================================= |==> WWN: 0x5000c5008c0db2c7 | PATH: /dev/disk/by-id/ata-ST2000NX0253_S4605EN0 |-- VOL: 132 pvc-94229d46-e381-4e3c-99a1-ddfe389d7839 352 MB nslices=5 nsegs=11 (3 ) nsnaps=1 DEVICE /dev/sdf on eqx01-flash15 0/100 vols | 0/119194 slices | 1.8 TB free of 1.8 TB NODE: ONLINE, RDVM: UP, DEV: READY ============================================================================================================================================================================================= |==> WWN: 0x5000c5008c0db9be | PATH: /dev/disk/by-id/ata-ST2000NX0253_S4605E90 DEVICE /dev/sdi on eqx01-flash15 0/100 vols | 0/119194 slices | 1.8 TB free of 1.8 TB NODE: ONLINE, RDVM: UP, DEV: READY ============================================================================================================================================================================================= |==> WWN: 0x5000c5008c0dbae3 | PATH: /dev/disk/by-id/ata-ST2000NX0253_S4605DY2 DEVICE /dev/sdg on eqx01-flash15 0/100 vols | 0/119194 slices | 1.8 TB free of 1.8 TB NODE: ONLINE, RDVM: UP, DEV: READY ============================================================================================================================================================================================= |==> WWN: 0x5000c5008c0ddd62 | PATH: /dev/disk/by-id/ata-ST2000NX0253_S4605D33 DEVICE /dev/sdh on eqx01-flash15 0/100 vols | 0/119194 slices | 1.8 TB free of 1.8 TB NODE: ONLINE, RDVM: UP, DEV: READY ============================================================================================================================================================================================= |==> WWN: 0x5000c5008c0df3ba | PATH: /dev/disk/by-id/ata-ST2000NX0253_S4605CZ7 DEVICE /dev/sdd on eqx01-flash15 0/100 vols | 0/119194 slices | 1.8 TB free of 1.8 TB NODE: ONLINE, RDVM: UP, DEV: READY ============================================================================================================================================================================================= |==> WWN: 0x5000c5008c101de8 | PATH: /dev/disk/by-id/ata-ST2000NX0253_S4605TKT DEVICE /dev/sdb on eqx01-flash15 1/100 vols | 20/57194 slices | 893.8 GB free of 894.3 GB NODE: ONLINE, RDVM: UP, DEV: READY ============================================================================================================================================================================================= |==> WWN: 0x500a075109604998 | PATH: /dev/disk/by-id/ata-Micron_M500_MTFDDAK960MAV_140109604998 |-- VOL: 1 file-collection-1631971248912.0798c2d5-332f-4c6f-96e6-8283a431851b 448 MB nslices=20 nsegs=14 (1 ) nsnaps=1 DEVICE /dev/sdb on eqx04-flash05 0/100 vols | 0/57194 slices | 894.3 GB free of 894.3 GB NODE: ONLINE, RDVM: UP, DEV: READY ============================================================================================================================================================================================= |==> WWN: 0x500a07510ec79d1f | PATH: /dev/disk/by-id/ata-Crucial_CT960M500SSD1_14330EC79D1F DEVICE /dev/sdh on eqx01-flash16 0/100 vols | 0/119194 slices | 1.8 TB free of 1.8 TB NODE: ONLINE, RDVM: UP, DEV: READY ============================================================================================================================================================================================= |==> WWN: 0x5000c5008c0d9e30 | PATH: /dev/disk/by-id/ata-ST2000NX0253_S4605EZW DEVICE /dev/sdf on eqx01-flash16 0/100 vols | 0/119194 slices | 1.8 TB free of 1.8 TB NODE: ONLINE, RDVM: UP, DEV: READY ============================================================================================================================================================================================= |==> WWN: 0x5000c5008c0dc21f | PATH: /dev/disk/by-id/ata-ST2000NX0253_S4605DSS DEVICE /dev/sdb on eqx01-flash16 0/100 vols | 0/119194 slices | 1.8 TB free of 1.8 TB NODE: ONLINE, RDVM: UP, DEV: READY ============================================================================================================================================================================================= |==> WWN: 0x5000c5008c0dd039 | PATH: /dev/disk/by-id/ata-ST2000NX0253_S4605DQ3 DEVICE /dev/sdg on eqx01-flash16 0/100 vols | 0/119194 slices | 1.8 TB free of 1.8 TB NODE: ONLINE, RDVM: UP, DEV: READY ============================================================================================================================================================================================= |==> WWN: 0x5000c5008c0dee42 | PATH: /dev/disk/by-id/ata-ST2000NX0253_S4605DCS DEVICE /dev/sdd on eqx01-flash16 0/100 vols | 0/119194 slices | 1.8 TB free of 1.8 TB NODE: ONLINE, RDVM: UP, DEV: READY ============================================================================================================================================================================================= |==> WWN: 0x5000c5008c0df26c | PATH: /dev/disk/by-id/ata-ST2000NX0253_S4605DB2 DEVICE /dev/sdc on eqx04-flash05 0/100 vols | 0/57194 slices | 894.3 GB free of 894.3 GB NODE: ONLINE, RDVM: UP, DEV: READY ============================================================================================================================================================================================= |==> WWN: 0x500a07510ee9a052 | PATH: /dev/disk/by-id/ata-Crucial_CT960M500SSD1_14280EE9A052 SHOWING 1 FILE COLLECTIONS IN THE CLUSTER SHOWING 1 BUNDLES IN THE CLUSTER sherlock produced results in 141 milliseconds (Sat Sep 18 11:45:50 PM 2021). |-- 3 nodes, 26 disks, 3 vols, 3 snapshots, 0 apps, 0 pods, 1 file-collections, 1 bundles, 3 users and 3 tenants were analyzed ============================== View Sherlock Command Options ============================== To view sherlock command options, run the following command: .. code-block:: text # sherlock --help **Resource Inspection Options** Using the **Resource Inspection Command Options**, you can provide comma-separated values of resources(objects) names. It enables you to view details of multiple objects at a time. ================================ ============================================== ``-a`` | ``--app NAME,...`` Displays applications information. ``-n`` | ``--node NAME,...`` Displays nodes information. ``-p`` | ``--pod NAME,...`` Displays Pods information. ``-v`` | ``--vol NAME,...`` Displays volumes information. ``-d`` | ``--dev NAME,...`` Displays devices information. ================================ ============================================== **Advisory Rebalancing Options** You can use the rebalancing command options to known the disks that are being overutilized or underutilized. Based on the report advice, you can make the adjustments ======================================= =============================================================== ``-D`` | ``--devs-needing-rebalance`` Displays information about devices that need rebalancing. ``-L`` | ``--vols-needing-rebalance`` Displays information about volumes that need rebalancing ``-Y`` | ``--dev-rebalance-advice DEV`` Provides advice on device rebalancing ``-X`` | ``--vol-rebalance-advice VOL`` Provides advice on volume rebalancing. ======================================= =============================================================== **Behavior Controlling Options** ================================ ==================================================================================================================================================================================== ``-U`` | ``--strict`` Mark resources that are not fully online as unhealthy. This option displays the resources (objects) that are partially healthy as unhelathy. ``-C`` | ``--cache`` Build and use cache to speed up queries. It caches resources once and use the same cache for subsequent queries. Use when you run Sherlock repeatedly run with different options. ``-H`` | ``--healthy`` Also show healthy resources. Displays healthy resources (objects) along with unhealthy objects. ``-V`` | ``--verbose`` Displays detailed report. ``-M`` | ``--html`` Print output in HTML format. Use this along with ``--outfile`` and provide a path to save an HTML file. You can open the HTML file in a browser. ``-O`` | ``--outfile`` Print output to specified file. Provide any file path to save a file. For example: ``/tmp/sherlock-output``. ``-K`` | ``--no-skip`` Don't skip unimportant resources to minimize output. ``-J`` | ``--scan-joblogs`` Scan job logs for errors. ``-S`` | ``--mon SECS`` Monitor the resource metrics for every interval. Use this option along with ``--app``, ``--pod`` or ``--vol``. ``--start TIME`` Start scanning jobs starting at this date/time (default: 72 hrs ago). ``--end TIME`` End scanning jobs at this date/time (default: now). ``--server PORT`` Run in server mode so you can operate this from a web brower. ``--prom`` Run in server mode to serve metrics in Prometheus format. ================================ ==================================================================================================================================================================================== **Device Options** ============================== ======================================================================== ``--dev all`` Displays information about all devices ``--dev full`` Displays information devices that are near full ``--dev NODENAME:,...`` Displays information devices of ``--dev wwn,...`` Displays information for devices with matching WWNs ``--dev NODENAME:DEVPATH,...`` Displays information for specific device ============================== ======================================================================== ================================== Access Sherlock On Web Server ================================== You can access the Sherlock tool on a web browser. You can provide an available port number between 1-65535. To set a port number to access the Sherlock tool for your cluster, run the following command: .. code-block:: text # sherlock --server **Example** .. code-block:: text # sherlock --server 45536 running the read_config now Running in server mode. Point your web browser to the following address: https://eqx01-flash15:45536 ================================== Check Application Health ================================== Using Sherlock you can check the health of deployed applications. The Sherlock command output provides the following details about an application: - Volumes or devices on which the application data is stored - Pods in which the application is running - Node on which the Pod is existing - Running application details - Failed jobs on the specific application **Prerequisite** You must have the application name for which you want to check the health. You can run the ``sherlock -H`` command to know the application name. To check the application health, run the following command: .. code-block:: text sherlock --app You can optionally use the ``-V`` option to view the details. **Example** .. code-block:: text sherlock --app mysql-test -V -H APPNAME: mysql-test STATE: ONLINE Robin Systems 1/1 vnodes healthy ============================================================================================================================================================================================================================================== APP HAS 1 VNODES: VNODEID 2: mysql-test.mysql.01 on centos-60-181 INST: ONLINE/INST: STARTED, NODE: ONLINE, RIO: UP |-- VOLID 4: mysql-test.mysql.01.data.1.3daab239-4327-4f00-873d-ffda3c9575f2 1 GB | |-- DEVID : /dev/sdd segs=centos-60-181 slices=14 rawspace=1 448 MB |-- VOLID 5: mysql-test.mysql.01.root_fs.1.58523556-a318-483c-9f8c-d2cd98ad6a32 10 GB | |-- DEVID : /dev/sdd segs=centos-60-181 slices=10 rawspace=10 320 MB | APP IS RUNNING ON THE FOLLOWING 1 NODES: |-- centos-60-181 RIO: UP | |-- mysql-test.mysql.01 ONLINE/STARTED | APP IS STORING DATA ON THE FOLLOWING 1 DEVICES: |-- DEVID 6: /dev/sdd on centos-60-181 2 vols | |-- VOLID 4: mysql-test.mysql.01.data.1.3daab239-4327-4f00-873d-ffda3c9575f2 448 MB nslices=1 nsegs=14 nsnaps=3 segspersnap=5 | |-- VOLID 5: mysql-test.mysql.01.root_fs.1.58523556-a318-483c-9f8c-d2cd98ad6a32 320 MB nslices=10 nsegs=10 nsnaps=3 segspersnap=1 | THERE ARE 23 FAILED JOBS TO INSPECT BETWEEN Fri May 3 01:22:23 AM 2019 - Fri May 10 01:22:23 AM 2019 |-- mysql-test.mysql.01 | |-- VnodeDelete jobid=98 state=10 error=1 start=Thu May 9 00:23:31 2019 end=Thu May 9 00:23:32 2019 | | predestroy vnodehook cmd bash /var/lib/robin/.file_object_cache/64f1ef8529796f8199a63eaf2e65365f/scripts/vnode_sample failed to execute | |-- VnodeDelete jobid=88 state=10 error=1 start=Thu May 9 00:20:37 2019 end=Thu May 9 00:20:43 2019 | | postdestroy vnodehook cmd bash /var/lib/robin/.file_object_cache/64f1ef8529796f8199a63eaf2e65365f/scripts/vnode_sample failed to execute | |-- mysql-test | |-- ApplicationDelete jobid=97 state=10 error=1 start=Thu May 9 00:23:31 2019 end=Thu May 9 00:23:32 2019 | | Job failed. One or more child jobs reported errors. Error: 'predestroy vnodehook cmd bash /var/lib/robin/.file_object_cache/64f1ef8529796f8199a63eaf2e65365f/scripts/vnode_sample failed to | | execute' | |-- ApplicationDelete jobid=87 state=10 error=1 start=Thu May 9 00:20:37 2019 end=Thu May 9 00:20:43 2019 | | Job failed. One or more child jobs reported errors. Error: 'postdestroy vnodehook cmd bash /var/lib/robin/.file_object_cache/64f1ef8529796f8199a63eaf2e65365f/scripts/vnode_sample failed to | | execute' | |-- ApplicationDelete jobid=92 state=10 error=1 start=Thu May 9 00:22:17 2019 end=Thu May 9 00:22:18 2019 | | Job failed. One or more child jobs reported errors. Error: 'predestroy vnodehook cmd bash /var/lib/robin/.file_object_cache/64f1ef8529796f8199a63eaf2e65365f/scripts/vnode_sample failed to | | execute' | |-- ApplicationDelete jobid=95 state=10 error=1 start=Thu May 9 00:22:59 2019 end=Thu May 9 00:23:00 2019 | | Job failed. One or more child jobs reported errors. Error: 'predestroy vnodehook cmd bash /var/lib/robin/.file_object_cache/64f1ef8529796f8199a63eaf2e65365f/scripts/vnode_sample failed to | | execute' | |-- mysql-test1 | |-- ApplicationCreate jobid=129 state=10 error=1 start=Thu May 9 03:50:54 2019 end=Thu May 9 03:50:54 2019 | | Invalid Zone Id and/or Bundle Id: 1/2 | |-- ApplicationCreate jobid=128 state=10 error=1 start=Thu May 9 03:50:08 2019 end=Thu May 9 03:50:08 2019 | | Invalid Zone Id and/or Bundle Id: 1/2 | sherlock produced results in 90 milliseconds (Fri May 10 01:22:23 AM 2019). |-- 3 nodes, 12 disks, 3 vols, 7 snapshots, 1 apps, 1 vnodes, 2 users and 1 tenants were analyzed ================== Check Node Health ================== When you check health of a node, you can find details about all the objects (Applications, Pods, voulmes, devices, file collection, and bundles) under the node. To check the health of a node, complete the following steps: 1 To get the list of nodes for selecting a desired node for health check, run the following command: .. code-block:: # sherlock -H -K 2. To know the health of a node and its details, run the following command: .. code-block:: # sherlock --node You can optionally use the ``-V`` option to view the details. **Example** .. code-block:: # sherlock -H -K No matching apps found No matching pods found SHOWING HEALTH OF 3/3 VOLUMES IN THE CLUSTER |-- VOLID 1: file-collection-1632045271349.5ff1f19f-937f-4ec1-a595-9d9df9d11d44, usage: 448 MB / 20 GB, 1 snapshots, resync progress: SYNCED, using 1 devices |-- VOLID 163: pvc-66646581-0210-46e2-b945-9ea880be38d7 , usage: 352 MB / 5 GB, 1 snapshots, resync progress: SYNCED, using 1 devices |-- VOLID 162: pvc-eb63979d-720e-41c9-808f-145306dc1259 , usage: 576 MB / 11 GB, 1 snapshots, resync progress: SYNCED, using 1 devices All volumes are healthy SHOWING HEALTH OF 3/3 NODES RUNNING IN THE CLUSTER |-- eqx04-flash05 ONLINE 0 errors, 0 warnings NODE: UP, AGENT: UP, IOMGR: UP, K8S Service(s): READY |-- eqx01-flash16 ONLINE 0 errors, 0 warnings NODE: UP, AGENT: UP, IOMGR: UP, K8S Service(s): READY |-- eqx01-flash15 ONLINE 0 errors, 0 warnings NODE: UP, AGENT: UP, IOMGR: UP, K8S Service(s): READY SHOWING HEALTH OF 26/26 DEVICES IN THE CLUSTER |-- /dev/sdh@eqx01-flash16 | 1/100 vols | 11/119194 slices | 1.8 TB free of 1.8 TB, NODE: ONLINE, RDVM: UP, DEV: READY | (WWN: 0x5000c5008c0d9e30 PATH: /dev/disk/by-id/ata-ST2000NX0253_S4605EZW) | |-- /dev/sde@eqx01-flash15 | 0/100 vols | 0/119194 slices | 1.8 TB free of 1.8 TB, NODE: ONLINE, RDVM: UP, DEV: READY | (WWN: 0x5000c5008c0db2c7 PATH: /dev/disk/by-id/ata-ST2000NX0253_S4605EN0) | |-- /dev/sdf@eqx01-flash15 | 0/100 vols | 0/119194 slices | 1.8 TB free of 1.8 TB, NODE: ONLINE, RDVM: UP, DEV: READY | (WWN: 0x5000c5008c0db9be PATH: /dev/disk/by-id/ata-ST2000NX0253_S4605E90) | |-- /dev/sdi@eqx01-flash15 | 0/100 vols | 0/119194 slices | 1.8 TB free of 1.8 TB, NODE: ONLINE, RDVM: UP, DEV: READY | (WWN: 0x5000c5008c0dbae3 PATH: /dev/disk/by-id/ata-ST2000NX0253_S4605DY2) | |-- /dev/sdg@eqx01-flash15 | 0/100 vols | 0/119194 slices | 1.8 TB free of 1.8 TB, NODE: ONLINE, RDVM: UP, DEV: READY | (WWN: 0x5000c5008c0ddd62 PATH: /dev/disk/by-id/ata-ST2000NX0253_S4605D33) | |-- /dev/sdh@eqx01-flash15 | 0/100 vols | 0/119194 slices | 1.8 TB free of 1.8 TB, NODE: ONLINE, RDVM: UP, DEV: READY | (WWN: 0x5000c5008c0df3ba PATH: /dev/disk/by-id/ata-ST2000NX0253_S4605CZ7) | |-- /dev/sdd@eqx01-flash15 | 0/100 vols | 0/119194 slices | 1.8 TB free of 1.8 TB, NODE: ONLINE, RDVM: UP, DEV: READY | (WWN: 0x5000c5008c101de8 PATH: /dev/disk/by-id/ata-ST2000NX0253_S4605TKT) | |-- /dev/sdb@eqx01-flash15 | 1/100 vols | 20/57194 slices | 893.8 GB free of 894.3 GB, NODE: ONLINE, RDVM: UP, DEV: READY | (WWN: 0x500a075109604998 PATH: /dev/disk/by-id/ata-Micron_M500_MTFDDAK960MAV_140109604998) | |-- /dev/sdf@eqx01-flash16 | 1/100 vols | 5/119194 slices | 1.8 TB free of 1.8 TB, NODE: ONLINE, RDVM: UP, DEV: READY | (WWN: 0x5000c5008c0dc21f PATH: /dev/disk/by-id/ata-ST2000NX0253_S4605DSS) | |-- /dev/sdb@eqx01-flash16 | 0/100 vols | 0/119194 slices | 1.8 TB free of 1.8 TB, NODE: ONLINE, RDVM: UP, DEV: READY | (WWN: 0x5000c5008c0dd039 PATH: /dev/disk/by-id/ata-ST2000NX0253_S4605DQ3) | |-- /dev/sdg@eqx01-flash16 | 0/100 vols | 0/119194 slices | 1.8 TB free of 1.8 TB, NODE: ONLINE, RDVM: UP, DEV: READY | (WWN: 0x5000c5008c0dee42 PATH: /dev/disk/by-id/ata-ST2000NX0253_S4605DCS) | |-- /dev/sdd@eqx01-flash16 | 0/100 vols | 0/119194 slices | 1.8 TB free of 1.8 TB, NODE: ONLINE, RDVM: UP, DEV: READY | (WWN: 0x5000c5008c0df26c PATH: /dev/disk/by-id/ata-ST2000NX0253_S4605DB2) | |-- /dev/sdi@eqx01-flash16 | 0/100 vols | 0/119194 slices | 1.8 TB free of 1.8 TB, NODE: ONLINE, RDVM: UP, DEV: READY | (WWN: 0x5000c5008c0fe71f PATH: /dev/disk/by-id/ata-ST2000NX0253_S46059W1) | |-- /dev/sde@eqx01-flash16 | 0/100 vols | 0/119194 slices | 1.8 TB free of 1.8 TB, NODE: ONLINE, RDVM: UP, DEV: READY | (WWN: 0x5000c5008c0feea2 PATH: /dev/disk/by-id/ata-ST2000NX0253_S46059R8) | |-- /dev/sdb@eqx04-flash05 | 0/100 vols | 0/57194 slices | 894.3 GB free of 894.3 GB, NODE: ONLINE, RDVM: UP, DEV: READY | (WWN: 0x500a07510ec79d1f PATH: /dev/disk/by-id/ata-Crucial_CT960M500SSD1_14330EC79D1F) | |-- /dev/sdc@eqx04-flash05 | 0/100 vols | 0/57194 slices | 894.3 GB free of 894.3 GB, NODE: ONLINE, RDVM: UP, DEV: READY | (WWN: 0x500a07510ee9a052 PATH: /dev/disk/by-id/ata-Crucial_CT960M500SSD1_14280EE9A052) | SHOWING 1 FILE COLLECTIONS IN THE CLUSTER |-- file-collection-1632045271349 Online 0 errors 0 warnings SHOWING 0 BUNDLES IN THE CLUSTER All bundles are available To see more details rerun with -V|--verbose option sherlock produced results in 158 milliseconds (Sun Sep 19 06:02:28 PM 2021). |-- 3 nodes, 26 disks, 3 vols, 3 snapshots, 0 apps, 0 pods, 1 file-collections, 0 bundles, 5 users and 5 tenants were analyzed **For this example, we have selected this node:eqx01-flash16** # sherlock --node eqx01-flash16 -H -V |-- eqx01-flash16 ONLINE 0 errors, 0 warnings NODE: UP, AGENT: UP, IOMGR: UP, K8S Service(s): READY ============================================================================================================================================================================================= 0 PODS ARE RUNNING ON THIS NODE 9 DEVICES ARE ATTACHED TO THIS NODE |-- DEVID 0: /dev/sdc INIT 14.9 GB free=14.9 GB (100%) 0/100 vols, 0/0 slices, 0 segs |-- DEVID 9: /dev/sdh READY 1.8 TB free=1.8 TB (100%) 1/100 vols, 11/119194 slices, 18 segs | |-- VOLID 162: pvc-eb63979d-720e-41c9-808f-145306dc1259 576 MB nslices=11 nsnaps=1 nsegs=18 nsegs_per_snap=2 | |-- DEVID 13: /dev/sdf READY 1.8 TB free=1.8 TB (100%) 1/100 vols, 5/119194 slices, 11 segs | |-- VOLID 163: pvc-66646581-0210-46e2-b945-9ea880be38d7 352 MB nslices=5 nsnaps=1 nsegs=11 nsegs_per_snap=3 | |-- DEVID 12: /dev/sdb READY 1.8 TB free=1.8 TB (100%) 0/100 vols, 0/119194 slices, 0 segs |-- DEVID 14: /dev/sdg READY 1.8 TB free=1.8 TB (100%) 0/100 vols, 0/119194 slices, 0 segs |-- DEVID 8: /dev/sdd READY 1.8 TB free=1.8 TB (100%) 0/100 vols, 0/119194 slices, 0 segs |-- DEVID 0: /dev/sda INIT 1.8 TB free=1.8 TB (100%) 0/100 vols, 0/0 slices, 0 segs |-- DEVID 10: /dev/sdi READY 1.8 TB free=1.8 TB (100%) 0/100 vols, 0/119194 slices, 0 segs |-- DEVID 11: /dev/sde READY 1.8 TB free=1.8 TB (100%) 0/100 vols, 0/119194 slices, 0 segs THERE ARE 1 FAILED JOBS TO INSPECT BETWEEN Sun Sep 12 05:58:15 PM 2021 - Sun Sep 19 05:58:15 PM 2021 |-- eqx01-flash16.robinsystems.com | |-- HostAddResourcePool jobid=30 state=10 error=1 start=Sun Sep 19 03:23:54 2021 end=Wed Dec 31 16:00:00 1969 | | Host 'eqx01-flash16.robinsystems.com' already has a resource pool 'default' | sherlock produced results in 166 milliseconds (Sun Sep 19 05:58:15 PM 2021). |-- 3 nodes, 26 disks, 3 vols, 3 snapshots, 0 apps, 0 pods, 1 file-collections, 0 bundles, 5 users and 5 tenants were analyzed ================= Check Pod Health ================= To check the health of a Pod, complete the following steps: 1 To get the list Pods, run the following command and choose a desired Pod from the list: .. code-block:: sherlock -V -H 2. To check the health of the Pod, run the following command: .. code-block:: # sherlock --pod You can optionally use the ``-V`` option to view the details. **Example** .. code-block:: # sherlock -V -H SHOWING HEALTH OF 3/3 APPLICATIONS RUNNING IN THE CLUSTER APPNAME: centos1 STATE: ONLINE tuser1 1/1 pods healthy KIND: ROBIN ============================================================================================================================================================================================= APP HAS 1 PODS: POD ID 187: centos1.server.01 on eqx04-flash05 INST: ONLINE/INST: STARTED, NODE: ONLINE, RIO: UP |-- VOLID 238: centos1.server.01.data.1.3a588402-0288-4921-a611-8c8b27e94313 1 GB | |-- DEVID : /dev/sdb (eqx04-flash05) segs=2 slices=1 rawspace=64 MB RDVM: UP |-- VOLID 237: centos1.server.01.block.1.0dd5e060-0e28-499c-a3f8-198e33b10851 1 GB | |-- DEVID : /dev/sdc (eqx04-flash05) segs=0 slices=1 rawspace=0 RDVM: UP | APP IS RUNNING ON THE FOLLOWING 1 NODES: |-- eqx04-flash05 RIO: UP | |-- centos1.server.01 ONLINE/STARTED | APP IS STORING DATA ON THE FOLLOWING 2 DEVICES: |-- DEVID 15: /dev/sdb on eqx04-flash05 1 vols | |-- VOLID 238: centos1.server.01.data.1.3a588402-0288-4921-a611-8c8b27e94313 64 MB nslices=1 nsegs=2 nsnaps=1 segspersnap=2 | |-- DEVID 16: /dev/sdc on eqx04-flash05 1 vols | |-- VOLID 237: centos1.server.01.block.1.0dd5e060-0e28-499c-a3f8-198e33b10851 0 nslices=1 nsegs=0 nsnaps=1 segspersnap=0 # sherlock --pod centos1.server.01 -H -V SHOWING HEALTH OF 1 PODS IN THE CLUSTER: o-- POD/VNODE ID 187: centos1.server.01 STARTED/ONLINE 1 CPU, 200 MB MEM NODE: UP, RIO: UP | |-- VOLID 238: centos1.server.01.data.1.3a588402-0288-4921-a611-8c8b27e94313 64 MB/1 GB nsnaps=1 | | |-- DEVID 15: /dev/sdb on eqx04-flash05 nsegs=2 nslices=1 64 MB | |-- VOLID 237: centos1.server.01.block.1.0dd5e060-0e28-499c-a3f8-198e33b10851 0/1 GB nsnaps=1 | | |-- DEVID 16: /dev/sdc on eqx04-flash05 nsegs=0 nslices=1 0 | THERE ARE 73 FAILED JOBS TO INSPECT BETWEEN Sun Sep 12 10:42:37 PM 2021 - Sun Sep 19 10:42:37 PM 2021 |-- | |-- HostProbe jobid=5 state=10 error=1 start=Sun Sep 19 02:52:57 2021 end=Sun Sep 19 02:52:57 2021 | | HTTPSConnectionPool(host='172.19.0.1', port=443): Max retries exceeded with url: /api/v1/namespaces?fieldSelector=metadata.name%3Drobin- | | admin&limit=0&timeoutSeconds=56 (Caused by NewConnectionError(': Failed to establish a | | new connection: [Errno 111] Connection refused')) | |-- HostProbe jobid=7 state=10 error=1 start=Sun Sep 19 02:53:21 2021 end=Sun Sep 19 02:53:21 2021 | | HTTPSConnectionPool(host='172.19.0.1', port=443): Max retries exceeded with url: /api/v1/namespaces?fieldSelector=metadata.name%3Drobin- | | admin&limit=0&timeoutSeconds=56 (Caused by NewConnectionError(': Failed to establish a | | new connection: [Errno 111] Connection refused')) | |-- HostProbe jobid=6 state=10 error=1 start=Sun Sep 19 02:53:12 2021 end=Sun Sep 19 02:53:12 2021 | | HTTPSConnectionPool(host='172.19.0.1', port=443): Max retries exceeded with url: /api/v1/namespaces?fieldSelector=metadata.name%3Drobin- | | admin&limit=0&timeoutSeconds=56 (Caused by NewConnectionError(': Failed to establish a | | new connection: [Errno 111] Connection refused')) | |-- HostAddResourcePool jobid=30 state=10 error=1 start=Sun Sep 19 03:23:54 2021 end=Wed Dec 31 16:00:00 1969 | | Host 'eqx01-flash16.robinsystems.com' already has a resource pool 'default' | |-- DockerRegistryRemove jobid=1870 state=10 error=1 start=Sun Sep 19 15:16:44 2021 end=Wed Dec 31 16:00:00 1969 | | Invalid registry ID '2', does not exist. | |-- TenantUserRemove jobid=2062 state=10 error=1 start=Sun Sep 19 17:52:01 2021 end=Wed Dec 31 16:00:00 1969 | | Delete the following objects or assign them to another tenant user before removing user 'tuser1': Bundles: robintest jobmgr-3.1 | |-- HostRemoveStorageRole jobid=2129 state=10 error=1 start=Sun Sep 19 18:53:57 2021 end=Wed Dec 31 16:00:00 1969 | | Disk 0x500a075109604998 is still part of 1 device sets | |-- HostRemoveRoles jobid=2127 state=10 error=1 start=Sun Sep 19 18:53:57 2021 end=Sun Sep 19 18:53:57 2021 | | Job failed. One or more child jobs reported errors. Error: Disk 0x500a075109604998 is still part of 1 device sets | |-- UserRemove jobid=2061 state=10 error=1 start=Sun Sep 19 17:52:01 2021 end=Sun Sep 19 17:52:02 2021 | | Job failed. One or more child jobs reported errors. Error: Delete the following objects or assign them to another tenant user before removing user 'tuser1': | | Bundles: robintest jobmgr-3.1 | |-- SetTag jobid=2256 state=10 error=1 start=Sun Sep 19 20:44:13 2021 end=Sun Sep 19 20:44:14 2021 | | Tags support single value per key and there is already present a tag 'nightly':'eqx01-flash15-Disk', please remove it first | |-- HostRemoveRoles jobid=2123 state=10 error=1 start=Sun Sep 19 18:49:05 2021 end=Sun Sep 19 18:49:05 2021 | | Job failed. One or more child jobs reported errors. Error: Disk 0x500a075109604998 is still part of 1 device sets | |-- HostRemoveStorageRole jobid=2125 state=10 error=1 start=Sun Sep 19 18:49:05 2021 end=Wed Dec 31 16:00:00 1969 | | Disk 0x500a075109604998 is still part of 1 device sets | |-- HostAddResourcePool jobid=2131 state=10 error=1 start=Sun Sep 19 18:54:04 2021 end=Wed Dec 31 16:00:00 1969 | | Cannot assign resource pool to host 'eqx01-flash15.robinsystems.com' which has a compute or storage role. | |-- SetTag jobid=2232 state=10 error=1 start=Sun Sep 19 20:10:56 2021 end=Sun Sep 19 20:10:56 2021 | | Tags support single value per key and there is already present a tag 'nightly':'non-default', please remove it first | |-- SetTag jobid=2254 state=10 error=1 start=Sun Sep 19 20:44:13 2021 end=Sun Sep 19 20:44:14 2021 | | Tags support single value per key and there is already present a tag 'nightly':'eqx01-flash15-Disk', please remove it first | |-- SetTag jobid=2258 state=10 error=1 start=Sun Sep 19 20:44:14 2021 end=Sun Sep 19 20:44:14 2021 | | Tags support single value per key and there is already present a tag 'nightly':'eqx01-flash15-Disk', please remove it first | |-- SetTag jobid=2257 state=10 error=1 start=Sun Sep 19 20:44:13 2021 end=Sun Sep 19 20:44:14 2021 | | Tags support single value per key and there is already present a tag 'nightly':'eqx01-flash15-Disk', please remove it first | |-- SetTag jobid=2255 state=10 error=1 start=Sun Sep 19 20:44:13 2021 end=Sun Sep 19 20:44:14 2021 | | Tags support single value per key and there is already present a tag 'nightly':'eqx01-flash15-Disk', please remove it first | |-- SetTag jobid=2260 state=10 error=1 start=Sun Sep 19 20:44:14 2021 end=Sun Sep 19 20:44:14 2021 | | Tags support single value per key and there is already present a tag 'nightly':'eqx01-flash15-Disk', please remove it first | |-- SetTag jobid=2259 state=10 error=1 start=Sun Sep 19 20:44:14 2021 end=Sun Sep 19 20:44:14 2021 | | Tags support single value per key and there is already present a tag 'nightly':'eqx01-flash15-Disk', please remove it first | |-- SetTag jobid=2261 state=10 error=1 start=Sun Sep 19 21:05:10 2021 end=Sun Sep 19 21:05:11 2021 | | Tags support single value per key and there is already present a tag 'nightly':'DiskAll', please remove it first | |-- UserAdd jobid=2329 state=10 error=1 start=Sun Sep 19 22:35:21 2021 end=Wed Dec 31 16:00:00 1969 | | A user by the name of 'tuser1' already exists | |-- TenantUserRemove jobid=2325 state=10 error=1 start=Sun Sep 19 22:24:23 2021 end=Wed Dec 31 16:00:00 1969 | | Delete the following objects or assign them to another tenant user before removing user 'tuser1': Namespace with deployed apps: t016-u000018 Bundles: | | centos 7 Applications: centos1, centos12, centos123 | |-- UserRemove jobid=2327 state=10 error=1 start=Sun Sep 19 22:32:35 2021 end=Sun Sep 19 22:32:35 2021 | | Job failed. One or more child jobs reported errors. Error: Delete the following objects or assign them to another tenant user before removing user 'tuser1': | | Namespace with deployed apps: t016-u000018 Bundles: centos 7 Applications: centos1, centos12, centos123 | |-- BundleRemove jobid=2323 state=10 error=1 start=Sun Sep 19 22:21:44 2021 end=Wed Dec 31 16:00:00 1969 | | Applications active for bundle, centos, with bundleid = 89 | |-- UserAdd jobid=2326 state=10 error=1 start=Sun Sep 19 22:27:06 2021 end=Wed Dec 31 16:00:00 1969 | | A user by the name of 'tuser1' already exists | |-- UserRemove jobid=2324 state=10 error=1 start=Sun Sep 19 22:24:23 2021 end=Sun Sep 19 22:24:23 2021 | | Job failed. One or more child jobs reported errors. Error: Delete the following objects or assign them to another tenant user before removing user 'tuser1': | | Namespace with deployed apps: t016-u000018 Bundles: centos 7 Applications: centos1, centos12, centos123 | |-- TenantUserRemove jobid=2328 state=10 error=1 start=Sun Sep 19 22:32:35 2021 end=Wed Dec 31 16:00:00 1969 | | Delete the following objects or assign them to another tenant user before removing user 'tuser1': Namespace with deployed apps: t016-u000018 Bundles: | | centos 7 Applications: centos1, centos12, centos123 | |-- robinte-backup-rw | |-- StorageRepoAdd jobid=1943 state=10 error=1 start=Sun Sep 19 16:08:10 2021 end=Wed Dec 31 16:00:00 1969 | | Storage repo with name 'robinte-backup-rw' already exists. | |-- MySqlOn | |-- ApplicationScalein jobid=330 state=10 error=1 start=Sun Sep 19 05:19:42 2021 end=Wed Dec 31 16:00:00 1969 | | Cannot scaledown to less than '1' nodes. multinode_min is set to '1' in the bundle definition. | |-- MariaOn | |-- ApplicationScalein jobid=249 state=10 error=1 start=Sun Sep 19 04:51:29 2021 end=Wed Dec 31 16:00:00 1969 | | Cannot scaledown to less than '1' nodes. multinode_min is set to '1' in the bundle definition. | |-- Conflue | |-- ApplicationScalein jobid=747 state=10 error=1 start=Sun Sep 19 06:47:59 2021 end=Wed Dec 31 16:00:00 1969 | | Cannot scaledown to less than '1' nodes. multinode_min is set to '1' in the bundle definition. | |-- ApplicationScalein jobid=748 state=10 error=1 start=Sun Sep 19 06:48:00 2021 end=Wed Dec 31 16:00:00 1969 | | Cannot scaledown to less than '1' nodes. multinode_min is set to '1' in the bundle definition. | |-- ApplicationScalein jobid=736 state=10 error=1 start=Sun Sep 19 06:46:15 2021 end=Wed Dec 31 16:00:00 1969 | | Cannot scaledown to less than '1' nodes. multinode_min is set to '1' in the bundle definition. | |-- ApplicationScalein jobid=743 state=10 error=1 start=Sun Sep 19 06:47:53 2021 end=Wed Dec 31 16:00:00 1969 | | Cannot scaledown to less than '1' nodes. multinode_min is set to '1' in the bundle definition. | |-- ApplicationScalein jobid=744 state=10 error=1 start=Sun Sep 19 06:47:55 2021 end=Wed Dec 31 16:00:00 1969 | | Cannot scaledown to less than '1' nodes. multinode_min is set to '1' in the bundle definition. | |-- ApplicationScalein jobid=745 state=10 error=1 start=Sun Sep 19 06:47:56 2021 end=Wed Dec 31 16:00:00 1969 | | Cannot scaledown to less than '1' nodes. multinode_min is set to '1' in the bundle definition. | |-- ApplicationScalein jobid=746 state=10 error=1 start=Sun Sep 19 06:47:57 2021 end=Wed Dec 31 16:00:00 1969 | | Cannot scaledown to less than '1' nodes. multinode_min is set to '1' in the bundle definition. | |-- NginxHo | |-- ApplicationSnapshot jobid=477 state=10 error=1 start=Sun Sep 19 06:04:36 2021 end=Wed Dec 31 16:00:00 1969 | | Cannot snapshot application 'NginxHo' in 'CREATE_FAILED' state | |-- ApplicationSnapshot jobid=478 state=10 error=1 start=Sun Sep 19 06:04:54 2021 end=Wed Dec 31 16:00:00 1969 | | Cannot snapshot application 'NginxHo' in 'CREATE_FAILED' state | |-- ApplicationScale jobid=479 state=10 error=1 start=Sun Sep 19 06:05:01 2021 end=Wed Dec 31 16:00:00 1969 | | Cannot scale application 'NginxHo' in 'CREATE_FAILED' state | |-- ApplicationCreate jobid=474 state=10 error=1 start=Sun Sep 19 05:49:13 2021 end=Sun Sep 19 06:04:23 2021 | | Job failed. One or more child jobs reported errors. Error: Job failed. One or more child jobs reported errors. Error: Timeout expired while waiting for pod | | desired phase 'Ready' current phase 'Pending' | |-- ApplicationScalein jobid=481 state=10 error=1 start=Sun Sep 19 06:08:23 2021 end=Wed Dec 31 16:00:00 1969 | | Cannot scale-in application 'NginxHo' in 'CREATE_FAILED' state | |-- CentosO | |-- ApplicationScalein jobid=168 state=10 error=1 start=Sun Sep 19 04:22:52 2021 end=Wed Dec 31 16:00:00 1969 | | Cannot scaledown to less than '1' nodes. multinode_min is set to '1' in the bundle definition. | |-- RedisCl | |-- ApplicationScalein jobid=457 state=10 error=1 start=Sun Sep 19 05:46:43 2021 end=Wed Dec 31 16:00:00 1969 | | Cannot scaledown to less than '6' nodes. multinode_min is set to '6' in the bundle definition. | |-- ApplicationScalein jobid=456 state=10 error=1 start=Sun Sep 19 05:46:42 2021 end=Wed Dec 31 16:00:00 1969 | | Cannot scaledown to less than '6' nodes. multinode_min is set to '6' in the bundle definition. | |-- ApplicationScalein jobid=458 state=10 error=1 start=Sun Sep 19 05:46:44 2021 end=Wed Dec 31 16:00:00 1969 | | Cannot scaledown to less than '6' nodes. multinode_min is set to '6' in the bundle definition. | |-- ApplicationScalein jobid=459 state=10 error=1 start=Sun Sep 19 05:46:46 2021 end=Wed Dec 31 16:00:00 1969 | | Cannot scaledown to less than '6' nodes. multinode_min is set to '6' in the bundle definition. | |-- ApplicationScalein jobid=460 state=10 error=1 start=Sun Sep 19 05:46:47 2021 end=Wed Dec 31 16:00:00 1969 | | Cannot scaledown to less than '6' nodes. multinode_min is set to '6' in the bundle definition. | |-- ApplicationScalein jobid=461 state=10 error=1 start=Sun Sep 19 05:46:48 2021 end=Wed Dec 31 16:00:00 1969 | | Cannot scaledown to less than '6' nodes. multinode_min is set to '6' in the bundle definition. | |-- NginxHo.nginx.01 | |-- VnodeDeploy jobid=480 state=10 error=1 start=Sun Sep 19 06:07:41 2021 end=Wed Dec 31 16:00:00 1969 | | Cannot perform Relocate Operation on Vnode 'NginxHo.nginx.01' of an application in failed state | |-- VnodeDeploy jobid=482 state=10 error=1 start=Sun Sep 19 06:08:28 2021 end=Wed Dec 31 16:00:00 1969 | | Cannot perform Repair Operation on Vnode 'NginxHo.nginx.01' of an application in failed state | |-- VnodeAdd jobid=476 state=10 error=1 start=Sun Sep 19 05:49:18 2021 end=Sun Sep 19 06:04:23 2021 | | Timeout expired while waiting for pod desired phase 'Ready' current phase 'Pending' | |-- Postgre | |-- ApplicationScalein jobid=411 state=10 error=1 start=Sun Sep 19 05:37:06 2021 end=Wed Dec 31 16:00:00 1969 | | Cannot scaledown to less than '1' nodes. multinode_min is set to '1' in the bundle definition. | |-- nginx | |-- RoleCreate jobid=475 state=10 error=1 start=Sun Sep 19 05:49:18 2021 end=Sun Sep 19 06:04:23 2021 | | Job failed. One or more child jobs reported errors. Error: Timeout expired while waiting for pod desired phase 'Ready' current phase 'Pending' | |-- mysql | |-- ApplicationSnapshot jobid=1190 state=10 error=1 start=Sun Sep 19 10:15:01 2021 end=Sun Sep 19 10:15:59 2021 | | Snapshotting application mysql failed | |-- ApplicationDataSnapshot jobid=1191 state=10 error=1 start=Sun Sep 19 10:15:30 2021 end=Sun Sep 19 10:15:31 2021 | | Failed to create snapshot: 529 Volume mysql.mysql.01.data.1.a0374e98-442b-46c2-8784-032461845f7f (103) snapshots is using 384.0MB which is above space limit | | 256.0MB. No new snapshot is allowed. | |-- ApplicationCreate jobid=1446 state=10 error=1 start=Sun Sep 19 13:11:57 2021 end=Wed Dec 31 16:00:00 1969 | | Non static IP allocations cannot be done from non-range(network) IP pools -> 'demo-ovs-pool-nw' | |-- ApplicationCreate jobid=1487 state=10 error=1 start=Sun Sep 19 13:22:31 2021 end=Wed Dec 31 16:00:00 1969 | | Static IP 10.10.0.10 is not a part of the IP pool demo-ovs-pool-nw associated with this application | |-- ApplicationCreate jobid=1536 state=10 error=1 start=Sun Sep 19 13:37:28 2021 end=Wed Dec 31 16:00:00 1969 | | Static IP 10.9.63.151 is not within the current Robin IP Pool ranges. Cannot allocate static IP | |-- ApplicationCreate jobid=1593 state=10 error=1 start=Sun Sep 19 13:53:41 2021 end=Wed Dec 31 16:00:00 1969 | | Static IP 10.9.105.64 is not a part of the IP pool demo-ovs-pool associated with this application | |-- Elastic | |-- ApplicationScalein jobid=903 state=10 error=1 start=Sun Sep 19 07:11:51 2021 end=Wed Dec 31 16:00:00 1969 | | Cannot scaledown to less than '1' nodes. multinode_min is set to '1' in the bundle definition. | |-- ApplicationScalein jobid=904 state=10 error=1 start=Sun Sep 19 07:11:52 2021 end=Wed Dec 31 16:00:00 1969 | | Cannot scaledown to less than '1' nodes. multinode_min is set to '1' in the bundle definition. | |-- ApplicationScalein jobid=905 state=10 error=1 start=Sun Sep 19 07:11:53 2021 end=Wed Dec 31 16:00:00 1969 | | Cannot scaledown to less than '1' nodes. multinode_min is set to '1' in the bundle definition. | |-- WordPre | |-- ApplicationScalein jobid=1023 state=10 error=1 start=Sun Sep 19 07:31:58 2021 end=Wed Dec 31 16:00:00 1969 | | Cannot scaledown to less than '1' nodes. multinode_min is set to '1' in the bundle definition. | |-- ApplicationScalein jobid=1022 state=10 error=1 start=Sun Sep 19 07:31:57 2021 end=Wed Dec 31 16:00:00 1969 | | Cannot scaledown to less than '1' nodes. multinode_min is set to '1' in the bundle definition. | |-- centos | |-- ApplicationCreate jobid=1742 state=10 error=1 start=Sun Sep 19 14:09:40 2021 end=Sun Sep 19 14:15:18 2021 | | Job failed. One or more child jobs reported errors. Error: Job failed. One or more child jobs reported errors. Error: time="2021-09-19T14:13:04-07:00" | | level=fatal msg="pulling image: rpc error: code = Unknown desc = Exception calling application: ErrorUnknown:StatusCode.UNKNOWN:Error response from daemon: | | Get https://artifactory.robinsystems.com/v2/robinsys/centos/manifests/7: unauthorized: BAD_CREDENTIAL" | |-- ApplicationCreate jobid=2330 state=10 error=1 start=Sun Sep 19 22:40:53 2021 end=Sun Sep 19 22:40:54 2021 | | Tenant Application count (3) equals or exceeds max_apps_per_tenant limit (2) | |-- server | |-- RoleCreate jobid=1743 state=10 error=1 start=Sun Sep 19 14:09:42 2021 end=Sun Sep 19 14:15:18 2021 | | Job failed. One or more child jobs reported errors. Error: time="2021-09-19T14:13:04-07:00" level=fatal msg="pulling image: rpc error: code = Unknown desc = | | Exception calling application: ErrorUnknown:StatusCode.UNKNOWN:Error response from daemon: Get | | https://artifactory.robinsystems.com/v2/robinsys/centos/manifests/7: unauthorized: BAD_CREDENTIAL" | |-- centos.server.01 | |-- VnodeAdd jobid=1744 state=10 error=1 start=Sun Sep 19 14:09:42 2021 end=Sun Sep 19 14:15:18 2021 | | time="2021-09-19T14:13:04-07:00" level=fatal msg="pulling image: rpc error: code = Unknown desc = Exception calling application: | | ErrorUnknown:StatusCode.UNKNOWN:Error response from daemon: Get https://artifactory.robinsystems.com/v2/robinsys/centos/manifests/7: unauthorized: | | BAD_CREDENTIAL" | |-- robi-bkup-rw | |-- StorageRepoAdd jobid=2142 state=10 error=1 start=Sun Sep 19 18:58:24 2021 end=Wed Dec 31 16:00:00 1969 | | Storage repo with supplied parameters already exists: robinte-backup-rw | |-- alpine-limit | |-- ImageAdd jobid=2276 state=10 error=1 start=Sun Sep 19 21:40:19 2021 end=Wed Dec 31 16:00:00 1969 | | Tenant ImageAdd count (2) equals or exceeds max_images_per_tenant limit (2) | |-- podlimit | |-- ApplicationCreate jobid=2322 state=10 error=1 start=Sun Sep 19 22:02:11 2021 end=Sun Sep 19 22:02:11 2021 | | Tenant Application count (3) equals or exceeds max_pods_per_tenant limit (3) | sherlock produced results in 169 milliseconds (Sun Sep 19 10:42:37 PM 2021). |-- 3 nodes, 26 disks, 9 vols, 9 snapshots, 3 apps, 3 pods, 1 file-collections, 1 bundles, 15 users and 16 tenants were analyzed =============================================== Check Volume Health =============================================== Sherlock provides you an option to check health of the Volumes attached to virtual nodes or pods. It is recommended to validate this check frequently. To check volume health attached to vNodes or Pods, complete the following steps: 1. Run the following command to get the list of volumes to choose for checking health. .. code-block:: text robin volume list 2. Run the following command to check health of the volume. .. code-block:: text # sherlock --vol -H -V **Example** .. code-block:: text # robin volume list ID | Name | Media | Type | BlockSz | Size | Psize | SnapLimit | Prot | Repl | Compress | QGroupID | CTime | Mount ---+--------------------------------------------------------------------+-------+------+---------+-------------+-----------+------------+------+------+----------+----------+------------+---------------------------------------------------------------------------------------------- 1 | file-collection-1557385703714.98f09770-bedd-496f-a7d1-1fd3d1ec840e | 72 | 0 | 4096 | 10737418240 | 268435456 | 0 | 0 | 1 | 1 | 1 | 1557385704 | [{'readonly': 0, 'state': 14, 'zoneid': 1556896675, 'nodeid': 1, 'mntpath': '/dev/vblock0'}] 4 | mysql-test.mysql.01.data.1.3daab239-4327-4f00-873d-ffda3c9575f2 | 72 | 0 | 4096 | 1073741824 | 469762048 | 214748364 | 0 | 1 | 0 | 4 | 1557390977 | [{'readonly': 0, 'state': 14, 'zoneid': 1556896675, 'nodeid': 2, 'mntpath': '/dev/vblock0'}] 5 | mysql-test.mysql.01.root_fs.1.58523556-a318-483c-9f8c-d2cd98ad6a32 | 72 | 0 | 4096 | 10737418240 | 335544320 | 2147483648 | 0 | 1 | 0 | 5 | 1557390977 | [{'readonly': 0, 'state': 14, 'zoneid': 1556896675, 'nodeid': 2, 'mntpath': '/dev/vblock1' # sherlock --vol file-collection-1557385703714.98f09770-bedd-496f-a7d1-1fd3d1ec840e -H -V SHOWING HEALTH OF 1/3 VOLUMES IN THE CLUSTER |-- VOLID 1: file-collection-1557385703714.98f09770-bedd-496f-a7d1-1fd3d1ec840e 256 MB / 10 GB 1 snapshots, using 1 devices | |-- DEVID 9: /dev/sdd on centos-60-182 using 256 MB/127 GB capacity, 8/10 slices, 8 segs, segspernap=1 RDVM: UP, DEV: READY | | (WWN: 0x60022480d380ceef6ddbcdfa327834ca PATH: /dev/disk/by-id/scsi-360022480d380ceef6ddbcdfa327834ca) | | | |-- SNAPSHOTS: 1 CREATED DEV OWN CLONES STATE SIZE | | |-- SNAPID 1: 1969/12/31 16:00:00 8 8 0 READY 256 MB | | | | All volumes are healthy sherlock produced results in 70 milliseconds (Fri May 10 01:46:43 AM 2019). |-- 3 nodes, 12 disks, 3 vols, 7 snapshots, 1 apps, 1 vnodes, 2 users and 1 tenants were analyze ===================== Check Devices Health ===================== To check health of a device, complete the following steps: 1. Run the following command to get the list of devices to choose for checking health. From the list select the desired device name. .. code-block:: text sherlock -H 2. Run the following command to check health of the volume. .. code-block:: text # sherlock --dev -H -V **Example** .. code-block:: text # sherlock -H No matching apps found No matching pods found SHOWING HEALTH OF 1/1 VOLUMES IN THE CLUSTER |-- VOLID 1: file-collection-1632045271349.5ff1f19f-937f-4ec1-a595-9d9df9d11d44, usage: 448 MB / 20 GB, 1 snapshots, resync progress: SYNCED, using 1 devices All volumes are healthy SHOWING HEALTH OF 2/2 NODES RUNNING IN THE CLUSTER |-- eqx01-flash15 ONLINE 0 errors, 0 warnings NODE: UP, AGENT: UP, IOMGR: UP, K8S Service(s): READY |-- eqx01-flash16 ONLINE 0 errors, 0 warnings NODE: UP, AGENT: UP, IOMGR: UP, K8S Service(s): READY SHOWING HEALTH OF 18/18 DEVICES IN THE CLUSTER |-- /dev/sdh@eqx01-flash16 | 0/100 vols | 0/119194 slices | 1.8 TB free of 1.8 TB, NODE: ONLINE, RDVM: UP, DEV: READY | (WWN: 0x5000c5008c0d9e30 PATH: /dev/disk/by-id/ata-ST2000NX0253_S4605EZW) | |-- /dev/sde@eqx01-flash15 | 0/100 vols | 0/119194 slices | 1.8 TB free of 1.8 TB, NODE: ONLINE, RDVM: UP, DEV: READY | (WWN: 0x5000c5008c0db2c7 PATH: /dev/disk/by-id/ata-ST2000NX0253_S4605EN0) | |-- /dev/sdf@eqx01-flash15 | 0/100 vols | 0/119194 slices | 1.8 TB free of 1.8 TB, NODE: ONLINE, RDVM: UP, DEV: READY | (WWN: 0x5000c5008c0db9be PATH: /dev/disk/by-id/ata-ST2000NX0253_S4605E90) | |-- /dev/sdi@eqx01-flash15 | 0/100 vols | 0/119194 slices | 1.8 TB free of 1.8 TB, NODE: ONLINE, RDVM: UP, DEV: READY | (WWN: 0x5000c5008c0dbae3 PATH: /dev/disk/by-id/ata-ST2000NX0253_S4605DY2) | |-- /dev/sdg@eqx01-flash15 | 0/100 vols | 0/119194 slices | 1.8 TB free of 1.8 TB, NODE: ONLINE, RDVM: UP, DEV: READY | (WWN: 0x5000c5008c0ddd62 PATH: /dev/disk/by-id/ata-ST2000NX0253_S4605D33) | |-- /dev/sdh@eqx01-flash15 | 0/100 vols | 0/119194 slices | 1.8 TB free of 1.8 TB, NODE: ONLINE, RDVM: UP, DEV: READY | (WWN: 0x5000c5008c0df3ba PATH: /dev/disk/by-id/ata-ST2000NX0253_S4605CZ7) | |-- /dev/sdd@eqx01-flash15 | 0/100 vols | 0/119194 slices | 1.8 TB free of 1.8 TB, NODE: ONLINE, RDVM: UP, DEV: READY | (WWN: 0x5000c5008c101de8 PATH: /dev/disk/by-id/ata-ST2000NX0253_S4605TKT) | |-- /dev/sdb@eqx01-flash15 | 1/100 vols | 20/57194 slices | 893.8 GB free of 894.3 GB, NODE: ONLINE, RDVM: UP, DEV: READY | (WWN: 0x500a075109604998 PATH: /dev/disk/by-id/ata-Micron_M500_MTFDDAK960MAV_140109604998) | |-- /dev/sdf@eqx01-flash16 | 0/100 vols | 0/119194 slices | 1.8 TB free of 1.8 TB, NODE: ONLINE, RDVM: UP, DEV: READY | (WWN: 0x5000c5008c0dc21f PATH: /dev/disk/by-id/ata-ST2000NX0253_S4605DSS) | |-- /dev/sdb@eqx01-flash16 | 0/100 vols | 0/119194 slices | 1.8 TB free of 1.8 TB, NODE: ONLINE, RDVM: UP, DEV: READY | (WWN: 0x5000c5008c0dd039 PATH: /dev/disk/by-id/ata-ST2000NX0253_S4605DQ3) | |-- /dev/sdg@eqx01-flash16 | 0/100 vols | 0/119194 slices | 1.8 TB free of 1.8 TB, NODE: ONLINE, RDVM: UP, DEV: READY | (WWN: 0x5000c5008c0dee42 PATH: /dev/disk/by-id/ata-ST2000NX0253_S4605DCS) | |-- /dev/sdd@eqx01-flash16 | 0/100 vols | 0/119194 slices | 1.8 TB free of 1.8 TB, NODE: ONLINE, RDVM: UP, DEV: READY | (WWN: 0x5000c5008c0df26c PATH: /dev/disk/by-id/ata-ST2000NX0253_S4605DB2) | |-- /dev/sdi@eqx01-flash16 | 0/100 vols | 0/119194 slices | 1.8 TB free of 1.8 TB, NODE: ONLINE, RDVM: UP, DEV: READY | (WWN: 0x5000c5008c0fe71f PATH: /dev/disk/by-id/ata-ST2000NX0253_S46059W1) | |-- /dev/sde@eqx01-flash16 | 0/100 vols | 0/119194 slices | 1.8 TB free of 1.8 TB, NODE: ONLINE, RDVM: UP, DEV: READY | (WWN: 0x5000c5008c0feea2 PATH: /dev/disk/by-id/ata-ST2000NX0253_S46059R8) **For this example, we used this device name: 0x500a075109604998** # sherlock --dev 0x500a075109604998 -H -V DEVICE /dev/sdb on eqx01-flash15 1/100 vols | 20/57194 slices | 893.8 GB free of 894.3 GB NODE: ONLINE, RDVM: UP, DEV: READY ============================================================================================================================================================================================= |==> WWN: 0x500a075109604998 | PATH: /dev/disk/by-id/ata-Micron_M500_MTFDDAK960MAV_140109604998 |-- VOL: 1 file-collection-1632045271349.5ff1f19f-937f-4ec1-a595-9d9df9d11d44 448 MB nslices=20 nsegs=14 (1 ) nsnaps=1 sherlock produced results in 130 milliseconds (Sun Sep 19 03:34:51 AM 2021). |-- 2 nodes, 18 disks, 1 vols, 1 snapshots, 0 apps, 0 pods, 1 file-collections, 0 bundles, 1 users and 1 tenants were analyzed ======================================= Check Devices Nearing Maximum Capacity ======================================= Using Sherlock you can check devices space usage statistics and identify if any attached device's capacity is nearing maximum capacity. If any device is nearing its maximum capacity, Robin recommends adding more devices to boost the performance. To check devicies nearing maximum capacity, run the following command: .. code-block:: # sherlock --dev full -H -V If there are any devices nearing to maximum capacity, the output will display. ================================= Find Devices With Rebalance Need ================================= To check the devices that might need rebalance, run the following command: .. code-block:: # sherlock --devs-needing-rebalance .. code-block:: # sherlock --devs-needing-rebalance SHOWING APPLICATIONS THAT NEED ATTENTION: All apps are healthy SHOWING PODS THAT NEED ATTENTION: All pods are healthy SHOWING UNHEALTHY VOLUMES THAT NEED ATTENTION All volumes are healthy SHOWING UNHEALTHY NODES THAT NEED ATTENTION: All nodes are healthy SHOWING UNHEALTHY DEVICES THAT NEED ATTENTION: All devices are healthy SHOWING UNAVAILABLE FILE COLLECTIONS THAT NEED ATTENTION: All file collection are available SHOWING UNAVAILABLE BUNDLES THAT NEED ATTENTION: All bundles are available Moving 4 vols, 20 slices and 256 segments: eqx04-flash05 /dev/sdb 894.3 GB/894.3 GB (free=100.0 %) slices= 0/57194 segs= 0/57194 vols= 0/100 [ 1.26 ] eqx04-flash05 /dev/sdc 894.3 GB/894.3 GB (free=100.0 %) slices= 0/57194 segs= 0/57194 vols= 0/100 [ 1.26 ] eqx04-flash05 /dev/sda 894.3 GB/894.3 GB (free=100.0 %) slices= 0/0 segs= 0/0 vols= 0/100 [ -1.00 ] eqx04-flash05 /dev/sdd 894.3 GB/894.3 GB (free=100.0 %) slices= 0/0 segs= 0/0 vols= 0/100 [ -1.00 ] eqx04-flash05 /dev/sde 59.6 GB/59.6 GB (free=100.0 %) slices= 0/0 segs= 0/0 vols= 0/100 [ -1.00 ] eqx04-flash05 /dev/dm-1 17.4 GB/17.4 GB (free=100.0 %) slices= 0/0 segs= 0/0 vols= 0/100 [ -1.00 ] eqx04-flash05 /dev/dm-2 35.7 GB/35.7 GB (free=100.0 %) slices= 0/0 segs= 0/0 vols= 0/100 [ -1.00 ] eqx04-flash05 /dev/dm-0 6.0 GB/6.0 GB (free=100.0 %) slices= 0/0 segs= 0/0 vols= 0/100 [ -1.00 ] ------------------------------------------------------------------------------------------------------------------------ eqx01-flash16 /dev/sdb 1.8 TB/1.8 TB (free=100.0 %) slices= 0/119194 segs= 0/119194 vols= 0/100 [ 1.26 ] eqx01-flash16 /dev/sdg 1.8 TB/1.8 TB (free=100.0 %) slices= 0/119194 segs= 0/119194 vols= 0/100 [ 1.26 ] eqx01-flash16 /dev/sdd 1.8 TB/1.8 TB (free=100.0 %) slices= 0/119194 segs= 0/119194 vols= 0/100 [ 1.26 ] eqx01-flash16 /dev/sdi 1.8 TB/1.8 TB (free=100.0 %) slices= 0/119194 segs= 0/119194 vols= 0/100 [ 1.26 ] eqx01-flash16 /dev/sde 1.8 TB/1.8 TB (free=100.0 %) slices= 0/119194 segs= 0/119194 vols= 0/100 [ 1.26 ] eqx01-flash16 /dev/sdf 1.8 TB/1.8 TB (free=100.0 %) slices= 5/119194 segs= 11/119194 vols= 1/100 [ 1.25 ] eqx01-flash16 /dev/sdh 1.8 TB/1.8 TB (free=100.0 %) slices= 11/119194 segs= 18/119194 vols= 1/100 [ 1.25 ] eqx01-flash16 /dev/sdc 14.9 GB/14.9 GB (free=100.0 %) slices= 0/0 segs= 0/0 vols= 0/100 [ -1.00 ] eqx01-flash16 /dev/sda 1.8 TB/1.8 TB (free=100.0 %) slices= 0/0 segs= 0/0 vols= 0/100 [ -1.00 ] ------------------------------------------------------------------------------------------------------------------------ eqx01-flash15 /dev/sde 1.8 TB/1.8 TB (free=100.0 %) slices= 0/119194 segs= 0/119194 vols= 0/100 [ 1.26 ] eqx01-flash15 /dev/sdf 1.8 TB/1.8 TB (free=100.0 %) slices= 0/119194 segs= 0/119194 vols= 0/100 [ 1.26 ] eqx01-flash15 /dev/sdi 1.8 TB/1.8 TB (free=100.0 %) slices= 0/119194 segs= 0/119194 vols= 0/100 [ 1.26 ] eqx01-flash15 /dev/sdg 1.8 TB/1.8 TB (free=100.0 %) slices= 0/119194 segs= 0/119194 vols= 0/100 [ 1.26 ] eqx01-flash15 /dev/sdh 1.8 TB/1.8 TB (free=100.0 %) slices= 0/119194 segs= 0/119194 vols= 0/100 [ 1.26 ] eqx01-flash15 /dev/sdd 1.8 TB/1.8 TB (free=100.0 %) slices= 0/119194 segs= 0/119194 vols= 0/100 [ 1.26 ] eqx01-flash15 /dev/sdb 893.8 GB/894.3 GB (free=100.0 %) slices= 20/57194 segs= 14/57194 vols= 1/100 [ 1.24 ] eqx01-flash15 /dev/sda 894.3 GB/894.3 GB (free=100.0 %) slices= 0/0 segs= 0/0 vols= 0/100 [ -1.00 ] eqx01-flash15 /dev/sdc 14.9 GB/14.9 GB (free=100.0 %) slices= 0/0 segs= 0/0 vols= 0/100 [ -1.00 ] Only unhealthy objects are shown. To see everything re-run with -H|--healthy option To see more details rerun with -V|--verbose option sherlock produced results in 131 milliseconds (Sun Sep 19 05:19:44 PM 2021). |-- 3 nodes, 26 disks, 3 vols, 3 snapshots, 0 apps, 0 pods, 1 file-collections, 1 bundles, 2 users and 2 tenants were analyzed