****************************** Runbooks ****************************** PP-44959 - csi-nodeplugin Pod in CrashLoopBackOff due to invalid config.ini =============================================================================== **Affected Version**: All versions Background ----------- During or after a Robin upgrade, one or more ``csi-nodeplugin`` pods can enter ``CrashLoopBackOff``. This occurs because the node configuration file ``/home/robinds/etc/robin/rcm/config.ini`` is inconsistent — a concurrent write during the agent bootstrap process produces a file with a duplicate ``[host]`` section. The CSI server process on the affected node cannot parse the malformed file and fails immediately on startup and loops in ``CrashLoopBackOff``. .. admonition:: Operational Impact CSI operations (volume mount/unmount) fail on affected node — new pods requiring Robin volumes cannot be scheduled there. Existing mounted volumes on affected nodes are unaffected. Detection Symptoms ------------------- Monitoring ^^^^^^^^^^^ After an upgrade, watch for: * ``csi-nodeplugin`` pod in ``CrashLoopBackOff`` with only ``2/4`` containers ready Cluster State ^^^^^^^^^^^^^ .. code-block:: shell # Check csi-nodeplugin pod status across all nodes kubectl get pods -n robinio -o wide | grep csi-nodeplugin Expected pattern — one or more pods stuck in ``CrashLoopBackOff`` at ``2/4``: .. code-block:: text csi-nodeplugin-robin- 4/4 Running 0 Xh 10.9.x.x csi-nodeplugin-robin- 2/4 CrashLoopBackOff N Xh 10.9.x.x Logs ^^^^^^^^^^^^^ .. code-block:: shell # Check the robin container logs inside the crashlooping pod kubectl logs -n robinio -c robin The definitive error is: .. code-block:: text configparser.DuplicateSectionError: While reading from '/etc/robin/rcm/config.ini' [line 65]: section 'host' already exists [PYI-7:ERROR] Failed to execute script 'server' due to unhandled exception! Mitigation ------------ **Step 1 — Delete the invalid config file on the affected host** .. code-block:: shell ssh root@ rm /home/robinds/etc/robin/rcm/config.ini **Step 2 — Bounce the robin-worker pod on the affected node** .. code-block:: shell kubectl delete pod -n robinio The new worker pod regenerates a clean ``config.ini`` on startup. Kubernetes will also automatically restart the ``csi-nodeplugin`` pod once the worker is healthy. **Step 3 — Verify recovery** .. code-block:: shell # All csi-nodeplugin pods should be 4/4 Running kubectl get pods -n robinio -o wide | grep csi-nodeplugin # Confirm the config file on each previously affected node now has exactly one [host] section grep -c '\[host\]' /home/robinds/etc/robin/rcm/config.ini # Expected output: 1 PP-44441 - robin-worker bootstrap exited after exhausting the max retries and iomgr didn't start as a result ============================================================================================================= **Affected Version**: Robin CNS v6.1.0+ Background ------------ A node was network partitioned while both the ``robin-worker`` and ``iomgr`` pods were starting, as a result of which the ``robin-worker-bootstrap`` process running in the ``robin-worker`` pod couldn’t connect to the k8s api-server. As we have set the maximum bootstrap retries to 5 by default, the ``worker-bootstrap`` process tried to connect to the k8s api-server for 5 times and eventually ``EXITED``. Due to this, the ``robin-agent`` process never started and it never registered with the robin-server. (``node.ini`` file was never populated as the host registration never happened). Due to the missing ``node.ini`` file, the ``iomgr-server-wrapper`` complained about the file and eventually failed after it exhausted the 120s timeout, this failure loop happened for 5 retries by the supervisord service and then the ``iomgr-server`` service also transitioned into ``EXITED`` state. Detection Symptoms ------------------- Monitoring ^^^^^^^^^^^^ Check the output of the following command - ``kubectl describe robincluster`` 1. robincluster CRD lists the problematic worker pod in Running state (0/1) 2. although robincluster CRD won’t list the problematic host in the host details section Cluster State ^^^^^^^^^^^^^^ Run the following commands to confirm this issue: .. code-block:: shell robin host list --> host missing in the output kubectl get pods -n robinio -o wide | grep robin-worker | grep kubectl exec -it -n robinio -- supervisorctl status robin-bootstrap (service would be in EXITED state) kubectl get pods -n robinio -o wide | grep robin-iomgr | grep kubectl exec -it -n robinio -- supervisorctl status iomgr-server (again this service would also be in EXITED state) Logs ^^^^^^^^^^^^ ``robin-worker-bootstrap`` logs on the affected host (check ``/home/robinds/var/log/robin/robin-worker-bootstrap.log``) .. code-block:: text Traceback (most recent call last): File "new_init.py", line 2901, in retry_setup File "new_init.py", line 1527, in agent_setup File "new_init.py", line 690, in get_token File "robin/rcm/kubernetes/robink8s/robink8s.py", line 378, in __init__ File "robin/rcm/kubernetes/robink8s/robink8s.py", line 477, in init_robink8s File "robin/rcm/kubernetes/robink8s/robink8s.py", line 451, in wait_for_api_server File "kubernetes/client/api/version_api.py", line 61, in get_code File "kubernetes/client/api/version_api.py", line 128, in get_code_with_http_info File "kubernetes/client/api_client.py", line 348, in call_api File "kubernetes/client/api_client.py", line 180, in __call_api File "kubernetes/client/api_client.py", line 373, in request File "kubernetes/client/rest.py", line 244, in GET File "kubernetes/client/rest.py", line 217, in request File "urllib3/request.py", line 77, in request File "urllib3/request.py", line 99, in request_encode_url File "urllib3/poolmanager.py", line 376, in urlopen File "urllib3/connectionpool.py", line 829, in urlopen File "urllib3/connectionpool.py", line 829, in urlopen File "urllib3/connectionpool.py", line 829, in urlopen File "urllib3/connectionpool.py", line 801, in urlopen File "urllib3/util/retry.py", line 594, in increment urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='10.96.0.1', port=443): Max retries exceeded with url: /version/ (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 101] Network is unreachable')) 2026-08-13 06:42:21,142 - MainThread - robin - ERROR - new_init: Robin Bootstrap failed after 5 attempt(s); exiting to allow leader transition Traceback (most recent call last): File "new_init.py", line 2947, in File "new_init.py", line 2895, in retry_setup Exception: Robin Bootstrap failed after 5 attempt(s); exiting to allow leader transition ``robin-iomgr-wrapper`` logs on the affected host (check ``/home/robinds/var/log/robin/iomgr-server-wrapper.log``) .. code-block:: text Thu Aug 13 07:28:39 PDT 2026: ROBIN config file /etc/robin/node.ini not found.Exiting Thu Aug 13 07:28:40 PDT 2026: Starting iomgr-server-wrapper Thu Aug 13 07:28:40 PDT 2026: Waiting for robin config file /etc/robin/node.ini Thu Aug 13 07:28:50 PDT 2026: Waiting for robin config file /etc/robin/node.ini Thu Aug 13 07:29:00 PDT 2026: Waiting for robin config file /etc/robin/node.ini Thu Aug 13 07:29:10 PDT 2026: Waiting for robin config file /etc/robin/node.ini Thu Aug 13 07:29:20 PDT 2026: Waiting for robin config file /etc/robin/node.ini Thu Aug 13 07:29:30 PDT 2026: Waiting for robin config file /etc/robin/node.ini Thu Aug 13 07:29:40 PDT 2026: Waiting for robin config file /etc/robin/node.ini Thu Aug 13 07:29:50 PDT 2026: Waiting for robin config file /etc/robin/node.ini Thu Aug 13 07:30:00 PDT 2026: Waiting for robin config file /etc/robin/node.ini Thu Aug 13 07:30:10 PDT 2026: Waiting for robin config file /etc/robin/node.ini Thu Aug 13 07:30:20 PDT 2026: Waiting for robin config file /etc/robin/node.ini Thu Aug 13 07:30:30 PDT 2026: Waiting for robin config file /etc/robin/node.ini Thu Aug 13 07:30:40 PDT 2026: Waiting for robin config file /etc/robin/node.ini Thu Aug 13 07:30:50 PDT 2026: ROBIN config file /etc/robin/node.ini not found.Exiting Mitigation ------------ **Step 1 - Bounce the affected robin-worker pod first and let the host be added in the robin host list** .. code-block:: shell kubectl delete pod -n robinio **Step 2 - Bounce the affected robin-iomgr pod on the same node and verify if the IOMgr service becomes ready in the robin host list --services** .. code-block:: shell kubectl delete pod -n robinio Final expected state: * ``robin host list`` - missing host should be present and also should be in ``Ready/ONLINE`` state * ``robin host list --services`` - IOMgr service should be in Ready state for the affected host (not CRIT) PP-44489 - Worker Pod Stuck in Terminating State and IOMgr Fails to Start Due to a Deadlock ============================================================================================ **Affected Version**: All versions Background ----------- When both the ``robin-worker`` and ``robin-iomgr`` pods on a node are restarted simultaneously — for example during a node restart, a pod bounce, or a chaos event — a circular deadlock can form if the worker pod happens to be in the middle of device discovery when IOMgr is not yet running. The worker pod cannot complete its termination because a storage command on the host is blocked waiting for I/O to complete, and that I/O never completes because IOMgr is not up. This creates a circular deadlock: 1. The old ``robin-worker`` pod is stuck in ``Terminating`` (blocked on a storage command that needs IOMgr). 2. The new ``robin-iomgr`` pod cannot pass its readiness check because it needs ``consul-client`` to be healthy. 3. ``consul-client`` runs inside the ``robin-worker`` pod — but the new worker pod cannot start until the old ``Terminating`` pod is gone. 4. The old ``Terminating`` pod never exits because the blocked storage command never finishes. The node remains stuck until the deadlock is broken manually. .. admonition:: Operational Impact The cluster reports ``Degraded`` and the affected node appears ``Notready*`` / ``UNREACHABLE``. Existing workloads on healthy nodes continue to run normally. Detection Symptoms -------------------- Monitoring ^^^^^^^^^^^^^^ Watch for: * A ``robin-worker`` pod on one node stuck in ``Terminating`` for more than a few minutes. * The ``robin-iomgr`` pod on the same node stuck at ``0/1 Running`` (readiness probe failing). * The cluster phase changing to ``Degraded`` with the affected node listed as unreachable. Cluster State ^^^^^^^^^^^^^^ .. code-block:: shell # Check for worker and iomgr pods stuck on the same node kubectl get pods -n robinio -o wide | grep -E "worker|iomgr" Expected pattern — one node shows its worker in ``Terminating`` and its iomgr at ``0/1 Running``: .. code-block:: text robin-iomgr- 0/1 Running 0 Xh 10.9.x.x robin-iomgr- 1/1 Running 0 Xh 10.9.x.x robin-worker- 1/1 Terminating 0 Xh 10.9.x.x robin-worker- 1/1 Running 0 Xh 10.9.x.x .. code-block:: shell # Confirm the affected node is unreachable robin host list The affected node shows ``Notready*`` under ``Status`` and ``UNREACHABLE`` under ``LastOpr``. .. code-block:: shell # Confirm all services on the affected node are down robin host list --service The affected node will show ``CRIT`` or ``DOWN`` for ``Iomgr``, ``ConCl``, ``RMon``, and ``RAgt`` — all services are dead because both the worker and iomgr are stuck. .. code-block:: shell # Confirm cluster is Degraded kubectl describe robincluster | grep -A5 "Phase:" .. code-block:: text Phase: Degraded phase_reason: Down worker services: Host : iomgr-server consul-client robin-agent monitor-server Down nodes: Logs ^^^^^^^^^^^^^^ **IOMgr pod logs** on the affected node — waiting indefinitely for consul: .. code-block:: shell kubectl logs -n robinio .. code-block:: text Call to local consul agent fails. Waiting for node to re-join the consul cluster This line will repeat on a ~3-seconds interval as long as the deadlock persists. Mitigation ----------- **Step 1 — Identify the stuck worker pod on the affected node** .. code-block:: shell kubectl get pods -n robinio -o wide | grep Terminating Note the pod name and the node it is on. **Step 2 — Force delete the stuck worker pod** .. code-block:: shell kubectl delete pod -n robinio --force This immediately removes the stuck pod, allowing a new worker pod to start. Once the new worker is up, ``consul-client`` becomes healthy, IOMgr passes its readiness check, and the blocked storage command on the host resolves — breaking the deadlock and returning the node to a healthy state. **Step 3 — Verify the deadlock is broken** .. code-block:: shell # New worker and iomgr should both be Running on the affected node kubectl get pods -n robinio -o wide | grep -E "worker|iomgr" # Node should return to Ready robin host list # Cluster phase should return to Ready kubectl describe robincluster | grep "Phase:" PP-40480 - Pod stuck in ContainerCreating due to node in Mount-Blocked state ============================================================================== **Affected Version**: All versions Background ----------- In rare scenarios, a node can get stuck in a ``mount-blocked`` state after a node re-registration event (such as a node restart or worker pod restart). During re-registration, a timing race can leave the node's mount gate permanently blocked — new volume mounts to that node are rejected until the mount state is cleared. Any pod scheduled on that node will remain stuck in ``ContainerCreating``. .. admonition:: Operational Impact Pods requiring volume mounts on the affected node cannot start. Existing mounted volumes and running workloads on the node are unaffected. Detection Symptoms -------------------- Monitoring ^^^^^^^^^^^^^^ Watch for: * One or more pods stuck in ``ContainerCreating`` for an extended period with no progress * ``kubectl describe pod`` showing repeated ``FailedAttachVolume`` or ``FailedMount`` warnings referencing ``STORMGR_NODE_BLOCK_MOUNT`` Cluster State ^^^^^^^^^^^^^^ .. code-block:: shell # Identify pods stuck in ContainerCreating kubectl get pods -A | grep ContainerCreating .. code-block:: shell # Confirm the mount-blocked error on the stuck pod kubectl describe pod -n Look for this error in the ``Events`` section: .. code-block:: text Warning FailedAttachVolume ... attachdetach-controller AttachVolume.Attach failed for volume "": rpc error: code = Aborted desc = Error: Job failed. One or more child jobs reported errors. Error: Failed to mount volume : Node > has mount_blocked STORMGR_NODE_BLOCK_MOUNT. No new mounts are allowed. Note the hostname inside ``>`` — this is the affected node. Logs ^^^^^^^^^^^^^^ VolumeMount Job log contains: .. code-block:: text ERROR - Failed to mount volume : Node > has mount_blocked STORMGR_NODE_BLOCK_MOUNT. No new mounts are allowed. Mitigation ------------ **Step 1 — Identify the affected node** The node hostname is embedded in the error message from ``kubectl describe pod``: .. code-block:: text Node > has mount_blocked STORMGR_NODE_BLOCK_MOUNT **Step 2 — Find the robin-worker pod on that node** .. code-block:: shell kubectl get pods -n robinio -o wide | grep robin-worker | grep **Step 3 — Delete the worker pod to trigger a re-registration** .. code-block:: shell kubectl delete pod -n robinio When the worker pod restarts and re-registers, it clears the blocked state on the node. New volume mounts to that node will be allowed again. **Step 4 — Verify recovery** .. code-block:: shell # Confirm the worker pod has restarted cleanly kubectl get pods -n robinio | grep robin-worker # Confirm the stuck pod is now running after next volume mount attempt kubectl get pod -n The previously stuck pod should transition from ``ContainerCreating`` to ``Running`` within a few minutes as the mount block should have been cleared. PP-42479 - Orphaned Robin Volume After Deleting a PVC During Hydration ====================================================================== **Affected Version**: Robin CNS v6.1.2 Background ---------- When a thick-clone PVC is deleted while its volume is still under hydration, the underlying Robin volume is not deleted because CSI doesn't issue volume deletion request — volume remains as an orphan in Robin's storage layer, consuming disk space but with no Kubernetes object referencing it. .. admonition:: Operational Impact The orphaned volume consumes storage capacity on the node until deleted. No running workloads are affected. Detection Symptoms ------------------ Monitoring ^^^^^^^^^^^^ Watch for storage capacity on a node being lower than expected after a PVC delete during a clone/hydration workflow. Cluster State ^^^^^^^^^^^^^^ .. code-block:: shell # List all Robin volumes and look for volumes with no corresponding PVC robin volume list An orphaned volume will appear as ``REGULAR`` type, ``ONLINE`` status, **without** being mounted or associated with any active workload. Cross-reference with PVCs: .. code-block:: shell # List all PVCs across all namespaces kubectl get pvc -A Any Robin volume whose name (``pvc-``) does not appear in the PVC list is a candidate orphan. Logs ^^^^^^^^^^^^^^ Look for the PVC deletion event but **absence** of a corresponding ``VolumeDelete`` job: .. code-block:: shell robin job list | grep You will see ``VolumeHydrate`` and ``VolumeCreate`` jobs completing, but no ``VolumeDelete`` job for that volume. Mitigation ---------- .. note:: Before deleting, confirm there is no active PVC or workload referencing the volume. Deletion is irreversible. **Step 1 — Confirm the volume has no active PVC** .. code-block:: shell kubectl get pvc -A | grep kubectl get pv Proceed only if both return empty. **Step 2 — Delete the orphaned volume** .. code-block:: shell robin volume delete **Step 3 — Confirm deletion** .. code-block:: shell robin volume list | grep The volume should no longer appear. PP-41599 - K8s Snapshot Succeeded Even When Robin Snapshot Failed ================================================================== **Affected Version**: Robin CNS v6.0.0+ Background ------------- In rare scenarios, if the RIO snapshot request takes a long time, RCM may incorrectly determine that the snapshot has succeeded while it is still in progress in SM. This can cause the Kubernetes snapshot to be marked as ready_to_use=True, even though the Robin snapshot subsequently fails. Robin will not have an entry for the failed snapshot. Operational Impact: Since the Kubernetes snapshot is incorrectly marked as ready_to_use=True, all clones created from this snapshot will fail. Detection Symptoms -------------------- The Kubernetes snapshot will be marked as READYTOUSE=True, but robin volume-snapshot list will not show an entry for the volumesnapshot. .. code-block:: text #kubectl get volumesnapshot NAME READYTOUSE SOURCEPVC SOURCESNAPSHOTCONTENT RESTORESIZE SNAPSHOTCLASS SNAPSHOTCONTENT CREATIONTIME AGE test-snapshot true test-pvc robin-snapshotclass snapcontent-a3c651da-4752-42c2-9f62-deaf5e940b55 19m 22m 1. Get the ``sourcesnapshotcontent`` from the Kubernetes VolumeSnapshot using the ``kubectl get volumesnapshot`` command mentioned above. 2. The ``sourcesnapshotcontent`` will have a value in the following format: ``snapcontent-`` For example: ``snapcontent-a3c651da-4752-42c2-9f62-deaf5e940b55`` 3. Extract the UUID portion after ``snapcontent-`` and use it to search the Robin volume snapshots: .. code-block:: shell robin volume-snapshot list | grep "73e9d974-d92f-4f26-aea1-11492a5f3279" **Example** .. code-block:: shell robin volume-snapshot list | grep "a3c651da-4752-42c2-9f62-deaf5e940b55" | 3f14a697a68811f199170de78cb14337 | snapshot-a3c651da-4752-42c2-9f62-deaf5e940b55 | ONLINE | 0 | 01 Sep 2026 21:39:12 | False | False | 4. If the ``robin volume-snapshot list`` command returns **no output**, check the Kubernetes VolumeSnapshot status: * If READYTOUSE``=True`` * No corresponding Robin volume snapshot is found, Then the environment is affected by this issue. **In short:** A Kubernetes snapshot marked ``READYTOUSE=True`` without a corresponding Robin volume snapshot indicates that we have hit this issue Additionally, Jobs for clones created from this snapshot will continue to fail with the following error: ``Snapshot for Volume already exists`` The kubectl describe pvc command will show the clone provisioning failing: .. code-block:: text Warning ProvisioningFailed 5m22s robin_csi-provisioner-robin-5c9fd966c9-txnjg_f4f55e15-a6af-4280-b758-74a549e6e36f rpc error: code = Aborted desc = Error: {"jobid":} Mitigation ------------ .. Note:: This mitigation will delete the k8s snapshot. The user will need to take a new snapshot and, if required, create a clone from it after completing the workaround steps. **Step 1 - Delete the clone PVC**: .. code-block:: shell kubectl delete pvc **Step 2 - Retrieve the VolumeSnapshotContent object associated with the VolumeSnapshot**: .. code-block:: shell kubectl get volumesnapshot -o jsonpath='{.status.boundVolumeSnapshotContentName}' **Step 3 - Remove the finalizer from the VolumeSnapshot object**: .. code-block:: shell kubectl patch volumesnapshot -p '{"metadata":{"finalizers":[]}}' --type=merge **Step 4 - Delete the VolumeSnapshot object**: .. code-block:: shell kubectl delete volumesnapshot **Step 5 - Remove the finalizer from the VolumeSnapshotContent object**: .. code-block:: shell kubectl patch volumesnapshotcontent -p '{"metadata":{"finalizers":[]}}' --type=merge **Step 6 - Delete the VolumeSnapshotContent object**: .. code-block:: shell kubectl delete volumesnapshotcontent PP-44832 - Drive ACCESS_FAILED and Host Partial State After Transient IOMgr Pod Restart ======================================================================================== **Affected Version**: Robin CNS v6.0.0+ Background ---------- In rare scenarios, a ``IOMgr`` pod restart could see drives to enter ``SUSPECTED_OFFLINE`` / ``ACCESS_FAILED`` states. The affected host transitions to ``Partial`` status and the cluster phase changes to ``Degraded``. The drive's state do not auto-recover even after the ``IOMgr`` service returns to a healthy (UP) state. **Root Cause**: During the IOMgr restart window, another process on host holds an exclusive lock on the block device file. During startup, iomgr process exhausts its retries(~38 seconds) to open the drive in exclusive mode. The device is marked ``ACCESS_FAILED`` in the iomgr state machine for that session. The condition self-corrects only on the next iomgr restart (when the stale lock holder is no longer present), but this does not recover the disk state in control plane because of the bug. .. admonition:: Operational Impact Existing workload continues to function. Volumes that were already allocated on the affected drives remain accessible. The cluster reports ``Degraded`` but services are not interrupted. Detection Symptoms ------------------ Monitoring ^^^^^^^^^^^ Watch for: * ``robincluster`` phase changing to ``Degraded`` * Host status transitioning to ``Partial`` for a node whose iomgr pod and service is **Running** and **UP** Cluster State ^^^^^^^^^^^^^^ Run the following commands to confirm this issue: .. code-block:: shell # Check for drives in ACCESS_FAILED state robin drive list | grep SUSPECTED_OFFLINE **Expected output** — one or more drives on the same host showing both ``SUSPECTED_OFFLINE`` and ``ACCESS_FAILED``: .. code-block:: text | | | default | | ... | SUSPECTED_OFFLINE | ACCESS_FAILED | ... .. code-block:: shell # Confirm the host is in Partial state robin host list The affected host will show ``Partial`` under the ``Status`` column while all other hosts show ``Ready``. .. code-block:: shell # Confirm IOMgr is healthy on the affected host (UP, not DOWN/CRIT) robin host list --service The affected host's ``Iomgr`` column will show ``UP`` — confirming this is a stuck state, not an active IOMgr failure. .. code-block:: shell # Confirm cluster phase kubectl describe robincluster **Expected output** .. code-block:: text Phase: Degraded Logs ^^^^^^^^^^^^^^ **IOMgr log** (``/var/log/robin/iomgr/iomgr.log`` on the IOMgr pod of the affected host): Look for the exclusive-open failure, which occurs during the restart window: .. code-block:: text ERROR [:dev_open():1402] - Failed to exclusively open dev /dev/disk/by-id/ after 15 retries for 38641948 us. 'fuser /dev/disk/by-id/' can give the list of processes accessing this device This error appears for each affected drive. The retry window is approximately 38 seconds. **Optional — identify the locking process** (run on affected host at time of incident): .. code-block:: shell fuser /dev/disk/by-id/ Mitigation ---------- .. note:: Existing I/O and workloads are not impacted. This procedure only restores cluster health reporting. **Step 1 — Identify all affected drives** .. code-block:: shell robin drive list | grep ACCESS_FAILED Note the ``WWN`` (second column) for each drive in ``ACCESS_FAILED`` state. **Step 2 — Unfault each affected drive** Run for each ```` identified above: .. code-block:: shell robin drive unfault Repeat for all drives listed. Example with two drives: .. code-block:: shell robin drive unfault 0xQEMU_QEMU_HARDDISK_CVLYCA079EHK241D robin drive unfault 0xQEMU_QEMU_HARDDISK_CVLYCA079EHK240D **Step 3 — Verify recovery** .. code-block:: shell # Drives should return to ONLINE state (no SUSPECTED_OFFLINE entries) robin drive list | grep -E "SUSPECTED_OFFLINE|ACCESS_FAILED" # Host should return to Ready robin host list # Cluster phase should return to Ready robin cluster info | grep Phase **Expected final state** * ``robin drive list`` — no drives in ``SUSPECTED_OFFLINE`` or ``ACCESS_FAILED`` * ``robin host list`` — previously ``Partial`` host now shows ``Ready`` * ``robin cluster info`` — ``Phase: Ready`` PP-44568 - Worker Pod CrashLoopBackOff Due to truncated kubeconfig.tmpl After Node Restart =========================================================================================== **Affected Version**: All versions (day-0 bug) Background ---------- After a node restart, the ``robin-worker`` pod on the restarted node can enter ``CrashLoopBackOff``. This occurs because the file ``/home/robinds/etc/robin/k8s/kubeconfig.tmpl`` is truncated to 0 bytes. The worker bootstrap reads this empty template, generates a broken kubeconfig, and fails to start. Every subsequent pod restart repeats the same cycle since the zero'ed file is not regenerated as long as it exists on disk. **Secondary effects**: While the worker pod is in ``CrashLoopBackOff``, the affected node's IOMgr readiness probe also fails (consul agent unreachable), drives on that host go ``OFFLINE``, and the host is marked ``Notready*`` / ``UNREACHABLE``. .. admonition:: Operational Impact Existing volumes remain accessible via replicas on healthy nodes. No data loss occurs, but write availability may be reduced depending on replication factor. Detection Symptoms ------------------ Monitoring ^^^^^^^^^^ Watch for: * ``robin-worker`` pod on one node stuck in ``CrashLoopBackOff`` with many restarts * Corresponding host showing ``Notready*`` * All services (``ConCl``, ``Iomgr``, ``RMon``, ``RAgt``) reporting ``CRIT`` for that host Cluster State ^^^^^^^^^^^^^ .. code-block:: bash # Identify the CrashLoopBackOff worker pod and affected node kubectl get pods -n robinio | grep worker Expected — one worker in ``CrashLoopBackOff``, the rest ``Running``: .. code-block:: text robin-worker- 0/1 CrashLoopBackOff 49 (18s ago) 4d 10.9.x.x .. code-block:: bash # Confirm host is unreachable robin host list Affected node shows ``Notready*`` under ``Status`` and ``UNREACHABLE`` under ``LastOpr``. .. code-block:: bash # Confirm all services are CRIT on that host robin host list --service # Confirm drives on the affected host are OFFLINE robin drive list | grep OFFLINE # Verify the kubeconfig template is 0 bytes on the affected host stat /home/robinds/etc/robin/k8s/kubeconfig.tmpl A truncated file shows ``Size: 0``. Logs ^^^^^^^^^^^^^ **Worker pod logs** (on the affected node): .. code-block:: bash kubectl logs -n robinio --previous Key error pattern: .. code-block:: text TypeError: 'NoneType' object is not subscriptable [PYI-64:ERROR] Failed to execute script 'new_init' due to unhandled exception! + [[ 1 != 0 ]] + echo 'Setting up cluster config failed' + exit 1 Setting up cluster config failed **IOMgr pod logs** (on the affected node — IOMgr readiness probe also failing): .. code-block:: bash kubectl logs -n robinio .. code-block:: text Call to local consul agent fails. Waiting for node to re-join the consul cluster **Bootstrap log** inside worker pod or at ``/var/log/robin/robin-worker-bootstrap.log``: .. code-block:: text File "new_init.py", line 2171, in setup_config_ini File "new_init.py", line 1455, in get_node_kube_install_info File "robin/rcm/kubernetes/robink8s/robink8s.py", line 378, in __init__ File "robin/rcm/kubernetes/robink8s/robink8s.py", line 477, in init_robink8s File "robin/rcm/kubernetes/robink8s/robink8s.py", line 454, in wait_for_api_server IndexError: tuple index out of range Mitigation ---------- **Step 1 — SSH to the affected host** .. code-block:: bash ssh root@ **Step 2 — Remove the truncated kubeconfig template** .. code-block:: bash rm /home/robinds/etc/robin/k8s/kubeconfig.tmpl **Step 3 — Delete the crashlooping worker pod to trigger a restart** .. code-block:: bash kubectl delete pod -n robinio The bootstrap process will detect the missing file, regenerate it correctly, and the pod will come up ``Running``. **Step 4 — Verify recovery** .. code-block:: bash # Worker pod should return to Running kubectl get pods -n robinio | grep worker # Host should return to Ready robin host list # Drives should return to ONLINE robin drive list | grep OFFLINE # Confirm template was regenerated with correct size (306 bytes) stat /home/robinds/etc/robin/k8s/kubeconfig.tmpl PP-44051 - Upgrade Stuck Due to IOMgr Failing because of control infra port missing in node.ini ================================================================================================ **Affected Version**: Robin CNS v6.0.0+ **Fixed In** Robin CNS v6.1.2 Background ---------- During a Robin upgrade, the ``iomgr-server`` may fail to start because it reads its configuration before the node has finished writing all required settings. Specifically, the ``control_infra_server`` port entry is absent from ``/etc/robin/node.ini`` at the time ``iomgr-server`` starts up. The process exhausts its retries and enters a ``FATAL`` state, causing the upgrade to stall. By the time the retries are exhausted, the port entry is already present in the file — the window is a timing issue during agent and iomgr bootstrap. Detection Symptoms ------------------ Monitoring ^^^^^^^^^^ During upgrade, watch for ``robin-iomgr`` pod staying in a non-ready state (``0/1 Running``) on a node while it is expected to be ``1/1 Running``. Cluster State ^^^^^^^^^^^^^ .. code-block:: bash # Check IOMgr pod status across all nodes kubectl get pods -n robinio -o wide | grep iomgr Expected during this issue — pod stuck at ``0/1 Running``, others healthy: .. code-block:: text robin-iomgr-bqzjw 1/1 Running 0 9h 10.9.x.x robin-iomgr-pkf69 1/1 Running 0 9h 10.9.x.x robin-iomgr-zs5mf 0/1 Running 0 8h 10.9.x.x Logs ^^^^ **IOMgr log** on the affected node (``/var/log/robin/iomgr.log``): Look for repeated failures to read the port, followed by the process entering a fatal state: .. code-block:: text INFO [iomgr:stormgr_load_system_config():74] - Reading system config file /etc/robin/node.ini ERROR [iomgr:stormgr_load_system_config():177] - Failed to read the Control Infra Server Port from config /etc/robin/node.ini: error=505 This error will repeat until retries are exhausted, after which the process exits: .. code-block:: text [iomgr@vnode-113-7 robin]$ supervisorctl status iomgr-server FATAL Exited too quickly (process log may have details) **Confirming the port is now present** (run on the affected host after the failure): .. code-block:: bash grep control_infra_server_port /etc/robin/node.ini If this returns a value (e.g., ``"control_infra_server_port": 29469``), the configuration is now correct and the IOMgr pod just needs to be restarted to pick it up. Mitigation ---------- .. note:: This issue is fixed in Robin CNS v6.1.2. If you are on an affected version and encounter upgrade stall, follow the steps below to resume it. **Step 1 — Identify the stuck IOMgr pod**: .. code-block:: bash kubectl get pods -n robinio -o wide | grep iomgr Note the pod name showing ``0/1 Running`` and its node. **Step 2 — Force delete the stuck IOMgr pod**: .. code-block:: bash kubectl delete pod -n robinio Kubernetes will immediately schedule a replacement pod on the same node. By the time the new pod starts, ``/etc/robin/node.ini`` already contains the correct port, so the iomgr starts successfully. **Step 3 — Verify all IOMgr pods are running**: .. code-block:: bash kubectl get pods -n robinio -o wide | grep iomgr All pods should show ``1/1 Running``. **Step 4 — Verify the upgrade resumes and completes**: .. code-block:: bash robin host list All hosts should return to ``Ready`` status with the upgraded version once the upgrade job proceeds to completion.