21. Runbooks¶

21.1. PP-44959 - csi-nodeplugin Pod in CrashLoopBackOff due to invalid config.ini¶

Affected Version: All versions

21.1.1. Background¶

During or after a Robin upgrade, one or more csi-nodeplugin pods can enter CrashLoopBackOff. This occurs because the node configuration file /home/robinds/etc/robin/rcm/config.ini is inconsistent — a concurrent write during the agent bootstrap process produces a file with a duplicate [host] section. The CSI server process on the affected node cannot parse the malformed file and fails immediately on startup and loops in CrashLoopBackOff.

Operational Impact

CSI operations (volume mount/unmount) fail on affected node — new pods requiring Robin volumes cannot be scheduled there. Existing mounted volumes on affected nodes are unaffected.

21.1.2. Detection Symptoms¶

21.1.2.1. Monitoring¶

After an upgrade, watch for:

csi-nodeplugin pod in CrashLoopBackOff with only 2/4 containers ready

21.1.2.2. Cluster State¶

# Check csi-nodeplugin pod status across all nodes
kubectl get pods -n robinio -o wide | grep csi-nodeplugin

Expected pattern — one or more pods stuck in CrashLoopBackOff at 2/4:

csi-nodeplugin-robin-<id>   4/4   Running            0   Xh   10.9.x.x   <healthy-node>
csi-nodeplugin-robin-<id>   2/4   CrashLoopBackOff   N   Xh   10.9.x.x   <affected-node>

21.1.2.3. Logs¶

# Check the robin container logs inside the crashlooping pod
kubectl logs -n robinio <crashlooping-csi-nodeplugin-pod> -c robin

The definitive error is:

configparser.DuplicateSectionError: While reading from '/etc/robin/rcm/config.ini' [line 65]: section 'host' already exists
[PYI-7:ERROR] Failed to execute script 'server' due to unhandled exception!

21.1.3. Mitigation¶

Step 1 — Delete the invalid config file on the affected host

ssh root@<affected-node>
rm /home/robinds/etc/robin/rcm/config.ini

Step 2 — Bounce the robin-worker pod on the affected node

kubectl delete pod -n robinio <robin-worker-pod-on-affected-node>

The new worker pod regenerates a clean config.ini on startup. Kubernetes will also automatically restart the csi-nodeplugin pod once the worker is healthy.

Step 3 — Verify recovery

# All csi-nodeplugin pods should be 4/4 Running
kubectl get pods -n robinio -o wide | grep csi-nodeplugin
# Confirm the config file on each previously affected node now has exactly one [host] section
grep -c '\[host\]' /home/robinds/etc/robin/rcm/config.ini
# Expected output: 1

21.2. PP-44441 - robin-worker bootstrap exited after exhausting the max retries and iomgr didn’t start as a result¶

Affected Version: Robin CNS v6.1.0+

21.2.1. Background¶

A node was network partitioned while both the robin-worker and iomgr pods were starting, as a result of which the robin-worker-bootstrap process running in the robin-worker pod couldn’t connect to the k8s api-server. As we have set the maximum bootstrap retries to 5 by default, the worker-bootstrap process tried to connect to the k8s api-server for 5 times and eventually EXITED. Due to this, the robin-agent process never started and it never registered with the robin-server. (node.ini file was never populated as the host registration never happened). Due to the missing node.ini file, the iomgr-server-wrapper complained about the file and eventually failed after it exhausted the 120s timeout, this failure loop happened for 5 retries by the supervisord service and then the iomgr-server service also transitioned into EXITED state.

21.2.2. Detection Symptoms¶

21.2.2.1. Monitoring¶

Check the output of the following command - kubectl describe robincluster

robincluster CRD lists the problematic worker pod in Running state (0/1)
although robincluster CRD won’t list the problematic host in the host details section

21.2.2.2. Cluster State¶

Run the following commands to confirm this issue:

robin host list --> host missing in the output

kubectl get pods -n robinio -o wide | grep robin-worker | grep <mising host>

kubectl exec -it -n robinio <robin-worker-pod> -- supervisorctl status robin-bootstrap (service would be in EXITED state)

kubectl get pods -n robinio -o wide | grep robin-iomgr | grep <missing host>

kubectl exec -it -n robinio <robin-iomgr-pod> -- supervisorctl status iomgr-server (again this service would also be in EXITED state)

21.2.2.3. Logs¶

robin-worker-bootstrap logs on the affected host (check /home/robinds/var/log/robin/robin-worker-bootstrap.log)

Traceback (most recent call last):
  File "new_init.py", line 2901, in retry_setup
  File "new_init.py", line 1527, in agent_setup
  File "new_init.py", line 690, in get_token
  File "robin/rcm/kubernetes/robink8s/robink8s.py", line 378, in __init__
  File "robin/rcm/kubernetes/robink8s/robink8s.py", line 477, in init_robink8s
  File "robin/rcm/kubernetes/robink8s/robink8s.py", line 451, in wait_for_api_server
  File "kubernetes/client/api/version_api.py", line 61, in get_code
  File "kubernetes/client/api/version_api.py", line 128, in get_code_with_http_info
  File "kubernetes/client/api_client.py", line 348, in call_api
  File "kubernetes/client/api_client.py", line 180, in __call_api
  File "kubernetes/client/api_client.py", line 373, in request
  File "kubernetes/client/rest.py", line 244, in GET
  File "kubernetes/client/rest.py", line 217, in request
  File "urllib3/request.py", line 77, in request
  File "urllib3/request.py", line 99, in request_encode_url
  File "urllib3/poolmanager.py", line 376, in urlopen
  File "urllib3/connectionpool.py", line 829, in urlopen
  File "urllib3/connectionpool.py", line 829, in urlopen
  File "urllib3/connectionpool.py", line 829, in urlopen
  File "urllib3/connectionpool.py", line 801, in urlopen
  File "urllib3/util/retry.py", line 594, in increment
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='10.96.0.1', port=443): Max retries exceeded with url: /version/ (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f336d8cb410>: Failed to establish a new connection: [Errno 101] Network is unreachable'))
2026-08-13 06:42:21,142 - MainThread - robin - ERROR - new_init: Robin Bootstrap failed after 5 attempt(s); exiting to allow leader transition
Traceback (most recent call last):
  File "new_init.py", line 2947, in <module>
  File "new_init.py", line 2895, in retry_setup
Exception: Robin Bootstrap failed after 5 attempt(s); exiting to allow leader transition

robin-iomgr-wrapper logs on the affected host (check /home/robinds/var/log/robin/iomgr-server-wrapper.log)

Thu Aug 13 07:28:39 PDT 2026: ROBIN config file /etc/robin/node.ini not found.Exiting
Thu Aug 13 07:28:40 PDT 2026: Starting iomgr-server-wrapper
Thu Aug 13 07:28:40 PDT 2026: Waiting for robin config file /etc/robin/node.ini
Thu Aug 13 07:28:50 PDT 2026: Waiting for robin config file /etc/robin/node.ini
Thu Aug 13 07:29:00 PDT 2026: Waiting for robin config file /etc/robin/node.ini
Thu Aug 13 07:29:10 PDT 2026: Waiting for robin config file /etc/robin/node.ini
Thu Aug 13 07:29:20 PDT 2026: Waiting for robin config file /etc/robin/node.ini
Thu Aug 13 07:29:30 PDT 2026: Waiting for robin config file /etc/robin/node.ini
Thu Aug 13 07:29:40 PDT 2026: Waiting for robin config file /etc/robin/node.ini
Thu Aug 13 07:29:50 PDT 2026: Waiting for robin config file /etc/robin/node.ini
Thu Aug 13 07:30:00 PDT 2026: Waiting for robin config file /etc/robin/node.ini
Thu Aug 13 07:30:10 PDT 2026: Waiting for robin config file /etc/robin/node.ini
Thu Aug 13 07:30:20 PDT 2026: Waiting for robin config file /etc/robin/node.ini
Thu Aug 13 07:30:30 PDT 2026: Waiting for robin config file /etc/robin/node.ini
Thu Aug 13 07:30:40 PDT 2026: Waiting for robin config file /etc/robin/node.ini
Thu Aug 13 07:30:50 PDT 2026: ROBIN config file /etc/robin/node.ini not found.Exiting

21.2.3. Mitigation¶

Step 1 - Bounce the affected robin-worker pod first and let the host be added in the robin host list

kubectl delete pod -n robinio <robin-worker-pod>

Step 2 - Bounce the affected robin-iomgr pod on the same node and verify if the IOMgr service becomes ready in the robin host list –services

kubectl delete pod -n robinio <robin-iomgr-pod>

Final expected state:

robin host list - missing host should be present and also should be in Ready/ONLINE state
robin host list --services - IOMgr service should be in Ready state for the affected host (not CRIT)

21.3. PP-44489 - Worker Pod Stuck in Terminating State and IOMgr Fails to Start Due to a Deadlock¶

Affected Version: All versions

21.3.1. Background¶

When both the robin-worker and robin-iomgr pods on a node are restarted simultaneously — for example during a node restart, a pod bounce, or a chaos event — a circular deadlock can form if the worker pod happens to be in the middle of device discovery when IOMgr is not yet running. The worker pod cannot complete its termination because a storage command on the host is blocked waiting for I/O to complete, and that I/O never completes because IOMgr is not up.

This creates a circular deadlock:

The old robin-worker pod is stuck in Terminating (blocked on a storage command that needs IOMgr).
The new robin-iomgr pod cannot pass its readiness check because it needs consul-client to be healthy.
consul-client runs inside the robin-worker pod — but the new worker pod cannot start until the old Terminating pod is gone.
The old Terminating pod never exits because the blocked storage command never finishes.

The node remains stuck until the deadlock is broken manually.

Operational Impact

The cluster reports Degraded and the affected node appears Notready* / UNREACHABLE. Existing workloads on healthy nodes continue to run normally.

21.3.2. Detection Symptoms¶

21.3.2.1. Monitoring¶

Watch for:

A robin-worker pod on one node stuck in Terminating for more than a few minutes.
The robin-iomgr pod on the same node stuck at 0/1 Running (readiness probe failing).
The cluster phase changing to Degraded with the affected node listed as unreachable.

21.3.2.2. Cluster State¶

# Check for worker and iomgr pods stuck on the same node
kubectl get pods -n robinio -o wide | grep -E "worker|iomgr"

Expected pattern — one node shows its worker in Terminating and its iomgr at 0/1 Running:

robin-iomgr-<id>    0/1   Running      0   Xh    10.9.x.x   <affected-node>
robin-iomgr-<id>    1/1   Running      0   Xh    10.9.x.x   <healthy-node>
robin-worker-<id>   1/1   Terminating  0   Xh    10.9.x.x   <affected-node>
robin-worker-<id>   1/1   Running      0   Xh    10.9.x.x   <healthy-node>

# Confirm the affected node is unreachable
robin host list

The affected node shows Notready* under Status and UNREACHABLE under LastOpr.

# Confirm all services on the affected node are down
robin host list --service

The affected node will show CRIT or DOWN for Iomgr, ConCl, RMon, and RAgt — all services are dead because both the worker and iomgr are stuck.

# Confirm cluster is Degraded
kubectl describe robincluster | grep -A5 "Phase:"

Phase:    Degraded
phase_reason:
  Down worker services:
    Host <affected-node>: iomgr-server consul-client robin-agent monitor-server
  Down nodes: <affected-node>

21.3.2.3. Logs¶

IOMgr pod logs on the affected node — waiting indefinitely for consul:

kubectl logs -n robinio <stuck-iomgr-pod>

Call to local consul agent fails. Waiting for node to re-join the consul cluster

This line will repeat on a ~3-seconds interval as long as the deadlock persists.

21.3.3. Mitigation¶

Step 1 — Identify the stuck worker pod on the affected node

kubectl get pods -n robinio -o wide | grep Terminating

Note the pod name and the node it is on.

Step 2 — Force delete the stuck worker pod

kubectl delete pod <stuck-worker-pod> -n robinio --force

This immediately removes the stuck pod, allowing a new worker pod to start. Once the new worker is up, consul-client becomes healthy, IOMgr passes its readiness check, and the blocked storage command on the host resolves — breaking the deadlock and returning the node to a healthy state.

Step 3 — Verify the deadlock is broken

# New worker and iomgr should both be Running on the affected node
kubectl get pods -n robinio -o wide | grep -E "worker|iomgr"

# Node should return to Ready
robin host list

# Cluster phase should return to Ready
kubectl describe robincluster | grep "Phase:"

21.4. PP-40480 - Pod stuck in ContainerCreating due to node in Mount-Blocked state¶

Affected Version: All versions

21.4.1. Background¶

In rare scenarios, a node can get stuck in a mount-blocked state after a node re-registration event (such as a node restart or worker pod restart). During re-registration, a timing race can leave the node’s mount gate permanently blocked — new volume mounts to that node are rejected until the mount state is cleared. Any pod scheduled on that node will remain stuck in ContainerCreating.

Operational Impact

Pods requiring volume mounts on the affected node cannot start. Existing mounted volumes and running workloads on the node are unaffected.

21.4.2. Detection Symptoms¶

21.4.2.1. Monitoring¶

Watch for:

One or more pods stuck in ContainerCreating for an extended period with no progress
kubectl describe pod showing repeated FailedAttachVolume or FailedMount warnings referencing STORMGR_NODE_BLOCK_MOUNT

21.4.2.2. Cluster State¶

# Identify pods stuck in ContainerCreating
kubectl get pods -A | grep ContainerCreating

# Confirm the mount-blocked error on the stuck pod
kubectl describe pod <stuck-pod> -n <namespace>

Look for this error in the Events section:

Warning  FailedAttachVolume  ...  attachdetach-controller
  AttachVolume.Attach failed for volume "<pvc-name>":
  rpc error: code = Aborted desc = Error: Job failed. One or more child jobs reported errors.
  Error: Failed to mount volume <pvc-name>: Node <default:<hostname>> has mount_blocked STORMGR_NODE_BLOCK_MOUNT. No new mounts are allowed.

Note the hostname inside <default:<hostname>> — this is the affected node.

21.4.2.3. Logs¶

VolumeMount Job log contains:

ERROR - Failed to mount volume <pvc-name>: Node <default:<hostname>> has mount_blocked STORMGR_NODE_BLOCK_MOUNT. No new mounts are allowed.

21.4.3. Mitigation¶

Step 1 — Identify the affected node

The node hostname is embedded in the error message from kubectl describe pod:

Node <default:<affected-node>> has mount_blocked STORMGR_NODE_BLOCK_MOUNT

Step 2 — Find the robin-worker pod on that node

kubectl get pods -n robinio -o wide | grep robin-worker | grep <affected-node>

Step 3 — Delete the worker pod to trigger a re-registration

kubectl delete pod <robin-worker-pod> -n robinio

When the worker pod restarts and re-registers, it clears the blocked state on the node. New volume mounts to that node will be allowed again.

Step 4 — Verify recovery

# Confirm the worker pod has restarted cleanly
kubectl get pods -n robinio | grep robin-worker
# Confirm the stuck pod is now running after next volume mount attempt
kubectl get pod <stuck-pod> -n <namespace>

The previously stuck pod should transition from ContainerCreating to Running within a few minutes as the mount block should have been cleared.

21.5. PP-42479 - Orphaned Robin Volume After Deleting a PVC During Hydration¶

Affected Version: Robin CNS v6.1.2

21.5.1. Background¶

When a thick-clone PVC is deleted while its volume is still under hydration, the underlying Robin volume is not deleted because CSI doesn’t issue volume deletion request — volume remains as an orphan in Robin’s storage layer, consuming disk space but with no Kubernetes object referencing it.

Operational Impact

The orphaned volume consumes storage capacity on the node until deleted. No running workloads are affected.

21.5.2. Detection Symptoms¶

21.5.2.1. Monitoring¶

Watch for storage capacity on a node being lower than expected after a PVC delete during a clone/hydration workflow.

21.5.2.2. Cluster State¶

# List all Robin volumes and look for volumes with no corresponding PVC
robin volume list

An orphaned volume will appear as REGULAR type, ONLINE status, without being mounted or associated with any active workload. Cross-reference with PVCs:

# List all PVCs across all namespaces
kubectl get pvc -A

Any Robin volume whose name (pvc-<uuid>) does not appear in the PVC list is a candidate orphan.

21.5.2.3. Logs¶

Look for the PVC deletion event but absence of a corresponding VolumeDelete job:

robin job list | grep <volume-name>

You will see VolumeHydrate and VolumeCreate jobs completing, but no VolumeDelete job for that volume.

21.5.3. Mitigation¶

Note

Before deleting, confirm there is no active PVC or workload referencing the volume. Deletion is irreversible.

Step 1 — Confirm the volume has no active PVC

kubectl get pvc -A | grep <volume-name>
kubectl get pv <volume-name>

Proceed only if both return empty.

Step 2 — Delete the orphaned volume

robin volume delete <volume-name>

Step 3 — Confirm deletion

robin volume list | grep <volume-name>

The volume should no longer appear.

21.6. PP-41599 - K8s Snapshot Succeeded Even When Robin Snapshot Failed¶

Affected Version: Robin CNS v6.0.0+

21.6.1. Background¶

In rare scenarios, if the RIO snapshot request takes a long time, RCM may incorrectly determine that the snapshot has succeeded while it is still in progress in SM. This can cause the Kubernetes snapshot to be marked as ready_to_use=True, even though the Robin snapshot subsequently fails. Robin will not have an entry for the failed snapshot. Operational Impact: Since the Kubernetes snapshot is incorrectly marked as ready_to_use=True, all clones created from this snapshot will fail.

21.6.2. Detection Symptoms¶

The Kubernetes snapshot will be marked as READYTOUSE=True, but robin volume-snapshot list will not show an entry for the volumesnapshot.

#kubectl get volumesnapshot
NAME                  READYTOUSE   SOURCEPVC          SOURCESNAPSHOTCONTENT   RESTORESIZE   SNAPSHOTCLASS         SNAPSHOTCONTENT                                    CREATIONTIME   AGE
test-snapshot         true         test-pvc                                               robin-snapshotclass   snapcontent-a3c651da-4752-42c2-9f62-deaf5e940b55   19m            22m

Get the sourcesnapshotcontent from the Kubernetes VolumeSnapshot using the kubectl get volumesnapshot command mentioned above.
The sourcesnapshotcontent will have a value in the following format: snapcontent-<UUID> For example: snapcontent-a3c651da-4752-42c2-9f62-deaf5e940b55

Extract the UUID portion after snapcontent- and use it to search the Robin volume snapshots:

robin volume-snapshot list | grep "73e9d974-d92f-4f26-aea1-11492a5f3279"

Example

robin volume-snapshot list | grep "a3c651da-4752-42c2-9f62-deaf5e940b55"
| 3f14a697a68811f199170de78cb14337 | snapshot-a3c651da-4752-42c2-9f62-deaf5e940b55 | ONLINE | 0 | 01 Sep 2026 21:39:12 | False | False |

If the robin volume-snapshot list command returns no output, check the Kubernetes VolumeSnapshot status:
- If READYTOUSE``=True``
- No corresponding Robin volume snapshot is found,

Then the environment is affected by this issue.

In short: A Kubernetes snapshot marked READYTOUSE=True without a corresponding Robin volume snapshot indicates that we have hit this issue

Additionally, Jobs for clones created from this snapshot will continue to fail with the following error:

Snapshot <snap-name> for Volume <clone-pvc> already exists

The kubectl describe pvc command will show the clone provisioning failing:

Warning  ProvisioningFailed    5m22s                  robin_csi-provisioner-robin-5c9fd966c9-txnjg_f4f55e15-a6af-4280-b758-74a549e6e36f  rpc error: code = Aborted desc = Error: {"jobid":<job-id>}

21.6.3. Mitigation¶

Note

This mitigation will delete the k8s snapshot. The user will need to take a new snapshot and, if required, create a clone from it after completing the workaround steps.

Step 1 - Delete the clone PVC:

kubectl delete pvc <clone-pvc>

Step 2 - Retrieve the VolumeSnapshotContent object associated with the VolumeSnapshot:

kubectl get volumesnapshot <snapshot-name> -o jsonpath='{.status.boundVolumeSnapshotContentName}'

Step 3 - Remove the finalizer from the VolumeSnapshot object:

kubectl patch volumesnapshot <snapshot-name> -p '{"metadata":{"finalizers":[]}}' --type=merge

Step 4 - Delete the VolumeSnapshot object:

kubectl delete volumesnapshot <snapshot-name>

Step 5 - Remove the finalizer from the VolumeSnapshotContent object:

kubectl patch volumesnapshotcontent <snapcontent-name> -p '{"metadata":{"finalizers":[]}}' --type=merge

Step 6 - Delete the VolumeSnapshotContent object:

kubectl delete volumesnapshotcontent <snapcontent-name>

21.7. PP-44832 - Drive ACCESS_FAILED and Host Partial State After Transient IOMgr Pod Restart¶

Affected Version: Robin CNS v6.0.0+

21.7.1. Background¶

In rare scenarios, a IOMgr pod restart could see drives to enter SUSPECTED_OFFLINE / ACCESS_FAILED states. The affected host transitions to Partial status and the cluster phase changes to Degraded. The drive’s state do not auto-recover even after the IOMgr service returns to a healthy (UP) state.

Root Cause: During the IOMgr restart window, another process on host holds an exclusive lock on the block device file. During startup, iomgr process exhausts its retries(~38 seconds) to open the drive in exclusive mode. The device is marked ACCESS_FAILED in the iomgr state machine for that session. The condition self-corrects only on the next iomgr restart (when the stale lock holder is no longer present), but this does not recover the disk state in control plane because of the bug.

Operational Impact

Existing workload continues to function. Volumes that were already allocated on the affected drives remain accessible. The cluster reports Degraded but services are not interrupted.

21.7.2. Detection Symptoms¶

21.7.2.1. Monitoring¶

Watch for:

robincluster phase changing to Degraded
Host status transitioning to Partial for a node whose iomgr pod and service is Running and UP

21.7.2.2. Cluster State¶

Run the following commands to confirm this issue:

# Check for drives in ACCESS_FAILED state
robin drive list | grep SUSPECTED_OFFLINE

Expected output — one or more drives on the same host showing both SUSPECTED_OFFLINE and ACCESS_FAILED:

<drive-id> | <wwn> | <hostname> | default | <devpath> | ... | SUSPECTED_OFFLINE | ACCESS_FAILED | ...

# Confirm the host is in Partial state
robin host list

The affected host will show Partial under the Status column while all other hosts show Ready.

# Confirm IOMgr is healthy on the affected host (UP, not DOWN/CRIT)
robin host list --service

The affected host’s Iomgr column will show UP — confirming this is a stuck state, not an active IOMgr failure.

# Confirm cluster phase
kubectl describe robincluster

Expected output

Phase:    Degraded

21.7.2.3. Logs¶

IOMgr log (/var/log/robin/iomgr/iomgr.log on the IOMgr pod of the affected host):

Look for the exclusive-open failure, which occurs during the restart window:

ERROR [:dev_open():1402] - Failed to exclusively open dev /dev/disk/by-id/<devpath> after 15 retries for 38641948 us.
'fuser /dev/disk/by-id/<devpath>' can give the list of processes accessing this device

This error appears for each affected drive. The retry window is approximately 38 seconds.

Optional — identify the locking process (run on affected host at time of incident):

fuser /dev/disk/by-id/<devpath>

21.7.3. Mitigation¶

Note

Existing I/O and workloads are not impacted. This procedure only restores cluster health reporting.

Step 1 — Identify all affected drives

robin drive list | grep ACCESS_FAILED

Note the WWN (second column) for each drive in ACCESS_FAILED state.

Step 2 — Unfault each affected drive

Run for each <wwn> identified above:

robin drive unfault <wwn>

Repeat for all drives listed. Example with two drives:

robin drive unfault 0xQEMU_QEMU_HARDDISK_CVLYCA079EHK241D
robin drive unfault 0xQEMU_QEMU_HARDDISK_CVLYCA079EHK240D

Step 3 — Verify recovery

# Drives should return to ONLINE state (no SUSPECTED_OFFLINE entries)
robin drive list | grep -E "SUSPECTED_OFFLINE|ACCESS_FAILED"
# Host should return to Ready
robin host list
# Cluster phase should return to Ready
robin cluster info | grep Phase

Expected final state

robin drive list — no drives in SUSPECTED_OFFLINE or ACCESS_FAILED
robin host list — previously Partial host now shows Ready
robin cluster info — Phase: Ready

21.8. PP-44568 - Worker Pod CrashLoopBackOff Due to truncated kubeconfig.tmpl After Node Restart¶

Affected Version: All versions (day-0 bug)

21.8.1. Background¶

After a node restart, the robin-worker pod on the restarted node can enter CrashLoopBackOff. This occurs because the file /home/robinds/etc/robin/k8s/kubeconfig.tmpl is truncated to 0 bytes. The worker bootstrap reads this empty template, generates a broken kubeconfig, and fails to start. Every subsequent pod restart repeats the same cycle since the zero’ed file is not regenerated as long as it exists on disk.

Secondary effects: While the worker pod is in CrashLoopBackOff, the affected node’s IOMgr readiness probe also fails (consul agent unreachable), drives on that host go OFFLINE, and the host is marked Notready* / UNREACHABLE.

Operational Impact

Existing volumes remain accessible via replicas on healthy nodes. No data loss occurs, but write availability may be reduced depending on replication factor.

21.8.2. Detection Symptoms¶

21.8.2.1. Monitoring¶

Watch for:

robin-worker pod on one node stuck in CrashLoopBackOff with many restarts
Corresponding host showing Notready*
All services (ConCl, Iomgr, RMon, RAgt) reporting CRIT for that host

21.8.2.2. Cluster State¶

# Identify the CrashLoopBackOff worker pod and affected node
kubectl get pods -n robinio | grep worker

Expected — one worker in CrashLoopBackOff, the rest Running:

robin-worker-<id>   0/1   CrashLoopBackOff   49 (18s ago)   4d    10.9.x.x   <affected-node>

# Confirm host is unreachable
robin host list

Affected node shows Notready* under Status and UNREACHABLE under LastOpr.

# Confirm all services are CRIT on that host
robin host list --service

# Confirm drives on the affected host are OFFLINE
robin drive list | grep OFFLINE

# Verify the kubeconfig template is 0 bytes on the affected host
stat /home/robinds/etc/robin/k8s/kubeconfig.tmpl

A truncated file shows Size: 0.

21.8.2.3. Logs¶

Worker pod logs (on the affected node):

kubectl logs -n robinio <crashlooping-worker-pod> --previous

Key error pattern:

TypeError: 'NoneType' object is not subscriptable
[PYI-64:ERROR] Failed to execute script 'new_init' due to unhandled exception!
+ [[ 1 != 0 ]]
+ echo 'Setting up cluster config failed'
+ exit 1
Setting up cluster config failed

IOMgr pod logs (on the affected node — IOMgr readiness probe also failing):

kubectl logs -n robinio <iomgr-pod-on-affected-node>

Call to local consul agent fails. Waiting for node to re-join the consul cluster

Bootstrap log inside worker pod or at /var/log/robin/robin-worker-bootstrap.log:

  File "new_init.py", line 2171, in setup_config_ini
  File "new_init.py", line 1455, in get_node_kube_install_info
  File "robin/rcm/kubernetes/robink8s/robink8s.py", line 378, in __init__
  File "robin/rcm/kubernetes/robink8s/robink8s.py", line 477, in init_robink8s
  File "robin/rcm/kubernetes/robink8s/robink8s.py", line 454, in wait_for_api_server
IndexError: tuple index out of range

21.8.3. Mitigation¶

Step 1 — SSH to the affected host

ssh root@<affected-node>

Step 2 — Remove the truncated kubeconfig template

rm /home/robinds/etc/robin/k8s/kubeconfig.tmpl

Step 3 — Delete the crashlooping worker pod to trigger a restart

kubectl delete pod -n robinio <crashlooping-worker-pod>

The bootstrap process will detect the missing file, regenerate it correctly, and the pod will come up Running.

Step 4 — Verify recovery

# Worker pod should return to Running
kubectl get pods -n robinio | grep worker
# Host should return to Ready
robin host list
# Drives should return to ONLINE
robin drive list | grep OFFLINE
# Confirm template was regenerated with correct size (306 bytes)
stat /home/robinds/etc/robin/k8s/kubeconfig.tmpl

21.9. PP-44051 - Upgrade Stuck Due to IOMgr Failing because of control infra port missing in node.ini¶

Affected Version: Robin CNS v6.0.0+

Fixed In Robin CNS v6.1.2

21.9.1. Background¶

During a Robin upgrade, the iomgr-server may fail to start because it reads its configuration before the node has finished writing all required settings. Specifically, the control_infra_server port entry is absent from /etc/robin/node.ini at the time iomgr-server starts up. The process exhausts its retries and enters a FATAL state, causing the upgrade to stall. By the time the retries are exhausted, the port entry is already present in the file — the window is a timing issue during agent and iomgr bootstrap.

21.9.2. Detection Symptoms¶

21.9.2.1. Monitoring¶

During upgrade, watch for robin-iomgr pod staying in a non-ready state (0/1 Running) on a node while it is expected to be 1/1 Running.

21.9.2.2. Cluster State¶

# Check IOMgr pod status across all nodes
kubectl get pods -n robinio -o wide | grep iomgr

Expected during this issue — pod stuck at 0/1 Running, others healthy:

robin-iomgr-bqzjw   1/1   Running   0   9h    10.9.x.x   <node-a>
robin-iomgr-pkf69   1/1   Running   0   9h    10.9.x.x   <node-b>
robin-iomgr-zs5mf   0/1   Running   0   8h    10.9.x.x   <affected-node>

21.9.2.3. Logs¶

IOMgr log on the affected node (/var/log/robin/iomgr.log):

Look for repeated failures to read the port, followed by the process entering a fatal state:

INFO [iomgr:stormgr_load_system_config():74] - Reading system config file /etc/robin/node.ini
ERROR [iomgr:stormgr_load_system_config():177] - Failed to read the Control Infra Server Port from config /etc/robin/node.ini: error=505

This error will repeat until retries are exhausted, after which the process exits:

[iomgr@vnode-113-7 robin]$ supervisorctl status
iomgr-server                     FATAL     Exited too quickly (process log may have details)

Confirming the port is now present (run on the affected host after the failure):

grep control_infra_server_port /etc/robin/node.ini

If this returns a value (e.g., "control_infra_server_port": 29469), the configuration is now correct and the IOMgr pod just needs to be restarted to pick it up.

21.9.3. Mitigation¶

Note

This issue is fixed in Robin CNS v6.1.2. If you are on an affected version and encounter upgrade stall, follow the steps below to resume it.

Step 1 — Identify the stuck IOMgr pod:

kubectl get pods -n robinio -o wide | grep iomgr

Note the pod name showing 0/1 Running and its node.

Step 2 — Force delete the stuck IOMgr pod:

kubectl delete pod <stuck-iomgr-pod> -n robinio

Kubernetes will immediately schedule a replacement pod on the same node. By the time the new pod starts, /etc/robin/node.ini already contains the correct port, so the iomgr starts successfully.

Step 3 — Verify all IOMgr pods are running:

kubectl get pods -n robinio -o wide | grep iomgr

All pods should show 1/1 Running.

Step 4 — Verify the upgrade resumes and completes:

robin host list

All hosts should return to Ready status with the upgraded version once the upgrade job proceeds to completion.