20. Release Notes

20.1. Robin Cloud Native Storage v6.1.2

20.1.1. Robin CNS v6.1.2 Release Notes

The Robin CNS v6.1.2-39 Release Notes document provides information about upgrade paths, improvements, fixed issues, and known issues.

Release Date: September 2, 2026

20.1.2. Infrastructure Versions

The following software applications are included in this CNS release:

Software Application

Version

RobinCluster CR

v2

Patroni

4.1.0

PostgreSQL

15.17

Zalando PostgreSQL Operator

1.15.1

Spilo

18:4.1-p2

20.1.3. Upgrade Paths

The following are the supported upgrade paths for Robin CNS v6.1.2-39:

  • Robin CNS v6.0.0-226 to Robin CNS v6.1.2-39

  • Robin CNS v6.1.0-203 to Robin CNS v6.1.2-39

  • Robin CNS v6.1.1-1 to Robin CNS v6.1.2-39

Note

  • If you use custom options in the RobinCluster CR during Robin CNS installation, you must first upgrade to Robin CNS v6.1.1 before upgrading to Robin CNS v6.1.2 or later to ensure that custom configurations migrate correctly.

  • After upgrading to Robin CNS v6.1.2, if you are using the Robin Client outside the robincli pod, you must upgrade to the latest version of the Robin Client.

  • If you have installed Robin CNS with the skip_postgres_operator parameter to use the Zalando PostgreSQL operator, then you must first upgrade the Zalando PostgreSQL operator to v1.11.0 or later before upgrading to Robin CNS v6.1.2.

20.1.4. Improvements

20.1.4.1. Enabled RobinCluster CRD v2 as default CRD

Starting with Robin CNS v6.1.2, the RobinCluster CRD v2 is enabled by default. This improvement ensures that all new and upgraded clusters utilize the RobinCluster CRD v2 as the default CRD.

Note

If you use custom options in the RobinCluster CR during Robin CNS installation, you must first upgrade to Robin CNS v6.1.1 before upgrading to Robin CNS v6.1.2 or later to ensure that custom configurations migrate correctly.

20.1.4.2. Support to update concurrent data transfers per device

Starting with Robin CNS v6.1.2, you can now configure the number of concurrent data transfers per device using the num_datatransfers_per_dev config parameter. The default value is 10 concurrent data transfers per device.

Data transfers include the following operations:

  • Resync of lagging replicas

  • Rebuild replicas

  • Clone hydration

To update the number of concurrent data transfers per device, run the following command:

# robin config update server num_datatransfers_per_dev <value>

20.1.5. Fixed Issues

Reference ID

Description

RSD-12239

The issue of RPC requests waiting indefinitely and blocking operations is fixed. All RPC requests now have default timeout values to prevent processes from hanging when a response is not received. You can also configure these default timeout values for your Robin CNS cluster if needed.

RSD-12293

The issue of volumes remaining in a DEGRADED state incorrectly after a node reboot is fixed. The issue was caused by a race condition between the monitor server detecting the I/O manager’s health and the I/O manager re-registering itself. The synchronization process is updated to ensure that volumes correctly transition to a healthy state once the I/O manager is active.

RSD-12397

The issue of nodes being incorrectly reported as FAULTED despite being fully operational is fixed. Previously, intermittent network disconnects caused the StorMgr to exhaust request retries prematurely. Now, the logic is updated to handle these network fluctuations correctly and maintain the accurate node state.

RSD-12074

When the IOMgr server exhausts its restart attempts, it exits the restart loop if Consul is unavailable. This issue is fixed. With this release, Robin CNS restarts the IOMgr server once the Consul is available.

RSD-12098

The robin-master Pod failed to recover automatically after a network partition. This issue is fixed. The robin-server now effectively manages temporary Kubernetes API server downtime during startup, preventing initialization failures.

RSD-12454

The issue of Robin CNS installation failing due to a node stuck in a the NOT_READY state is fixed. This issue occurred because the iomgr-server-wrapper reached its retry limit and entered a FATAL state while waiting for the rio.cfg file to be created during agent registration.

RSD-12309

When you try to expand a volume, in rare scenarios where a network error occurs within a small race window, the Kubernetes API may report a successful resize while the underlying block device or filesystem inside the Pod remains at its original size. This issue is fixed.

PP-44055

When upgrading to Robin CNS versions prior to Robin CNS v6.1.2, some nodes got stuck on an older version due to the robin-worker registration error. This issue is fixed.

PP-44051

When upgrading to Robin CNS versions prior to Robin CNS v6.1.2, the upgrade process is stuck because the iomgr-server fails to retrieve the control_infra_server port from the node.ini file. This issue occurred because the iomgr-server attempted to start before the robin-worker agent finished updating the node.ini file with the conrol_infra_server information. This issue is fixed.

RSD-12003

During a drive evacuation, if a Pod needs to fail over to another node, it gets stuck in the ContainerCreating state for the duration of the evacuation. This issue occurred because the VolumeEvacuate job held the volume lock, blocking mount operations required for Pod failover. This issue is fixed.

RSD-12618

During a volume unmount operation, a race condition between a configfs rescan triggered by the Robin agent and the concurrent unmount in IOMgr can leave stale configfs entries. This stale entry caused subsequent volume mount and unmount operations to fail until the node was rebooted to clean up the cruft sys config. This issue is fixed..

RSD-12673

The tlshd.log file in the /var/log/robin/nodeplugin/ directory was consuming more disk space due to missing log rotation. This issue is fixed. The ktlshd process running inside the nodeplugin Pod now rotates logs and maintains a maximum of 10 files at 10 MB each.

PP-43216

The issue of a node remaining stuck in the PROBE_PENDING state indefinitely instead of being marked as UNREACHABLE is fixed. Previously, if a Robin CNS service failed, the node was marked as PROBE_PENDING and the node’s watcher was disabled. Robin CNS now maintains a reachability state that ensures heartbeat probes continue until the node is explicitly marked as UNREACHABLE.

RSD-12706

With Robin CNS v6.1.2, a new deploy_snapshot_crd option in the RobinCluster custom resource is added. When set to 1, Robin CNS deploys the Kubernetes Volume Snapshot CRDs (volumesnapshots, volumesnapshotcontents, volumesnapshotclasses) during installation. By default, these CRDs are not deployed, allowing K8s environments that already ship with their own snapshot CRDs to avoid version conflicts or configuration drift. If this option is updated on a running cluster, restart the robin-operator for the change to take effect.

RSD-12734

During thick VM cloning using the VM CDI (Container Data Importer) workflow, after data is fully copied to the clone volume, the temporary PVC is intentionally deleted and the underlying PV is reassigned to a new PVC. The stale volume watcher job interpreted this PV reassignment as an orphaned volume signal and issued a delete against the fully hydrated volume, resulting in a failure for the cloned VM. This issue is fixed.

20.1.6. Known Issues

Reference ID

Description

PP-44832

Symptom

In rare scenarios, an IOMgr Pod restart might cause drives to enter SUSPECTED_OFFLINE and ACCESS_FAILED states. Because of this issue, the host status changes to the Partial state, and the cluster phase to the Degraded state. The drives do not automatically recover, even after the IOMgr service returns to a healthy state.

Workaround

Unfault drives that are stuck in the ACCESS_FAILED state:

# robin drive unfault <wwn>

PP-41599

Symptom

When creating a clone from a volume snapshot, it might fail and the robin job info command displays the following error:

Failed to create snapshot: 541 Vol <pvc_name>.

The kubectl get volumesnapshot command output displays that the volume snapshot is successful (READYTOUSE: true), but the robin volume-snapshot list command displays no volume snapshot.

Workaround

  1. If a clone PVC was created from the snapshot, then delete the clone PVC:

    # kubectl delete pvc <clone-pvc>

  2. Retrieve the VolumeSnapshotContent object from VolumeSnapshot:

    # kubectl get volumesnapshot <snapshot-name>
-o jsonpath='{.status.boundVolumeSnapshotContentName}'

  3. Remove the finalizer from the VolumeSnapshot object:

    # kubectl patch volumesnapshot <snapshot-name>
-p '{"metadata":{"finalizers":[]}}' --type=merge

  4. Delete the VolumeSnapshot object:

    # kubectl delete volumesnapshot <snapshot-name>

  5. Remove the finalizer from the VolumeSnapshotContent object:

    # kubectl patch volumesnapshotcontent <snapcontent-name>
-p '{"metadata":{"finalizers":[]}}' --type=merge

  6. Delete the VolumeSnapshotContent object:

    # kubectl delete volumesnapshotcontent <snapcontent-name>

PP-44568

Symptom

After a node restart, the robin-worker Pod might enter a CrashLoopBackOff state and and the kubectl logs -n robinio <robin-worker> command displays the following error:

TypeError: 'NoneType' object is not subscriptable [PYI-64:ERROR]
Failed to execute script 'new_init' due to unhandled exception!

This issue occurs because the kubeconfig.tmpl file in the /home/robinds/etc/robin/k8s/ directory is truncated to 0 bytes, and the bootstrap process currently verifies only the file’s existence rather than its content or size.

Workaround

  1. Identify the worker pod that in the CrashLoopBackOff state and the node it is running on:

    kubectl get pods -n robinio -o wide | grep worker

  2. Identify a healthy worker pod on a different node

    HEALTHY_POD=$(kubectl get pods -n robinio -o wide
| grep worker | grep Running
| head -1 | awk '{print $1}')

echo "Will copy config from: $HEALTHY_POD"

  3. Replace the broken kubeconfig file from the healthy worker pod identified in the above step:

    Note

    This command must be run on the affected node. The kubectl cp destination is the local filesystem and running it from a different node will write to the wrong host.

    ssh root@<affected-node>

# Copy the populated kubeconfig from the healthy pod
#directly to the affected node's hostPath.

kubectl cp -c robinrcm robinio/${HEALTHY_POD}:/root/.kube/config \
/home/robinds/root/.kube/config

# Verify it's non-zero
wc -c /home/robinds/root/.kube/config

  4. Remove the truncated kubeconfig.tmpl file also from the /home/robinds/etc/robin/k8s/ directory on the affected node:

    # rm /home/robinds/etc/robin/k8s/kubeconfig.tmpl

  5. Delete the robin-worker pod that was in the CrashLoopBackOff` state to trigger a restart:

    # kubectl delete pod -n robinio <robin-worker>

    The bootstrap process will detect the missing file, regenerate it correctly, and the pod will come up Running state.

PP-44489

Symptom

During an upgrade from the supported Robin CNS version to Robin CNS v6.1.2, the robin-worker Pod might get stuck in the Terminating state due to pending I/O operations. This issue occurs because the IOMgr Pod cannot start until it detects a healthy consul-client. Simultaneously, the consul-client cannot start while the robin-worker Pod remains stuck in the Terminating state.

Workaround

Bounce the affected robin-worker Pod using the --force option:

# kubectl delete pod <robin_worker> --force -n robinio

PP-44476

Symptom

In rare scenarios, you might observe that one of the Pods is stuck in the ContainerCreating state, and the kubectl describe pod command displays the following volume mount error:

Failed to mount volume pvc-d16fa6b1-5bcb-4c69-805d-ab4df9018cee: Node <default:vnode-87-237> has mount_blocked STORMGR_NODE_BLOCK_MOUNT. No new mounts are allowed.

Workaround

Bounce the robin-worker Pod running on the affected node.

# kubectl delete pod <robin_worker> --force -n robinio

PP-44441

Symptom

When installing Robin CNS v6.1.2, during the network partition, one of the nodes might fail to join the cluster. The iomgr-server-wrapper.log displays the following error, indicating that the node configuration is missing:

Waiting for robin config file /etc/robin/node.ini
ROBIN config file /etc/robin/node.ini not found.

Workaround

  1. Verify that the robin-bootstrap service in the robin-worker Pod is in the EXITED state:

    # kubectl exec -it -n robinio <robin-worker-pod> -- bash

  2. Bounce the robin-worker Pod and wait for the host to be added to the robin host list:

    # kubectl delete pod -n robinio <robin-worker-pod>

  3. Bounce the robin-iomgr Pod on the same node and verify if the IOMgr service becomes ready in the robin host list --services:

    # kubectl delete pod -n robinio <robin-iomgr-pod>

PP-44959

Symptom

When upgrading from a supported Robin CNS version to Robin CNS v6.1.2, the csi-nodeplugin Pod might enter a CrashLoopBackOff state. The kubectl logs -n robinio csi-nodeplugin-robin-schcq command displays the following error:

configparser.DuplicateSectionError: While reading from ‘/etc/robin/rcm/config.ini’ [line 65]: section ‘host’ already exists

During the agent bootstrap process, simultaneous write operations can corrupt the /home/robinds/etc/robin/rcm/config.ini file. As a result, it creates duplicate host sections in the config.ini file.

Workaround

  1. Delete the corrupt config.ini file from the /home/robinds/etc/robin/rcm/ directory:

    # rm -rf /home/robinds/etc/robin/rcm/config.ini

  2. Bounce the robin-worker Pod on that node:

    # kubectl delete pod -n robinio <robin-worker>

PP-42418

Symptom

If you try to delete a volume snapshot after a failed clone operation, the kubectl delete volumesnapshot command might hang indefinitely.

This issue occurs because the Kubernetes finalizer provisioner.storage.kubernetes.io/volumesnapshot-as-source-protection is not removed if the associated Persistent Volume Claim (PVC) is deleted while in an unbound or unprovisioned state.

For more information, see SnapshotFinalizerController

Note

Make sure that you have already deleted the underlying snapshot in Robin.

When this issue occurs, the following warning appears:

Snapshot is being used to restore a PVC.

Workaround

To delete the snapshot, manually remove the finalizer by following these steps:

  1. Remove the finalizer from the snapshot by running the following command:

    # kubectl patch volumesnapshot <snapshot_name> \
-p '{"metadata":{"finalizers":[]}}' \
--type=merge

    Replace snapshot_name with the name of your snapshot.

  2. Delete the snapshot:

    # kubectl delete volumesnapshot <snapshot_name> -n <namespace>

PP-34414

Symptom

In a rare scenario, the IOMGR service might fail to open devices in the exclusive mode when it starts as other processes are using these disks. You might observe the following issue:

  • All app Pods restart, and some app Pods get stuck in the ContainerCreating state.

To confirm the above issue, complete the following steps:

  1. Check for the EVENT_DISK_FAULTED event type in the disk events:

    # robin event list --type EVENT_DISK_FAULTED

  2. If you see the disk is faulted error, check the IOMGR logs for dev_open() error and Failed to exclusively open message on the node where disks are present.

    # cat iomgr.log.0 | grep <device>
| grep "dev_open"

  3. If you see the Device or resource busy error in the log file, use fuser command with the device path to confirm whether the device is in use:

    # fuser /dev/disk/by-id/scsi-SATA_
Micron_M500_MTFD_1401096049D5

Workaround

If the device is not in use, restart the IOMGR service on the respective node:

# supervisorctl restart iomgr-server

PP-42723

Symptom

During the Robin CNS v6.1.2 installation process, the robin-patroni-pre-install-hook Pod might go into the Pending state and the installation gets stuck. If you observe this issue, apply the following workaround.

Workaround

  1. Check the robin-operator logs for the following message:

    Failed to deploy patroni helm chart.

  2. Find the node that has the patroni-pre-install-hook.

    # kubectl get node -l=patroni-pv-node=true

  3. Identify which node is hosting the robin-patroni-pre-install-hook Pod that is stuck in the Pending status.

    # kubectl get pods -n robinio | grep pre-install

  4. Run the following command to remove the has-run-patroni-pre-install-job label on the node associated with the pending Pod to allow it to reschedule.

    # kubectl label node <node-name> has-run-patroni-pre-install-job-

PP-45004

Symptom

Robin CNS cannot push the Gin audit logs to Google due to an API endpoint issue. Robin CNS does not generate any alerts for this issue, and cluster administrators might not be aware that audit logs are not being delivered to Google. You can check audit logs in the server.log file at the /var/log/robin/server/ location.

Workaround

Resolve the API endpoint issue to restore the Gin audit logs delivery.

PP-42479

Symptom

If you delete a thick-clone PVC while its volume is still hydrating, the underlying Robin volume is not automatically deleted. This issue occurs because the CSI driver does not issue the required volume deletion request. The orphaned volume remains in the Robin storage layer without any associated Kubernetes object.

Workaround

Manually remove the orphaned Robin volume.

  1. List all Robin volumes and identify the volume that is not mounted:

    # robin volume list

    Note

    An orphaned volume appears as a REGULAR type and ONLINE status, but it does not have a mount path or be associated with any active workload.

  2. Verify that no PVC or PersistentVolumes (PV) exist for this volume:

    # kubectl get pvc -A | grep <volume-name>
# kubectl get pv <volume-name>

    Proceed to the next step only if both commands return no results.

  3. Delete the orphaned volume:

    # robin volume delete <volume-name>

  4. Verify that the volume no longer appears:

    # robin volume list | grep <volume-name>

20.1.7. Technical Support

Contact Technical support for any assistance.