*************************
Upgrading Robin Platform
*************************

Upgrading a Robin cluster involves several steps, updating multiple components and rigorous checks before after the process is complete to ensure all the necessary components are functioning appropriately. In order to ease this process, Robin ships two native scripts, the installer script and the ``GoROBIN`` utility tool, that can complete the upgrade process with as little as one command.

Review the details below and follow the necessary instructions given to upgrade a Robin cluster in a fast, reliable and painless manner.

Robin Upgrade Path Matrix
============================

+--------------------------------------+-----------------------------------------+
|   **Robin Upgrade**                  |       **Kubernetes Upgrade**            |
+--------------------------------------+-----------------------------------------+
| **5.3.3-115(HF4) -> 5.3.5-159(HF2)** |     **1.18.6 -> 1.19.6 -> 1.20.5**      |
+--------------------------------------+                                         |
| **5.3.5-159(HF2) -> 5.3.5-207(HF3)** |                                         |
+--------------------------------------+-----------------------------------------+

.. Note:: Please use this as a reference document only for performing upgrades to the 5.3.5 GA version of Robin.

Pre Robin Upgrade Checklist
===========================

* If you are planning to use the Robin installer script to upgrade a Robin cluster ensure it is present on all nodes within the cluster alongside the Kubernetes/Robin Images tar files. If you are planning on using GoROBIN to perform the upgrade, this is not necessary.
* Credentials for the super admin within the cluster (both username and password) must be used for all upgrade steps.
* All nodes in the cluster must be correctly NTP time synced. If this is not the case, it could cause the upgade to fail.
* If a Kubernetes upgrade is required as part of the upgrade flow, any pods with a Pod Disruption Budget (PDB) configured must either be deleted beforehand or the associated PDB(s) must be configured such that the respective pods can be evicted cleanly while draining the node via ``kubeadm`` during the Kubernetes upgrade.
* No control plane operations, including those associated with applications and volumes, can be performed by any user(s) until the entire upgrade process has been completed successfully.
* Any taint on master nodes other than ``node-role.kubernetes.io/master:NoSchedule`` need to be cleared.

Manual upgrade via the Robin installer
=======================================

Load Upgrade Images
^^^^^^^^^^^^^^^^^^^

This step downloads all the required Kubernetes/Robin images with regard to the upgrade process and can be executed by issuing the command described below.

.. Note:: This step needs to be executed on all nodes within the Robin cluster and can be run in parallel on all of the aforementioned nodes.

.. code-block:: text

        # ./<Robin Installer> prepare-upgrade-hosts --robin-image-archive=<Location for Robin Images tar file> --k8s-image-archive=<Location for Kubernetes Images tar file>


Pre Robin Upgrade I
^^^^^^^^^^^^^^^^^^^

This step ensures all file collections within the cluster are offline as well as disabling Robin autopilot and HA before starting the upgrade. It can be executed with the command described below.

.. Note:: This step needs to be executed only on the Master manager node of the Robin cluster.

.. code-block:: text

        # ./<Robin Installer> pre-robin-upgrade --username=<Robin Admin Username> --password=<Robin Admin Password>


Pre Robin Upgrade II
^^^^^^^^^^^^^^^^^^^^^^^^^^^^

This step ensures all nodes within the cluster are ready for the upgrade. This involves making sure Robin watchdog is stopped on every node and all PGSQL instances on master nodes are synced. It can be executed with the command described below.

.. Note:: This step needs to be executed on all nodes within the Robin cluster and can be run in parallel on all of the aforementioned nodes.

.. code-block:: text

        # ./<Robin Installer> pre-robin-upgrade-all-hosts --username=<Robin Admin Username> --password=<Robin Admin Password>


Robin Upgrade
^^^^^^^^^^^^^^^^^^^^^^^^^^^^

This step upgrades the Calico, Multus, CSI, Robin master and Robin worker pods in the cluster. It can be executed with the command described below.

.. Note:: This step needs to be executed only on the Master manager node of the Robin cluster.

.. code-block:: text

        # ./<Robin Installer> upgrade-robin --username=<Robin Admin Username> --password=<Robin Admin Password>


.. Note:: It is also possible to upgrade the Robin Worker and CSI pods individually using the respective ``upgrade-robin-workers`` and ``upgrade-csi`` options in GoROBIN.


Kubernetes Upgrade
^^^^^^^^^^^^^^^^^^

A Kubernetes upgrade is only required if indicated for your selected upgrade path in the matrix detailed `here <upgrade.html#robin-upgrade-path-matrix>`_. The Kubernetes upgrade steps listed below need to be executed
for every higher Kubernetes version in a sequential manner. For example when upgrading Kubernetes from 1.16.3 to 1.18.6, the command(s) listed below need to be executed twice;
once when upgradring Kubernetes from 1.16.3 to 1.17.5 and another time when upgrading Kubernetes from 1.17.5 to 1.18.6. In order for the the Kubernetes upgrade for the cluster to be successful, it needs to be run and complete
with no errors on the Master manager node of the cluster. After this is complete, the upgrade can be performed on the remaining two control plane nodes in the cluster followed by worker nodes sequentially. The Kubernetes upgrade step can be executed with the command described below.

.. Note:: The Kubernetes upgrade step is not necessary for all Robin upgrades but instead is only required for a select few upgrade paths.

.. code-block:: text

        # ./<Robin Installer> upgrade-k8s --username=<Robin Admin Username> --password=<Robin Admin Password> --k8s-upgrade-version=<K8S version> --reserved-cpus <Reserved cpus Kubelet option. Pass the cpus to be reserved by kubelet and not used for guaranteed PODs> --topology-manager-policy <Topology manager policy. Valid values none,best-effort,restricted and single-numa-node. Default: best-effort> --cpu-manager-policy <CPU manager policy. Valid values none or static. Default: static>

.. Note:: The upgrade procedure on control plane nodes should be executed serially. The first node that is upgraded must have the ``/etc/kubernetes/admin.conf`` file present. The upgrade procedure on worker nodes can be executed parallelly, however caution must be taken to ensure that there are enough nodes not in the process of being upgraded to host existing workloads at all times.


Post Robin Upgrade I
^^^^^^^^^^^^^^^^^^^^^^^^^^^^

This step ensures all the nodes are in a healthy state after the upgrade and can be executed by issuing the command described below.

.. Note:: This step needs to be executed on all nodes within the Robin cluster and can be run in parallel on all of the aforementioned nodes.

.. code-block:: text

        # ./<Robin Installer> post-robin-upgrade-all-hosts --username=<Robin Admin Username> --password=<Robin Admin Password>


Post Robin Upgrade II
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

This step ensures all file collections are brought online and re-enables Robin HA and Autopilot. It can be executed with the command described below.

.. Note:: This step needs to be executed only on the Master manager node of the Robin cluster.

.. code-block:: text

        # ./<Robin Installer> post-robin-upgrade --username=<Robin Admin Username> --password=<Robin Admin Password>


Automated Upgrade via GoROBIN
==================================

All the upgrade steps listed above can be performed in an automated manner using the Robin utility script, GoROBIN. There are two options available when using this utility; namely one can either perform the entire upgrade with a single command or break it down into stages.

.. Note:: Performing upgrades using GoROBIN requires root access to all nodes within the Robin cluster. This is because the script initiates an SSH session to access the aforementioned nodes and run commands during the upgrade process. Also, it is necessary that all nodes in the Robin cluster are reachable before starting the upgrade process using GoROBIN. Moreover, if all hosts in the Robin cluster have been configured to allow passwordless SSH access, users can utilize the ``--sshnopwd`` option in all commands to indicate no root password is needed for access.

Prepare cluster for upgrade via GoROBIN
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

In order to prepare a cluster for an upgrade, users can copy the GoROBIN tar onto all nodes within the cluster beforehand with the below command. This is recommended when upgrading clusters that contain 15 or more nodes since the transfer of the tarfile to all nodes can take considerable amount of time.

.. code-block:: text

        # ./gorobin onprem copy-gorobin-tar --hosts <IP address of the Robin Primary Master Node> --gorobintar <Upgrade version GoROBIN tar> --robin-admin-user <Robin Super Admin Username> --robin-admin-passwd <Robin Super Admin Password>

.. Note:: This step is entirely optional and should only be run in the interest of time.

Single command upgrade via GoROBIN
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Utilizing the below command enables Robin users to perform all the steps listed as part of manual upgrade with one action. Moreover, this command also performs the Kubernetes upgrade if it is required for the Robin upgrade.

.. code-block:: text

        # ./gorobin onprem upgrade-cluster --hosts <IP address of the Robin Primary Master Node> --gorobintar <Upgrade version GoROBIN tar> --robin-admin-user <Robin Super Admin Username> --robin-admin-passwd <Robin Super Admin Password> --reserved-cpus <Reserved cpus Kubelet option. Pass the cpus to be reserved by kubelet and not used for guaranteed PODs> --topology-manager-policy <Topology manager policy. Valid values none,best-effort,restricted and single-numa-node. Default: best-effort> --cpu-manager-policy <CPU manager policy. Valid values none or static. Default: static> 

.. Note:: If the GoROBIN tar is already copied to all nodes within the cluster use the ``--skip-gorobintar-push`` parameter to avoid repeating the process. Also specify the ``--dir`` option to specify the destination tar folder if GoROBIN tar was manually copied to all nodes. ``--reserved-cpus`` option will set the same value for reserved CPUs on all nodes in the cluster during upgrades. Users should also be able to specify reserved CPUs for master nodes using ``--reserved-cpus-masters`` and for worker nodes using ``--reserved-cpus-workers`` options respectively. If none of these options are used; default value of 0-3 as reserved CPUs shall be configured on all nodes in the cluster.

In addition, if the GoROBIN tar is present on an external server/artifactory, its associated URL can be specified via the ``--tar-url`` parameter. This results in the tarfile being downloaded on each node in the cluster by the GoROBIN script. An example of the parameter's usage is shown below.

.. code-block:: text

        # ./gorobin onprem upgrade-cluster --hosts <IP address of the Robin Primary Master Node> --robin-admin-user <Robin Super Admin Username> --robin-admin-passwd <Robin Super Admin Password> --tar-url <URL for GoROBIN tarfile in artifactory>

.. Note:: Ensure that the URL specified is accessible from all hosts in the Robin cluster before starting the upgrade process.

GoROBIN can also be used to perform upgrades on hosts with different login credentials using a simple JSON file similar to the one shown below. Users can specify host SSH usernames, passwords and ports via this JSON file.
Make sure to specify the ``role`` attribute for each of the master nodes within the cluster. Specify the value as 'primary' for the Master manager within the cluster and 'secondary' for the other master nodes. The ``role`` attribute does not need to be specified for worker nodes
in the cluster. User should also be able to specify custom values for reserved CPUs on each node in the cluster by providing an optional parameter ``reserved_cpus_host`` per host (If no value gets sepcified; this value will be default to 0-3). In addition, the path to the host PEM file can be specified within the JSON file instead of the SSH password. An example of this is also shown below.

.. code-block:: json

    {
        "10.10.10.2":{
            "password":"xxxxxxxxx",
            "user":"root",
            "port":"22",
            "role":"secondary",
            "reserved_cpus_host":"<cpus>"
        },
        "10.10.10.3":{
            "password":"/home/host-2.pem",
            "user":"root",
            "port":"22",
            "role":"primary",
            "reserved_cpus_host":"<cpus>"
        },
        "10.10.10.4":{
            "password":"/home/host-3.pem",
            "user":"root",
            "port":"22",
            "role":"secondary",
            "reserved_cpus_host":"<cpus>"
        },
        "10.10.10.5":{
            "password":"/home/host-4.pem",
            "user":"root",
            "port":"22",
            "reserved_cpus_host":"<cpus>"
        }
        "10.10.10.6":{
            "password":"/home/host-5.pem",
            "user":"root",
            "port":"22",
            "reserved_cpus_host":"<cpus>"
        }
    }

.. Note:: For upgrading non-HA clusters specify the role of the primary Robin master node as ``non_ha_primary`` within the JSON file instead of ``primary``. Do not specify a role for any other nodes as no secondary masters will be configured for non-HA clusters.

To specify the hosts login credentials via a JSON file, issue the following GoROBIN command to execute the upgrade for the cluster:

.. code-block:: text

        # ./gorobin onprem upgrade-cluster --hosts-json <Hosts login credentials and ports JSON file path> --gorobintar <Upgrade version GoROBIN tar> --robin-admin-user <Robin Super Admin Username> --robin-admin-passwd <Robin Super Admin Password>

.. note:: Only one of either the ``--hosts-json`` or ``--hosts`` options can be specified.

Multistage upgrade via GoROBIN
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Using GoROBIN, users are able to split their upgrade process into four main parts: ``pre-upgrade`` , ``upgrade``, ``kubernetes-upgrade`` and ``post-upgrade``. Given the flexibility this feature provides, the user inherently gains more control over the upgrade process. 

Listed below are the four GoROBIN options that correspond to each of the aforementioned sections. These commands must be executed sequentially in order to result in a successful upgrade.

.. Note:: For each of the given commands GoROBIN will copy the necessary Robin tar files to all nodes in the Robin cluster while executing the upgrade process. The user can skip this copying process (assuming the required files are already present on the respective machines) using the ``--skip-gorobintar-push`` parameter. Similarly, the ``--hosts-json`` option can be used for each of the commands in order to specify the login credentials of each host via a JSON file.

Pre-Robin Upgrade
------------------

This step will perform all the pre-upgrade steps on the Robin cluster and can be executed with the following command:

.. code-block:: text

        # ./gorobin onprem pre-upgrade-robin --hosts <IP address of the Robin Primary Master Node> --gorobintar <Upgrade version GoROBIN tar> --robin-admin-user <Robin Super Admin Username> --robin-admin-passwd <Robin Super Admin Password>


Robin Upgrade
--------------

This step will perform the actual upgrade on the Robin cluster and can be executed with the following command:

.. code-block:: text

        # ./gorobin onprem upgrade-robin --hosts <IP address of the Robin Primary Master Node> --gorobintar <Upgrade version GoROBIN tar> --robin-admin-user <Robin Super Admin Username> --robin-admin-passwd <Robin Super Admin Password> --topology-manager-policy <Topology manager policy. Valid values none,best-effort,restricted and single-numa-node. Default: best-effort> --cpu-manager-policy <CPU manager policy. Valid values none or static. Default: static>


Kubernetes Upgrade
------------------

This step performs the Kubernetes upgrade on the Robin cluster. If there are multiple Kubernetes versions involved in the Kubernetes upgrade then the command detailed below needs to be executed for each intermediate Kubernetes version.

.. Note:: This step can be skipped for upgrade paths where a Kubernetes upgrade is not required. ``--reserved-cpus`` option will set the same value for reserved CPUs on all nodes in the cluster during upgrades. Users should also be able to specify reserved CPUs for master nodes using ``--reserved-cpus-masters`` and for worker nodes using ``--reserved-cpus-workers`` options respectively. If none of these options are used; default value of 0-3 as reserved CPUs shall be configured on all nodes in the cluster.

.. code-block:: text

        # ./gorobin onprem upgrade-k8s --hosts <IP address of the Robin Primary Master Node> --gorobintar <Upgrade version GoROBIN tar> --robin-admin-user <Robin Super Admin Username> --robin-admin-passwd <Robin Super Admin Password> --k8s-upgrade-version <K8S Version i.e 1.17.5> --reserved-cpus <Reserved cpus Kubelet option. Pass the cpus to be reserved by kubelet and not used for guaranteed PODs> --topology-manager-policy <Topology manager policy. Valid values none,best-effort,restricted and single-numa-node. Default: best-effort> --cpu-manager-policy <CPU manager policy. Valid values none or static. Default: static>


Post-Robin Upgrade
-------------------

This step will perform all post upgrade steps on the Robin cluster and can be executed with the following command:

.. code-block:: text

        # ./gorobin onprem post-upgrade-robin --hosts <IP address of the Robin Primary Master Node> --gorobintar <Upgrade version GoROBIN tar>  --robin-admin-user <Robin Super Admin Username> --robin-admin-passwd <Robin Super Admin Password>


Post Upgrade Cluster Audit
===========================

After the upgrade process is complete, users can obtain a brief summary of all the updated components and services by executing the command shown below.

.. note:: This command can be run from any node with the Robin cluster using the installer script.

.. code-block:: text

        # ./<Robin Installer> audit-cluster --username=<robin admin username> --password=<robin admin password>


Upgraded Component Versions
==================================

+----------------------------+----------------------------------------+
|        **Component**       |          **Upgraded Version**          |
+----------------------------+----------------------------------------+
|       **Kubernetes**       |          **1.20.5**                    |
+----------------------------+----------------------------------------+
|       **cri-tools**        |          **1.21.0**                    |
+----------------------------+----------------------------------------+