************************************************************************************************* Configuring Bundles ************************************************************************************************* Robin's manifest file has several attributes that can be used to manipulate aspects of a container including the compute and storage resources allocated to the containers, the order in which containers are spawned etc. This section will cover the most commonly used attributes via a generic example alongside a real world use case. A complete list of attributes can be found here `here `_. ---------------------------------------------------------------------------------------------- Image ---------------------------------------------------------------------------------------------- ROBIN bundle allows users to specify image and runtime engine options for a role. The mandatory options for image are - name: Name of the image. - version: Version of the image - engine: Type of runtime (docker, kvm, lxc) There are further options available at the image section level specific to engine. .. Note:: For the ``docker`` runtime, the combination of the name and version attributes should point to an image in the registry. With regard to the other runtime operations, the name and version attributes of the image should match those used when the image was registered with Robin via the ``robin image add`` command. The following is an example of image attributes available for a KVM based application bundle. .. code-block:: yaml image: name: vnf1 version: "3.0-1094" engine: kvm ostype: linux osvariant: rhel7 graphics: "none" imgsize: "40G" ---------------------------------------------------------------------------------------------- Compute Resources ---------------------------------------------------------------------------------------------- Compute resources such as memory, CPU, hugepages, GPU etc are allocated at the container level. When specifying the number of cores or millicores (fractional value of a CPU) for the CPU(s), you can also specify the type of CPUs that should be picked; options include: ``non isolated``, ``isolated-shared``, ``isolated-dedicated``. Their distinctions and how to specify them are detailed below: - **Non Isolated** - This option indicates that the physical CPUs to be used for an actual deployment of the bundle should be from the non-isolated pool of CPUs on a host. It can be specified by setting the ``nonisol`` attribute to true and the ``reserve`` attribute to false. - **Isolated Shared** - This option indicates that the physical CPUs to be used for an actual deployment of the bundle should be from the isolated pool of CPUs on a host. With this option, eventhough the allocated CPU(s) are isolated from kernel processes, it can still be utilized by other application deployments (hence the term sharing). It can be specified by setting the ``isol`` attribute to true and the ``reserve`` attribute to false. - **Isolated Dedicated** - This option indicates that the physical CPUs to be used for an actual deployment of the bundle should be from the isolated pool of CPUs on a host. With this option, the allocated CPU(s) are not only isolated from kernel processes, but also other application deployments (hence the term dedicated). It can be specified by setting both the ``isol`` and ``reserve`` attributes to true. In addition, when specifying GPU requests, the GPU resource type can be specified. If no type is specified then only non-partitioned GPU devices will be selected for allocation. As a result, if MIG devices need to be allocated for a container the resource type must be specified. With the configuration given below Robin will allocate 2GB memory, 4.03 isolated-shared CPU cores, hugepages and a single GPU (matching the type specified) to a container. .. code-block:: yaml compute: memory: 2048M hugepages_1gi: 1G hugepages_2mi: 1G cpu: reserve: false cores: 4.03 isol: true gpu: count: 1 type: "nvidia-a100" If no values are specified for the compute attribute in the manifest, Robin will allocate 1GB of Memory and 1 non-isolated CPU core (as they are default values) to the relevant containers. Below is a manifest file for an Nginx bundle with the compute attribute specified: .. code-block:: yaml name: Nginx Server version: "1.12" description: Nginx roles: [nginx] nginx: name: "nginx server" image: name: nginx version: 1.12-alpine engine: docker compute: memory: 2048M cpu: reserve: false cores: 2 isol: true Below is from a manifest file for a KVM based vRAN bundle with the compute attributes specified. The following application requires: - 18 cores: out of which Core IDs 2-17 should be mapped to RT (Realtime) scheduling priority and FIFO policy - 24G in 1Gi-Hugepages that should not be shared and should be locked - KVM memory balloon device with the specified attibutes - KVM emulatorpin cpuset outside the CPUset allocated to the KVM POD in order to ensure the emulator does not interfere with the application - 4 cache ways allocated exclusively for this POD's cpuset (Intel Cache Allocation Technology) .. code-block:: yaml compute: memory: 4M hugepages_1gi: 24G hugepages_locked: "yes" hugepages_noshared: "yes" memballoon: "model=none,stats.period=10" cpu: reserve: true isol: true cores: 18 rt_cores: 2-17 rt_sched: fifo rt_priority: 1 emulatorpin: outside_cpuset cache_ways: 4 kvmopts: "mode=host-passthrough,require=tsc-deadline,check=none" .. Note:: The following attributes will only work on Realtime kernel: ``rt_cores``, ``rt_sched``, ``rt_priority``. For cache_ways, the profiles/masks for each NUMA node should be configured beforehand. ---------------------------------------------------------------------------------------------- Storage Resources ---------------------------------------------------------------------------------------------- Robin offers several attributes within the manifest file to define and allocate storage to a container. With the configuration given below Robin will allocate a 20GB SSD block device, add it to a container and mount it on the path /opt/data. .. code-block:: yaml storage: - type: data media: ssd path: /opt/data size: 20G .. Note:: If the storage attribute is not specified, then Robin will not allocate any storage for the container. Below is a manifest file for an Nginx bundle with the storage attribute specified: .. code-block:: yaml name: Nginx Server version: "1.12" description: Nginx roles: [nginx] nginx: name: nginx server image: name: nginx version: 1.12-alpine engine: docker compute: memory: 20484M cpu: reserve: false cores: 2 storage: - type: data media: ssd path: /opt/data size: 20G Ephemeral Storage ^^^^^^^^^^^^^^^^^ If non-persistent temporary storage is required by a container, Robin supports the provisioning of Application Ephemeral Volumes (AEVs) which are mounted within the relevant container at a specified path. These volumes are similar to regular volumes in every aspect except for the fact they are removed when a container is deleted or stopped. More details on the concept of AEVs can be found `here `_. AEVs can be specified within the manifest file like any other resource albeit within a different section. With the configuration given below Robin will allocate a 1GB SSD volume without any affinity to the compute host of the container and mount it on the path /mnt/scratch. .. code-block:: yaml ephemeral_storage: - media: SSD size: 1073741824 compute_storage_affinity: false path: "/mnt/scratch" .. Note:: The ephemeral storage attribute is not a mandatory attribute and thus need not be specified. Below is a manifest file for an Nginx bundle with the ephemeral storage attribute specified: .. code-block:: yaml name: Nginx Server version: "1.12" description: Nginx roles: [nginx] nginx: name: nginx server image: name: nginx version: 1.12-alpine engine: docker compute: memory: 20484M cpu: reserve: false cores: 2 storage: - type: data media: ssd path: /opt/data size: 20G ephemeral_storage: - media: SSD size: 1073741824 compute_storage_affinity: false path: "/mnt/scratch" ---------------------------------------------------------------------------------------------- Container Based Environment Variables ---------------------------------------------------------------------------------------------- When deploying an application, the its behavior often needs to be modified in order to run seamlessly given specific environment configurations. Depending on the workload, the defaults associated with it can be updated by editign the application's configuration file, setting up environment variables or invoking the application with certain command line options. This is not an issue for legacy baremetal deployments, however, for docker deployments it is sometimes complicated to pass through runtime parameters especially those driven via user input. To solve this problem Robin ensures variables are available within containers via the environment variable construct. As a result, these environment variables can be used by any process within the container to configure the application. In the guide below we change the default value for worker connections in the configuration file of an Nginx application. **1. Create the manifest file** Construct the below manifest file for the Ngnix Application Bundle. In particular note the values in the ``env`` section: .. code-block:: yaml name: Nginx Server version: "1.12" description: Nginx roles: [nginx] nginx: name: nginx server image: name: nginx version: 1.12-alpine engine: docker compute: memory: 20484M cpu: reserve: false cores: 2 storage: - type: data media: ssd path: /opt/data size: 20G env: WORKER_CONNECTIONS: type: Number value: 4096 label: "Worker Connections" mandatory: true **2. Edit the entrypoint** When the containers are deployed with using the aforementioned manifest file, Robin will set the environment variable WORKER_CONNECTIONS within the container. Now add the below line to the docker entrypoint script to ensure that Nginx configuration file is updated with the new value of worker connections whenever a container is spawned. .. code-block:: text sed -i "/worker_connections =/c\worker_connections ${WORKER_CONNECTIONS}" /etc/nginx/nginx.conf ---------------------------------------------------------------------------------------------- Data Locality Policies ---------------------------------------------------------------------------------------------- Robin enables scaling compute and storage independant of each other. This means it is possible that the compute resources of a container are allocated from one host whilst it's storage is provided by another host. Whilst this is adequate in most scenarios, it can be in problematic for performant applications. As a result, Robin allows users to enforce compute and storage affinity within the manifest itself so every instance of the application has this feature enabled by default. With this option enabled, Robin allocates both storage and compute resources from the same host and thus meets the need for improved performance. This attribute is defined by the ``affinity`` keyword and has three values: host, rack, or datacenter. An example manifest with the ``affinity`` attribute set is shown below: .. code-block:: yaml name: Nginx Server version: "1.12" description: Nginx roles: [nginx] nginx: name: nginx server image: name: nginx version: 1.12-alpine engine: docker compute: memory: 20484M cpu: reserve: false cores: 2 storage: - type: data media: ssd path: /opt/data size: 20G affinity: host env: WORKER_CONNECTIONS: type: Number value: 4096 label: "Worker Connections" mandatory: true ---------------------------------------------------------------------------------------------- Application Variables ---------------------------------------------------------------------------------------------- In certain situations every container that is deployed as part of a workload might need access to certain variables (including application details) for configuration purposes. With the configuration given below, every container will have access to ``var1`` and ``var2``. .. code-block:: yaml appvars: var1: "{APP_NAME}}-val1" var2: "{{APP_NAME}}-val2" Below is a manifest file for an Nginx bundle with application variables specified: .. code-block:: yaml name: Nginx Server version: "1.12" description: Nginx roles: [nginx] appvars: var1: "{APP_NAME}}-val1" var2: "{{APP_NAME}}-val2" nginx: name: nginx server image: name: nginx version: 1.12-alpine engine: docker compute: memory: 20484M cpu: reserve: false cores: 2 storage: - type: data media: ssd path: /opt/data size: 20G affinity: host env: WORKER_CONNECTIONS: type: Number value: 4096 label: "Worker Connections" mandatory: true ---------------------------------------------------------------------------------------------- Network Resources ---------------------------------------------------------------------------------------------- Applications such as Oracle-RAC sometimes require multiple NICs per POD - one private NIC and another public NIC. With the manifest attribute given below, each POD for the respective role can be assigned multiple NICs. .. Note:: In order to actually utilize this option, the underlying Robin installation must be configured to support multiple NICs. For more information on this, please read the installation guide detailed `here `_. .. code-block:: yaml multi_ips: true Below is a manifest file for an Nginx bundle wherein which each pod for the ``nginx server`` role can be assigned multiple NICs: .. code-block:: yaml name: Nginx Server version: "1.12" description: Nginx roles: [nginx] appvars: var1: "{APP_NAME}}-val1" var2: "{{APP_NAME}}-val2" nginx: name: nginx server multi_ips: true image: name: nginx version: 1.12-alpine engine: docker compute: memory: 20484M cpu: reserve: false cores: 2 storage: - type: data media: ssd path: /opt/data size: 20G affinity: host env: WORKER_CONNECTIONS: type: Number value: 4096 label: "Worker Connections" mandatory: true ---------------------------------------------------------------------------------------------- Service Construction ---------------------------------------------------------------------------------------------- In Kubernetes, a Service is an abstraction which defines a logical set of Pods and a policy by which to access them. The set of Pods targeted by a Service is usually determined by a selector, although this is an optional enhancement. Services are useful as they provide abstractions between application stacks with regards to accessing a set a PODs via a static IP Address as opposed to directly contacting the PODs via their own IP Addresses which could change in certain scenarios such the redeployment of a POD. Consider an Elasticsearch application comprised of master and ingest nodes alike. Whilst the actual Pods that compose the backend set may change, the frontend clients like Kibana should not need to be aware of that, nor should they need to keep track of the set of backends themselves. The Service abstraction enables this decoupling by providing the frontend clients a network service they can always query regardless of the state of the backend. Robin supports three different types of services: NodePort, ClusterIP and LoadBalancer. Any combination of these services can be created at the Role level whilst Vnode level services can only be of type NodePort. In order to define a Vnode level service, the ``scope`` attribute must be specified with a value of "pod". If the aforementioned scope is not specified, a Role level service will be created as part of the application creation. In addition, any labels and/or annotations that need to be added to each of these Kubernetes service objects can be specified as key-value pairs within the service definition (see below for an example) via the ``label`` and ``annotations`` attributes. Moreover, alongside the ability to set the protocol (either "TCP", "UDP" or "HTTP") and the corresponding port, the custom static port on which the relevant service will be on exposed on can specified in the definition; namely via the ``node_port`` attribute for both LoadBalancer and NodePort services. If no port is specified, Kubernetes will dynamically select an available port for the respective services to be exposed on. .. Note:: For applications created with IP-Pool(s) backed by the OVS CNI driver, Vnode level services will not be created. This is because in the case of OVS, the subnets of the host on which the pod(s) are deployed will be used. Below is a manifest file for an Nginx bundle wherein a Vnode level NodePort service and two Role level services (one ClusterIP and the other a LoadBalancer) have been defined for "nginx" role: .. code-block:: yaml name: Nginx Server version: "1.12" description: Nginx roles: [nginx] appvars: var1: "{APP_NAME}}-val1" var2: "{{APP_NAME}}-val2" nginx: name: nginx server multi_ips: true image: name: nginx version: 1.12-alpine engine: docker compute: memory: 20484M cpu: reserve: false cores: 2 storage: - type: data media: ssd path: /opt/data size: 20G affinity: host env: WORKER_CONNECTIONS: type: Number value: 4096 label: "Worker Connections" mandatory: true services: - type: NodePort labels: name: demo-nodeport annotations: role: nginx scope: pod ports: - port: 80 protocol: TCP name: web node_port: 30001 target_port: 2000 - type: ClusterIP labels: name: demo-clusterip annotations: role: nginx ports: - port: 80 protocol: TCP name: web - type: LoadBalancer labels: name: demo-loadbalancer annotations: role: nginx ports: - port: 80 protocol: TCP name: web node_port: 30002 Bundle users can also specify custom service names as a part of service definition in bundle manifest configs. Robin Macros are also supported as a part of custom service names specified under ``services`` section. These can be configured by adding ``name`` as a part of the service spec defined in the bundle manifest. When creating bundle applications; K8S services that get created will use the user specified custom service name. This is not a mandatory configuration for service specs defined in the bundle manifest. Users should also be able to create headless services as a part of Robin bundle deployment by specifying ``clusterIP: None`` as a part of the ClusterIP service type definition as shown in the manifest spec below. Apart from headless services; users should also be able to specify custom value for ``clusterIP`` parameter as a part of ClusterIP service spec definition. When user specifies custom value for ``clusterIP``, Robin platform will create clusterIP service with the user specified value during bundle application deployment. By default, K8S picks any random free IP from the cluster CIDR configured during Robin/K8S cluster installation while creating clusterIP type services. .. Note:: Bundle users must make sure that they use the clusterIP custom value from the cluster CIDR subnet configured for their Robin cluster .. code-block:: yaml name: Nginx Server version: "1.12" description: Nginx roles: [nginx] appvars: var1: "{APP_NAME}}-val1" var2: "{{APP_NAME}}-val2" nginx: name: nginx server multi_ips: true image: name: nginx version: 1.12-alpine engine: docker services: - type: NodePort name: "{{APP_NAME}}-np-custom-svc" labels: name: demo-nodeport annotations: role: nginx scope: pod ports: - port: 80 protocol: TCP name: web node_port: 30001 target_port: 2000 - type: ClusterIP clusterIP: None name: cl-custom-svc-name labels: name: demo-clusterip annotations: role: nginx ports: - port: 80 protocol: TCP name: web - type: ClusterIP clusterIP: 172.19.159.99 labels: name: demo-clusterip annotations: role: nginx ports: - port: 80 protocol: TCP name: web - type: LoadBalancer name: lb-custom-svc-name labels: name: demo-loadbalancer annotations: role: nginx ports: - port: 80 protocol: TCP name: web node_port: 30002 ---------------------------------------------------------------------------------------------- Skip Headless Service ---------------------------------------------------------------------------------------------- By default, Robin creates a headless service for every role within an application. With the manifest attribute given below set to true, the creation of the Kubernetes headless service will be skipped when the application is deployed. Below is a manifest file for an Nginx bundle where the ``skip_headless_service`` option is specified global level as well as at the role level. .. code-block:: yaml name: Nginx Server version: "1.12" skip_headless_service: true roles: [nginx] nginx: name: nginx server skip_headless_service: true .. Note:: If the option is specified at the global level it applies to all roles within the bundle. ---------------------------------------------------------------------------------------------- Specifying Sidecar Containers ---------------------------------------------------------------------------------------------- A Kubernetes Pod can be composed of one or more application containers. A sidecar is a utility container in the Pod with the purpose of supporting the main container. Generally, sidecar container is reusable and can be paired with numerous type of main containers. With the configuration given below, a sidecar container named "busyside1" will be created. .. Note:: When defining sidecontainers to be spawned within an Application Bundle, one should define the resource limits and storage in the native Kubernetes manner instead of utilizing the Robin notation. .. code-block:: yaml sidecars: - name: busyside1 image: k8s.gcr.io/busybox command: - sleep - "3000" resources: limits: memory: 200Mi requests: cpu: 100m memory: 100Mi volumeMounts: - name: vol-cm1 mountPath: /cm1 - name: vol-cm2 mountPath: /cm2 - name: vol-cm3 mountPath: /cm3 securityContext: allowPrivilegeEscalation: true Below is a manifest file for an Nginx bundle wherein which a sidecar will be created to access each instance of the ``nginx server`` role: .. code-block:: yaml name: Nginx Server version: "1.12" description: Nginx roles: [nginx] appvars: var1: "{APP_NAME}}-val1" var2: "{{APP_NAME}}-val2" nginx: name: nginx server multi_ips: true image: name: nginx version: 1.12-alpine engine: docker compute: memory: 20484M cpu: reserve: false cores: 2 storage: - type: data media: ssd path: /opt/data size: 20G affinity: host env: WORKER_CONNECTIONS: type: Number value: 4096 label: "Worker Connections" mandatory: true services: - type: NodePort scope: pod ports: - port: 80 protocol: TCP name: web - type: ClusterIP ports: - port: 80 protocol: TCP name: web - type: LoadBalancer ports: - port: 80 protocol: TCP sidecars: - name: busyside1 image: k8s.gcr.io/busybox command: - sleep - "3000" resources: limits: memory: 200Mi requests: cpu: 100m memory: 100Mi volumeMounts: - name: vol-cm1 mountPath: /cm1 - name: vol-cm2 mountPath: /cm2 - name: vol-cm3 mountPath: /cm3 securityContext: allowPrivilegeEscalation: true ---------------------------------------------------------------------------------------------- Dependencies between Roles ---------------------------------------------------------------------------------------------- When deploying multi-tiered applications or applications with several services such as Hadoop, the containers/services needs to be spawned in certain order to successfully start the application. Robin, by default, parallelly creates all the containers necessary for an application stack. In order to override this behavior and specify the right order in which the containers should be created Robin provides the 'depends' attribute in the manifest. This ensures the correct hierarchy is maintained as it enforeces that the containers of particular role are created only after all the containers of the role it depends on are successfully spawned. With the configuration given below, all the containers of ``role2`` will only be created after those of ``role1`` are brought up. .. code-block:: yaml role2: name: role2 depends: [role1] Below is a manifest file for a Wordpress bundle wherein which the containers for the ``wordpress`` role are spawned after those of the ``mysql`` role. .. code-block:: yaml name: WordPress version: "1.0" roles: [mysql, wordpress] serialize: true icon: wordpress-logo-stacked-rgb.png mysql: name: mysql image: name: mysql version: "5.7" engine: docker storage: - media: ssd path: "/var/lib/mysql" type: data wordpress: name: wordpress image: name: robinsys/wordpress version: "1.0" engine: docker depends: [mysql] storage: - media: ssd type: data path: "/var/www/html" ---------------------------------------------------------------------------------------------- Defining Affinity/Anti-Affinity Policies ---------------------------------------------------------------------------------------------- Affinity Rules ^^^^^^^^^^^^^^^ Affinity rules determine the manner in which containers (and their underlying storage volumes) are allocated across the physical infrastructure. These rules inform the orchestrator whether or not particular Vnodes should be placed within the same physical boundary, across separate boundaries, or placed in accordance with infrastructure tags. Each rule is constructed with a set of constraints which can be defined by the ``constraints`` attribute shown below. .. code-block:: yaml affinityrules: - name: "same host" constraints: - type: infrastructure target: host - name: "model a or model b" constraints: - type: tag target: host tags: model: [a, b], - name: "same rack" constraints: - type: infrastructure target: "rack" There are three separate contraints in the above example: - "same host" - All instances affected by this constraint must be created on the same host. - "model a or model b" - All instances affected by this constraint must be created on a host with one of the following tags: ``model=a`` or ``model=b`` - "same rack" - All instances affected by this constraint must be placed within the same rack. As shown above, each constraint comprises of the following fields: - Type: As per its name it is category/type of the constraint. Valid values are either ``tag`` or ``infrastructure``. - Target: The level at which the constraint is enforced. Valid values include ``host``, ``lab``, ``rack``, or ``datacenter``. Note: ``host`` is the only valid target for tag based constraints. - Tags: This field is online valid if the type of the constraint is ``tag``. The target must have (or not have) the tag values specified. Infrastructure based constraints are usually set to enforce that containers grouped by the target specified. As a result, they can be used to ensure that containers are placed within the same host/rack/lab/datacenter. Tag based constraints are usually set to force containers to be placed on certain infrastructure pieces due to the granular level of control they provide. After defining the affinity rules they must be referenced (via the ``affinityrules`` attribute) in the Application Bundle such that they actually impact Vnodes. Detailed below are the two available choices. Intra-Role Affinity ==================== Affinity rules can be set to impact all Vnodes for a particular role whilst leaving instances of other roles constraint free. With the configuration given below, all Vnodes of ``role1`` must conform to the first and second rules we defined in the previous example. .. code-block:: yaml role1: affinityrules: ["same host", "model a or model b"] Inter-Role Affinity ==================== Affinity rules can be set to impact all Vnodes of multiple roles. In order to specify affinity rules between roles the ``roleaffinity`` attribute needs to be utilized. This attribute is defined by three properties: - Name: A descriptor for the affinity rule. This is used mostly as a reference. - Rules: The values specified here are a reference to the rules we originally created - Roles: The values specified here are the names of the roles that are to be constrained by the affinity rule. With the configuration given below, all Vnodes of ``role1`` must be on the same rack as all of the instances of ``role2``. .. code-block:: yaml roleaffinity: - name: "same rack role affinity" affinityrules: ["same rack"] roles: [role1, role2] Anti-Affinity Policies ^^^^^^^^^^^^^^^^^^^^^^ To enforce anti-affinity policies i.e. to ensure containers are spawned on different infrastructure pieces, a hyphen (indicating negation) can be put at the beginning of the target value when defining the constraints. The application of the rules follow the same behavior and structure when implementing affinity policies. An example is given below. .. code-block:: yaml affinityrules: - name: "different host" constraints: - type: infrastructure target: -host The above constraint enforces that any Vnodes associated with it must all be spawned on different hosts. ------------------------------------ Enabling Horizontal Pod Autoscaling ------------------------------------ Horizontal Pod Autoscaling (HPA) is a Kubernetes construct that allows for more efficient utilization of resources and allows for applications (and their respective pods) to be scaled in sync with dynamic usage/load on the application. More information on the advantages and considerations of HPA can be found `here `_. Robin extends the aformentioned Kubernetes HPA infrastructure and allows user to configure the HPA at the role level within the bundle manifest file. If enabled HPA will manipulate every pod deployed for a particular role. .. Note:: This feature is only supported on Robin version 5.2.4 and above. Mandatory Attributes ^^^^^^^^^^^^^^^^^^^^ In order to enable HPA for a particular role, the following attributes need to be set and added within the defintion of the target role. - multinode: true - multinode_max: - multinode_min: (optional, default: 1) - scalein: enabled (If not set, it is enabled by default) - scaleout: enabled (If not set, it is enabled by default) - restart_on_scalein: false - restart_on_scaleout: false - hpa: Examples of what can be specified for the ``hpa`` attribute are detailed in the next section. .. Note:: HPA cannot be enabled for roles which require thier respective pods to be restarted whenever a scale-in or scale-out operation occurs. This is enforced in order to avoid any application downtime due to automated operations driven by the autoscaler. Specifying the HPA configuration ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Detailed below are some examples of rules/metrics with limits that can be set for the autoscaler. If the specified threshold is met or crossed, HPA will kick in and ensure that there are enough pods available to make sure the appropriate constraints are met. For ease of use, Robin utilizes the same syntax as Kubernetes HPA when declaring a configuration. Additional details on the syntax can be found `here `_. CPU usage based autoscaling ============================ The following is an example of a HPA configuration where the average CPU usage maintained across pods will be 80% of 1 CPU. .. Note:: In this example the unit "m" is an abbreviation for milli, so 1000m is equivalent to 1. .. code-block:: yaml hpa: metrics: - type: Pods pods: metric: name: cpu_usage target: type: AverageValue averageValue: 800m Below is a manifest file for a Wordpress bundle wherein which the ``mysql`` role is autoscaled (up to a maximum of 5 pods) based on the CPU usage of its spawned instances. .. code-block:: yaml name: WordPress version: "1.0" roles: [mysql, wordpress] serialize: true icon: wordpress-logo-stacked-rgb.png mysql: multinode: True multinode_min: 1 multinode_max: 5 restart_on_scalein: false restart_on_scaleout: false name: mysql image: name: mysql version: "5.7" engine: docker storage: - media: ssd path: "/var/lib/mysql" type: data hpa: metrics: - type: Pods pods: metric: name: cpu_usage target: type: AverageValue averageValue: 800m Memory usage based autoscaling =============================== The following is an example of a HPA configuration where the average memory usage maintained across pods will be 50% of total pod memory. .. code-block:: yaml hpa: metrics: - type: Resource resource: name: memory target: type: Utilization averageUtilization: 50 Below is a manifest file for a Wordpress bundle wherein which the ``mysql`` role is autoscaled (up to a maximum of 5 pods) based on the memory usage of its spawned instances. .. code-block:: yaml name: WordPress version: "1.0" roles: [mysql, wordpress] serialize: true icon: wordpress-logo-stacked-rgb.png mysql: multinode: True multinode_min: 1 multinode_max: 5 restart_on_scalein: false restart_on_scaleout: false name: mysql image: name: mysql version: "5.7" engine: docker storage: - media: ssd path: "/var/lib/mysql" type: data hpa: metrics: - type: Resource resource: name: memory target: type: Utilization averageUtilization: 50 Custom metrics based autoscaling ================================= The following is an example of a HPA configuration which is based off a custom metric that exposes the number of requests handled per second per pod. With the configuration below HPA will try to maintain an average of 1 request per second for all pods. .. Note:: In this example 1000m is the equivalent to a single request. .. code-block:: yaml hpa: metrics: - type: Pods pods: metric: name: http_requests_per_second target: type: AverageValue averageValue: 1000m Below is a manifest file for a Wordpress bundle wherein which the ``mysql`` role is autoscaled (up to a maximum of 5 pods) based on the number of requests handled per second per pod. .. code-block:: yaml name: WordPress version: "1.0" roles: [mysql, wordpress] serialize: true icon: wordpress-logo-stacked-rgb.png mysql: multinode: True multinode_min: 1 multinode_max: 5 restart_on_scalein: false restart_on_scaleout: false name: mysql image: name: mysql version: "5.7" engine: docker storage: - media: ssd path: "/var/lib/mysql" type: data hpa: metrics: - type: Pods pods: metric: name: http_requests_per_second target: type: AverageValue averageValue: 1000m Combining constraining metrics =============================== The above defined metrics can be combined by simply specifying them one after the other in the manifest within the ``metrics`` section. The following is an example of a HPA configuration that will try to maintain an average of 1 request per second for all pods as well as ensuring that the average CPU usage maintained across pods will be 80% of 1 CPU. .. Note:: When combining metrics the number of desired replicas is calculated for each metric independently and the max of all the calculated replicas is set as the target to be reached with regard to autoscaling. .. code-block:: yaml hpa: metrics: - type: Pods pods: metric: name: http_requests_per_second target: type: AverageValue averageValue: 1000m - type: Pods pods: metric: name: cpu_usage target: type: AverageValue averageValue: 800m Below is a manifest file for a Wordpress bundle wherein which the ``mysql`` role is autoscaled (up to a maximum of 5 pods) based on the number of requests handled per second per pod as well as the average CPU usage across all pods. .. code-block:: yaml name: WordPress version: "1.0" roles: [mysql, wordpress] serialize: true icon: wordpress-logo-stacked-rgb.png mysql: multinode: True multinode_min: 1 multinode_max: 5 restart_on_scalein: false restart_on_scaleout: false name: mysql image: name: mysql version: "5.7" engine: docker storage: - media: ssd path: "/var/lib/mysql" type: data hpa: metrics: - type: Pods pods: metric: name: http_requests_per_second target: type: AverageValue averageValue: 1000m - type: Pods pods: metric: name: cpu_usage target: type: AverageValue averageValue: 800m Available Metrics ^^^^^^^^^^^^^^^^^^ By default a certain set of metrics are made available for the Kubernetes HPA to leverage. The complete list of readily available metrics can be found by running the following command: ``kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1``. Detailed below are some pod metrics that can be utilized: - cpu_load_average_10s - memory_rss - spec_cpu_shares - threads - cpu_system - fs_writes - spec_cpu_period - spec_memory_reservation_limit_bytes - fs_writes_bytes - memory_failcnt - spec_memory_limit_bytes - start_time_seconds - fs_reads_bytes - last_seen - cpu_cfs_throttled - cpu_user - file_descriptors - tasks_state - memory_usage_bytes - processes - memory_mapped_file - memory_swap - memory_max_usage_bytes - threads_max - spec_cpu_quota - cpu_usage - cpu_cfs_throttled_periods - memory_failures - cpu_cfs_periods - memory_cache - spec_memory_swap_limit_bytes - fs_reads - memory_working_set_bytes - sockets Creating custom metrics ^^^^^^^^^^^^^^^^^^^^^^^^ In certain scenarios, there might be no pre-determined metric that is suitable for a particular use case. In order to solve this a metric will have to be added manually. Detailed below is a set of steps that can be followed in order to create a custom metric that can be used by HPA and is scraped by Prometheus. **Deploy Prometheus** In order to actually process the metric to be defined we will utilize Prometheus as it provides an easy mechanism via which to expose metrics that need to be utilized. Robin enables users to deploy Prometheus in a covienent manner as the application is used as a general monitoring solution for Robin. Issue the following command to deploy Prometheus: .. code-block:: text # robin metrics start --media HDD --resource-pool default --faultdomain disk --replicas 3 --retention-period 2 Job: 74 Name: MetricsStartCollect State: PREPARED Error: 0 Job: 74 Name: MetricsStartCollect State: DONE Error: 0 Job: 74 Name: MetricsStartCollect State: AGENT_WAIT Error: 0 Job: 74 Name: MetricsStartCollect State: NOTIFIED Error: 0 Job: 74 Name: MetricsStartCollect State: COMPLETED Error: 0 .. Note:: More details on Prometheus support within Robin are provided `here `_. **Expose the necessary metrics** The metrics that need to be scraped by Prometheus, should be exposed from within the application at the ``/metrics`` endpoint. For more details on standard procdures and formats with regards to exposing metrics , review the guide `here `_. **Instruct Prometheus to scrape the exposed metrics** In order to actually have the metrics that are now exposed by the application scraped by Prometheus, annotations will have to be specified for each pod within the target role. This can be easily achieved by specifying the annotations at the role level within the Application Bundle manifest. The following is an example of the necessary annotations: .. code-block:: yaml annotations: prometheus.io/scrape: "\"true\"" prometheus.io/port: "\"8080\"" prometheus.io/scheme: "http" Below is a manifest file for a Wordpress bundle wherein which all the pods spawned as part of the ``mysql`` role will have the specified annotations. .. code-block:: yaml name: WordPress version: "1.0" roles: [mysql, wordpress] serialize: true icon: wordpress-logo-stacked-rgb.png mysql: multinode: True multinode_min: 1 multinode_max: 5 restart_on_scalein: false restart_on_scaleout: false name: mysql image: name: mysql version: "5.7" engine: docker storage: - media: ssd path: "/var/lib/mysql" type: data annotations: prometheus.io/scrape: "\"true\"" prometheus.io/port: "\"8080\"" prometheus.io/scheme: "http" **Update the prometheus-adapter configmap** The metric will have to also be defined in the ``prometheus-adapter`` configmap within the ``kube-system`` namespace. As a result, the configmap will have to be updated with a configuration that contains the query which summarizes the metric you want to use. The following is an example configuration of the ``http_requests_per_second`` metric we utilized above. .. code-block:: yaml - seriesQuery: 'http_requests_total' resources: overrides: kubernetes_namespace: resource: namespace kubernetes_pod_name: resource: pod name: matches: "^(.*)_total" as: "${1}_per_second" metricsQuery: (sum(rate(<<.Series>>{<<.LabelMatchers>>}[1m])) by (<<.GroupBy>>)) .. Note:: For more information on the rules constraining configuration specification, read the documentation `here `_. In order to confirm that your metric has been successfully added, verify it is displayed in the output of ``kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1``. For the configuration specified, an example is shown below: .. code-block:: text # kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1 | jq | grep pod | grep http_requests_per_second "name": "pods/http_requests_per_second", .. Note:: The metric name here does not match the one specified above as prometheus converts absolute values into a rate i.e. total per second which in turn modifies the metric name to make it more self-explanatory. **Verify the metrics are available** In order for the metric to actually be useful and reflect the current state of the pod, Prometheus needs to be able to scrape the information exposed in the previous step of this guide. In order to confirm this is the case, visit the Prometheus endpoint and make sure the metric you have exposed is available for use. Issue the following command to attain the Prometheus endpoint: .. code-block:: text # robin metrics status Prometheus: -- Status........ Running -- URL........... http://10.10.2.33:30423 Grafana: -- Status........ Running -- URL........... https://10.10.2.33:32533 **Utilize the metric** The metric can now be specified for use within the manifest as if it were a standard metric exposed to the Kubernetes HPA.