3. Configuring Bundles

Robin’s manifest file has several attributes that can be used to manipulate aspects of a container including the compute and storage resources allocated to the containers, the order in which containers are spawned etc. This section will cover the most commonly used attributes via a generic example alongside a real world use case. A complete list of attributes can be found here here.

3.1. Image

ROBIN bundle allows users to specify image and runtime engine options for a role. The mandatory options for image are

  • name: Name of the image.

  • version: Version of the image

  • engine: Type of runtime (docker, kvm, lxc)

There are further options available at the image section level specific to engine.

Note

For the docker runtime, the combination of the name and version attributes should point to an image in the registry. With regard to the other runtime operations, the name and version attributes of the image should match those used when the image was registered with Robin via the robin image add command.

The following is an example of image attributes available for a KVM based application bundle.

image:
    name: vnf1
    version: "3.0-1094"
    engine: kvm
    ostype: linux
    osvariant: rhel7
    graphics: "none"
    imgsize: "40G"

3.2. Compute Resources

Compute resources such as memory, CPU, hugepages, GPU etc are allocated at the container level. When specifying the number of cores or millicores (fractional value of a CPU) for the CPU(s), you can also specify the type of CPUs that should be picked; options include: non isolated, isolated-shared, isolated-dedicated. Their distinctions and how to specify them are detailed below:

  • Non Isolated - This option indicates that the physical CPUs to be used for an actual deployment of the bundle should be from the non-isolated pool of CPUs on a host. It can be specified by setting the nonisol attribute to true and the reserve attribute to false.

  • Isolated Shared - This option indicates that the physical CPUs to be used for an actual deployment of the bundle should be from the isolated pool of CPUs on a host. With this option, eventhough the allocated CPU(s) are isolated from kernel processes, it can still be utilized by other application deployments (hence the term sharing). It can be specified by setting the isol attribute to true and the reserve attribute to false.

  • Isolated Dedicated - This option indicates that the physical CPUs to be used for an actual deployment of the bundle should be from the isolated pool of CPUs on a host. With this option, the allocated CPU(s) are not only isolated from kernel processes, but also other application deployments (hence the term dedicated). It can be specified by setting both the isol and reserve attributes to true.

In addition, when specifying GPU requests, the GPU resource type can be specified. If no type is specified then only non-partitioned GPU devices will be selected for allocation. As a result, if MIG devices need to be allocated for a container the resource type must be specified.

With the configuration given below Robin will allocate 2GB memory, 4.03 isolated-shared CPU cores, hugepages and a single GPU (matching the type specified) to a container.

compute:
    memory: 2048M
    hugepages_1gi: 1G
    hugepages_2mi: 1G
    cpu:
        reserve: false
        cores: 4.03
        isol: true
    gpu:
        count: 1
        type: "nvidia-a100"

If no values are specified for the compute attribute in the manifest, Robin will allocate 1GB of Memory and 1 non-isolated CPU core (as they are default values) to the relevant containers.

Below is a manifest file for an Nginx bundle with the compute attribute specified:

name: Nginx Server
version: "1.12"
description: Nginx
roles: [nginx]
nginx:
   name: "nginx server"
   image:
       name: nginx
       version: 1.12-alpine
       engine: docker
   compute:
       memory: 2048M
       cpu:
           reserve: false
           cores: 2
           isol: true

Below is from a manifest file for a KVM based vRAN bundle with the compute attributes specified. The following application requires:

  • 18 cores: out of which Core IDs 2-17 should be mapped to RT (Realtime) scheduling priority and FIFO policy

  • 24G in 1Gi-Hugepages that should not be shared and should be locked

  • KVM memory balloon device with the specified attibutes

  • KVM emulatorpin cpuset outside the CPUset allocated to the KVM POD in order to ensure the emulator does not interfere with the application

  • 4 cache ways allocated exclusively for this POD’s cpuset (Intel Cache Allocation Technology)

compute:
    memory: 4M
    hugepages_1gi: 24G
    hugepages_locked: "yes"
    hugepages_noshared: "yes"
    memballoon: "model=none,stats.period=10"
    cpu:
        reserve: true
        isol: true
        cores: 18
        rt_cores: 2-17
        rt_sched: fifo
        rt_priority: 1
        emulatorpin: outside_cpuset
        cache_ways: 4
        kvmopts: "mode=host-passthrough,require=tsc-deadline,check=none"

Note

The following attributes will only work on Realtime kernel: rt_cores, rt_sched, rt_priority. For cache_ways, the profiles/masks for each NUMA node should be configured beforehand.

3.3. Storage Resources

Robin offers several attributes within the manifest file to define and allocate storage to a container. With the configuration given below Robin will allocate a 20GB SSD block device, add it to a container and mount it on the path /opt/data.

storage:
    - type: data
      media: ssd
      path: /opt/data
      size: 20G

Note

If the storage attribute is not specified, then Robin will not allocate any storage for the container.

Below is a manifest file for an Nginx bundle with the storage attribute specified:

name: Nginx Server
version: "1.12"
description: Nginx
roles: [nginx]
nginx:
   name: nginx server
   image:
       name: nginx
       version: 1.12-alpine
       engine: docker
   compute:
       memory: 20484M
       cpu:
           reserve: false
           cores: 2
   storage:
       - type: data
         media: ssd
         path: /opt/data
         size: 20G

3.3.1. Ephemeral Storage

If non-persistent temporary storage is required by a container, Robin supports the provisioning of Application Ephemeral Volumes (AEVs) which are mounted within the relevant container at a specified path. These volumes are similar to regular volumes in every aspect except for the fact they are removed when a container is deleted or stopped. More details on the concept of AEVs can be found here. AEVs can be specified within the manifest file like any other resource albeit within a different section. With the configuration given below Robin will allocate a 1GB SSD volume without any affinity to the compute host of the container and mount it on the path /mnt/scratch.

ephemeral_storage:
    - media: SSD
      size: 1073741824
      compute_storage_affinity: false
      path: "/mnt/scratch"

Note

The ephemeral storage attribute is not a mandatory attribute and thus need not be specified.

Below is a manifest file for an Nginx bundle with the ephemeral storage attribute specified:

name: Nginx Server
version: "1.12"
description: Nginx
roles: [nginx]
nginx:
   name: nginx server
   image:
       name: nginx
       version: 1.12-alpine
       engine: docker
   compute:
       memory: 20484M
       cpu:
           reserve: false
           cores: 2
   storage:
       - type: data
         media: ssd
         path: /opt/data
         size: 20G
   ephemeral_storage:
       - media: SSD
         size: 1073741824
         compute_storage_affinity: false
         path: "/mnt/scratch"

3.4. Container Based Environment Variables

When deploying an application, the its behavior often needs to be modified in order to run seamlessly given specific environment configurations. Depending on the workload, the defaults associated with it can be updated by editign the application’s configuration file, setting up environment variables or invoking the application with certain command line options. This is not an issue for legacy baremetal deployments, however, for docker deployments it is sometimes complicated to pass through runtime parameters especially those driven via user input. To solve this problem Robin ensures variables are available within containers via the environment variable construct. As a result, these environment variables can be used by any process within the container to configure the application.

In the guide below we change the default value for worker connections in the configuration file of an Nginx application.

1. Create the manifest file

Construct the below manifest file for the Ngnix Application Bundle. In particular note the values in the env section:

name: Nginx Server
version: "1.12"
description: Nginx
roles: [nginx]
nginx:
   name: nginx server
   image:
       name: nginx
       version: 1.12-alpine
       engine: docker
   compute:
       memory: 20484M
       cpu:
           reserve: false
           cores: 2
   storage:
       - type: data
         media: ssd
         path: /opt/data
         size: 20G
   env:
       WORKER_CONNECTIONS:
           type: Number
           value: 4096
           label: "Worker Connections"
           mandatory: true

2. Edit the entrypoint

When the containers are deployed with using the aforementioned manifest file, Robin will set the environment variable WORKER_CONNECTIONS within the container. Now add the below line to the docker entrypoint script to ensure that Nginx configuration file is updated with the new value of worker connections whenever a container is spawned.

sed -i "/worker_connections  =/c\worker_connections ${WORKER_CONNECTIONS}" /etc/nginx/nginx.conf

3.5. Data Locality Policies

Robin enables scaling compute and storage independant of each other. This means it is possible that the compute resources of a container are allocated from one host whilst it’s storage is provided by another host. Whilst this is adequate in most scenarios, it can be in problematic for performant applications. As a result, Robin allows users to enforce compute and storage affinity within the manifest itself so every instance of the application has this feature enabled by default. With this option enabled, Robin allocates both storage and compute resources from the same host and thus meets the need for improved performance.

This attribute is defined by the affinity keyword and has three values: host, rack, or datacenter. An example manifest with the affinity attribute set is shown below:

name: Nginx Server
version: "1.12"
description: Nginx
roles: [nginx]
nginx:
   name: nginx server
   image:
       name: nginx
       version: 1.12-alpine
       engine: docker
   compute:
       memory: 20484M
       cpu:
           reserve: false
           cores: 2
   storage:
       - type: data
         media: ssd
         path: /opt/data
         size: 20G
         affinity: host
   env:
       WORKER_CONNECTIONS:
           type: Number
           value: 4096
           label: "Worker Connections"
           mandatory: true

3.6. Application Variables

In certain situations every container that is deployed as part of a workload might need access to certain variables (including application details) for configuration purposes. With the configuration given below, every container will have access to var1 and var2.

appvars:
    var1: "{APP_NAME}}-val1"
    var2: "{{APP_NAME}}-val2"

Below is a manifest file for an Nginx bundle with application variables specified:

name: Nginx Server
version: "1.12"
description: Nginx
roles: [nginx]
appvars:
    var1: "{APP_NAME}}-val1"
    var2: "{{APP_NAME}}-val2"
nginx:
   name: nginx server
   image:
       name: nginx
       version: 1.12-alpine
       engine: docker
   compute:
       memory: 20484M
       cpu:
           reserve: false
           cores: 2
   storage:
       - type: data
         media: ssd
         path: /opt/data
         size: 20G
         affinity: host
   env:
       WORKER_CONNECTIONS:
           type: Number
           value: 4096
           label: "Worker Connections"
           mandatory: true

3.7. Network Resources

Applications such as Oracle-RAC sometimes require multiple NICs per POD - one private NIC and another public NIC. With the manifest attribute given below, each POD for the respective role can be assigned multiple NICs.

Note

In order to actually utilize this option, the underlying Robin installation must be configured to support multiple NICs. For more information on this, please read the installation guide detailed here.

multi_ips: true

Below is a manifest file for an Nginx bundle wherein which each pod for the nginx server role can be assigned multiple NICs:

name: Nginx Server
version: "1.12"
description: Nginx
roles: [nginx]
appvars:
    var1: "{APP_NAME}}-val1"
    var2: "{{APP_NAME}}-val2"
nginx:
   name: nginx server
   multi_ips: true
   image:
       name: nginx
       version: 1.12-alpine
       engine: docker
   compute:
       memory: 20484M
       cpu:
           reserve: false
           cores: 2
   storage:
       - type: data
         media: ssd
         path: /opt/data
         size: 20G
         affinity: host
   env:
       WORKER_CONNECTIONS:
           type: Number
           value: 4096
           label: "Worker Connections"
           mandatory: true

3.8. Service Construction

In Kubernetes, a Service is an abstraction which defines a logical set of Pods and a policy by which to access them. The set of Pods targeted by a Service is usually determined by a selector, although this is an optional enhancement. Services are useful as they provide abstractions between application stacks with regards to accessing a set a PODs via a static IP Address as opposed to directly contacting the PODs via their own IP Addresses which could change in certain scenarios such the redeployment of a POD.

Consider an Elasticsearch application comprised of master and ingest nodes alike. Whilst the actual Pods that compose the backend set may change, the frontend clients like Kibana should not need to be aware of that, nor should they need to keep track of the set of backends themselves. The Service abstraction enables this decoupling by providing the frontend clients a network service they can always query regardless of the state of the backend.

Robin supports three different types of services: NodePort, ClusterIP and LoadBalancer. Any combination of these services can be created at the Role level whilst Vnode level services can only be of type NodePort. In order to define a Vnode level service, the scope attribute must be specified with a value of “pod”. If the aforementioned scope is not specified, a Role level service will be created as part of the application creation. In addition, any labels and/or annotations that need to be added to each of these Kubernetes service objects can be specified as key-value pairs within the service definition (see below for an example) via the label and annotations attributes. Moreover, alongside the ability to set the protocol (either “TCP”, “UDP” or “HTTP”) and the corresponding port, the custom static port on which the relevant service will be on exposed on can specified in the definition; namely via the node_port attribute for both LoadBalancer and NodePort services. If no port is specified, Kubernetes will dynamically select an available port for the respective services to be exposed on.

Note

For applications created with IP-Pool(s) backed by the OVS CNI driver, Vnode level services will not be created. This is because in the case of OVS, the subnets of the host on which the pod(s) are deployed will be used.

Below is a manifest file for an Nginx bundle wherein a Vnode level NodePort service and two Role level services (one ClusterIP and the other a LoadBalancer) have been defined for “nginx” role:

name: Nginx Server
version: "1.12"
description: Nginx
roles: [nginx]
appvars:
    var1: "{APP_NAME}}-val1"
    var2: "{{APP_NAME}}-val2"
nginx:
   name: nginx server
   multi_ips: true
   image:
       name: nginx
       version: 1.12-alpine
       engine: docker
   compute:
       memory: 20484M
       cpu:
           reserve: false
           cores: 2
   storage:
       - type: data
         media: ssd
         path: /opt/data
         size: 20G
         affinity: host
   env:
       WORKER_CONNECTIONS:
           type: Number
           value: 4096
           label: "Worker Connections"
           mandatory: true
   services:
   - type: NodePort
     labels:
       name: demo-nodeport
     annotations:
       role: nginx
     scope: pod
     ports:
     - port: 80
       protocol: TCP
       name: web
       node_port: 30001
       target_port: 2000
   - type: ClusterIP
     labels:
       name: demo-clusterip
     annotations:
       role: nginx
     ports:
     - port: 80
       protocol: TCP
       name: web
   - type: LoadBalancer
     labels:
       name: demo-loadbalancer
     annotations:
       role: nginx
     ports:
     - port: 80
       protocol: TCP
       name: web
       node_port: 30002

Bundle users can also specify custom service names as a part of service definition in bundle manifest configs. Robin Macros are also supported as a part of custom service names specified under services section. These can be configured by adding name as a part of the service spec defined in the bundle manifest. When creating bundle applications; K8S services that get created will use the user specified custom service name. This is not a mandatory configuration for service specs defined in the bundle manifest. Users should also be able to create headless services as a part of Robin bundle deployment by specifying clusterIP: None as a part of the ClusterIP service type definition as shown in the manifest spec below. Apart from headless services; users should also be able to specify custom value for clusterIP parameter as a part of ClusterIP service spec definition. When user specifies custom value for clusterIP, Robin platform will create clusterIP service with the user specified value during bundle application deployment. By default, K8S picks any random free IP from the cluster CIDR configured during Robin/K8S cluster installation while creating clusterIP type services.

Note

Bundle users must make sure that they use the clusterIP custom value from the cluster CIDR subnet configured for their Robin cluster

name: Nginx Server
version: "1.12"
description: Nginx
roles: [nginx]
appvars:
    var1: "{APP_NAME}}-val1"
    var2: "{{APP_NAME}}-val2"
nginx:
   name: nginx server
   multi_ips: true
   image:
       name: nginx
       version: 1.12-alpine
       engine: docker
   services:
   - type: NodePort
     name: "{{APP_NAME}}-np-custom-svc"
     labels:
       name: demo-nodeport
     annotations:
       role: nginx
     scope: pod
     ports:
     - port: 80
       protocol: TCP
       name: web
       node_port: 30001
       target_port: 2000
   - type: ClusterIP
     clusterIP: None
     name: cl-custom-svc-name
     labels:
       name: demo-clusterip
     annotations:
       role: nginx
     ports:
     - port: 80
       protocol: TCP
       name: web
   - type: ClusterIP
     clusterIP: 172.19.159.99
     labels:
       name: demo-clusterip
     annotations:
       role: nginx
     ports:
     - port: 80
       protocol: TCP
       name: web
   - type: LoadBalancer
     name: lb-custom-svc-name
     labels:
       name: demo-loadbalancer
     annotations:
       role: nginx
     ports:
     - port: 80
       protocol: TCP
       name: web
       node_port: 30002

3.9. Skip Headless Service

By default, Robin creates a headless service for every role within an application. With the manifest attribute given below set to true, the creation of the Kubernetes headless service will be skipped when the application is deployed.

Below is a manifest file for an Nginx bundle where the skip_headless_service option is specified global level as well as at the role level.

name: Nginx Server
version: "1.12"
skip_headless_service: true
roles: [nginx]
nginx:
   name: nginx server
   skip_headless_service: true

Note

If the option is specified at the global level it applies to all roles within the bundle.

3.10. Specifying Sidecar Containers

A Kubernetes Pod can be composed of one or more application containers. A sidecar is a utility container in the Pod with the purpose of supporting the main container. Generally, sidecar container is reusable and can be paired with numerous type of main containers. With the configuration given below, a sidecar container named “busyside1” will be created.

Note

When defining sidecontainers to be spawned within an Application Bundle, one should define the resource limits and storage in the native Kubernetes manner instead of utilizing the Robin notation.

sidecars:
- name: busyside1
  image: k8s.gcr.io/busybox
  command:
    - sleep
    - "3000"
  resources:
    limits:
      memory: 200Mi
    requests:
      cpu: 100m
      memory: 100Mi
  volumeMounts:
  - name: vol-cm1
    mountPath: /cm1
  - name: vol-cm2
    mountPath: /cm2
  - name: vol-cm3
    mountPath: /cm3
  securityContext:
    allowPrivilegeEscalation: true

Below is a manifest file for an Nginx bundle wherein which a sidecar will be created to access each instance of the nginx server role:

name: Nginx Server
version: "1.12"
description: Nginx
roles: [nginx]
appvars:
    var1: "{APP_NAME}}-val1"
    var2: "{{APP_NAME}}-val2"
nginx:
   name: nginx server
   multi_ips: true
   image:
       name: nginx
       version: 1.12-alpine
       engine: docker
   compute:
       memory: 20484M
       cpu:
           reserve: false
           cores: 2
   storage:
       - type: data
         media: ssd
         path: /opt/data
         size: 20G
         affinity: host
   env:
       WORKER_CONNECTIONS:
           type: Number
           value: 4096
           label: "Worker Connections"
           mandatory: true
   services:
       - type: NodePort
         scope: pod
         ports:
         - port: 80
           protocol: TCP
           name: web
       - type: ClusterIP
         ports:
         - port: 80
           protocol: TCP
           name: web
       - type: LoadBalancer
         ports:
         - port: 80
           protocol: TCP
         sidecars:
         - name: busyside1
           image: k8s.gcr.io/busybox
           command:
             - sleep
             - "3000"
           resources:
             limits:
               memory: 200Mi
             requests:
               cpu: 100m
               memory: 100Mi
           volumeMounts:
           - name: vol-cm1
             mountPath: /cm1
           - name: vol-cm2
             mountPath: /cm2
           - name: vol-cm3
             mountPath: /cm3
     securityContext:
       allowPrivilegeEscalation: true

3.11. Dependencies between Roles

When deploying multi-tiered applications or applications with several services such as Hadoop, the containers/services needs to be spawned in certain order to successfully start the application. Robin, by default, parallelly creates all the containers necessary for an application stack. In order to override this behavior and specify the right order in which the containers should be created Robin provides the ‘depends’ attribute in the manifest. This ensures the correct hierarchy is maintained as it enforeces that the containers of particular role are created only after all the containers of the role it depends on are successfully spawned. With the configuration given below, all the containers of role2 will only be created after those of role1 are brought up.

role2:
    name: role2
    depends: [role1]

Below is a manifest file for a Wordpress bundle wherein which the containers for the wordpress role are spawned after those of the mysql role.

name: WordPress
version: "1.0"
roles: [mysql, wordpress]
serialize: true
icon: wordpress-logo-stacked-rgb.png

mysql:
    name: mysql
    image:
        name: mysql
        version: "5.7"
        engine: docker
    storage:
        - media: ssd
          path: "/var/lib/mysql"
          type: data

wordpress:
  name: wordpress
  image:
        name: robinsys/wordpress
        version: "1.0"
        engine: docker
  depends: [mysql]
  storage:
      - media: ssd
        type: data
        path: "/var/www/html"

3.12. Defining Affinity/Anti-Affinity Policies

3.12.1. Affinity Rules

Affinity rules determine the manner in which containers (and their underlying storage volumes) are allocated across the physical infrastructure. These rules inform the orchestrator whether or not particular Vnodes should be placed within the same physical boundary, across separate boundaries, or placed in accordance with infrastructure tags. Each rule is constructed with a set of constraints which can be defined by the constraints attribute shown below.

affinityrules:
   - name: "same host"
      constraints:
         - type: infrastructure
           target: host
   - name: "model a or model b"
      constraints:
         - type: tag
           target: host
           tags:
               model: [a, b],
    - name: "same rack"
      constraints:
          - type: infrastructure
            target: "rack"

There are three separate contraints in the above example:

  • “same host” - All instances affected by this constraint must be created on the same host.

  • “model a or model b” - All instances affected by this constraint must be created on a host with one of the following tags: model=a or model=b

  • “same rack” - All instances affected by this constraint must be placed within the same rack.

As shown above, each constraint comprises of the following fields:

  • Type: As per its name it is category/type of the constraint. Valid values are either tag or infrastructure.

  • Target: The level at which the constraint is enforced. Valid values include host, lab, rack, or datacenter. Note: host is the only valid target for tag based constraints.

  • Tags: This field is online valid if the type of the constraint is tag. The target must have (or not have) the tag values specified.

Infrastructure based constraints are usually set to enforce that containers grouped by the target specified. As a result, they can be used to ensure that containers are placed within the same host/rack/lab/datacenter.

Tag based constraints are usually set to force containers to be placed on certain infrastructure pieces due to the granular level of control they provide.

After defining the affinity rules they must be referenced (via the affinityrules attribute) in the Application Bundle such that they actually impact Vnodes. Detailed below are the two available choices.

3.12.1.1. Intra-Role Affinity

Affinity rules can be set to impact all Vnodes for a particular role whilst leaving instances of other roles constraint free. With the configuration given below, all Vnodes of role1 must conform to the first and second rules we defined in the previous example.

role1:
    affinityrules: ["same host", "model a or model b"]

3.12.1.2. Inter-Role Affinity

Affinity rules can be set to impact all Vnodes of multiple roles. In order to specify affinity rules between roles the roleaffinity attribute needs to be utilized. This attribute is defined by three properties: - Name: A descriptor for the affinity rule. This is used mostly as a reference. - Rules: The values specified here are a reference to the rules we originally created - Roles: The values specified here are the names of the roles that are to be constrained by the affinity rule.

With the configuration given below, all Vnodes of role1 must be on the same rack as all of the instances of role2.

roleaffinity:
    - name: "same rack role affinity"
      affinityrules: ["same rack"]
      roles: [role1, role2]

3.12.2. Anti-Affinity Policies

To enforce anti-affinity policies i.e. to ensure containers are spawned on different infrastructure pieces, a hyphen (indicating negation) can be put at the beginning of the target value when defining the constraints. The application of the rules follow the same behavior and structure when implementing affinity policies. An example is given below.

affinityrules:
   - name: "different host"
      constraints:
         - type: infrastructure
           target: -host

The above constraint enforces that any Vnodes associated with it must all be spawned on different hosts.

3.13. Enabling Horizontal Pod Autoscaling

Horizontal Pod Autoscaling (HPA) is a Kubernetes construct that allows for more efficient utilization of resources and allows for applications (and their respective pods) to be scaled in sync with dynamic usage/load on the application. More information on the advantages and considerations of HPA can be found here. Robin extends the aformentioned Kubernetes HPA infrastructure and allows user to configure the HPA at the role level within the bundle manifest file. If enabled HPA will manipulate every pod deployed for a particular role.

Note

This feature is only supported on Robin version 5.2.4 and above.

3.13.1. Mandatory Attributes

In order to enable HPA for a particular role, the following attributes need to be set and added within the defintion of the target role.

  • multinode: true

  • multinode_max: <max number of pods HPA can create for the role>

  • multinode_min: <min number of pods HPA needs to maintain for the role> (optional, default: 1)

  • scalein: enabled (If not set, it is enabled by default)

  • scaleout: enabled (If not set, it is enabled by default)

  • restart_on_scalein: false

  • restart_on_scaleout: false

  • hpa: <list of metrics and their target values for HPA to act on>

Examples of what can be specified for the hpa attribute are detailed in the next section.

Note

HPA cannot be enabled for roles which require thier respective pods to be restarted whenever a scale-in or scale-out operation occurs. This is enforced in order to avoid any application downtime due to automated operations driven by the autoscaler.

3.13.2. Specifying the HPA configuration

Detailed below are some examples of rules/metrics with limits that can be set for the autoscaler. If the specified threshold is met or crossed, HPA will kick in and ensure that there are enough pods available to make sure the appropriate constraints are met. For ease of use, Robin utilizes the same syntax as Kubernetes HPA when declaring a configuration. Additional details on the syntax can be found here.

3.13.2.1. CPU usage based autoscaling

The following is an example of a HPA configuration where the average CPU usage maintained across pods will be 80% of 1 CPU.

Note

In this example the unit “m” is an abbreviation for milli, so 1000m is equivalent to 1.

hpa:
  metrics:
  - type: Pods
    pods:
      metric:
        name: cpu_usage
      target:
        type: AverageValue
        averageValue: 800m

Below is a manifest file for a Wordpress bundle wherein which the mysql role is autoscaled (up to a maximum of 5 pods) based on the CPU usage of its spawned instances.

name: WordPress
version: "1.0"
roles: [mysql, wordpress]
serialize: true
icon: wordpress-logo-stacked-rgb.png

mysql:
    multinode: True
    multinode_min: 1
    multinode_max: 5
    restart_on_scalein: false
    restart_on_scaleout: false
    name: mysql
    image:
        name: mysql
        version: "5.7"
        engine: docker
    storage:
        - media: ssd
          path: "/var/lib/mysql"
          type: data
    hpa:
      metrics:
      - type: Pods
        pods:
          metric:
            name: cpu_usage
          target:
            type: AverageValue
            averageValue: 800m

3.13.2.2. Memory usage based autoscaling

The following is an example of a HPA configuration where the average memory usage maintained across pods will be 50% of total pod memory.

hpa:
  metrics:
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 50

Below is a manifest file for a Wordpress bundle wherein which the mysql role is autoscaled (up to a maximum of 5 pods) based on the memory usage of its spawned instances.

name: WordPress
version: "1.0"
roles: [mysql, wordpress]
serialize: true
icon: wordpress-logo-stacked-rgb.png

mysql:
    multinode: True
    multinode_min: 1
    multinode_max: 5
    restart_on_scalein: false
    restart_on_scaleout: false
    name: mysql
    image:
        name: mysql
        version: "5.7"
        engine: docker
    storage:
        - media: ssd
          path: "/var/lib/mysql"
          type: data
    hpa:
      metrics:
      - type: Resource
        resource:
          name: memory
          target:
            type: Utilization
            averageUtilization: 50

3.13.2.3. Custom metrics based autoscaling

The following is an example of a HPA configuration which is based off a custom metric that exposes the number of requests handled per second per pod. With the configuration below HPA will try to maintain an average of 1 request per second for all pods.

Note

In this example 1000m is the equivalent to a single request.

hpa:
  metrics:
  - type: Pods
    pods:
      metric:
        name: http_requests_per_second
      target:
        type: AverageValue
        averageValue: 1000m

Below is a manifest file for a Wordpress bundle wherein which the mysql role is autoscaled (up to a maximum of 5 pods) based on the number of requests handled per second per pod.

name: WordPress
version: "1.0"
roles: [mysql, wordpress]
serialize: true
icon: wordpress-logo-stacked-rgb.png

mysql:
    multinode: True
    multinode_min: 1
    multinode_max: 5
    restart_on_scalein: false
    restart_on_scaleout: false
    name: mysql
    image:
        name: mysql
        version: "5.7"
        engine: docker
    storage:
        - media: ssd
          path: "/var/lib/mysql"
          type: data
    hpa:
      metrics:
      - type: Pods
        pods:
          metric:
            name: http_requests_per_second
          target:
            type: AverageValue
            averageValue: 1000m

3.13.2.4. Combining constraining metrics

The above defined metrics can be combined by simply specifying them one after the other in the manifest within the metrics section. The following is an example of a HPA configuration that will try to maintain an average of 1 request per second for all pods as well as ensuring that the average CPU usage maintained across pods will be 80% of 1 CPU.

Note

When combining metrics the number of desired replicas is calculated for each metric independently and the max of all the calculated replicas is set as the target to be reached with regard to autoscaling.

hpa:
  metrics:
  - type: Pods
    pods:
      metric:
        name: http_requests_per_second
      target:
        type: AverageValue
        averageValue: 1000m
  - type: Pods
    pods:
      metric:
        name: cpu_usage
      target:
        type: AverageValue
        averageValue: 800m

Below is a manifest file for a Wordpress bundle wherein which the mysql role is autoscaled (up to a maximum of 5 pods) based on the number of requests handled per second per pod as well as the average CPU usage across all pods.

name: WordPress
version: "1.0"
roles: [mysql, wordpress]
serialize: true
icon: wordpress-logo-stacked-rgb.png

mysql:
    multinode: True
    multinode_min: 1
    multinode_max: 5
    restart_on_scalein: false
    restart_on_scaleout: false
    name: mysql
    image:
        name: mysql
        version: "5.7"
        engine: docker
    storage:
        - media: ssd
          path: "/var/lib/mysql"
          type: data
    hpa:
      metrics:
      - type: Pods
        pods:
          metric:
            name: http_requests_per_second
          target:
            type: AverageValue
            averageValue: 1000m
      - type: Pods
        pods:
          metric:
            name: cpu_usage
          target:
            type: AverageValue
            averageValue: 800m

3.13.3. Available Metrics

By default a certain set of metrics are made available for the Kubernetes HPA to leverage. The complete list of readily available metrics can be found by running the following command: kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1. Detailed below are some pod metrics that can be utilized:

  • cpu_load_average_10s

  • memory_rss

  • spec_cpu_shares

  • threads

  • cpu_system

  • fs_writes

  • spec_cpu_period

  • spec_memory_reservation_limit_bytes

  • fs_writes_bytes

  • memory_failcnt

  • spec_memory_limit_bytes

  • start_time_seconds

  • fs_reads_bytes

  • last_seen

  • cpu_cfs_throttled

  • cpu_user

  • file_descriptors

  • tasks_state

  • memory_usage_bytes

  • processes

  • memory_mapped_file

  • memory_swap

  • memory_max_usage_bytes

  • threads_max

  • spec_cpu_quota

  • cpu_usage

  • cpu_cfs_throttled_periods

  • memory_failures

  • cpu_cfs_periods

  • memory_cache

  • spec_memory_swap_limit_bytes

  • fs_reads

  • memory_working_set_bytes

  • sockets

3.13.4. Creating custom metrics

In certain scenarios, there might be no pre-determined metric that is suitable for a particular use case. In order to solve this a metric will have to be added manually. Detailed below is a set of steps that can be followed in order to create a custom metric that can be used by HPA and is scraped by Prometheus.

Deploy Prometheus

In order to actually process the metric to be defined we will utilize Prometheus as it provides an easy mechanism via which to expose metrics that need to be utilized. Robin enables users to deploy Prometheus in a covienent manner as the application is used as a general monitoring solution for Robin. Issue the following command to deploy Prometheus:

# robin metrics start --media HDD --resource-pool default --faultdomain disk --replicas 3 --retention-period 2
Job:  74 Name: MetricsStartCollect  State: PREPARED        Error: 0
Job:  74 Name: MetricsStartCollect  State: DONE            Error: 0
Job:  74 Name: MetricsStartCollect  State: AGENT_WAIT      Error: 0
Job:  74 Name: MetricsStartCollect  State: NOTIFIED        Error: 0
Job:  74 Name: MetricsStartCollect  State: COMPLETED       Error: 0

Note

More details on Prometheus support within Robin are provided here.

Expose the necessary metrics

The metrics that need to be scraped by Prometheus, should be exposed from within the application at the /metrics endpoint. For more details on standard procdures and formats with regards to exposing metrics , review the guide here.

Instruct Prometheus to scrape the exposed metrics

In order to actually have the metrics that are now exposed by the application scraped by Prometheus, annotations will have to be specified for each pod within the target role. This can be easily achieved by specifying the annotations at the role level within the Application Bundle manifest. The following is an example of the necessary annotations:

annotations:
    prometheus.io/scrape: "\"true\""
    prometheus.io/port: "\"8080\""
    prometheus.io/scheme: "http"

Below is a manifest file for a Wordpress bundle wherein which all the pods spawned as part of the mysql role will have the specified annotations.

name: WordPress
version: "1.0"
roles: [mysql, wordpress]
serialize: true
icon: wordpress-logo-stacked-rgb.png

mysql:
    multinode: True
    multinode_min: 1
    multinode_max: 5
    restart_on_scalein: false
    restart_on_scaleout: false
    name: mysql
    image:
        name: mysql
        version: "5.7"
        engine: docker
    storage:
        - media: ssd
          path: "/var/lib/mysql"
          type: data
    annotations:
        prometheus.io/scrape: "\"true\""
        prometheus.io/port: "\"8080\""
        prometheus.io/scheme: "http"

Update the prometheus-adapter configmap

The metric will have to also be defined in the prometheus-adapter configmap within the kube-system namespace. As a result, the configmap will have to be updated with a configuration that contains the query which summarizes the metric you want to use. The following is an example configuration of the http_requests_per_second metric we utilized above.

- seriesQuery: 'http_requests_total'
  resources:
    overrides:
      kubernetes_namespace:
        resource: namespace
      kubernetes_pod_name:
        resource: pod
  name:
    matches: "^(.*)_total"
    as: "${1}_per_second"
  metricsQuery: (sum(rate(<<.Series>>{<<.LabelMatchers>>}[1m])) by (<<.GroupBy>>))

Note

For more information on the rules constraining configuration specification, read the documentation here.

In order to confirm that your metric has been successfully added, verify it is displayed in the output of kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1. For the configuration specified, an example is shown below:

  # kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1 | jq  | grep pod | grep http_requests_per_second
"name": "pods/http_requests_per_second",

Note

The metric name here does not match the one specified above as prometheus converts absolute values into a rate i.e. total per second which in turn modifies the metric name to make it more self-explanatory.

Verify the metrics are available

In order for the metric to actually be useful and reflect the current state of the pod, Prometheus needs to be able to scrape the information exposed in the previous step of this guide. In order to confirm this is the case, visit the Prometheus endpoint and make sure the metric you have exposed is available for use. Issue the following command to attain the Prometheus endpoint:

    # robin metrics status
Prometheus:
-- Status........ Running
-- URL........... http://10.10.2.33:30423

Grafana:
-- Status........ Running
-- URL........... https://10.10.2.33:32533

Utilize the metric

The metric can now be specified for use within the manifest as if it were a standard metric exposed to the Kubernetes HPA.