Running real-time workloads

Configuring a cluster for real-time workloads
Configuring a virtual machine for real-time workloads
Additional resources

You can configure an OKD cluster to run real-time virtual machine (VM) workloads that require low and predictable latency. OKD provides the Node Tuning Operator to implement automatic tuning for real-time and low latency workloads.

Configuring a cluster for real-time workloads

You can configure an OKD cluster to run real-time workloads.

Prerequisites

You have access to the cluster as a user with cluster-admin permissions.
You have installed the OpenShift CLI (oc).
You have installed the Node Tuning Operator.

Procedure

Label a subset of the compute nodes with a custom role, for example, worker-realtime:
```
$ oc label node <node_name> node-role.kubernetes.io/worker-realtime=""
```
You must use the default master role for single-node OpenShift and compact clusters.

Create a new MachineConfigPool manifest that contains the worker-realtime label in the spec.machineConfigSelector object:

Example MachineConfigPool manifest

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfigPool
metadata:
  name: worker-realtime
  labels:
    machineconfiguration.openshift.io/role: worker-realtime
spec:
  machineConfigSelector:
    matchExpressions:
      - key: machineconfiguration.openshift.io/role
        operator: In
        values:
          - worker
          - worker-realtime
  nodeSelector:
    matchLabels:
      node-role.kubernetes.io/worker-realtime: ""

You do not need to create a new MachineConfigPool manifest for single-node OpenShift and compact clusters.

If you created a new MachineConfigPool manifest in step 2, apply it to the cluster by using the following command:
```
$ oc apply -f <real_time_mcp>.yaml
```

Create a PerformanceProfile manifest that applies to the labeled nodes and the machine config pool that you created in the previous steps:

Example PerformanceProfile manifest

apiVersion: performance.openshift.io/v2
kind: PerformanceProfile
metadata:
  name: profile-1
spec:
  cpu:
    isolated: 4-39,44-79
    reserved: 0-3,40-43
  globallyDisableIrqLoadBalancing: true
  hugepages:
    defaultHugepagesSize: 1G
    pages:
    - count: 8
      size: 1G
  realTimeKernel:
    enabled: true
  workloadHints:
    highPowerConsumption: true
    realTime: true
  nodeSelector:
    node-role.kubernetes.io/worker-realtime: ""
  numa:
    topologyPolicy: single-numa-node

Apply the PerformanceProfile manifest:
```
$ oc apply -f <real_time_pp>.yaml
```
The compute nodes automatically reboot twice after you apply the MachineConfigPool and PerformanceProfile manifests. This process might take a long time.
Retrieve the name of the generated RuntimeClass resource from the status.runtimeClass field of the PerformanceProfile object:
```
$ oc get performanceprofiles.performance.openshift.io profile-1 -o=jsonpath='{.status.runtimeClass}{"\n"}'
```
Set the previously obtained RuntimeClass name as the default container runtime class for the virt-launcher pods by editing the HyperConverged custom resource (CR):
```
$ oc patch hyperconverged kubevirt-hyperconverged -n kubevirt-hyperconverged \
    --type='json' -p='[{"op": "add", "path": "/spec/defaultRuntimeClass", "value":"<runtimeclass_name>"}]'
```
Editing the HyperConverged CR changes a global setting that affects all VMs that are created after the change is applied.
If your real-time-enabled compute nodes use simultaneous multithreading (SMT), enable the alignCPUs feature gate by editing the HyperConverged CR:
```
$ oc patch hyperconverged kubevirt-hyperconverged -n kubevirt-hyperconverged \
    --type='json' -p='[{"op": "replace", "path": "/spec/featureGates/alignCPUs", "value": true}]'
```
Enabling alignCPUs allows OKD Virtualization to request up to two additional dedicated CPUs to bring the total CPU count to an even parity when using emulator thread isolation.

Configuring a virtual machine for real-time workloads

You can configure a virtual machines (VM) to run real-time workloads.

Prerequisites

Your cluster is configured to run real-time workloads.
You have installed the virtctl tool.

Procedure

Create a VirtualMachine manifest to include information about CPU topology, CRI-O annotations, and huge pages:

Example VirtualMachine manifest

apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: realtime-vm
spec:
  template:
    metadata:
      annotations:
        cpu-load-balancing.crio.io: disable (1)
        cpu-quota.crio.io: disable (2)
        irq-load-balancing.crio.io: disable (3)
    spec:
      domain:
        cpu:
          dedicatedCpuPlacement: true
          isolateEmulatorThread: true
          model: host-passthrough
          numa:
            guestMappingPassthrough: {}
          realtime: {}
          sockets: 1 (4)
          cores: 4 (5)
          threads: 1
        devices:
          autoattachGraphicsDevice: false
          autoattachMemBalloon: false
          autoattachSerialConsole: true
        ioThreadsPolicy: auto
        memory:
          guest: 4Gi
          hugepages:
            pageSize: 1Gi (6)
        terminationGracePeriodSeconds: 0
# ...

1	This annotation specifies that load balancing is disabled for CPUs that are used by the container.
2	This annotation specifies that the CPU quota is disabled for CPUs that are used by the container.
3	This annotation specifies that interrupt request (IRQ) load balancing is disabled for CPUs that are used by the container.
4	The number of sockets inside the VM.
5	The number of cores inside the VM. This must be a value greater than or equal to `1`.
6	The size of the huge pages. The possible values for x86-64 architectures are `1Gi` and `2Mi`. In this example, the request is for 4 huge pages of size 1 Gi.

Apply the VirtualMachine manifest:
```
$ oc apply -f <file_name>.yaml
```
Configure the guest operating system. The following example shows the configuration steps for a Fedora 8 operating system:
1. Run the following command to connect to the VM console:
  $ virtctl console <vm_name>
2. Configure huge pages by using the GRUB boot loader command-line interface. In the following example, 8 1G huge pages are specified.
  $ grubby --update-kernel=ALL --args="default_hugepagesz=1GB hugepagesz=1G hugepages=8"
3. To achieve low-latency tuning by using the cpu-partitioning profile in the TuneD application, run the following commands:
  $ dnf install -y tuned-profiles-cpu-partitioning
  $ echo isolated_cores=2-9 > /etc/tuned/cpu-partitioning-variables.conf
  The first two CPUs (0 and 1) are set aside for house keeping tasks and the rest are isolated for the real-time application.
  $ tuned-adm profile cpu-partitioning
Restart the VM to apply the changes.

Configuring a cluster for real-time workloads

Configuring a virtual machine for real-time workloads

Additional resources