Containers offer a lightweight virtualization of resources and thus also an isolation of processes. Containers are based on a common operating system (keyword: immutable system) and the applications in the container bring the additionally required software components themselves. The necessary mechanisms for isolation, such as namespaces and control groups, are provided by the Linux kernel.
In order to ensure the real-time capability of the system and thus also of the applications in the containers, the Preempt-RT Patch (https://wiki.linuxfoundation.org/realtime/start) is used.
Container
Linutronix evaluated two different approaches. In a first step, two different runtime systems for containers, namely Docker and podman, were examined. Then, in a second step, an orchestration system, namely Kubernetes, was examined for its properties in relation to the deployment of real-time processes. [Explanation: Kubernetes (sometimes referred to as K8s) is a popular open-source platform that orchestrates container runtime systems across a cluster of network resources. Docker was used as the runtime system].
Special permissions and capabilities are required to run Linux real-time applications in containers:
- SYS_NICE: Assignment of real-time priorities
- CAP_IPC_LOCK: Memory Locking
- CAP_NET_ADMIN Socket permissions
- SYS_RAWIO: MSRs, ….
With these permissions, the real-time applications can be executed isolated in the system (i.e. in the container). RT measurements with cyclictest showed that the difference between a native execution and the one in the container is negligible. The cyclictest latency, measured directly on the hardware and in the container, was about 23us on the test system. Thus, the container did not stand out in the measurement.
Figure 1: Latencies
Docker or podman are used to run an application in a container on a specific device. To do this, they fetch the corresponding images from a repository located centrally on the network (see Fig. 2). However, an application can also consist of several containers ("microservices"), which have defined communication relationships with each other. To roll out these containers and their configuration in a network or on an edge machine, an orchestration tool is very helpful. This is where Kubernetes comes into play as a tool, for example.
Figure 2: Deployment
Kubernetes and Real-Time
Kubernetes, a Google development, has been designed for cloud services, not real-time applications. Scheduling requirements between general-purpose and real-time applications differ. For example, real-time applications, especially in the network domain, require dedicated CPU cores and memory in combination with PREEMPT_RT. Consequently, multiple real-time applications must not be run on the same CPUs under any circumstances, otherwise delays may occur. Kubernetes supports several quality of service classes. The "Guaranteed" class allows the specification of fixed resources in terms of the number of CPUs and main memory for pods and real-time containers, respectively. Therefore, this mechanism is used. Furthermore, Kubernetes allows the allocation of applications to specific nodes via a "NodeSelector" mechanism. The Image Registry provides the container images with the real-time applications. These can be stored there directly by the application developers. In addition, there is a central Kubernetes master instance in the network, which coordinates the container deployment.
For Kubernetes to run correctly on Linux with PREEMPT_RT, additional changes were needed. This concerns in particular the "Memory Resource Controller". This was adapted for PREEMPT_RT and ported to the current stable branch of Linux. This allows Kubernetes to be used on PREEMPT_RT without any discernible problems.
On modern multi-core CPUs, one can now also imagine a combination of pure Linux applications and containers and virtual machines. With the help of kubevirt (https://github.com/kubevirt/kubevirt), virtual machines (VM) can also be managed with Kubernetes.
On a 4 core machine, we set up a system as follows:
CPU #0 - used only for Linux housekeeping and for the k8s daemons.
CPU 1-3 is used for containers and/or the VM. Linux's own KVM is used as the hypervisor. The CPUs are isolated by means of optimized profiles and the assignment of the pods to the "QoS guaranted" class. The latency in the container remains the same as in the previous approach, the latency in the VM is significantly higher and is max. 54 µsec.
The diagrams in Fig. 3 show from left to right the latencies in the bare metal (i.e. on Linux), in the container and in the VM
Figure 3: Latencies in comparison
And here comes the latency in the virtual machine:
Figure 4: Latency in the Virtual Machine
It must be noted in this case that the virtual CPU for real-time must be assigned to exactly one physical CPU and vice versa. The remaining CPUs (here CPU 2 and 3) are freely available and can be used by k8s as required. However, if a container with real-time tasks is to be used here, a physical CPU must also be assigned exactly for this container (isolate).
Conclusion
Applications that are directly based on the kernel with Preempt-RT as well as containerized applications that are also based on a kernel with Preempt-RT allow hard real-time applications. No differences are noticeable in measurements with the generally accepted and proven cyclictest.
Kubernetes can be used for the distribution of real-time capable containers if some minor bugs are fixed and it is ensured that only one real-time container with an exactly assigned core is used per physical CPU.
If virtual machines are also to be used, they can also be managed under kubernetes with the help of kubevirt. For real-time tasks, however, it should be noted that exactly one VM is assigned to a physical core here. And even then, the results delivered by cyclictest are significantly worse (factor of two).
Alternating assignments of real-time tasks to physical CPUs (e.g. load-dependent distribution) cause a significant increase in latencies and should therefore be avoided.
In the next article we will look at the real-time capability of a network (Ethernet) under Linux. This is of great interest for automation technology especially in the context of TSN (Time Sensitive Network) and for example OPC UA Pub/Sub.
For more information don't hesitate to contact us by email to info@linutronix.de