This is part 1 from a two-part blog series. We start by talking about standards and technical documentation. The second part will be a "lessons learned" including some implementation details.
Cyber-Security is of growing interest for industrial automation. While often neglected in the past and covered up with air-gapped infrastructure, current trends like Industrial IoT, Industry 4.0 and mass customization rely on interconnection of previously isolated communication islands. Without a defense-in-depth approach, attacks to this critical infrastructure are unavoidable. This and similar transformations in other areas called the EU into action to formulate the Cyber-Resilience Act (CRA).
Motivated by these developments, the company Linutronix, mostly known for their involvement in the PREEMPT_RT patches enabling real-time behavior in Linux systems, set out on a journey to better support their industrial customers in the development of cyber-resilient products. An important milestone was reached in May 2024, when Linutronix received their IEC 62443-4-1 certificate for adopting a secure development lifecycle process. This blog post will describe the way towards the next milestone – the certification of a IEC 62443-4-2 compliant component – from an engineering perspective.
The IEC 62443-4-2 standard lists technical security requirements for control system components. The number and content of requirements depends on the targeted security level and on the component type (one of software application, host device, embedded device or network device). To serve as a blueprint for future customer products, our goal was to build an embedded device with security level 2. For embedded devices, security level 2 implies 61 requirements, where 4 are not applicable to the product. In the end Linutronix achieved not only an embedded device approved by TÜV SÜD, but also a long-term maintained software stack and a collection of patterns that can be reused to support customers in similar cases.
We developed an immutable device [1] called “Secure Beacon”. It comes with a Web UI and a multi-color LED that blinks in configurable patterns. The LED blinking serves as a placeholder for an actual industrial application like a motor controller, a sensor or a PLC. Immutable here means that all programs and most configuration resides on a read-only vendor partition. Mutable configuration, such as the user-selected LED pattern or network settings, are well-specified exceptions stored on an encrypted read-write partition.
This software stack is now the core of our Industrial Grade Linux Operating System (IGLOS [2]), making it long-term maintained and reusable for a large variety of other products that are in need of a stable and secure operating system basis.
Maintenance has two flavors: In the simple case, Linutronix continuously integrates stable updates from Debian. In the slightly more demanding case where we have to maintain custom builds (example: hardware related adaptions in U-Boot, Linux kernel), we rebase and rebuild our patches continuously on upstream LTS releases. In any case Linutronix contributes fixing issues with an upstream first policy.
Why IEC 62443? The European Commission recently requested 15 horizontal and 26 vertical harmonized standards for the CRA [3]. Complying with them gives manufacturers the highest legal confidence, namely a “presumption of conformity” [4]. The standards will have lyric titles such as "European standard(s) on making products with digital elements available on the market without known exploitable vulnerabilities", or "European standard(s) on essential cybersecurity requirements for operating systems". Notably these harmonized standards won’t be fully available until 2027. So we could twiddle our thumbs… or better, since CLC/TC 65X (Source: Mirroring committee for IEC in Europe) was acknowledged as relevant by CEN-CENELEC [5], follow IEC 62443 as the most likely CRA approximation for OT components. For more details see blog posts from Sarah Fluchs [6]. She is a member of the EU Expert Group on Cybersecurity of Products with Digital Elements [7]. Her blog posts provide an excellent up to date insider view on the standardization progress.
Working with a Secure Lifecycle means to extend traditional software engineering activities like requirements engineering or testing to apply security best practices and guidelines. Some activities that might otherwise be considered optional become mandatory, like SBOM creation, key management and 3rd party software component assessment. And it adds “new” activities like threat modeling, long-term vulnerability and patch management. This implies to track a good amount of information that must be turned into technical documentation. Such documentation is mandatory per IEC 62443 4-1 (for example: SM-1 wants configuration management and requirements traceability, SI-1 wants traceability of security design to source code) and also by CRA Article 31 [8]. It’s less of a burden as it may sound.
The team needs a digital collaboration platform anyway. If done right, documentation becomes mostly a side effect from using this platform.
Developing secure software implies that every piece of work (and that’s not only source code…) has to be reviewed and discussions as well as conclusions have to be tracked in an integrity-protected and access-controlled change log. By using a software version control system such as Git, backed by a system such as GitLab, we already have an appropriate tool that is routinely used to develop source code. So why not extend the same workflow to the other software engineering activities? After all, requirements are just textual statements like source code. Test cases are often just expressed as text in some kind of domain specific language and software architecture and design documentation is also mostly text enriched with vector graphics. We would just need suitable tooling that provides the needed structure, convenience and traceability.
But Linutronix are certainly not the first ones with this requirement and also not the first who thought about using text files backed by Git. For example, the Zephyr RTOS project approaches certification for the IEC 61508 standard. They're using a Git based workflow and an open-source tool called StrictDoc to document safety requirements [9]. This was the promising reference we needed for a start. Conversation with the StrictDoc authors and a look at their Python code base [10] showed that the project is well designed, maintained and tested and that we would also be able to contribute fixes and features where required (Examples: Jinja2 autoescaping, roles for file relation, Robot Framework integration). So, we found a tool to go forward and could start to document our work items in a traceable way.
Traceability helps to keep track of what has been done and what not and is the basis to decide if an increment is ready to be released. Which test cases verify a given requirement? Where is the code that implements a given requirement? Which threats are known for a given interface? Which security mechanism mitigates a threat? When StrictDoc parses a set of sdoc text files, it internally builds up the traceability index, which represents all these relations as directed acyclic graph. Initially the tool was created to handle requirements only, but meanwhile custom grammar elements are first-class which means we can create holistic documentation. Historically, different documentation tools and templates evolved for different engineering activities like Requirements Engineering, Architecture and Design. Testing or Security Engineering. For example the Volere requirements template [11], the arc42 software architecture template [12] and OWASP threat modeling samples [13] all provide super helpful guides for a specific activity. In fact, “Secure Beacon" documentation started off as arc42-clone that was converted to StrictDoc syntax. But it's hard to find examples that manage to integrate all the resulting documentation islands consistently and meaningful. This makes it hard to establish fully traceability. Therefore, we extended the original arc42 structure and integrated the needed documentation for all other areas, together with source and test code (Git submodules or Android repo provide a way to have source and test code integrated with the documentation), into one single document base and are thus able to establish connections from everything to everything else.
StrictDoc’s file traceability features is in particular handy to keep documentation minimal. For example, you can directly link requirements to D-Bus policy definitions or to test code of the Robot Framework, instead of adding an intermediate layer of redundant information to the documentation. It’s also useful to link design to implementation. But since open source code should not be tainted with custom annotations, where to place a link? That’s where StrictDoc’s language-aware parsing feature becomes handy. We can define a forward relation by file name, class name and function name [14]. Look at this example:
[DESIGN]
UID: DESIGN-UEFI-IMAGE-AUTH
TITLE: Platform independent authentication of software images during boot
DESCRIPTION: >>>
... description with flow and state charts of the boot process ...
<<<
RELATIONS:
- TYPE: File
VALUE: sources/u-boot/lib/efi_loader/efi_image_loader.c
FUNCTION: efi_image_authenticate
As you see, only our documentation got edited. U-Boot source code remains clean. With a recent StrictDoc feature one can also include test reports and automatically connect test results back to requirements [15].
Our internal collaboration workflow can be roughly summarized as follows
- Any team member may add new content by editing *.sdoc source files with a text editor or by using the web UI from StrictDoc server for convenience. Usually documentation is ahead of implementation, but we also accept to improve documentation in hindsight.
- Commit and push to GitLab for review. This triggers a pipeline that calls strictdoc export. The export command verifies constraints and consistency and renders StrictDoc files, source code and test reports to HTML. The pipeline then publishes it on an internal web server.
- Review, fix, approve. Reviewers can browse the recently rendered per-user draft, or use strictdoc diff to get a nice semantic diff locally.
- Pipeline runs again and now publishes to the web server directory that holds the latest common revision. Despite merged, the STATUS field still is ForReview.
- There are out-of-band bulk reviews on a set of already integrated changes carried out by named experts (for example security engineers and test engineers). Only their agreement can change the STATUS field of items to Reviewed.
The clear and accessible rendering of StrictDoc provides a hassle-free exchange of the documentation between all involved parties, including management, development and external auditors.
Therefore, external collaboration with TÜV SÜD mostly evolved around this documentation. Usually compliance statements (see traceability diagram) served as entry points during gap analysis and during the final audit. From there we navigated the traceability graph up and down as needed to support discussion and to provide evidences.
Apart from rendering the documentation into HTML or PDF, StrictDoc also supports exporting to ReqIF for integration into other requirements engineering software.
The documentation now is an appreciated part of our day-to-day work. It supports discussions among different people, helps us to manage complexity and to find good solutions.
Conclusion
As shown, IEC 62443 4-2 is the best readily available standard to prepare for the CRA assessment of OT components. Linutronix built a product on top of their new long term maintained IGLOS software stack and got certified successfully. We're thus convinced that reusing IGLOS and sharing our experience will significantly simplify the prove of conformity for customer’s future products. The foundation of such certification is the documentation. As we have demonstrated, developing a IEC 62443 4-2 certified component doesn't require specialized and expensive tooling for executing and documenting a compliant development process. While it takes effort, documentation is less cumbersome if it's part of the developers natural git workflow. StrictDoc thus became a perfect companion. The resulting documentation served as main discussion point during certification and is appreciated by the team to discuss day-to-day work.
With IGLOS, Linutronix is able to support a wide range of customers in building secure products, expanding their services, including setting up customized Board Support Packages and supporting the use of Real-Time Linux, to cover all aspects of open-source solutions in industry.
Stay tuned for next week, when Linutronix will publish "lessons learned" from implementing IEC 62443-4-2 security features as second part of this blog.
Authors:
Tobias Deiminger is a software engineer with background in communication systems at Linutronix and contributor to several open-source projects. He recently worked on implementing the IEC 62443 security standard at Linutronix.
Florian Kauer is a software engineer specialized in embedded systems for reliable and secure networks. During his time in academia, he dealt with low-latency audio transmission and scalability of wireless industrial control networks. After developing cloud systems for life-science applications, he joined Linutronix to develop open-source networking and security solutions.
References:
[1] https://www.linutronix.de/blog/Industrial-Control-Systems-Platform-Strategy-Security
[2] https://www.linutronix.de/company/news/IGLOS-by-Linutronix
[4] https://www.european-cyber-resilience-act.com/Cyber_Resilience_Act_Article_27.html
[7] https://fluchsfriction.medium.com/
[8] https://www.european-cyber-resilience-act.com/Cyber_Resilience_Act_Article_31.html
[9] https://docs.zephyrproject.org/latest/safety/safety_requirements.html
[10] https://github.com/strictdoc-project/strictdoc
[11] https://www.cs.uic.edu/~i440/VolereMaterials/templateArchive16/c%20Volere%20template16.pdf
[12] https://arc42.org/overview
[13] https://owasp.org/www-community/Threat_Modeling_Process#sample-scope-the-work