The subject of this blog post is embedded Linux distribution engineering: the process which starts with an assortment of software components and ends with a (hopefully successful) product containing that software on the market. What happens in between? Let's take a look at this picture:
Distribution engineering is applying tools, processes and configuration to software components in order to produce the items on the right side (which in turn are required to ship the actual product). The central question is: who is taking care of the part in the middle? There are, by and large, two possible answers to it:
1. A distribution vendor: a company or organisation that delivers tools, processes and configuration (and, optionally, the artifacts on the right side of the picture that those three items produce) to the product company through a commercial supplier relationship. It can be distinguished between commercial distros and community distros like Debian.
2. The product company itself: as part of the product engineering, the company creates and maintains its own distribution
Let's consider the pros and cons of both options.
Distribution vendors
There are several advantages of having a distribution vendor that supplies the distro (distro is a short-hand for 'distribution', it's less letters to type, so let's use that).
- vendor takes care of making it. This is often underappreciated, but creating scalable tools and processes that generate a complete operating system, and that make it possible to rapidly make changes to the system (and have confidence that nothing broke) is not a simple task. There are an order of magnitude more moving parts than there are in a single software component.
- vendor (usually) knows what they're doing. This goes hand in hand with the above: doing a good job requires expertise in distro making. Maybe you as a product company don't have that expertise. And if you try to grow that, you'll quickly find out that hiring good distro engineers is harder than hiring good software component developers, and growing expertise in house takes time and patience, and senior, experienced mentors too.
- product company can focus its often very limited resources on the actual product. The distro is an enabler to the product functionality; it's not in itself where the value add is.
There are several disadvantages too:
- getting a distro from a vendor requires a commercial relationship or a committment in a community. You have to pay the vendor. Maybe making your own distro is actually less costly in the end? You have to decide how much effort you want to invest.
- there is a significant risk of 'vendor lock-in' - a situation where switching over to a different vendor (or to building your own distro) becomes extremely costly and time consuming. This can happen without any 'evil intent' by the vendor, and simply due the nature of what they deliver. Building your own product and its delivery process is necessarily tightly coupled to the vendor distribution and prepackaged solutions associated with it, and if you have to change the latter, you will have to throw away much of the former too, and start over.
- there is a significant loss of flexibility and control over the software stack in your own product. You have to take what the distro vendor delivers and accept it. The two companies agendas are not the same, and this may cause conflicts between what you need and what they supply.
- ability to customize the distribution to your particular needs is very limited. And so is ability to fix bugs in the distro: it's all done by the vendor, and that has potential to create friction, delays and frustration. Write your acceptance and support criteria carefully in the areas that particularly matter to the functionality of your product in the short, medium and long term.
Example vendor distro: ELBE by Linutronix
ELBE is a Debian-based distribution tailored for embedded specific use cases. What does that mean?
- binary packages come from upstream Debian project, an open source community takes care of producing them, and just about everything under the sun is available and (reasonably) tested. This is in contrast to some other vendors that do in-house component packaging.
- With ELBE, you can build each component of the distro yourself and thus customize it exactly to your own needs (as in Yocto). The behavior of each components can be defined by e.g. compiler options and also the size of the target distro can be defined.
- ELBE adds a customized bootloader and kernel for the specific hardware used in a product. Upstream Debian doesn't offer that, as it runs primarily on commodity PC hardware.
- ELBE also adds an update mechanism and tools for embedded products. Upstream Debian updates are based on packages and package managers; this isn't robust enough outside of desktop computers, and ELBE is offering whole-image updates.
- extended security support. Upstream Debian provides fixes for some (limited) amount of years; ELBE offers more on top of that.
Make your own distribution
So what, then, are the advantages and drawbacks of not having a vendor? In a way, they're a mirror image of having a vendor, but let's list them with a brief explanation for each.
Advantages:
- there is complete control over the full software stack in the product. You can make it behave exactly the way you want. Customizing the distro is as easy as it gets, as the distribution is made by you.
- there are no vendors and no contracts with them. You interact directly with open source community when something needs to be fixed, or you can develop the fixes and improvements yourself when needed. You can still bring in consultants to help, if in-house developer capacity or expertise is missing, but it's not a permanent relationship.
The disadvantages however are no less significant, and need to be carefully considered:
- you need to employ in-house distro experts, and ensure they don't leave the company, as they hold key knowledge to making your product. They're not easy to find or train.
- just as important is this: you need in-house distro infrastructure. This is harder to set up than the typical software component infrastructure. Things like integration pipelines, testing rigs (on-hardware testing is essential) are all firmly in the 'advanced software engineering' bucket and need to be taken seriously from the start. There is also an ongoing maintenance cost for that infrastructure: cloud providers will send significant bills.
- and finally, there's the need to have the right mindset too. As a product company you need to see the in-house distro engineering activity as an 'internal product' in itself, and not as a semi-neglected 'side gig' of the actual product.
Example: the Yocto project
What is the Yocto project? It's not a distribution, but a set of tools, and metadata for making your own distribution. It's not a commercial vendor either, but an open source project managed by Linux Foundation. Many companies ship successful products with it. What are the other key points to consider?
- Yocto builds all of the software that goes into a distribution from source code. This is key to being able to inspect it, and change it, perhaps many years after the product was shipped.
- Yocto is not easy to learn. The problem it solves is very complex, and this reflects in the steepness of the learning curve. You also have to make all the choices about how your distribution is configured (there are reasonable starting points though).
- Linutronix can help! We can offer initial Yocto-based distro setup according to the list of your requirements. We also offer Yocto trainings, and can help you with maintaining and updating Yocto-based distributions.
Making the choice
So how do you choose between the two options? Saying 'it depends' is perhaps not really helpful, but it really does depend on the particular circumstances and available resources at your company. There are some key points to consider in that decision though:
- the decision has to happen at the start. You can't defer it: product development needs a distro to put into the actual hardware. Sometimes the quick 'proof of concept' becomes the base of the eventual product; not making an explicit, informed choice and going down the wrong path may become an enormous source of regret, process inefficiency and delays in delivering products later on.
- the consequences of that decision are far reaching. Just about everything about making a product is very different with these two choices.
- switching later is very costly and time consuming. You have throw away a lot of software and in house expertise and start over - and doing that will frustrate your own engineers as well.
There are also technical points to consider, here's a quick sample of what can come up:
- is "standard upstream distro + vendor tweaks" delivery ok for your purposes? Will it scale and adapt with your features, teams, products and overall organization?
- are there license restrictions in your industry? Can you ship products with GPLv3 software in them?
- how much control over the software stack do you need vs. how much resources can be allocated to that? Are you fine adding a 'distro vendor' to your supply chain?
- how are you going to handle the situations when the component versions in the distro are too old or too new? Are configuration choices by the distro vendor fine? (for example, Debian is systemd-based, and that's not negotiable). Do you need tight control over resource consumption (disk, memory) or are the choices made by the distro vendor good enough?
Finally
Let us emphasize: Linutronix has no preference. We believe that both options in distro engineering are equally valid, and lead to successful products. There is no 'better' choice. We can support you and offer our experience and expertise in both.