Embedded development is not like web development

When I wrote Embedded programming is like web development, I thought it would be a pretty uncontroversial article—I expected the response to be along the lines of "geez manuel, that's pretty obvious."

It turns out that that was far from the case—most feedback I got was that both were significantly different. This puzzles me deeply: I switch back and forth between both domains weekly, yet I feel like I am using the same techniques and writing almost identical code.

To gain a better perspective, I wrote up all the reasons I could think of why embedded development is not like web development. I hope that this gives some context to the previous article and helps me provide more clarity moving forward.

Skills of a bare-metal embedded developer

I will use bare-metal microcontroller-oriented work as a stand-in for "embedded development." While a fair amount of embedded development these days involves Linux system programming, cloud computing, graphical UIs, and data processing, I don't want to blur the boundaries too much and muddy my examples.

If you do bare-metal programming, you need to know a certain set of tools and practical skills:

• It helps to learn how your CPU works. It is frequent to change architectures on different projects, and there is a wide range of peripheral buses and architectures that need to be relearned. While programming languages stay the same, there is a lot of datasheet reading involved.
• You must be comfortable with various physical tools: oscilloscope, logic analyzer, soldering iron, and breadboards.
• You need to know at least rudimentary electronics to pinpoint electrical bugs and know how to use the oscilloscope. More advanced electronics knowledge is useful for building peripheral circuits.
• Running and debugging your program requires a debugging probe.
• You have to be able to read and step through assembly language. The compiler output often demands special scrutiny.
• You will have to deal with intermediate board versions that are often broken in subtle ways (wrong traces, signal noise, bad power supply).
• You usually work with real-world prototypes (for example, a set of motors on your desk instead of the real machine). These not only exhibit their own quirky bugs but often are incomplete.

The code and runtime requirements are very different:

• Codebases are usually small and self-contained.
• There is low code reuse, and open-source libraries are fairly uncommon (although that is changing).
• Much of the code is interfacing with peripherals and bus protocols, requiring a lot of reading and comprehending datasheets.
• Memory management is typically manual, and memory is scarce: dynamic allocation and fancy data structures are usually out.
• Parts of the system have real-time constraints, often hard real-time.
• Doing IO usually involves dealing with interrupts, having to program peripherals by reading and writing memory-mapped IO, and often using DMA and timers. This is asynchronous by nature; there is often no OS to provide blocking semantics.
• There is no or limited persistent storage.
• Debugging usually perturbs the system badly enough that elaborate debugging strategies have to be devised.
• There is not that much material out there to learn from. Communities are fragmented and scarce (this has changed over the years, with the influx of developers growing up on Arduino and now with the Rust embedded movement).

Skills of a web full-stack developer

Similarly, web development has its own tools, technologies, workflows, and constraints.

• A lot of different technologies are involved in a single project. You will need to know some CSS, HTML, SQL, and Linux skills in addition to "pure" programming. You will often interface with a plethora of external APIs.
• There is a lot of code reuse. You often work against big frameworks like Django, React, and Vue. These frameworks move quickly, and there is always some new hype around the corner.
• There are a lot of tools: you manage packages, minify, lint, type check, bundle, transpile your front end, migrate your database, scaffold new components, run development containers, do remote debugging, and deploy and monitor your services in production. These tools evolve at the speed of the modern open-source community, which can be overwhelming.
• Programming trends and techniques evolve often; there is a lot of hype-driven iteration. Conversely, there are a lot of learning materials, conferences, and communities out there to help you stay up to date.
• Your code interfaces with a lot of third-party systems and services. Even a simple web application usually deals with a web server, a cache, a database, a load balancer, cloud storage, a logging service, and more. More complex systems deal with container orchestration, microservices, message buses, secrets management, and edge computing.
• Much time is spent working with UX, graphic design, and marketing, so much so that some of these functions are sometimes part of engineering teams.
• Your system runs on pretty beefy machines (browser, cloud servers). You can be fairly loose with runtime and memory requirements and use a lot of "layered" frameworks.
• While asynchronous code is fairly common, you are offered a lot of synchronous APIs to simplify your code, especially in backend work.
• Products tend to be significantly more complicated, with complex UX workflows, large-scale deployments, and the integration of many services.
• The actual runtime environment of your system is often significantly different from the development or staging environment. There will be emergent behavior at scale that can't easily be replicated on a developer's machine.
• Products are continuously observed and maintained while they are running. There usually is a dedicated ops team responsible for maintaining the application's runtime.

So, why did I say that embedded development is just like web development?

I think that the day-to-day of a programmer differs significantly depending on which domain they work with. The set of skills, the onboarding, and the learning experience are tremendously different. You can't just put a junior embedded developer before a web application and have them contribute meaningfully.

However, once you become a seasoned developer (at a senior+/staff level), your work becomes much more abstract: you think in systems more than in the minutiae of actual code.

In fact, with the decreasing amount of coding you do, you can quickly become rusty. It's been years since I have written a kernel driver or a bare-metal firmware, and I have to fight with webpack every time I get back into front-end development. My CSS knowledge—poor in the best of times—evaporates every few weeks. However, I know that I can pick up whatever current knowledge is required to build them, and I know where they fit in the system and what tradeoffs can be made. I can also provide guidance and advice to more junior developers on the team and help them grow.

Overall, higher-level patterns transpose very effectively from one domain to the other. Someone knowledgeable about "real-time" WebRTC, WebSockets, HTTP, and gRPC will have no trouble understanding the nuances of USB or how a network protocol might fit the problem at hand. Similarly, a frontend developer that is knowledgeable in building complex asynchronous applications using state reducers, promises, and state machines will have no problem designing and code reviewing some more intricate firmware—even writing one after having been shown the nitty-gritty of flashing and debugging by an experience embedded develop.

I hope to give concrete code examples to make this admittedly abstract statement much more tangible in my next few posts.