Sidney Harris [1]. © ScienceCartoonsPlus.com — reproduced under license.
I think of this cartoon whenever an architecture diagram for an automated driving system contains a box with the word Safety in its name, or several of them scattered through the stack, accompanied by a paragraph explaining how they keep the vehicle safe if something goes wrong. The boxes and their explanation raise more questions than they answer, and the unanswered ones matter in a direction most readers do not expect.
The capability of the safety system decides what the vehicle is allowed to do when nothing has gone wrong at all. It constrains the top speed. It constrains the permitted weather. It decides whether an urban operational design domain is feasible. It shows up in headcount, in the validation budget, and in how quickly anything can be deployed. The safety system arrives on the agenda as a safety topic and lands as a product specification.
If you build these systems, or fund them, or audit them, you will eventually have to answer the questions below. The punch line of the cartoon is aimed at the architecture descriptions, not at the people who create them. Almost everything in this post has been argued at length inside safety programs by engineers who lost sleep over it. The problem is that the arguments stay in the room and the diagrams are what the rest of us get.
The pattern
The safety content of these architecture diagrams usually consists of three things.
There is a fallback system, variously called the secondary, redundant, failsafe, or satellite system, which performs the dynamic driving task when the primary driving system is no longer performing it correctly. It may have its own sensors and its own route to the actuators, or it may share some of both.
There is checking, sometimes called monitoring, which validates what a driving subsystem has produced, constrains it, or raises a flag about its health. Checking is a responsibility rather than a place: it may sit inside the primary or stand apart with its own sensing, and the fallback will have checkers of its own.
And there is arbitration, which decides whose outputs reach the downstream systems, and what the actuators do when they receive nothing valid.
This post is about the first of the three: what the fallback needs to be capable of, and what that capability quietly decides about everything upstream of it. What a checker’s approval can establish, and who gets to overrule whom, is a different problem with its own failure modes, which I will take up in a subsequent blog post.
These patterns show up in modern end-to-end architectures as well as older rule-based ones. For example, NVIDIA’s NDAS Alpamayo architecture [2], TTTech’s reference architecture for Level 4 [3], Plus AI’s SuperDrive [4], and the classical modular stack published by the UNIST lab in Korea [5], which sets Safe Planning and Safe Controller alongside the nominal blocks.
Most descriptions say what the fallback is for, not what it can recover from: which vehicle states, entered under which failures, with how much sensing, control authority, and time remaining once the failure has happened. Call it the fallback’s recovery envelope. It is not a topology question, so no diagram will show it, and it is not a one-line answer either, because it is not a capability but a relation between a failure set, an operational domain, and a set of permitted states.
That recovery envelope decides the rest of the product. So let us start with an apparently straightforward question: what should the fallback system do?
The fallback capability ladder
Once the primary system is no longer performing the driving task correctly, the fallback is expected to execute a minimum risk maneuver and reach a minimum risk condition. Those phrases conceal a wide range of engineering.
The simplest thing the fallback could do is apply a fixed deceleration profile until the vehicle stops, open loop. A step up is to stop in lane without hitting the vehicle ahead. A step up from there is to stop on the shoulder when already traveling in the shoulder-adjacent lane, which requires knowing where the shoulder is and whether it is occupied by a vehicle or a person. A step up is to change lanes in order to reach it. A step up is to keep driving, slowly, to reach a shoulder that is not immediately available, possibly through a tunnel or across a bridge or a highway exit. Above all of that sits completing the mission in a degraded state (which is about service continuity rather than a minimum risk maneuver; I mention it here for completeness).
Each step up that ladder is a small increment in behavior and a disproportionate one in architecture. A fallback system that blindly applies the brakes bears no relation to one that can perceive, plan, and steer its way to a stopping place. The fallback can be so simple that it cannot help in the situations that it will actually encounter, or so capable that it eclipses the primary system it was built to back up.
The ladder is not a ranking by safety. A lane change towards the shoulder can introduce more danger than it removes. Continuing through a tunnel to find a stopping place may be right for one failure mode and indefensible for another. Climbing the ladder buys capability, and capability has to be spent deliberately on the failures you actually intend to handle, in the domain you actually intend to operate in.
Which step of the ladder is good enough also depends on how often you expect to need it: the fallback demand rate. This rate should be estimated but the error bars on it will be wide. So the fallback capabilities should be selected based on what cannot be ruled out by the estimate, rather than an expected demand rate.
Even the simplest fallbacks at the bottom of the ladder constrain the system’s fault-free operation.
Stopping in lane
The severity of stopping in lane is largely a function of the speed of the traffic around the vehicle. A vehicle stopped in a traffic jam is close to the ambient condition. The same vehicle stopped in free-flowing highway traffic is a stationary obstacle in a stream closing on it at highway speed, with correspondingly less time for anyone to react. So what a stop-in-lane-only fallback constrains is the set of roads and traffic conditions the vehicle may operate in.
Regulators worked through the same tradeoff. In the working discussions behind UN Regulation 157 [6], which governs automated lane keeping on divided highways (ALKS), the question of raising the permitted speed above 60 km/h brought two available levers into the open. One was to require lane-change capability as part of the fallback’s minimum risk maneuver, so the vehicle could clear the travel lanes. The other was to let a vehicle without lane-change capable fallback operate only where stopping in lane was acceptable, which meant a speed cap, the shoulder-adjacent lane, surrounding traffic moving at a similar speed, and further conditions besides. Japan’s submission argued that lane-change capability was indispensable to any considered speed increase. These were working documents rather than settled text, and I cite them for the shape of the reasoning, not as a statement of what the regulation now requires. The principle is the one I care about: either upgrade the fallback, or restrict what the primary is allowed to do. Not every objection demands a more capable fallback.
Aviation settled into the same shape decades ago and wrote it into ETOPS [7]. Twin-engine airliners were historically barred from routes taking them more than sixty minutes’ flying time from an adequate diversion airport, measured at the approved one-engine-inoperative cruise speed. That limit has since been extended to 180 minutes and beyond, depending on the aircraft. The capability of the aircraft with both engines running is irrelevant to that calculation. What matters is what it can do after one engine has failed, and how long it must sustain that state before it reaches somewhere to land. ETOPS is not seen as the safety department interfering with route planning. Manufacturers and airlines argue hard for extensions, backed by adequate justification. The fallback’s capability lands as a negotiable property of the product, like range or payload.
Stopping on the shoulder
How safe is it to stop on the shoulder, as the minimum risk condition?
Researchers at Impact Research, analyzing United States federal crash data [8], estimated 566 deaths and roughly 14,000 injuries per year between 2016 and 2018 in crashes involving disabled vehicles where low conspicuity was a likely factor. Slightly more than half of those deaths were not other road users but the vehicle’s own occupants, struck after getting out to attend to it. Those figures cover roadside scenarios generally rather than just shoulder stops, but the direction is clear: the minimum risk condition is not a risk-free condition.
A reasonable objection is that an automated vehicle stopped on the shoulder is better placed than a disabled one. It has hazard lights, it knows it has stopped, it can call for help, and a fleet can dispatch a retrieval crew. That is true where those capabilities survive the failure that stopped it, and it changes the magnitude rather than the category. The occupants can still decide to get out. The crew that arrives still has to stand next to a stationary object on a live road.
So the thing we call a minimum risk condition is a state with a duration attached. Exposure begins when the vehicle stops on the shoulder and does not end until the vehicle and any occupants are appropriately secured. The safety analysis should account for the risk accumulated over that interval.
The tail wagging the dog?
When a program selects its fallback architecture, the instinct is to analyze how the primary might fail and in which conditions, assess how bad the consequences would be, and scope the fallback to match. The design loop runs from the primary to the fallback. In operation it runs the other way: the fallback constrains the primary, which looks like the tail wagging the dog, but only if you think the primary is the dog.
Suppose the primary is excellent. It drives at highway speeds, handles rain and construction and school zones, and operates safely around pedestrians. Should it be permitted to do any of that unless the fallback can survive being handed control in such situations?
The fallback does not have to sustain normal operation or complete the mission. It needs to handle the state it inherits at the moment it takes over, and reach a minimum risk condition from there. The state it inherits is whatever the primary was doing when it stopped being trustworthy (factoring in any diagnostics latency), in whatever circumstances the vehicle happened to be in.
The useful formulation is: the primary system should be permitted to choose only those states that the fallback can recover from. Note that other road users can put the vehicle into states it did not select and would not have selected. Also, the fallback’s recovery is scoped to the failures the safety argument claims to cover, the assumed behavior of everyone else, and the sensing, authority, and time that remain once the failure has occurred. The value of this formulation is that it is a design principle the program can embrace, rather than a hard guarantee that is anyway not achievable.
One of the programs I worked on was upgrading an AV developed for urban driving to operate at highway speeds. The primary system was outfitted with the needed longer-range sensors and performed admirably. But until the fallback was equipped with long-range sensing, the primary was not permitted to drive at highway speeds without a human safety driver onboard. The same fallback was not qualified for rain, so the primary did not drive in rain without the human safety driver, even though it could.
Neither restriction was in the requirements at the start. I was leading safety on that program, and both came out of a hazard analysis my team ran, without the primary’s team in the room. Nobody disputed the conclusion once it was on the table. The problem was that it arrived when the primary’s capability was nearly built, rather than alongside the decision to build it, and that it arrived from one team rather than from both. A constraint that two teams derive together becomes an upfront requirement and a property of the product. The same constraint derived in isolation and handed over is news, and news of this kind is expensive to deliver no matter how well founded it is. I have written elsewhere about what happens when teams derive their artifacts in parallel rather than jointly [9]; this is the same failure on a slower fuse. We caught it before the hardware was locked down, which was luck rather than process.
The capability you intend to sell is on the plan. The capability that lets you ship it has to be discovered.
That is a resource allocation problem long before it is a safety problem. Without a sufficiently capable fallback, new capability in the primary cannot be deployed under the existing safety argument. A fallback that has to recognize and avoid pedestrians needs its own perception work, its own data, its own test campaign, and the people and budget and calendar to do it all. That work does not have to be wholly separate from the primary’s, but it does need a validation argument of its own. It needs to appear in the headcount plan and the validation budget, in lock-step with the primary’s development.
Capable fallback or constrained primary
Now take that reasoning into a city.
My position, and I am aware it is not a comfortable one, is that an AV whose fallback cannot detect and avoid a pedestrian has no business operating where pedestrians are, no matter how good the primary is. By avoid I mean reach a collision-free stopped state from the states the primary is permitted to be in. Applied strictly, that is a demanding requirement for essentially every urban deployment operating today.
The strongest counterargument is not that this is too expensive. It is that I am pointing at the wrong lever. If the primary is constrained so that handover can only ever occur from states where straight-line deceleration is sufficient, a simple braking fallback becomes adequate by construction. Limit the speed, widen the gaps, restrict the geometry, and the fallback’s job shrinks until it fits what the fallback can do. That is a serious argument but it mandates a potentially tricky obligation: the constraint has to survive the failure. If the component enforcing the constraint of safe spacing or bounded speed is the primary, and the failure you are covering is an arbitrary malfunction of the primary, you cannot assume the constraint still holds at the moment you need it. Enforcement has to sit somewhere the failure does not reach, with enough response-time margin to matter. If that enforcement sits in the fallback, you are back to a more capable fallback of a different kind, with the added complexity of when/where/whether the fallback should intervene in the primary’s operation.
So the choice is not between a capable fallback and an incapable one. It is how you divide the budget between the fallback’s capability, the primary’s permitted behavior, and the residual risk you decide to carry with an argument behind it. The programs that worry me are not the ones that choose a simpler fallback; they are the ones that do not make the division deliberately, and end up with a primary permitted to enter states that nothing downstream can recover from. The division is not something an outsider can read off a block diagram.
Being more explicit
Every decision in this post gets made whether or not anyone makes it deliberately. Build a fallback with limited sensing range and you have chosen a speed and a set of roads for the primary. Build one that cannot see a pedestrian and you have excluded an urban domain, or accepted a risk you may not have written down. Leave the fallback out of the headcount plan and you have set a deployment date that depends on capability nobody is building.
An architecture diagram records none of this. It records topology, and topology is not the part anyone is arguing about.
So here is what I would ask of anyone publishing a safety architecture description. Which of the three did you choose, and deliberately: a fallback built to match what the primary is allowed to do, a primary held back to match what the fallback can recover from, or a residual risk you have decided to carry? Why is that sufficient for the domain you intend to operate in, how did you validate it, and what evidence would cause you to reconsider it? Not the implementation or trade secrets. Just the claim, the reasoning, and what would change your mind.
That would say more about a program’s safety posture than any diagram and its accompanying paragraph can say on their own. And that is what being more explicit in step two looks like, for a safety architecture description.
References
All links verified 24 September 2026.
- Sidney Harris, "Then a miracle occurs." Reproduced under license from ScienceCartoonsPlus.com. ↩
- NVIDIA, NDAS Alpamayo automated driving architecture, GTC 2025 session DD40000. The architecture diagram appears from 25:41 to 26:18. nvidia.com ↩
- TTTech Auto, "E/E architectures on the way to Level 4," presenting an architecture attributed to Hermann Kopetz. trustmotion.com ↩
- Plus AI, "Inside Plus's AV2.0 approach: SuperDrive." plus.ai ↩
- UNIST High-assurance Mobility Control Lab, autonomous driving research. hmc.unist.ac.kr ↩
- UN R157 Special Interest Group working documents, including the Japan submission UNR157-08-06 and the options presentation UNR157-08-09. Working documents from the 2021 discussions; proposals under discussion, not adopted text. Consolidated adopted text: R157r1e. ↩
- FAA Advisory Circular 120-42B, Extended Operations (ETOPS and Polar Operations). Guidance on compliance rather than the regulation itself. faa.gov ↩
- R. Spicer et al., "Frequency and cost of crashes, fatalities, and injuries involving disabled vehicles," Impact Research, 2020. Figures as given in the abstract and results. impactresearchinc.com ↩
- Sagar Behere, "The traffic light that didn't exist." ↩
- L. Bainbridge, "Ironies of automation," Automatica, vol. 19, no. 6, 1983, pp. 775–779. doi.org ↩