Toyota unintended acceleration and the big bowl of "spaghetti" code (2013)

Posted by SoKamil 2 days ago

Comments

Comment by userbinator 2 days ago

I still believe that the actual cause was tin whiskers, but all the RoHS lobbying buried the evidence.

http://nepp.nasa.gov/whisker/reference/tech_papers/2011-NASA...

Comment by Aeglaecia 2 days ago

im sure that any reasonably charismatic software engineer could scare the shit out of a judge/jury based on code analysis ... perusing the linked nasa document, recurrence odds of this particular failure mode do seem significantly greater than the odds of a cosmic bit flip ...

Comment by thebruce87m 2 days ago

> Studies by IBM in the 1990s suggest that computers typically experience about one cosmic-ray-induced error per 256 megabytes of RAM per month.

https://www.scientificamerican.com/article/solar-storms-fast...

Just to give perspective on the bit flip probability. ECC ftw!

Comment by nsoqm 2 days ago

Is this was true, wouldn’t any modern computer would be crashing several times a month? And please don’t tell me “oh but it is”, because it is not.

Comment by userbinator 2 hours ago

It's not true.

I have left memtest86+ running on a few dozen GB of memory for several days during burn-in testing, definitely more than enough to pass the "once per 256MB per month" threshold, and did not encounter any errors.

Comment by ashirviskas 1 day ago

Most of the RAM may not be critical enough to crash the whole system. Just some random app you have open or a browser tab. So even if it is true, most bit flips should not crash a system.

Comment by nsoqm 1 day ago

Yes, I know that. So why aren’t my applications or tabs crashing at least once a day?

Comment by scraptor 1 day ago

Servers with ECC generally report zero recoverable memory errors until the chip starts failing, at which point there are increasingly many. Therefore the average server experiences zero cosmic ray related memory errors during its lifetime, despite having many times more memory than 256MB.

Comment by crote 2 days ago

How many of those errors vould result in a full system crash, though? And how many of them are just going to cause silent and mostly-harmless data corruption?

After all, was the error in the first line a typo on my side, or a single-bit upset?

A while ago some researchers registered off-by-one-bit domain name typos, which due to physical key positioning were unlikely to be the result of genuine mistyping. I can't find a reference right now, but I recall them getting quite a lot of queries!

Comment by SoKamil 1 day ago

Write a Bit Flip simulator and report your observations ;)

Comment by Aloha 2 days ago

I'm a very large believer that aggressive RoHS regulations have created more intractable problems than they have solved.

Comment by ta20240528 2 days ago

Sure, let's let them put lead back in paint.

Comment by Aloha 1 day ago

There is a large gap between "maybe trying to take lead out of solder causes more issues that it solves" and "lead paint is good"

Comment by 2 days ago

Comment by brettermeier 1 day ago

Stupid

Comment by pengaru 2 days ago

"supply voltage to the electronic control system was purposely lowered and perturbed to simulate bad alternator and/or battery system. The result from the manipulation of supply voltage was rather astonishing. The control systems seemed to work even with the perturbed supply voltage but not correctly. As a matter of fact, it seemed to cause the sudden unintended acceleration repeatedly. The supply voltage to the ECU can be disturbed by minor mishap in the alternator output function and possibly by the overload of ever increasing use of electric devices in the vehicle by the driver. In any case, the current study showed the reproduction of the sudden unintended acceleration when the supply voltage changes abruptly by sudden drop of the alternator output voltage or by overload of the electric devices."

https://www.sciencedirect.com/science/article/abs/pii/S03790...

Comment by M95D 2 days ago

Couldn't read the article, only the summary, but it sounds like glitching by the description. I expect the ECU lowers 12V to 5V or 3.3V by using a buck converter which includes a filter capacitor. To glitch the CPU, the 12V would need to drop well below 5V to have any effect. I don't see how this could happen. If the battery is weak enough to drop below 9V under any conditions other than a short, that car won't even start. My suspicion is that they glitched the ECU power supply directly, not the 12V input - the summary doesn't say.

My conclusion is that it's mosty (scientific) clickbait.

Comment by Glawen 2 days ago

So they drop the voltage to mimic an engine cranking, and they are surprised that the ECU behaves like it is cranking. When engine cranks, voltage drops below minimum voltage required by ECU to keep SW running (SW resets). To counter these, ECUs keep outputs to the max. Normally I would expect an electrical loop with the crank signal though.

Comment by PhotonHunter 2 days ago

Was there ever a recall of the ECU, and if not, why did the UA events go away? Were UA events more common at higher elevations, where there would be more cosmic ray activity?

This story is like Baba Yaga, it comes out from the shadows to scare people every now and then, but Barr’s theory has the interesting property that the ECU would be cleared by the error and so there could never be evidence of the event as he postulated.

Comment by 1970-01-01 1 day ago

There was an ECU update but not a recall. There's thousands of these old ECUs driving cars that are still on the road. Yet somehow they never misbehaved after the mass hysteria and legal circus was over and forgotten. Hmm.

Comment by stackghost 2 days ago

Safety Research Systems, the author of TFA, is a for-profit company whose income is based on lawsuits.

Make of that what you will.

Comment by throwaway81523 2 days ago

Philip Koopman was an expert witness against Toyota (which is a bit questionable to me) and he has some stuff on his website about the case too.

https://betterembsw.blogspot.com/search/label/Toyota%20UA

Comment by qchris 2 days ago

Related to [1]; this topic was discussed earlier today (perhaps inspiring this submission?) in a HN thread on C++ coding standards for the F-35 JSF (search "spaghetti").

[1] https://news.ycombinator.com/item?id=46183657

Comment by gnabgib 2 days ago

Popular in 2015:

(96 points, 106 comments) https://news.ycombinator.com/item?id=10437117

(152 points, 145 comments) https://news.ycombinator.com/item?id=9643204

Comment by LanceH 2 days ago

Ah yes, where Toyota was found guilty of not being a US company.

The only thing they did in the recall was the same floor mat anchor as so many other cases.

"NASA engineers found no electronic flaws in Toyota vehicles capable of producing the large throttle openings required to create dangerous high-speed unintended acceleration incidents. The two mechanical safety defects identified by NHTSA more than a year ago – “sticking” accelerator pedals and a design flaw that enabled accelerator pedals to become trapped by floor mats – remain the only known causes for these kinds of unsafe unintended acceleration incidents. Toyota has recalled nearly 8 million vehicles in the United States for these two defects." -- transportation.gov

Cosmic rays and other wild theories over the simple theory of driver error. Even with a stuck throttle, the brakes will still stop a car (not to mention shifting into neutral still works).

Comment by SV_BubbleTime 2 days ago

Ok, but their engine controller was found to have 12,000 global variables and no one could ever say conclusively that the pedal issue was real or not.

The issue was not that no one found the flaw, it’s that no one could prove it wasn’t there.

Comment by majormajor 2 days ago

>The issue was not that no one found the flaw, it’s that no one could prove it wasn’t there.

Are cars since then required to have formally verified codebases, or is "no one could prove [there are no bugs]" still true?

---

Trying to evaluate what happened based on observation of events alone and stats, in absence of a formal proof of issue or non-issue... the cars didn't just disappear overnight so if there was such an issue... where did it go?

Comment by LanceH 1 day ago

That software was never fixed. Those cars were still on the road after the "scandal". The problem went away somehow.

Comment by SV_BubbleTime 1 day ago

This isn’t true.

Toyota issued multiple engine controller updates. All mfgs do, all the time.

There are no changelogs.

It would also matter what their typical car lifecycle is, it could have been just before refresh so only effected a couple years.

It could have also been bad floor mats.

We’ll never know - but the point is, that their code was so bad you COULD never know.

Comment by LanceH 1 day ago

They also never caught the clowns in the forest stealing children, nor the satanic groups sacrificing babies in the 80's.

Comment by McGlockenshire 2 days ago

> no one could ever say conclusively that the pedal issue was real or not

You should ask a mechanic's opinion.

Comment by 2 days ago

Comment by behringer 2 days ago

Over 9000!!??

Comment by jiggawatts 2 days ago

Common in some real time systems with minimal or no usage of the stack.

Comment by behringer 1 day ago

Yes. In fact I would trust such a system far more since a more procedural approach to the system would be much, much easier to spot bugs.

Comment by SV_BubbleTime 1 day ago

My man. As someone who works on Autosar controllers, your trust is extremely misplaced.

Comment by Gibbon1 1 day ago

I'm suspicious that a lot of those 12,000 are constants. Just based on how I think those guys operate.

You and I would change a constant and recompile. They will just splat location 0x239A

Comment by cwmoore 2 days ago

[flagged]

Comment by Denatonium 2 days ago

Not to mention that in an emergency, you can always turn the key to kill the engine, and then put it back into pre-igntion (to unlock the steering column). You won't have power-assisted braking or power-steering, but with a bit of adrenaline-fueled strength, it is definitely preferable to being in a car that is stuck accelerating.

Comment by PhotonHunter 2 days ago

The service brakes of anything short of a supercar are sufficient to stop a car at WOT.

Comment by joecool1029 2 days ago

Well, they didn’t here, also: https://en.wikipedia.org/wiki/Brake_fade

Comment by jjav 2 days ago

Brakes will always overpower the engine unless the braking system is severly damaged. This is simple physics. Cars decelerate far faster than they accelerate, which is to say, the brakes can generate far more horsepower than the engine can.

(Apparently the Rimac Nevera, with about 2000hp, can accelerate faster than it brakes. So that one might be the only exception. So unless you're driving a 2000hp car, the brakes will always overpower the engine, that is not debatable.)

Brake fade is irrelevant here. Brakes fade when overheated beyond their operating range, either due to fluid boiling and/or the pads overheating. This is nearly impossible to achieve in street driving, but can be experienced on the race track. None of the claimed acceleration accidents involved extreme repeated braking prior to the incident.

Comment by mmooss 2 days ago

That's a lot of thought and action in a unexpected and very fast-moving situation. I don't think that's a realistic expectation, except perhaps for trained personnel like airplane pilots.

Comment by LanceH 1 day ago

Stomp on brakes is pretty basic, and all that was ever needed for overpowering a prius's engine/motor.

This "scandal" was never about mechanical failures. It was almost certainly about driver error and mass hysteria.

As for Toyota settling, had this been Ford or Chevy, the government wouldn't have had the appetite to go after them for what was always a non-issue. It was just less expensive for Toyota to fix floor mats and pay a billion to put it all behind them.

Comment by helterskelter 2 days ago

Key?

Comment by chneu 2 days ago

Long press the start/stop button.

Comment by laweijfmvo 2 days ago

shift into neutral

Comment by ehnto 2 days ago

It was a 2005 model, so it should have been possible. However the article isn't super clear on where exactly the software is running, and the transmission controller and engine control unit can be interlinked in various ways. Especially more modern vehicles, it would be entirely possible to write code that disallowed shifting if it was an automatic. We have no idea just how poorly orchestrated this system was and what features were affected.

I don't know enough about 2005 Camry's though, so I wouldn't speculate much further than that.

Comment by fnord77 2 days ago

> Other egregious deviations from standard practice were the number of global variables in the system. (A variable is a location in memory that has a number in it. A global variable is any piece of software anywhere in the system can get to that number and read it or write it.) The academic standard is zero. Toyota had more than 10,000 global variables.

Comment by jdlshore 2 days ago

My understanding is that global variables are more common in embedded systems because it provides memory determinism.

Comment by Glawen 2 days ago

Yep that's the standard in embedded on bare metal without memory allocation. There is a mechanism in place to synchronise data during interrupts, so it's not really direct write. Usually also coupled with a two complement variable or similar to make sure memory is not corrupted for safety critical data.

Comment by monegator 2 days ago

nah, by looking ad other people's firmware it's because most of my embedded colleagues are goats stuck in pre-ANSI C. You can always declare a "global" static, so it's not on the stack or heap, and access that via functions.

Nothing wrong with source-file-level statics, you're bound to use them

Comment by 1 day ago

Comment by supahfly_remix 2 days ago

Does anyone know where one could obtain the firmware for this? It might be interesting to reverse engineer.

Comment by altairprime 2 days ago

It’s available in various archives of the Toyota TechStream pre-2024 editions, in some sort of weird encrypted file format that can be trivially decrypted; I haven’t tried myself but the ECU I work with isn’t encrypted in-vehicle. I’ve spent five or six years in Ghidra with various hybrid Subaru-Toyota ECUs from 2013-2020 and I wonder what kind of source control practices result in the massive function spaghetti that must have produced in this SH-2A code; I can see where Toyota bolted their direct injection runloop into Subaru’s. So, yeah, if you’re curious, the firmware’s out there, if you’ve got a few years to spare and an absolutely ridiculous amount of patience (and a solid grasp of CAN bus messaging protocols, which you’ll need to identify code blocks and variables and such!)

“The Car Hacker’s Handbook” may be of interest as a first step review, but honestly I just dove in with Ghidra and just .. didn’t ever stop. YMMV :)

Comment by supahfly_remix 1 day ago

lol, thanks for the reply. Yes, I was thinking of looking at it, but your comment dissuaded me. Out of curiosity, did you try running ECU code through an LLM and asking questions of it, and, if so, how did that work out?

Comment by altairprime 1 day ago

No.