Hacker News new | past | comments | ask | show | jobs | submit login
Coding Error Forces 2019 Subaru Ascents to be Scrapped (ieee.org)
124 points by georgecmu on Sept 23, 2018 | hide | past | favorite | 43 comments



Subaru indicated in the recall that “there is no physical remedy available; therefore, any vehicles found with missing welds will be destroyed.”

Something doesn't feel right, when the cost of adding the missing welds is somehow higher than the cost of a whole new vehicle...

If missing welds were discovered on an airliner, for example, the fix would almost certainly be to add them, and not replace the whole plane.


Disclaimer: I work for a Subaru competitor, any opinions are solely my own.

1. There are conditions where you cannot repair a vehicle and then sell it as a new vehicle (eg when you have to cut and weld). In these cases it's actually more cost effective to scrap it and rebuild it. Quite likely, most of the components of these vehicles will be used again on a vehicle with the same VIN. Scrap doesn't necessarily mean crush and recycle.

2. The weld process may only be certified on bare, unpainted sheet metal. After the paint process, basically none of the sheet metal is bare. Even the unpainted parts are coated with a pre-primer. Fun fact: sheet metal must be welded in the plant within a month of being stamped. (Aftersales consumer repairs are all manual work, so the same restriction doesn't apply)

3. As another commentor stated, even if Subaru had to eat the entire cost of production for these ~300, that is much better than a lawsuit. 300*50,000 = 15 million (probably more like 7-10 million). A single lawsuit could blow that away.

4. Cars are not really like airplanes. Airplanes have replaceable body components. Cars don't really have replaceable body components. A whole airframe for a large aircraft is probably worth millions. A welded auto body costs less than 5k in the manufacturing plant. A plane may take between a few days to a month to manufacture. A car plant makes hundreds to a thousand cars in a day.


Theres also generally a lot more space to work in a plane

Spot welds can also end up blind but I would say the painted part is the most problematic bit. And it's not worth the effort of trying to strip down the car to then put it back together again. And then be sure you got the process 100% right.

Seems more likely to remove all the valuable bits of interior, engine, suspension, etc and then roll a new body. If they even bother doing that.


A fun anecdote to explain the absurdity that is Auto manufacture.

When Ford made it's own vehicles in Australia, the engines were made in Melbourne, while the body and final assembly of the two was in Geelong. That's an hour away on the busiest freeway in the country

A friend is a process Engineer and did a big investigation into what they could do when the engines were late arriving due to traffic jams. Of course the whole thing was Just In Time because reasons, so they needed the engines to arrive just at the right time.

At the end of a multi-year study it was concluded the most cost-effective thing to do when the engines were not ready to put into cars that were ready for them, was..... let the cars fall off the end of the line, crush them, and eventually crush the engines when they showed up.

Any attempt to re-integrate them, or salvage stuff, or push them back on the start of the line, or anything else you want to think of costs more money than the above solution. They ran the numbers every which way, and as shocked as they were, it's the truth.

It was a rumor for a long time that every single time the production line was stopped, even for five seconds, it cost $1Mil. It was investigated, and found to be higher than that.


Sounds like having the plants separated by a road with traffic jams was very expensive.

Probably way more expensive than colocating the two factories.

JITs are really susceptible to shock.


Kind of makes you understand why Ford would no longer be making vehicles in Australia...


Even Just In Time systems need small buffers (inventory) to absorb random noise.


Getting to bare metal would require so much of the vehicles interior to be removed, the chances of doing it without damaging said interior would be next to none, then doing the spot welds depending how close they are to the exterior could affect the paint, which would in turn require repainting, which would in turn require stripping the whole vehicle back to get the whole paint job even [edit: to get a perfect finish on a portion of a repair indistinguishable from the rest of a car is pretty hard]

To to go from "as new" to "Bare metal" to back again would be a logistical nightmare. There'd be so many little things "oops, damaged that plastic piece with a screw driver...binned !". And even if you did manage to get the welds in, you'd probably have to re-schedule a whole bunch of new parts for production, just for that batch.

The cost to pull that off would outweigh a new car - as it would all be skilled automotive labour (remembering you couldn't reverse the robotics that assembled part of the car in the first place).

People would be expecting to buy new cars that would have to be perfect. Even if by some miracle you did pull off getting the cars back to perfect condition, imagine if (when) the new owners found out?

291 would represent x%age of cars for that model year. Just bin them.


Conceivably, they may have to dismantle the entire vehicle to get to a hidden weld, which could require compromising the structural integrity of the vehicle if dismantling is equivalent to cutting at any step in the dismantling process.


My first thought was that these weld points must be inaccessible without cutting the vehicle open. A weld point may be surrounded by metal structures that are themselves welded in place.

I'd love to hear from a mechanical engineer how likely it is that a structural weld point would be buried this way.


Hidden welds are very common, although I don't know if these exact missing ones are.

That said, cutting out entire pieces and replacing them is not uncommon; here's an interesting set of slides on replacing an entire B-pillar (the part missing the welds in these Subarus) on a Toyota:

http://www.ciclink.com/archive/technicalpres/2010-7-Techpres...

Note how the official repair instructions (surprised they even still publish these instead of "scrap the whole thing") literally say to cut the old one out and weld a new one in.


I'm guessing the welds aren't all that hidden, since the recall notice mentions that people will be inspecting the vehicles for them and if any disassembly is required it presumably won't take all that long since it mentions the B-pillar --- a part which isn't buried in a bunch of other stuff, unlike e.g. under the seats or the firewall.

In any case, a weld done right won't weaken the structure --- they can be as strong or even stronger than the original material. It can be a bit surprising, especially to anyone who learns to weld for the first time and then tests his/her joints to destruction, to see just how strong they are.


The hole needed for inspection (if any; inspection need not be visual) can be a lot smaller than what’s needed for adding a weld.


My initial thoughts were that the omission may have affected the integrity of the bodyshell or it requires a factory jig to ensure correct alignment for the weld.

Perhaps scrapping the 293 cars is cheaper and simpler than one potential lawsuit though.


> If missing welds were discovered on an airliner

Airliners are riveted together, as rivets are more inspectable and reliable.


Too much risk for the manufacturer. These welds are part of the unibody - what protects you in a crash.


I still remember when I started playing around with computers, 40 years ago in a Tandy store on TRS-80, you were told something like "it doesn't matter what you do on the computer, you can't hurt it or damage anything". Now everyday when I work (on networking and security for large enterprises) I constantly am thinking about how I have to really make sure the process I am about to execute isn't going to break something or disrupt someone's business operation. Rarely do I have to contemplate that what I get wrong might kill someone though!


Here’s an interesting series of articles on Risks to the Public by the ACM Special Interest Group for Software Engineering: https://dl.acm.org/results.cfm?query=Risks+to+the+public&Go....


This was really an inspection error. A human can miss a weld, too, so manufacturing lines rely on inspections. Someone was signing off on work that wasn't done. Whether a human or a robot was supposed to do that work is a nit.


From TFA: “[Subaru] found an Ascent with the missing welds during a routine sampling inspection on 21 July”

The affected vehicles were built between 13th and 21st.

Not sure how long QA takes for a vehicle, but they found it within a week.


sampling inspection. Not a hard software check: `assert ( numWelds == 30);`

Or a camera that takes a picture of each vehicles undercarriage. They probably thought that was unnecessary until now.


Could have been an error of requirements causing both the robot control code and any automated testing to be incorrectly implemented.


Yep, someone forgot to put those welding points and others didn't observe them missing.


Literally the whole focus of the article is that the machines had some programming error causing them to not complete the welds sometimes (but not always) and that this was found during a regular inspection.


No. It was found during a sample inspection. That’s too late.


I'm interested why over-automation caused so many problems in manufacturing plants.

I would think that once an assembly line has been set, full automation would be completely ideal and perfect. Is it because plants are always changing and getting tweaked, and changes require software updates, which invariably have bugs and take time to adapt to changes?


> I'm interested why over-automation caused so many problems in manufacturing plants.

A robot will uncritically do what you tell it.

An experienced welder will tell you you're an idiot when you're an idiot.


So is the issue over-automation or bad QA after the system is set up?


I would imagine theyd try to automate QA too, akin to unit testing

You could imagine a situation where the automated QA is trusted and does its job in sets, with humans interspersed to backup the QA programs. But one of the QA checks is actually buggy and OK’s the missing weld, but the full batch of work covers it up before reaching a human for the double-check

In which case, both issues are at fault: too much reliance on the QA program, and allowing too much work to be done before it reached a human (bad double-check setup)


I imagine if you have a list of welding points and feed that list to both the assembly and inspection robots, stuff like this happens.

Especially in case the robots share code and e.g. have the same parse error when loading your list of welding points.


> A robot will uncritically do what you tell it

And might well muck it up one time out of 50.


It's basically because these production lines are incredibly complex and getting tiny details right is important.

When you have humans running a line, they can notice and adapt to the countless possible issues that occur (like "the alignment here is often slightly out" or "tightening X here causes problems for the people in the next station"). Their top speed is theoretically less than a automated line, but they are quite adaptable and robust.

Since automated lines are less adaptable and robust, so you have to get them just right. Any tiny little issue can cause surprising problems down the line and implementing fixes is quite slow because you can't just describe the problem and required fix to a human and leave them to work it out.

I think the best way to set up a production line is to exclusively use humans at the beginning, and carefully introduce automation once the process is understood and stable


humans are a robust people


My guess is that the complexity of integrating sensor information and maintaining calibration isn't refined to the tolerances needed. In theory, you only need to detect physical resistance and visual variations (not like the assembly line workers are licking the bolts). In practice, you can have a series of bolts within tolerance limits, but causing a warp an attached piece that lies outside the preceding sensors' detection ranges. A human might step back and see that the door is bending weirdly, the door bend might not be noticed for several stations, causing more disruption.

That being said, car parts in general seems so unnecessarily complex. Too many curves, too little compartmentalization and modularity, etc. At this point in history, I would expect the average person to be capable of practically disassembling most cosmetic pieces in under an hour and with little research, but it's an obnoxious process of hunting down hidden screws and plastic clips in precise sequences. If cars were built in a reasonable manner, it should have been trivial to repurpose most of the parts in these vehicles.


I think for many operations automation can be perfect, but other operations are really challenging to automate well today.

However, I do believe if you were to design the vehicle for optimal automated assembly throughout, it could significantly change the situation.

But a design optimized for 100% automated assembly is probably not going to be as servicable. Imagine for example if wiring harnesses were built as rigid modules which were captive within the unibody when it was welded together. This could probably simplify the automated assembly, eliminating a step of snaking a flexible bundle of wires through passages, but then nobody is ever going to be replacing the wiring harness module.

I do think things can be done in the vehicle to improve the automation without compromising servicability, but it might not be economical, or maybe it just requires more control over the entire supply chain to tailor everything to the vehicle's assembly process.


One of the limit is reliability. Robot don’t self maintain... The more elaborate the task, the more complicated the maintenance to ensure acceptable failure rate. At some point you spend as much time fixing as doing the job. Especially in cases like here where the throughput target is craaaaazy.


They may be referring to the actual process of perfecting the automation. Bespoke CI/CD isn’t an instantaneous thing of beauty and efficiency gains.


Coding error? Maybe... but I would bet it all started in business where some management decision was made to replace "Bob" who programmed the bots for years was replaced by a cheaper less experienced person, all in efforts to reduce IT costs. So let's rename this thread, "Business Management decision forces 2019 Subaru Ascents to be Scrapped".


Despite being the title of the article it isn’t what it seems. The coding error was in the production of the cars which led to them needing to be destroyed.

The coding error didn’t literally send them to the crusher instead of the showroom floor as the title implies.


I was picturing some oversized Subaru-branded Rube Goldberg machine, with the final step being a shiny car being tossed into a crusher.


Now that’s news!


Ok, we've revised the title to disambiguate it.


When you move your plants to the US.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: