Hacker News new | past | comments | ask | show | jobs | submit login
737 Max: 1960s Design, 1990s Computing Power and Paper Manuals (nytimes.com)
118 points by bushido on April 9, 2019 | hide | past | favorite | 127 comments



Why exactly does this author think an airplane needs a lot of computing power? Airplane software does not deal with especially large amounts of data or intense computations. Faster hardware does not provide an inherent safety benefit.

There is a large degree of coupling between avionics hardware and software amd updating that software is prohibitively expensive. There's just no point in doing it unless you get a tangible benefit.

Furthermore, avionics systems take a long time to develop are not often rebuilt from scratch, so the hardware naturally lags far behind the computing power we're used to.

The recycling of avionics systems is definitely fueled by a desire to reduce costs. However, it's also important to note that older, mature systems with a history of service inherently offer a degree of safety compared to something which is untested.

The issue this article complains about can easily be seen as a safety feature.

To be fair, the author does spend most of the article talking about missing features and design issues rather than harping on "processing power". It is a poor title.


Wait until they finds out what software powers the nuclear weapons of 90%+ of the world, top-tier fighter jets, "critical" infrastructure like power (including nuclear), and countless financial/banking systems.

To be fair the article doesn't critique using 1990s (or earlier) hardware/software. It's merely a clickbait headline. The only reference is dismissed by the very next sentence as insignificant [Boeing could be replaced with any ~~expert~~ somewhat-knowledgable person on the subject]:

> The flight-control computers have roughly the processing power of 1990s home computers. A Boeing spokesman said the aircraft was designed with an appropriate level of technology to ensure safety.

This only adds to my lowered standards when approaching NYT articles over the years. They've long been abandoning being correct for being entertaining at a seemingly accelerating rate. Which is too bad considering plenty of people continue to hold them to a higher credibility over the typical clickbaiter sites. Considering they are a subscription service I'm not convinced this is justified.


I remember working in the early 90s on a training simulation of the reactor safety shutdown equipment for a UK AGR reactor - this device was pretty critical as it as responsible for sequencing the shutdown operations during a scram.

It was designed in the 1960s using technology they were completely confident would work and keep working - so it was basically 1940s electro-mechanical technology: relays, worm gears etc.

I don't think they really trusted electronics at that point, let alone computing devices for something so super critical.

No idea how that is done now though!


> To be fair the article doesn't critique using 1990s (or earlier) hardware/software. It's merely a clickbait headline.

The "1990s home computers" power was simply an illustration of Boeing just trying to do the minimum work to move forward. Other examples that, while not bad per se, seem to indicate shortcuts:

> If one of two sensors malfunctioned, the system could struggle to know which was right. Airbus addressed this potential problem on some of its planes by installing three or more such sensors.

> Most new Boeing jets have electronic systems that take pilots through their preflight checklists, ensuring they don’t skip a step and potentially miss a malfunctioning part. On the Max, pilots still complete those checklists manually in a book.

> A second electronic system found on other Boeing jets also alerts pilots to unusual or hazardous situations during flight and lays out recommended steps to resolve them. On 737s, a light typically indicates the problem and pilots have to flip through their paper manuals to find next steps. [...] Boeing decided against adding it to the Max because it could have prompted regulators to require new pilot training, according to two former Boeing employees involved in the decision.

All of these things still get the job done, but they seem to indicate that Boeing simply wanted to do one last turn of the crank with the 737 design to compete with the A320neo, and may have rushed thing too much.


The NYT fired basically their entire editorial staff in 2017. The accelerated degradation has been noticable.


This is a good and comprehensive article. You and the grandparent comment are focusing on one minor component and using it to raise FUD about the rest of the article and indeed the entire NYT now. It's a lazy form of debate. HN used to be better than this.


I'm just providing facts. NYT fired a ton of editors, because they're expensive.[1]

They've also hired younger, cheaper writers.

As a result, the overall quality of the product has fallen, noticeable.

[1] https://www.google.com/amp/s/deadline.com/2017/06/new-york-t...


I never said the article was bad. I said the title was bad and specifically addressed the complaint in the title which brought me to the article in the first place.


Everybody's talking about how we're going to pay for good content. How will newspapers operate in the future?

We don't need to pay for content. The world is full of people who will write for nothing more than getting their name out there. What we need are good editors. The news media are firing the very people who could get them out of this mess they ended up in.


I pay for good content, lots of writers/producers use crowdfunding for a paycheck. Papers like NYTimes have slid down to clickbait titles and Buzzfeed type articles to get readers. The war for media control is corporate vs independent.


I disagree. We've had independents, corporations, and mixed-mode content creation and delivery for many, many decades.

What we used to have that we don't have any more is a forced-delay, quality-control, gatekeeping function. This was fulfilled by the slowness of manufacturing printed content and good editors. Turns out that this is a place where cutting out the middleman, the wisdom of crowds, and automation worked against our best interests.


What? I googled, and found this: "The Times newsroom staff peaked in early 2008 at about 1,332 employees, [...] still employs about 1,300 in its newsroom."

What are the correct numbers?


> To be fair the article doesn't critique using 1990s (or earlier) hardware/software.

I actually edited in an extra paragraph at the end of my original post because I wanted to at least credit the author for talking more about design issues than "processing power" in the body of the article. The title is very bad, though.


>Why exactly does this author think an airplane needs a lot of computing power?

I think the way you criticize the article is about as faulty as the article itself.

This piece is about detailing how the 737 is an outdated design that got patched and patched till it more or less broke. The processing power is just /one/ aspect and is given as a point of reference, like the 1968 photo.

I am no expert in aviation, but I guess it's quite a safe guess that fly-by-wire systems need more processing power than mechanical/analog designs. And that a glass cockpit with a trouble database needs more than a blinking LED and a paper handbook.

Nowhere in the article does the author claim the 737 MAX crashed because it had too weak CPUs. It is, however, one indication of an obsolete design - newer jets like the A320 or the 787 have more beefy processors for a reason.

I know I worked in 1-dimensional transportation and their 486-based onboard-processors felt cramped, the software engineers had to hack around their limitations and they wished they could switch to something newer. I doubt 3-dimensional transportation is easier to implement.


Thank you for taking the time to write this critique of the parent. You're totally correct.


> This piece is about detailing how the 737 is an outdated design that got patched and patched till it more or less broke. The processing power is just /one/ aspect and is given as a point of reference, like the 1968 photo.

I address that in my post and I do not fault the author for pointing out design issues. My issue is with the implication that processing power is an issue.

> Nowhere in the article does the author claim the 737 MAX crashed because it had too weak CPUs. It is, however, one indication of an obsolete design - newer jets like the A320 or the 787 have more beefy processors for a reason.

Newer jets have faster processors because they're built on a newer system. If you're developing a new system, you might as well throw in some faster hardware. The cost of the hardware itself is nothing compared to the overhead development costs for avionics.


>If you're developing a new system, you might as well throw in some faster hardware.

From my experience, this is not how the development of transportation systems work. They throw in faster hardware if they need it, as it is insanely expensive to do so. And I still believe glass cockpits and fly-by-wire need more computing power.


Yes, the 486 or 586 are far more powerful processors than what you would actually need for everything but the cockpit UI. You could probably get away with a Z80 or something similar. The flight control computers in our space probes frequently have a similar level of functionality simply because it's more than enough.

The only thing a more modern processor buys you is more memory, a faster tick rate for your RTOS, potentially an MMU if it doesn't cause substantial timing skew, and high-precision floating point. In the case of an avionics system I would imagine the requirements for fixed-point arithmetic are very well-understood at this point, so floating point doesn't buy you a whole lot and potentially creates new and exciting problems.

So I would say that 90's-era computers are plenty powerful for what they're used for.


IIRC, the Airbus A320 uses several older CPUs like that, but also Motorola 68k CPUs (sometimes as a mean to have 2 functionally identical systems with pieces from different manufacturers and different implementations).

I would not be surprised if they still use 68k in the A320 NEO.

Between specification, implementation, validation and testing, the cycles in the aircraft industry can be quite long. 5 to 10 years usually. And it's also quite costly to validate a brand new system. That being said, given the safety requirement, it's for the better.


The goddamn space shuttle ran on 386's, and that was an upgrade from 8086's


i think the space shuttle used a couple of AP101's, which were 16 bit machines, with core memory, programmed in JOVIAL...


I think the author was referring to processors handling UI displays rather than a computer system that has only the processing power to only handle a blinking LED.

For example if a UI display showed:

"Warning: MCAS overriding pitch up. Disable?"

Is better than a LED #234 and LED 412 are on. Consult manual for instructions.

I would imagine that a lot of the fly by wire or control systems nowadays are either analog systems or very very simple digital systems (with extreme redundancy).


It's a moot point anyway - the could not change the UI beyond converting physical dials to virtual ones, because that would have required retraining which they wanted to avoid.

Even if they could have changed the UI they still needed to hide the existence of the MCAS system, again to avoid retraining, so it's doubtful such a clear an helpful error message would be allowable.


It isn't a moot point, it is literally the point of the article. The whole article is explaining the history of the 737 and explaining why they designed the max to avoid retraining. It mentions digital UIs, fly-by-wire, and other designs found in modern jets to paint the picture that the Max was successful in avoiding retraining, but that requirement restricted the engineers and led to a potentially compromised design.


You might be right. However, the author didn't show any evidence that the UI was limited by the available processing power. 1990s computers were very capable of displaying advanced graphic interfaces.


I understood the author the same way you did


We sent people to the moon in 1969. We can definitely live with 90s computing power when it's enough.


Modern aviation is characterized by huge networks of sensors as most flight management systems are, in fact, compute driven and, since they are real time, higher compute power reduces latency. Clearly, this version of the Boeing 737 was produced with cost containment as the first priority.


Is there any evidence that latency is an issue?


The "1990s Computing Power" comment is ignorantly harsh. The MAX (to the best of my knowledge) uses a similar AMD29050-based architecture as the 777 (where it was pioneered) and the 737-800/900. It's amply fast and very very reliable, hardware and software (setting MCAS aside which is new to MAX).

Any sane person would prefer the solid well-proven choice over the bleeding edge. People used to make similar comments about the AGC and the Space Shuttle GPCs e.g. that such-and-such-a-PC-was-faster. Yeah, so what?


I'm not sure I agree here.

There exist possibilities for far more safety with more compute.

For example, when a mechanical failure occurs (for example an engine explodes and partially falls off) the flight characteristics of the plane change dramatically. Current systems fall back to human control, and hope that human will be able to figure out how to control a now very "unique" machine.

Future systems could dynamically create new flight models based on collected data to be able to fly the machine in entirely new ways.

For an example of this, did you know it's possible to fully control a quadcopter even with 3 out of 4 rotors broken and no working fins/flaps? [1]. That sort of 'fly it how it has never been done before' is out of reach of human control, and might save lives.

[1]:. https://www.ethz.ch/en/news-and-events/eth-news/news/2013/12...


> For an example of this, did you know it's possible to fully control a quadcopter even with 3 out of 4 rotors broken and no working fins/flaps?

The linked article does not support your point, it shows a quadcopter losing one rotor which then goes into a kind of 'controlled fall', two (or even three) rotors is never mentioned and even the one rotor missing scenario involves the whole craft spinning (see video on the page).


This tech is just now making it into military aircraft in the F-35 (it’s designed to compensate for failures or combat damage while keeping the plane’s handling relatively consistent), so I would guess it’ll be a while before we see a civilian version.


"Figure out a way to fly the thing in the new, crippled configuration" is hard. Figure out a way that's better than what the human pilot came up with may be even harder.


It's hard, but it's a very measurable problem.

You randomly deform the airframe in a physics simulator, then get a real pilot to try to fly the simulated damaged airframe to the nearest airport.

You then get an AI to try doing the same.

As soon as the AI manages to get to the airport more often than the human, the tech will save lives, on average.

My guess is a PhD student could come up with an AI passing the above test inside a year. Yet aviation standards are stringent enough AI will be decades away from production use, if ever.


Your link to ZTE trumpeting the success of an algorithm helping an actual physical model of a quadcopter with a disabled rotor operating in lab conditions to crashland a bit more softly might be interpreted as a hint that the general case of safely landing a stricken aircraft is the sort of problem that probably isn't solved by a single PhD in a year...

Keeping those with enough hubris to say "yeah, shouldn't take long if we've got some sort of sort of simulator to devise scenarios for a neural network to overfit to" about a wide range of failure modes that an entire field has spent decades studying well away from safety critical systems might be one of the more underappreciated aspects of tight regulation.


The political problem is that people still trust humans a lot more than they trust machines. So passengers would be nervous about being flown by an AI, even if the AI was safer.

There's some weird psychology behind this: human pilots/drivers/etc are seen by passengers as a personal proxy, with agency over any situation.

If you take away the proxy you take away the illusion of agency. Humans really do not like being put into situations where they believe they have no agency at all.

As AI gets smarter, this will become more and more of a problem, until there some kind of cultural shift because AI is obviously safer it's not even a question any more.

This may or may not happen.


There also exists possibilities for far more danger. As you increase compute, you can run more code, and build more complicated interdependent components that are more difficult to test, especially for the sort of unexpected scenarios that "dynamic" behaviour would warrant. You would basically have a whole new plane - the opposite of what the 737 MAX 8 was intended to be.


The "1990's Computer Power" comment is factually correct.

According to wikipedia, the AMD29000 processor was first released in 1988 and "In late 1995 AMD dropped development of the 29k..."

I'm not commenting on whether that's appropriate to be running your aircraft or not, but it does very much seem to be a 1990's CPU running approx 50mhz.


> The "1990s Computing Power" comment is ignorantly harsh.

I don't think so. You build a plane that's expected to run for decades -- if you put a 2010 computer in there now, it'll be 20 years old in 2030.

That 1990's computer was put there in the 90's, does that really make it more reliable?


The amount of processing that those computers do won’t increase over time, it stays the same. So why use something you won’t need?

Also, planes are extremely expensive so they don’t get thrown away, they get constant upgrades. The U2 spy plane is a good example: a 1950/60s airframe running with more modern instruments. So if the need arise they will upgrade those CPUs on the 737s.


Sometimes even the instruments are maintained. I know a consultant whose job is to repair bespoke factory and laboratory microcontrollers in situ. When we hired him he spent two weeks covering our lab in blueprints, two weeks troubleshooting, and one week programming. 25K to fix the device, but much cheaper than a new one.


> So why use something you won’t need?

The whole point of this issue that the technology and cockpit design is purposely 30 years old to prevent the need for re-certification not for safety issues.

I'm not suggesting technology be used that isn't needed; I'm just suggesting using the technology to make air travel safer and easier for pilots.


That so-called 1990s computer is probably a year old, embedded in a Honeywell module that’ll be available new for the next 15 years.


Of course. It's not literally a 1990's computer. But the idea that it's just somehow superior because the design is 3 decades old doesn't make any sense. 30 years ago it would have been new technology; so where was that same argument then?


It's had 3 decades of experience and bugfixes.


If new developing is happening, as it was in this case, that doesn't really matter. This problem was amplified by maintaining old technology, not mitigated by it.


I don’t agree. The failure(s) weren’t in the old stuff, they were in the poor and ultimately unsafe cover-up (for lack of a better word) of the new.

The CG change from the new engines specifically, being masked by new software, MCAS. The only old part culpable is the legacy 737 limit of 2 AoA sensors (with only 1 used for MCAS input (a new thing)).


If they had used the new technology available in other jets -- the pilots wouldn't have been sifting through paper manuals to find the problem. The would have an actual display.

And if they weren't trying so hard to purposely avoid modernizing the cockpit, the issue would have been avoided altogether.


The problem was the MCAS. That's new technology.


Kind of unfair. Do they really think that an aircraft should be powered by the same software or hardware that powers a modern computer or smartphone? An aircraft needs incredible reliability. You don’t get that with modern software/hardware architectures.


Yeah, the B-52, adopted in 1952, is planned to be is service through the 2050s. An airframe design expected to last for over 100 years. Keep in mind that flight itself is barely 100 years old.

And paper manuals? Are they expecting to use iPads for in flight documents or something? Shall we compare failure states of paper vs tablets?


"And paper manuals? Are they expecting to use iPads for in flight documents or something? Shall we compare failure states of paper vs tablets?"

That cat is already out of the bag. "According to the FAA, Class 1, Class 2 and Class 3 EFB may act as a substitute for the paper manuals that pilots are otherwise required to carry with them. "

https://en.m.wikipedia.org/wiki/Electronic_flight_bag


That article says that other plans will display the error and the recommended checklist automatically, which certainly seems faster than flipping through a bunch of books trying to find a specific page based off of some lights.


A lecturer of mine once said - knowing what kind of people we teach here for programmers, I am scared of going to the doctor.

The less software is in something(and less cloudy/iot) the better. And the more I program the more I prefer stupid and mechanical things when reliability is on the line.

A paper manual is always there for you.


> And the more I program the more I prefer stupid and mechanical things when reliability is on the line.

Magic is great. Until it's not.


>A paper manual is always there for you.

Then have the paper manual as a choice and/or backup.

If the programmers working on aircraft are so incompetent that they can't get a the digital manual correct, than they have no business programming any of the other multitude of systems that are critical to fly-by-wire systems.


And the time it takes to realize your iPad has run out of battery or you accidentally open an advertisement before grabbing your paper manual is too long.

The manuals have never been an issue. In fact, having a physical, rigorous checklist has been shown to improve the likely hood of a successful outcome.


iPad (and others) based electronic flight bags are already in place on passenger airlines. Your points are valid, but the debate is pretty much over.


It has to be physical for the checklist to work?


The app for my bicycle computer will be obsolete before my bike. On the other hand, my car computer has been running flawlessly for 25 years without any updates.


There's an old joke, if Microsoft made cars... https://www.hcs.harvard.edu/pnw/microsoftjoke.htm


Actually, yes, lots of pilots prefer ipads. They are lighter, for one, and for many tasks faster than paging through paper.


>They are lighter, for one, and for many tasks faster than paging through paper.

And are them in your opinion also as reliable as paper manuals?


No, they're not. They can run out of battery, have OS issues, and so on.


There are still DC-3s from the early '30s in service.


The old saying is "the only replacement for a DC-3 is another DC-3"

Though I think most of them that are flying are C-47's from the war era.


Mass media in general is stressing too much the fact that 737 MAX is allegedly based on dated designs.

I think it's fair to say that MAX versions present some compromise solutions (like the now-infamous MCAS, which is there to compensate for the "unnatural" bigger engines). But I think that is not the main point. They would be good solutions it they worked as intended.

There are some other more fundamental and more daunting, afaik unanswered questions.

Like, why does such a critical system like MCAS take only a single AoA sensor as input, when there are two sensors available? Specially considering that the inputs from both are hardware-available to MCAS (the new software version is going to take data from both).

Boeing affirmed in its manuals that the elevators would be able to compensate for the trimmed vertical stabilizers. Now the preliminary report in the Ethiopian's crash shows that the pilots wheren't able to perform such compensation, even by pulling the control columns all the way back.

Those and some other issues are much more critical.


>Like, why does such a critical system like MCAS take only a single AoA sensor as input, when there are two sensors available? Specially considering that the inputs from both are hardware-available to MCAS (the new software version is going to take data from both).

Are you saying the answer in the article didn't give enough information?

"Airbus addressed this potential problem on some of its planes by installing three or more such sensors. Former Max engineers, including one who worked on the sensors, said adding a third sensor to the Max was a nonstarter. Previous 737s, they said, had used two and managers wanted to limit changes.

The angle of attack sensor, bottom, on a Boeing 737 Max 8.CreditRuth Fremson/The New York Times “They wanted to A, save money and B, to minimize the certification and flight-test costs,” said Mike Renzelmann, an engineer who worked on the Max’s flight controls. “Any changes are going to require recertification.” Mr. Renzelmann was not involved in discussions about the sensors."


No it doesn’t answer the question. If there are two sensors already installed, not requiring change, how come the MCAS only uses one, not both?


Instrumentation on an aircraft is usually designed with a pilot and copilot set. Everything down to the pitot tubes which feed the information is unique to that side of the aircraft. This harks back to the day when it was the only sensible solution when your guages are actually just directly reading airpressure differences.

Clearly these signals can be cross connected, because that's the solution Boeing are testing at the moment, but it's outside the normal design of aircraft systems.


Uhm.. I don't know.. I'm aware this is the case of airspeed indicators and many others, but, for instance, the autopilot is fed with readings of Pitot tubes from both sides and it disengages when the sensors disagree beyond sensible thresholds. Besides, such issue is brought clearly to the knowledge of pilots.

In the MCAS case, however, we are speaking about a computer which not only interferes in the flight controls, but also does that in a way impossible to override and it's too difficult for the pilots to spot the root cause.


To be clear, I think MCAS was a massive engineering failure for Boeing, I'm just pointing out why I think they didn't automatically say "hey, we've got two sensors, lets compare the readings before we act on it".

However, for the autopilot example, I don't think you are quite right. Typically there are multiple autopilots (2 or 3 is quite common), and each is driven by a different set of flight data. In some scenarios multiple autopilots are engaged at same time, for example during CATIII auto-landings but as far as I'm aware that is the exception rather than the rule.

It's true that the autopilot may disconnect if there is a warning like 'IAS Disagree', I suspect this is driven by a separate monitoring process though, rather than being an integral part of the system.


The article also states that 2 sensors is not enough as then you have two sensors that can disagree with no way to figure out the correct one.


Yes but at least you know the reading is faulty and avoid applying dangerous commands. Now I don't know if the stall it is meant to avoid is a greater risk than MCAS pushing the plane in the wrong direction.


Stalls can be extremely difficult to recover from as it means the wings have lost lift, and therefore the control surfaces (which you need to regain stability) have reduced or even complete loss of effect - a so-called 'deep stall'. As bad as MCAS is, it could theoretically (in practise, couldn't) be switched off in this scenario and the plane would be flyable. In the imagined scenario where the plane pitched up and began to stall, Beoing's logic is that without MCAS, the plane would be essentially doomed. Air France 447 crashed due to a (pilot-induced) deep stall; it was otherwise stable at cruising altitude.

The root cause is without doubt relying on a single sensor, and then downplaying the importance of the system so that nobody opted for the additional expense of the extra sensor. Boeing also have to answer for their lack of transparency; their flight control logic has always left the pilot fully in control of the plane, and can override any automatic system. This sets them apart from Airbus, which under almost all circumstances will defer to the computer.

In ways, the 737 MAX crashes are the antithesis of the 447 crash - the pilots thought they were in full command of the plane, whereas an automatic system designed to protect them malfunctioned, versus the pilots in the Air France plane believed the computer would protect them from exceeding the plane's capabilities, whereas the plane's computers could not get reliable data and passed full control to the pilots.


The analysis is not as simple as asking whether a stall is more or less dangerous than an MCAS failure. Firstly, MCAS does not prevent a stall; it is intended to make it harder to accidentally stall (and no sane pilot would deliberately stall an airliner in normal operations), in order to compensate for the design change that made it easier to do so. When considering alternatives such as whether to disable MCAS on a sensor discrepancy, one should ask both how likely each possible scenario in each alternative is, and how much risk it adds.

Where the risk analysis seems to have gone most wrong is that Boeing apparently grossly underestimated the difficulty of both figuring out what actions were needed to respond to the symptoms of MCAS failure, and to perform them. I don't know whether it was a significant factor in the former, but when the AofA sensor failed, it caused the stick shaker, as well as MCAS, to kick in.

The other mistake in analysis seems to be that when the power of MCAS was increased after initial flight testing, the additional risk it created was not properly taken into account. In particular, the ability of MCAS to drive the trim all the way forward appears to have been an unintended and overlooked side-effect of one design change.


It doesn't answer the question because it's still a speculation.

Full investigation is hopefully going to tell us what has really happened.


>Like, why does such a critical system like MCAS take only a single AoA sensor as input

The classic approach is to have three sensors, so in case one fails you can know which one. Having two only indicates something is wrong but is not useful on the fly.


Of course, even with triple-redundant systems failures can still occur.

Air France 447 [1] three independent air data systems, two of them failed due to environmental conditions

XL Airways 888T [2] three independent AOA sensors, two failed because the plane was washed without the right covers in place

US Airways 1549 [3] two independent engines, both disabled by bird strike at the same time (No fatalities)

Qantas Flight 72 [4] three independent inertial reference units, bug in voting system if a single sensor's output had multiple spikes 1.2 seconds apart (no fatalities)

An in the data centre, no amount of power-supply redundancy will save you if a technician pulls out the power cables on the wrong server :)

[1] https://en.wikipedia.org/wiki/Air_France_Flight_447#cite_ref... [2] https://en.wikipedia.org/wiki/XL_Airways_Germany_Flight_888T [3] https://en.wikipedia.org/wiki/US_Airways_Flight_1549 [4] https://en.wikipedia.org/wiki/Qantas_Flight_72


Three with a disagree algorithm is definitely what I’d expect out of such a critical system, but two with signal averaging would still be much better than just one.


Or alternatively with two, and disabling MCAS if they disagree, seems a better solution than having one and having no way to tell if it is working (keeping in mind both can still fail simultaneously). Not an ideal solution but better.


This is one of the features of the MCAS software update.


Unless I am misunderstanding what signal averaging is (quite possible) isn't it possible that in situations where the average of a signal is still going to crash the plane, a 50/50 guess is actually more likely to end up with a better chance?

If true, it's possible signal averaging isnt necessarily the best choice


Averaging would be disastrous in this case. You don't want to use faulty data to average inputs to a flight control computer.


> The classic approach is to have three sensors

Which can also fail:

In 2008, on a customer-acceptance flight of an Airbus A320, two of the angle-of-attack sensors froze and those two sensors then outvoted the third. When the pilots went to demonstrate the stall-prevention system, they were not aware of the malfunctioning sensors. The plane crashed, killing the seven people on board.

The same problem arose again on a 2014 Airbus A321 Lufthansa flight leaving Spain. Eight minutes after takeoff, two of the angle-of-attack sensors froze at the same pitch. This time, after a drop in altitude, the pilots were able to regain control and complete the flight. [1]

I don't think the fundamental problem with MCAS was the number of sensors, but that it was too difficult for the pilots to override MCAS when it faulted.

1. https://www.seattletimes.com/business/boeing-aerospace/a-lac...


Boeing has been using only two for quite a time. Having a failed and a working one would simply indicate that something is wrong, but this information is valuable anyway -- it could be used to prevent MCAS from engaging based on wrong data, exactly one of the features the new software update is bringing.

Besides, those AoA sensors are EXTREMELY reliable. So reliable that some have raised the hypothesis that the real problem is not in the sensors themselves, but in some piece of hardware or software between them and the flight computers.

It seems plausible to me since failures in those sensors are too rare in the other planes but, despite that, they allegedly failed in two 737 Max 8s and in a really short timespan.


They designed it precisely like this because it's a dated design. To use two sensors (or three) would have required new cockpit components and new pilot training, which is what they were trying to avoid by reusing the same Type certificate originally from 1968.


Hm. I hesitate to defend Boeing here, but I think the outset of this article is a bit unfair.

> Pilots start some new Boeing planes by turning a knob and flipping two switches.

> The Boeing 737 Max, the newest passenger jet on the market, works differently. Pilots follow roughly the same seven steps used on the first 737 nearly 52 years ago: Shut off the cabin’s air-conditioning, redirect the air flow, switch on the engine, start the flow of fuel, revert the air flow, turn back on the air conditioning, and turn on a generator.

So? What does this have to do with anything? Is the goal to produce an airplane where pilots press a button "fly to destination", and the plane does it?

> The strategy, to keep updating the plane rather than starting from scratch, offered competitive advantages. Pilots were comfortable flying it, while airlines didn’t have to invest in costly new training for their pilots and mechanics. For Boeing, it was also faster and cheaper to redesign and recertify than starting anew.

> But the strategy has now left the company in crisis, following two deadly crashes in less than five months.

How was it the strategy to keep updating the plane that left to this crisis? The strategy itself is not to blame here, and I very much like the idea to gradually improve a proven model. It was a bad execution of this strategy that left the company in crisis.

In Germany, the national train agency Deutsche Bahn (and its predecessors) basically had a policy for nearly a century to order rolling stock that was designed to be produced for around 40 years. During this production run, the model was gradually improved. Some of the rolling stock designed in the 50ies is still in use, and quite reliable at that [0]. During the 90ies, agency and industry switched to a policy where basically every train generation was newly developed from scratch (for example, the ICE high speed trains). Guess what - you can channel a lot of public money into private hands that way, but bleeding edge technology is not what you want or need when you are trying to build a reliable transportation system.

[0] https://de.wikipedia.org/wiki/N-Wagen


> So? What does this have to do with anything? Is the goal to produce an airplane where pilots press a button "fly to destination", and the plane does it?

When you're nose down heading into the ground, taking your hand off the stick to adjust the trim can get panic-y. Especially since the MCAS may be working against your trim adjustments.

The design of the Max line seems to be just enough compete with the A320neo, and just enough to not "need" re-certification. But these two justs, together, may have cost several hundred people their lives.


When it comes to flight computers it makes sense to be conservative when it comes to computing power - the goal should be simple, proven, and functional. But when it comes to diagnostics why not use more computing power? As the article states A second electronic system found on other Boeing jets also alerts pilots to unusual or hazardous situations during flight and lays out recommended steps to resolve them. Seems better than flipping through a paper manual when you've only a minute or two to save the plane.


Because new diagnostics mean new pilot training, meaning the new plane is not economical to buy when compared to the competition, meaning no plane at all.

Those other Boeing jets required costly new training. Boeing were trying to build an update to the 737 that did not require this.


The thing that baffles me - why the hell has there never been any innovation in the form of making it easier to diagnose a problem and move through the necessary steps in a checklist beyond fumbling through a paper manual? In a situation where seconds count, it seems illogical that this has never been improved upon. Didn’t at least one of these planes crash because the pilots are believed to have only gotten X number of steps through a list of possible reasons?


As mentioned in the article, other Boeing jets do automatically present a checklist on the display when they detect a problem:

> A second electronic system found on other Boeing jets also alerts pilots to unusual or hazardous situations during flight and lays out recommended steps to resolve them.

The 737 is stuck in the past because given the choice, nobody wants to have to retrain their pilots on more modern systems.


>The 737 is stuck in the past because given the choice, nobody wants to have to retrain their pilots on more modern systems.

Ding ding ding!


Why is anyone being given the choice?


I remember hearing about the US Airways flight that had an emergency landing in the Hudson river (the film 'Sully' is based on this). The plane sunk faster than it should have because the checklist item 'close the vents' was too far down to be completed before the water landing. In response, the checklist was edited and the item was moved further to the top.

Apologies for not finding an exact link, but here is the wikipedia article for the flight: https://www.wikiwand.com/en/US_Airways_Flight_1549


Air France flight 447 had too many alarms go off, and the pilots could not cope. Turns out that getting mid-flight problem diagnostics right is a really difficult problem.

Now, in the Air France flight, part of the problem was the system trying harder than it should have: it would have sufficed to alert the pilots to the lack of air speed indication and let them figure out what to do, but instead the computers cascade alarm after alarm.

You can see the problem: various pieces of the system were engineered separately, and there was no single system that could have suppressed the downstream alarms so that the pilots could focus on first on understanding the first alarm. Then again, that too might not have been a good design: perhaps a stall alarm should take precedence, say, over a frozen pitot alarm.


There is a caveat to the Air France 447 crash, that the pilots apparently relied too heavily on the autopilot. When the alarms came on mid-flight, neither pilot was familiar with the current state of the plane and found it more difficult to diagnose the problems. One of the recommendations to come out of that crash was that pilots on long flights should occasionally switch back to manual control to keep themselves synced with the aircraft's state.


The entire aviation business and safety structure is build on extraordinary rigidity to processes and approaches. In addition, automation is viewed as a threat to multiple worker groups.

Sometimes this is justified -certainly the MCAS incident is a example. Changes like the MCAS should never have been allowed to change a fundamental part of the airplane interface without considerably more processes being met. That said, sometimes opposition to enhancements is trade unionists being trade unionists.


The newer airbuses actually do have electronic checklists including - for at least some steps - automatic detection that the step has been completed. You can see an example of the electronic checklist in this simulator video: https://www.youtube.com/watch?v=jzjyr0UZKRw

Even on aircraft without fully integrated checklists, many operators have switched to using iPads instead of paper manuals as well.


So do the new platform (e.g. not 737) Boeings. The retention of paper on 737MAX is all about not changing anything to avoid having pilots require simulator training. It's also why it has a glass cockpit that mostly shows digitized versions of the 60s gauges.


Does anyone else here feel aeroplane software should be mandated as being open source? Boeing would have never even dreamed of releasing the 737-Max software in the state it was in and the more eyes you have on things like this the better.


Do you think there are a massive amount of developers out there who are at the same time avionics experts? I just don't see how being opensource would help. You wouldn't be able to even test these software projects because it requires and emulator or actual hardware to run on that you do not have. What would it chage if it was opensource?


I think the argument you're making is completely false.

First Premise is that there re not many avionics expers - I agree so having more of those experts able to look at review and learn from different implementations would be a good thing.

Second premise is that you need to be an avionics expert to review and improve avionics software, I'm not convinced, there are many clever people out there.

Third premise you can't test or run the software becuase there is no emulator; maybe if the software was open source someone would start writing an emulator - maybe an emulator could be open sourced?

Fourth point; nothing would change if it was open source - I think the quality of code that would be released would go up and that's the main thing I'm talking about, embarrassment would be the optimal solution to something like this - I'd be very suprised if Boeing would release software that used a single sensor as input to a critical system like this.


I see so basically it would mean no benefits at all. Thanks for confirming.

Now the other problem that I have this in this thread. The Boeing issue is not a software issue at all. Even if we chose the best case scenario for the software it would be still a physical design & lack of pilot education problem.


Airbus agrees Boeing should make its software opensource.


Yes, I feel this way. Public skies, public code.


The entire management chain that dictated these design compromises should be surgically removed from Boeing and prosecuted for negligence.

Of course, that will never happen.


This doesn't get said enough.

Within days of the Lion Air crash, MCAS was strongly suspected to be the problem. And yet Boeing management allowed 157 more people to die in the Ethiopian Airlines crash. And then the Boeing CEO still tried to tell everyone that it was a safe airplane.

The entire top management of Boeing, and the entire board of directors, should be perp-walked out the door.

Of course, that will never happen.


>In the recent crashes, investigators believe the MCAS malfunctioned and moved a tail flap called the stabilizer,

Fact checking at the NYT seems to be dead. The MCAS moves the entire stabilizer -- which is not a "flap" in any sense. Can they not find a pilot or some other knowledgeable person to read through an article like this before publishing it?


I'd rather take a well tested i386 than a new processor that crashes Linux with microcode errors.

Paper manuals actually instill confidence. Should pilots contact Stackoverflow in an emergency?


I guess pilot retraining will always be time consuming, but can anything be done about recertification so that new designs don't face this issue?


Good grief. No!

Because as we are literally discovering, not knowing the details of how and why the automatic stabilization software is driving your trim wheel can kill you. The biggest single part of the failure chain here was the fact that this should have been an invasive change requiring recertification as a different aircraft and retraining for pilots instead of pretending that it was just like a 737-600.


You are on my page. Type certification is the process to discover issues like this.

I'm not familiar with pilot training and certification but it seems to me that even if the 737 MAX was type certified certifying a pilot with previous 737 experience would have been just supplemental training. Ultimately a nothing burger.

I've seen a lot of cases where people get bent thinking of all the bad things that will happen because they think customers won't like some change. Always in the end if it's not a show stopper the customers just lump it.


I'm sure this conversation will be looked at during discovery in any future lawsuits:

> Boeing also designed the system to rely on a single sensor — a rarity in aviation, where redundancy is common. Several former Boeing engineers who were not directly involved in the system’s design said their colleagues most likely opted for such an approach since relying on two sensors could still create issues. If one of two sensors malfunctioned, the system could struggle to know which was right.

> Airbus addressed this potential problem on some of its planes by installing three or more such sensors. Former Max engineers, including one who worked on the sensors, said adding a third sensor to the Max was a nonstarter. Previous 737s, they said, had used two and managers wanted to limit changes.


Three angle of attack sensors should have never been something that could be overridden by cost savings measures. The engineers wanted three sensors and the management said no without ensuring all planes sold had sufficient mitigating controls. We need justice.


>> “They wanted to A, save money and B, to minimize the certification and flight-test costs"

This seems to be the root cause of the problem. Competition is not always good.


I think Boeing was in a wrong mindset: certification was seen as a burden instead of being seen as an help to build a better aircraft. But maybe it was not only the fault of Boeing. The certification process of FAA is perhaps not friendly enough toward big air-frame changes (like putting the wings slightly higher to make room for the bigger reactor). Here, the certification process has not identified that a failure of one AOA sensor was able to cause a difficult to avoid crash. Maybe we should allow the use of less reliable source of data (like GPS, ...) each time a sensor has a problem.


Except rumors from media there is no clear explanation about what is going on inside Boeing. I just hope Boeing management is not the old Microsoft management.


While i386 may be enough to run a plane, what other software or hardware of the same age is still in use?


The Windows XP of passenger jets.


Windows XP was actually pretty modern for its time. Maybe you mean Windows ME. ;)


Thank you for reminding me of something I'd rather not remember.

One thing I remember is Microsoft tried repeatedly with Win95, 98 and ME to get USB support to work correctly. Failing each time and having to start over. Much to the horror of peripheral manufacturers. With Windows XP that finally mostly worked.


[flagged]


I'm on the fence on how bad the regulatory system is. Boeing has a massive incentive to make safe aircraft (they're seeing it first-hand right now), and modern commercial aircraft might be too complex for the FAA to effectively certify without input from manufacturers.


I vehemently disagree on the too complex front. If there is any inability to X going on in the FAA it is because of a culture of continual budget cuts hamstringing the ability of the agency to acquire and keep talent.

The current incarnation of the FAA is a joke. This is not some bureaucratic paper pusher organization. This is the agency responsible for the overall architecture and requirements that historically made the American Aerospace industry a force to be reckoned with when they weren't being blindsided or hamstrung by excessive political pressure by industry.

My proof?

There is nothing complicated, about the trim system of a 737 MAX. I've sat down and done a bit of research, reverse engineered the basic mechanical components of the system, and I'm just some bloke who had a few hours to kill on the weekend.

It's a highly geared mechanical gear train, with a secondary set of electrical motors, connected to a jackscrew, which drives the horizontal stabilizers while feeding back electrical actuation information back to the pilot's via the mechanical gear train. At some point, they wired in extra electrical connections from whatever black box MCAS is implemented in into the command circuit for the electric trim motor, and called it a day.

There is nothing complicated there. It's a basic composition of simple machines. The complicated part is the implementation and logistics details, in addition to making sure at some point, all those pesky details are written down, all those integrated systems are tested/verified to be of sufficient quality, and that each component has an audit trail to be referenced should investigation be necessary.

All that extra information is required specifically because it has been found to lead to increased safety over time. The FAA's duty is to ensure at a minimum, every player is living up to and respecting that standard. Something the history of budget cuts and delegation of authority to manufacturers has very clearly jeopardized.

I don't like the FAA. I'll never even be able to take the physical pilot training to fly because of the draconian nature of their medical certification regulations, but I can't even argue that they are unjustified in terms of the catastrophic outcomes that are prevented by having them there.

It is possible to hold a manufacturer to standards via external impartial audit.

It is difficult to do so if you can't even support the billets for a set of Subject Matter Experts who don't need to resort to alternative revenue streams so they can concentrate on doing one thing and doing it well.

There just needs to be the realization that at some point there must be a figure in the relationship between operator and manufacturer with the capacity, nay, responsibility to stop something which has not met the minimum standards of airworthyness.

It ain't physics that's the problem, it's the people. Machines don't lie. Even when malfunctioning, they're telling the truth in as much as the physical principles they are subject to will allow.


That's an excuse, not a reason. If you don't understand regulatory capture, then you're not paying attention.




Consider applying for YC's Spring batch! Applications are open till Feb 11.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: