Private pilot here. This story buries the lede. There were tail strikes, which can be serious, but usually aren't, and in this case weren't. But the reason there were tail strikes is because software told the pilots to take off with throttle settings that were too low, and that could very easily have ended catastrophically. If takeoff power settings are too low and the pilots don't notice in time, you can end up in an unrecoverable state where the aircraft cannot take off but there is not enough runway left to abort. The passengers and crew on these flights got very, very lucky that it was "just" a tail strike.
>> The data “confirms that the airplane was safely airborne with runway remaining and at an altitude by the end of the runway that was well within regulatory safety margins,” said the union’s Alaska unit chair, Will McQuillen, in a statement.
Sure, but you're missing the point. This particular bug produced an erroneous output that resulted in a tail strike. A different bug, with all the other circumstances unchanged, could have produced an erroneous output that resulted in a catastrophe. One should not shrug this off as "oh, it was just a tail strike, therefore it was no big deal." It was very nearly a very big deal.
If the data was too far off it is likely the pilots would have noticed it. This was a situation where it was off, but still "sane," so went unnoticed.
The airline completely grounded their entire fleet while they investigated this issue, so I don't really understand why you're trying to act like nobody took this seriously. The article makes it quite clear this was handled as a serious problem and steps taken.
The Alaska Airlines' pilots union is both qualified to have an opinion AND looking out primarily for their members (i.e., pilots), and they had this to say:
> The data “confirms that the airplane was safely airborne with runway remaining and at an altitude by the end of the runway that was well within regulatory safety margins,” said the union’s Alaska unit chair, Will McQuillen, in a statement.
If the union was unhappy with the remediation steps taken, they'd say so. That is quite literally their primary function.
> This was a situation where it was off, but still "sane,"
No. The margin for error in jets is very small. Jets are much less forgiving than Cessna 172s. The difference between a tail strike and a failed takeoff is very small, easily under the threshold of human perception until it's too late. Also, the whole point of using this software in the first place is so that every takeoff is done right on the hairy edge of minimal performance. It is a catastrophe waiting to happen. This was a wakeup call.
> If the union was unhappy with the remediation steps taken, they'd say so.
It is quite possible that no one in the union appreciates the broader implications. Pilots are generally not software engineers.
So your point is that professional airline pilots, their union, or an airline are less qualified to understand the safety implications of this incident than a PPL because that PPL is also a software engineer? Are you familiar with the Dunning–Kruger effect[0] and how it may be playing a role in your perspective of this incident?
Not just for that reason. I'm also not bound by the political constraints that union leaders are. They have to pick their battles. I'm free to say whatever I want about this with little risk to my career.
But yes, I think I understand software -- and particularly software risks -- better than most pilots, and I think I understand flying better than most software engineers, so I think that makes my opinion better informed than most. (It is nonetheless just an opinion.)
> Are you familiar with the Dunning–Kruger effect
Yes. Are you familiar with the ad hominem fallacy?
It doesn't apply. When an argument is backed by someone's qualifications (in this case PPL and software developer), discussing the merit of those qualifications for the discussion domain is absolutely relevant. Your first reply to this thread started off with:
> Private pilot here.
And now that I'm calling into question how relevant that is, or more to the point if you're in a position to know better than the pilot union/airline/etc, it is suddenly Ad hominem.
Cannot have it both ways, either your qualifications make your opinion more relevant or they don't. But pick a lane.
The implication with calling it Ad hominem though is that your qualifications aren't relevant, which I happen to agree with, and actually that was my point.
When you claim he is affected by dunning kreuger you are by definition attacking the person instead of the argument. Hence "ad hominem"
Then when stating the authority of pilots and their unions you are not attacking the argument with a counter argument. Instead you claim you are right because of the authoroty of the union. Hence argument from authority.
You don't seem to understand that Dunning-Kreuger effect. Let me quote from the article that you cited:
"The Dunning–Kruger effect is a cognitive bias whereby people with low ability, expertise, or experience regarding a certain type of task or area of knowledge tend to overestimate their ability or knowledge."
In other words, by invoking D-K, you have implicitly said that you think I am a person "with low ability, knowledge, or expertise" regarding the matter at hand. But I am a licensed private pilot with 30 years of flying experience, and a software engineer with a Ph.D. and over 40 years of professional experience, including over 15 years working at NASA and at an aerospace startup I co-founded. Those are all just facts. In the face of those facts, an implicit and unsupported assessment that I am a person of "low ability, expertise, or experience" is an ad hominem, essentially saying, "Yeah, you have all these credentials, but you still might be an idiot." Well, yeah, I might be. There are lot of idiot pilots out there, and there are a lot of idiot software engineers out there, and I may well be a member of both groups. One of the big problems with idiocy is that it does not yield readily to introspection. (And you might want to mull that over a few times before you respond.)
As long as I'm pointing out your logical fallacies, here's another one:
> your point is that professional airline pilots, their union, or an airline are less qualified to understand the safety implications of this incident
No, that is not my point, which makes this a straw-man [1]. The fact that neither the union nor the airline have publicly expressed the same concern that I have in no way implies a lack of expertise on their part. There are many other possibilities, one of which is that they agree with me, but they are savvy enough to realize that there is nothing to be gained by saying so publicly.
> But I am a licensed private pilot with 30 years of flying experience
So you have the lowest pilot qualification who can even solo, and no jet qualifications at all. But you know better than the pilots union at a commercial airline, who are run by commercial airline pilots? Or the FAA who haven't stepped in after TWO reportable events? So, yes, absolutely Dunning–Kruger applies here, and the fact you don't seem to see that is a problem.
Heck, you think "I am a software developer" means anything in this context is also a mystery to me.
> In the face of those facts, an implicit and unsupported assessment that I am a person of "low ability, expertise, or experience"
You are though. You only have PPL, and seemingly don't even know what it is that you lack in terms of expertise. The fact you think you know better than people who are demonstratively more qualified to have an opinion is problematic, it is textbook Dunning–Kruger.
If you knew more you'd be embarrassed but what you're saying here.
> The fact that neither the union nor the airline have publicly expressed the same concern that I have in no way implies a lack of expertise on their part.
They did express that concern, it is in the article. They even shut down the airline for a period because they were so concern at not insignificant cost to them.
> you have the lowest pilot qualification who can even solo
First, that's not true. Student pilots can fly solo, and there is also a sport pilot rating with is in some sense "lower" than a PPL. So the PPL is actually right in the middle of the primary rating hierarchy.
But more importantly, your characterization of a PPL as the "lowest pilot qualification" implies a pretty profound misunderstanding of how pilot ratings work and in particular how they relate to operational knowledge. It is possible to fly a jet with a PPL, and it is possible to get an ATP without ever flying anything beyond a Piper Arrow. There are also all kinds of orthogonal ratings like an instrument rating, seaplane rating, multi-engine rating, and instructor rating. And absolutely none of those things have anything to do with flying jets. For that you have "type ratings" [1], which are more or less orthogonal to all of the others.
> You only have PPL
That is also not true. I have additional ratings beyond the PPL, I just haven't mentioned them because they are not relevant. What is relevant is that I actually fly airplanes, and so in general I probably understand the process of flying an airplane better than someone who does not fly airplanes, and I have actually written and deployed software in mission-critical situations, and so in general I probably understand mission-critical software and its attendant risks better than someone who has not.
> They even shut down the airline for a period
No, "they" did not shut down the airline. A single individual, Bret Payton, shut it down.
> If you knew more you'd be embarrassed but what you're saying here.
Could be. But at least I know the difference between "but" and "about".
I agree with almost all that you said, but I seem to remember that Air Florida flight 90 crashed killing almost everyone aboard because the throttle settings were too low and no one figured it out until they were airborne and out of ground effect.
Looks like lots of people were very very luck here. Plus finding a bug and correction to production in 5 hours? That is at the same time good and bad..
The article downplayed the seriousness of a tail strike, but they can be quite serious. A tail strike can damage not just the skin of the aircraft, but also the aft pressure bulkheads of some aircraft. A damaged pressure bulkhead could cause the cabin to suddenly lose pressurization at altitude, with potentially major consequences, especially on a flight across open ocean.
(not in the aviation industry, just a fan, so grain of salt needs to be applied)
Aircraft age isn't in years, but in number of pressurisations (takeoff/landing cycles). A low cost airline optimising their schedule to squeeze every second of their airframe with constant short distance hopping around would result in much more of those compared to a legacy airline's widebody that is flown daily on a very long flight. So it depends on aircraft type/size and usage patterns, but Ryanair's 737 will be able to handle less years than DHL's 767Fs.
So for the 747 in the story, 22 years is on the long end, but not unheard of.
Do the climates a plane flies in and out of make a difference too, or is this such a minor factor compared to the act of pressurization that it doesn't matter? I remember flying out of Chicago once when I was a kid and having to wait for us to be coated in some special fluid because it was so cold that day that the takeoff was going to be especially stressful on the aircraft otherwise.
It depends, but usually after a certain number of cycles, there will be a larger maintenance check (D-check, see https://en.m.wikipedia.org/wiki/Aircraft_maintenance_checks). This will include removing all paint and performing lots of testing for cracks. It's possible to extend the life of an aircraft as long as it is economically viable (by replacing parts and performing the necessary structural repairs).
As a layperson, "they landed immediately and required inspection before being allowed to fly again" (not a direct quote, but it represents what the article said), seems pretty severe to me.
When people do this in cars, they might get worse mileage or track funny down the road. When the aviation industry just glances and says "Ah, it's fine", people die.
I agree. But this article describes thoroughly and repeatedly that no one said "Ah, it's fine."
You said, "The article downplayed the seriousness of a tail strike." I was pointing out that the thing they described multiple times did not sound like downplaying to me, as a layperson. "They immediately landed," sounds serious. "They grounded the fleet," sounds serious. "The planes required inspection before being allowed to fly," sounds serious. Nothing in the article reads to me, as a layperson, as if they were saying a tail strike is not serious.
I just re-read the thread and I realize that I took your comment "seemed pretty severe to me" incorrectly. I thought you were saying that it was an overreaction, but rather you were saying that it was a strong enough reaction and that my assessment of them not taking the tail strikes seriously was incorrect.
In that case, we're both right about different things. I was referring to the article, and you were referring to the airline.
The airline absolutely did the right thing and took the tail strikes seriously. As for the article, it focused on the strong reaction to the error in the software. The article did not discuss the serious consequences of a tail strike. Considering that multiple aircraft (and many lives) have been lost due to tail strikes, I was surprised that it was not mentioned.
Ultimately this would be true for most anomalies. It's one of the reasons commercial air travel is as safe as it is.
In a car, you hit a bump, you keep going. In a plane, something unexpected happens and that plane is grounded. The utility function is just too skewed to take even a slight risk in this regard.
It's a matter of regulation and policy. I was on a regional bus recently in the rain (Ontario's GO Transit) and a small car next to us hydroplaned and bumped the side of the bus. The driver immediately
- Pulled over onto the shoulder,
- Passed around a pad to collect everyone's contact information,
- Summoned a replacement bus, and
- Called the provincial police to escort us along the shoulder to the replacement bus.
In the moment, all of this felt a little over the top, and the total delay of about 20mins total was pretty frustrating given that I ended up missing a connection. But at the end of the day, it builds my trust in the safety of that system that the fitness of any particular piece of equipment is not up to the individual drivers to judge— part of paying for a ticket on the state-run transit system is expecting a higher level of safety than, say, what you might get on a Chinatown bus line (https://en.wikipedia.org/wiki/Chinatown_bus_lines#Accidents). And I think I was also glad that it hadn't happened to me in my personal vehicle, where I would have to be the one to assess fitness and then deal with the insurance and so-on in the aftermath.
We could certainly operate an aviation industry with a lot less regulation and a lot less care for safety. Operated that way, air travel would be noticeably cheaper. But it would also be more dangerous.
What’s the alternative? Land again and don’t look at it?
Inspections after abnormals are fairly common and many amount to nothing more than “inspected tail skid and aft fuselage, no defects founds, return to service”.
I was not suggesting an alternative. The comment I responded to started with, "The article downplayed the seriousness of a tail strike."
I was pointing out that the article described a course of action that, in fact, sounds serious to a layperson. It seems to me that "land and have a required inspection" is an indication of seriousness, not an indication of downplaying the severity.
> On the morning of Jan. 26, as two Alaska Airlines flights from Seattle to Hawaii lifted off six minutes apart, the pilots each felt a slight bump and the flight attendants at the back of the cabin heard a scraping noise.
> ...a software bug was sending bad takeoff weight data to its crews...
> ...To determine the thrust and speed settings for takeoff, Alaska’s pilots and others use a performance calculation tool supplied by a Swedish company called DynamicSource.
It delivers a message to the cockpit with crucial weight and balance data, including how many people are on board, the jet’s empty and gross weight and the position of its center of gravity.
In a cockpit check before takeoff, this data is entered into the flight computer to determine how much thrust the engines will provide and at what speed the jet will be ready to lift off.
> [!!] the bug [..] only presented when many aircraft at the same time were using the system.
Yes, they are done in the operations center. The weight of the aircraft empty, how much fuel has been loaded, how many passengers are boarding, the baggage weight, any cargo carried, the airport altitude, weather, runway length, and a great many other variables, plus a safety margin, all come from a variety of sources. Why would it be done on the aircraft, and how?
I have the idea that a captain is responsible for its flight, and thus want to have as much information as possible. I think flights have their own full manifest anyway, so they have all the information you mentioned?
As to "how", I'm not sure I understand the issue, can't the same computations be done locally? The computations you mention doesn't sound too computing intensive.
I concur with sibling "Why does this seem so absurd to you?". You probably know real reasons for this not to be done from aircraft, and we're curious to see some.
The pilot arrives about when you do, or is already on board from the previous flight.
Sure, the pilot could plug those numbers into the same computer the dispatcher does before they board, or disembark and use the computer and re-board, but it would be one more thing to do before you can start boarding. They could install that computer on every plane, but that's one more piece of equipment to go wrong/need to be certified, etc...
Why not have a helper who has access to networked systems with all of the cargo and passenger information available do it for the pilot, and access to complex software that they are specifically trained in, and have the calculations printed out and done for the pilot to review? Well, that's exactly what they do.
The pilot could probably do the calculations by hand (it is something you have to demonstrate knowledge of for small plane licenses, which all these pilots have), but that would take an hour at least.
While the calculations aren't compute intensive, there are a LOT of variables. Off the top of my head: passenger count, altitude and temperature at both departure and destination, cargo, cargo distribution, fuel, fuel distribution, reserve fuel needed for the weight carried and destination and routing.
What you end up with is that you want someone whose job it is to exclusively know how to correctly load the planes, and brief the pilots on how they've done it.
So what happens is exactly that. There's a room full of people for any given airline who figure out the weather, the cargo they need to to move, how full flights are, etc. and do all the math and hand the pilot a ream of paperwork that says that we loaded the plane factoring all these things in, here's the proof of work. And the pilot says: looks good. A person with more tools and information and expertise than me on loading has proven they did their job, I trust them in the same way that I trust the mechanic who rebuilt this engine, and the fuel guy who pumped the correct amount of fuel into the correct tanks, and the flight crew who have checked that the doors are latched and armed.
Everything on a commercial airliner is designed to fly safely and subject to FAA regulations. That includes (usually, the 737Max shows there are problems) high standards for the software on board. Once in the air, all the software and computational power, plus whatever networking hardware and software would be needed to talk the multiple systems needed for the computations, is just dead weight, and excess complexity.
Which would you rather have, a software error on the ground that results in a minor incident, or a software error while in flight, resulting in a possibly fatal crash?
The pilots have the necessary information and know how to do the calculation with pencil and paper if necessary. But computers can do the calculation faster and with less chance of error. And a computer in a vendor's data center is easier to maintain than a computer in every plane.
Are these calculations made/provided using metric units in American airports? If not, that's pretty much a death blow against doing them on-aircraft, as any international system has to support two different modes of ingesting data.
Edit: And even still, international regulations will vary and so the aircraft carrying two different sets of regulations seems complicated I would think?
To reduce the risk of this sort of thing happening. I imagine aircraft software standards are a bit better than whatever standards this software uses, if it failed simply due to high load.
> and how?
I'm pretty sure the computer on an aeroplane can handle this simple computation.
Pilots carried a paper performance manual for the aircraft and did the necessary calculations during preflight, I believe - though they stopped doing this because the computerised figures were much more accurate and reliable.
Having no knowledge about how the system works, I could imagine the bug being that one of the calculations timed out under load and wasn't caught, leading to the output not being added to the final calculation.
Where is it mentioned that it's Fortran? and what do you mean by "It’s Fortran so fixed array sizes might be to blame". Who says one cannot have dynamic array sizes in Fortran? You can have fixed array size in any language.
A concurrency bug that got fixed in production 5 hours later. It must be very unusual code or some people are excellent at debugging concurrency problems.
Article says that bug was introduced in an update that morning. Somewhere there was a developer familiar with the code and exactly what changed. Wouldn't be surprised if he smacked himself on the forehead and knew immediately what the issue was when he heard about the bug.
This sort of thing - in my uninformed opinion - should not be a service, and should be validated strictly as though it is part of the avionics software. After all it sends data that directly affects the power settings the pilots will use just as if those power settings were directly affected by the output of the program. That makes it an integral part of the plane to me.
I believe the tool takes data from multiple sources including airline reservation systems, baggage handling systems, refuelling systems and cabin crew confirming people are correctly seated on handheld devices and validates it for consistency before performing a standardised set of calcs. In which case the safety argument for having that airgapped from the flight computer, which receives only a few final calculated values (manually input by a pilot who also can potentially spot outliers) is probably stronger than the safety argument for integrating everything or manually inputting a large number of values into the flight computer (creating potential remote attack vectors and increased likelihood of human error respectively).
Would have thought lower than ideal safety margins on takeoff speed (the calculations were apparently <20% off the true value) were a more concerning problem
That is a very concise way of conveying a lot of information about the weather. It doesn't take long to learn it, and at a glance I can tell you that the weather is kind of shitty for flying at KAUS, and I can plug it into any flight app and read it in plain text if I really want.
Excuse me sir... that language is simply inappropriate here!
The fact that a lot of this output is almost complete inscrutable means you have to build a tool or something to translate it from that back to something a human can quickly read/grasp... and translation mistakes can happen there too :/
Right. Even if the developer knew exactly what he did -- any sane provider would put the updated update through a full QC cycle, which would take a while I'd figure.
Someone forgot their 'if err != nil { return err }' check from their microservice call (whenever it happens to be overloaded) and just plowed through with some intermediate weight component being zero?
It could be something as pedestrian as the software writing and later reading back from /tmp/somerandomfile. Or retrying with similar, predictable names.
Yeah the more I think about it the more this makes sense to me.
Both were lighter than expected (so if it was a swap with another aircraft we wouldn't necessarily know - possible that 4 aircraft got the wrong data and we don't know the other two).
But it feels more likely that the sums just failed and they incorrectly caught the failures instead of letting it fall out. catch(e){} is a bug factory for sure.
I've thought about this - if this is the case, they are extremely lucky because different planes could have significantly different payloads. Of course, then the TLAR would signal an issue.
Also this would mean some other aircraft had a particularly powerful departure, since both aircraft had lower-than-true weight calcs.
It's almost like we should try to stick to languages where creating concurrency bugs is nearly impossible, thereby pre-empting an entire class of bug that might have resulted in deaths
any functional one with immutable data would suffice. my current favorite is Elixir, but others fit this bill.
There is still the matter of dealing with mutable stores like databases, but well-written code would make that impossible as well, plus permit tests to run concurrently
If the plane was told how much weight was onboard, and it knows what throttle it was at... then it should know what acceleration it was expecting.
If it accelerated slower than expected, then there must be something wrong! So abort the takeoff...
Planes should have alerts for this kind of thing. If, 1 second after throttle is applied, the plane hasn't achieved the predicted speed, then the takeoff should be aborted - by this stage, you'll be well under the maximum abort speed, so you can stay safely on the ground and figure out whats wrong. Being overweight is just one possible cause - it might instead be that an engine isn't producing sufficient thrust, and that would probably doom the plane.
Given that both of these planes took-off well within the acceptable safety margin, I’m not sure you can say that the planes deviated from their expected acceleration profiles enough for a model to confidently say there was an error significant enough to justify an abort.
Ultimately the issue was noticed not because a single plane suffered a failure (one that’s expected to occur occasionally, and where multiple mitigations exist to ensure the failures are safe), but because two planes one after the other suffered an identical failure, a failure that’s only expected to occur rarely.
So ultimately the unexpected event was two closely spaced expected failures. Not the expected failures themselves, which kinda indicates that aborting to prevent the expected failures would have quite a high false positive rate. Aborts and alerts with high false positive rates are themselves safety issues which need to be avoided, so a trade off needs to be made. The fact that the individual tail strikes themselves were completely safe, if not ideal, and the larger issue was rapidly recognised and corrected, strongly indicates the right trade offs have been made.
FWIW, it takes way more than 1 second for the engines to spool up. And an engine not producing sufficient thrust would not doom the plane -- as long as the airplane makes it to V1 (decision speed), it should be able to fly with a single engine failure after that point. And the pilots should already have talked through the takeoff and come up with a go / no-go based on distance: "if we're after this point and we haven't reached V1, we'll abort".
One detail: V1 is the last-chance speed for being able to abort the takeoff within the remaining runway length, but it is predicated on the weight, among other things, being correct. If the airplane is heavier than the figure used in calculating it, you will not be able to stop in time from V1 (you will reach it later and further down the runway, yet need more distance to stop.)
Having said that, a speed-at-distance rule like the one you give here, especially if it was evaluated at a distance before you expect to reach V1, would be perfectly good. There is a rule-of-thumb for small-plane pilots saying you should have reached 80% of the rotation speed before you are halfway down the runway (though that speed is also weight-dependent - as demonstrated in these incidents.) There is a tacit assumption here that your deceleration after choosing to abort will exceed your acceleration up to that point.
True, but the spool up rate and thrust produced during spool up is all predictable.
Basically, I'm saying the plane should have an internal mathematical model of what it expects to happen, and anytime that sufficiently differs from reality, it should trigger an abort - especially if you're still on the ground at the time!
This parameter is displayed on some aircraft, notably French (Mirage 2000 & Rafale) which display the longitudinal acceleration (labelled Jx) in the headup display. Part of the takeoff procedure for the pilot is to confirm the Jx is nominal with regards to the takeoff parameters (loadout, density altitude, etc.).
Hopefully you mean tell the pilots to abort and not start the process itself. What if there's something on the runway or the runway is too short or slick for the plane to decelerate? There's differences between conditions and runways that would have to be accounted for and then you'd have to add a lot more capabilities to the plane's sensors for every other possible issue.
It's often a judgement call when things don't happen quite right and I don't think our technology is anywhere near ready to hand the abort process over to the plane's programming.
No that simple ; what is wind speed and direction for instance, which changes acceleration?
Air density will also affect, although that's easier to account for.
These things all have to be taken into account in the original calculation for how much power to use. It is impossible to determine the answer to that question without having the information necessary to calculate an expected acceleration profile for the takeoff.
If you are using approximations in that calculation (such as the average weight of a passenger), then that same approximation can be used in calculating the acceleration profile, and if you are falling significantly short, then it shows that the calculation of the power setting needed for a safe takeoff is unsound in some way. That alone is sufficient to justify an abort.
Some recent Airbus aircraft do have some take-off performance monitoring, though it's not extensive. If the expected acceleration differs enough from the actual acceleration of the aircraft, you'll get a "T.O. ACCELERATION DEGRADED" ECAM message and should reject the take-off.
I have no idea whether Boeing has the same. It's an emerging area, sometimes called a Take Off Performance Monitoring System.
Let’s go back even further: plane is loaded up, fueled and doors are closed. Why is the plane not capable of weighing itself? It seems entirely reasonable, even in an environment where weight savings are important, for the plane to sport the capability to have weight sensing mechanisms on its landing gear to get actual weight.
I have often wondered that myself, and I looked into it a little bit, finding that there have been ideas for actually weighing airplanes on the ramp going back to the 1930s, but nothing seems to have come of it.
One possible reason might be a usually well-justified concern over adding complexity which could, if it fails, be worse than the problem it solves - but what would it take to measure the load on each undercarriage leg? something to measure its extension? Strain gauges on the structure the legs are attached to? I would really appreciate the input from an aero structures engineer here (paging Walter Bright...)
One other issue this would help is with is ensuring that the center of gravity is within limits (after the crash of Air Midwest Flight 5481, where this was a significant cause, the rule for estimating passenger weights was revised.)
On the other hand, an acceleration test will detect a lot of other problems beyond being overweight. There was a Citation crash near Hartford CT last year where it is suspected that the parking brake did not fully disengage. Why not have both, if they are feasible?
It's difficult because any minute amount of wind will affect the final results, and you don't have the full wind profile over the entire aircraft to compensate for it. Easier and safer to track what's on board and calculate the a/c performance based on that + margins.
I take your point; I have no idea of the magnitude of this effect in proportion to the total weight. I do wonder if a combination of the airspeed instrument readings, their fluctuation, and fluctuations in the measured weight could be used to estimate the influence of the wind (an application for machine learning?) but even so, that would mean that it would no longer be a simple solution.
What I’m talking about is just a way to provide something besides a pilot’s hunch to compare against weight data from flight crews. If it was so windy that it would catastrophically throw off the comparison, I doubt they’d even let the planes take off.
That's not how it works. Airliners can safely take off into strong headwinds. A wind blowing across the wings will produce significant lift. In theory it might be possible to compensate for that effect, but in practice it's just too much hassle.
I wondered the same thing. Most metros perform load weighing to adjust the effort. Why can’t the same tech be applied to planes? It may not capture all the data needed, but it wouldn’t be off by 20 thousand pounds and it could be incorporated into a feedback loop, to increase effort if the plane isn’t getting the expected lift.
I think the short answer to this is that because of the forces involved on the landing gear, the load cells would need to be recalibrated more often than would economically be feasible.
Pilots are meant to have a rule of thumb where they set a stopwatch and whatever takeoff power they are using they should reach 80kts within a certain time period, can’t remember exactly but I think it’s about 30 seconds. If not they should reject takeoff before they reach the ‘high speed realm’. Possibly the newest Airbus A350 might have what you describe already, but they don’t tend to back port this stuff to the older systems in older airframe families.
> Possibly the newest Airbus A350 might have what you describe already, but they don’t tend to back port this stuff to the older systems in older airframe families.
It's not that they don't tend to, it's that airlines don't want this — training management and recertification on behavioral changes is costly for them. See Southwest influencing 737 MAX development—MCAS was a result to try to reduce the characteristic changes between aircraft & engine variants. (Band-aid driven development!)
I sincerely hope that the 737 MAX incidents change the industry mentality to encourage safety features to be backported to older airframes, even if it seems untenable now. For example, the 737 MAX uses older crew alerting methods—annunciator lights—not EICAS, which can be thought of as a centralized actionable list of anomalies.
MCAS was used to make the aircraft conform to the regulations, it could not be omitted. It is intended to increase the control force at a point in order to assure control forces steadily increase into a stall. (See 14 CFR 25.203 -- Stall characteristics. (FAR 25.203))
The fact that it was not communicated in the training and manuals was a result of lobbying by the airlines.
> Peyton added that even though the update to the DynamicSource software had been tested over an extended period, the bug was missed because it only presented when many aircraft at the same time were using the system.
There are only two hard problems in computer science:
Here's a bit of detail. The software was supposed to calculate the speed based on the weight of the aircraft and determine if the available runways were adequate.
The nature of the bug was that if it couldn't find a usable runway, it would lower the weight until it did fit the runway it could find.
That's right, the software would say, "oh, the aircraft can't take off on the runway with this weight, let me just lower the weight so it can". Apparently the runway data was missing or wrong, but instead of saying, "no, you can't take off from there" the software said "here's a solution allowing you to take off, but at this lower weight".
I'm not sure how/if that was caused by a concurrency bug, but that's what I know.
If you are flying out of Denver (high altitude) on a summer day, you need a TON of runway to get off the ground. There are some aircraft that simply can't take off fully loaded from high altitude with the runways that exist.
The solution is to not take off fully loaded (lower the weight) So you either don't fully fuel up, or you remove cargo/passengers. It's very common, and by design. The software exists so that you know not to fully load the cargo bays if the conditions don't allow it (cargo makes airlines a lot of money).
Yes, those runways are called "hot and high". Another one is Albuquerque. Without knowing more I can't say why the software would return a result using a different weight from what was input without clearly alerting the humans of that change, or else return an error with no attempt to give performance parameters at all.
I wonder if that is aviation language instead of software language. Permanently repaired like replacing damaged aircraft skin so that no further action is required, as opposed to grinding off the jagged edges and patching the opening, but still requiring a real repair after flying back to a maintenance depot.
We currently use a complex system of trying to weigh everything that goes onto the plane - baggage, cargo, fuel, and estimate the weight and distribution of the people.
Sure, estimates are good... But why not just weigh the whole plane just before takeoff?
Simple strain gauges in the suspension struts would give you all the info you need. And then you can still use the old system of weighing all the stuff you put on the plane as a backup/sanity check if you want to.
My father used to work in a manufacturing plant where they would weigh the trucks coming in and out. The scales had to be in a fully enclosed garage or any breeze would throw the reading completely off.
I have a hunch that this same issue might be even more of a problem for an airplane.
I don’t think the plane needs a high level precision for the weight. I’m sure it could be off by a couple thousand pounds without any issue, and winds wouldn’t affect the weight that much.
a 130kt (~the stall speed) wind over the wings would pretty much remove the entire weight of the aircraft.
Okay, lift is proportional to the square of the airspeed, but it's still going to be a huge factor and make it really unreliable
My impression is there is a sensor to accomplish this already.
Source: Just supposition, but I was on a completely full flight (on an Embraer 175) last week and they held us on the tarmac while ground crews drove a 200lb ballast weight to us which was placed in the forward cargo area. To decide this was an issue after having pulled away from the gate implies to me that the issue was raised by instruments on the plane.
Generally the computer / flight crew does those calculations based on location of passengers, cargo and fuel. They probably missed something and requested more weight.
Source: I was personally loading cargo on an Embraer 175 two days ago. Also this link gives a good overview of weight sensors in planes.
Where would these struts go? I assume in the landing gear suspension, and if so does that provide enough feedback to adequately understand the load and distribution? That's usually only 3 points of contact across the whole of the aircraft, isn't it?
You can still work out the total weight and the location of the center of mass (in 2D) from the strain on the three landing gear struts. It's a linear system of 3 equations and 3 unknowns.
As another commenter has pointed out, there may be an issue with wind, as the whole point of the airplane is that it efficiently converts a relative air velocity into an apparent change in weight.
Fair enough, I assumed there might need to be more consideration in where the weight in in terms of where along the wing it is, stuff like that, but I guess not.
Only a few planes have landing-gear based weight systems, mostly large military transports that won't be necessarily able to estimate weight well under fire.
> the update to the DynamicSource software had been tested over an extended period, the bug was missed because it only presented when many aircraft at the same time were using the system
Seems scary that something like the weight of a single airplane depends upon the amount of planes using some external system. So it seems that they not only discovered this particular bug by "luck" but that they've discovered a whole new vector of bugs.
> even though the update to the DynamicSource software had been tested over an extended period, the bug was missed because it only presented when many aircraft at the same time were using the system.
Errrr what?
How do you even get this type of bug?! These scare me more than just a calculation error
Edit: on second thought this looks more like of some race condition on inputting the data, or maybe some query that fails and gets replaced by a 0
22 minutes! They figured this out and issued a patch in 22 minutes!
Is anyone else just absolutely astounded by how short that response time was?
>That morning, a software bug in an update to the DynamicSource tool caused it to provide seriously undervalued weights for the airplanes.
The real issue question here is how often do they patch this software? Airline control software should not be optimized for feature velocity. It should be optimized for quality and change really slowly.
Ideally, we would have as few updates to this software as possible. We're the pilots informed that the software had updated that morning?
I worry that our increasing comfort with updating constantly broken software bleeds into systems like this that matter for people's lives. Apple has me updating almost weekly because they can't seem to write exploit free software, but my iPhone is not going to kill me. That attitude is not Ok for airplanes.
They didn’t issue a software patch in 22 minutes. It sounds like 22 minutes was the time it took the airline to figure out it was a software bug and tell pilots not to trust the software.
The article says the software bug was fixed in 5 hours, but it’s not clear to me if that was a bug fix or a rollback.
> but my iPhone is not going to kill me. That attitude is not Ok for airplanes.
This is the key point. I imagine the stakeholders here do not think the weight software is that critical, when in fact it could be. This is a common issue with complex systems.
Even though the software does not directly interface with the plane, its outputs are used to make key decisions on takeoff. Normally, there would be a large margin of error allowed, but in this case, due to the desire for max efficiency, the behavior of the aircraft is sensitive to the outputs.
The solutions are to (1) go back to having a large error margin, which of course should be cross-checked for sanity by the pilots, or (2) consider that the software is safety-critical, and should meet the same quality standards as the in-built flight systems.
I'm astounded that it was a problem to begin with. The only reason they managed to fix it so quickly is because they messed it up in the first place and shipped it to production without a stress test.
Further to the ideal, even if they did have to update the software, Alaska would choose to run both old and new concurrently until they're satisfied with the results. If the new code is producing numbers that are wildly off, and that isn't expected, abandon the update.
So the bug told pilots the aircraft was lighter than it was. I wonder if that would have also factored into the fuel load calculations? It seems like it could have the potential for the crew to take off with less fuel than they need.
Possibly, but aircraft carry a significant amount of extra fuel anyway. (That's how you can have a jet holding for 2 hours at a fix unexpectedly because of weather or whatever.)
Author of a fuel estimation program for 747 freighters here...
Yes, they take a significant amount of extra fuel, but that significant amount is very carefully calculated because carrying more than legally required adds up to a loss quickly, especially if your competitors are sharper at calculating these figures.
The are ~100 parameters that go into these calculations, ranging from almanacs with tables of various trade winds and densities to engine types, consumption at various altitudes and power settings and so on. And then the required hold time, first alternate, second alternate and legal reserve required on landing.
Getting that code certified still makes me break out in sweat, but good thing that this kind of thing is reviewed very strictly. That also makes me wonder how wise it was to have the software this article is about as a SaaS rather than something more integrated and less easy to mess up using an update. Maybe doing it as a SaaS is a kind of loophole that avoids certification? (not current on this stuff).
That's super interesting - I didn't realize that certification would be required for this.
I would suspect that certification would break down at some point; does a plain old calculator need to be certified? Certainly not, but I'd imagine the FAA or whoever if asked would prefer pilots use calculators than do everything by hand.
I'm curious about EFBs as well - I run W&B for my little bugsmasher in Foreflight. Has that gone through certification? Certainly someone from the FAA wouldn't have to review every release for that if so...
So when does certification come into play versus it just being another tool that pilots are responsible for validating?
Good question, I don't know the answer. But I can give you some more info on this project: the company, one of the largest freighters operating out of Schiphol already had such software but they had lost the source code. The assignment was to create a new piece of software that produced exactly identical output. That was the bar to clear. If not for a friend (who got me the job in the first place!) who managed to decrypt the original (it was apparently written in some variation of BASIC with an obfuscation layer but once you had it reversed it would actually list as source) I would likely have never managed to complete the assignment. But once I had that in hand it was much, much easier because I could read all of the original tables.
One of the bigger problems I encountered was that the original software wasn't perfect, so the first step was to replicate it, bugs and all and only then to fix the bugs. So the FAA never entered into it but the Dutch equivalent did. Also, I think there is a big difference in stuff in use in GA versus that which is used for commercial aviation. We're talking early 90's or so, so the details are a bit hazy by now.
Interesting job though, that's for sure, lots of good lessons from that one. I'd never been sent back four or five times by an auditor because my work wasn't up to their standards and every time they sent me back they had a valid point, it wasn't bs just to check a box somewhere, they had valid concerns and I had to rework some of it quite extensively. At the level of "I don't care if the answer is correct, I need to understand why it is correct".
Id love to have a simple FAA rule that says the following:
Every time there is a plane inspection, software update, at least 1 random person of the mechanical or software team for each party that directly touched either the plane or ground systems must be a passenger on the maiden operating flight for that plane/airport.
This would apply to all parties involved.
I'm sure the FAA can compensate and the FAA already has rules that are far costlier than this
I do wonder if in all of the cost cutting some of these massive companies have focused on in previous years, we are going to see more and more dangerous bugs like this come out of the webwork as a result of refusal to pay their best engineers above market rate.
Anecdotal, and probably just a result of software being more widespread in general, but it feels like incidents like these are becoming more frequent.
I read the title and for a minute I thought it must have been a type of beetle, maybe some wasp? Then, trying to figure out how the pumping along the runway is relevant. Maybe it fell out of the luggage compartment? And, where does Alaska Airlines fly that they could pick up suck insects?
Finally I read the article. That was a bucket of cold water...
It’s really interesting, that airlines use a lot of externally computed information, just to copy it over to the planes systems manually. I’ve heard about issues with that before, because some airlines used unsuitable systems for calculations.
It’s really interesting, that there are no standardized systems, that interface with the planes directly.
I wonder how common it is for a bug to be finally detected by the manufacturer months or years after users were impacted by it and thought it was fishy.
I wonder what design practices extend that intervening time, and which ones shorten it.
>A pilot at American Airlines, which uses the same DynamicSource performance data tool, and who also spoke anonymously because he didn’t have authorization, explained that the computer then calculates just the right amount of engine thrust so the pilots don’t use more than necessary.
>“The goal is to lower the power used on takeoff,” he said. “That reduces engine wear and saves money” on fuel and maintenance.
Cost cutting can do more harm than good, and occasionally puts lives in danger - they need to teach that in MBA classes.
Your advice is good, but the spirit is wrong.
They do that to save fuel not just to cut costs but also to reduce consumption, which is net positive if you factor in environmental conerns. Some locations require it for noise abatement. Also, reducing the wear on the engine means more cycles and less maintenance/chance of engine damage. These parameters are well tested and are very well understood.
Planes take off at mere 150kts, compared to what their cruise at, which is more than 250 knots. If you can take off at lesser thrust, it's not bad to do it. It's not like Airlines are making huge profits out of it too.
Upvoted - I see what you are saying and I am mostly convinced, except:
> It's not like Airlines are making huge profits out of it too.
True, but airline margins are also very thin, so there is some incentive to "pinch pennies." And I do not put it past management to cost cut as much a possible for those sweet, sweet quarterly stock increases.
Given the recent record of lately in terms of safety, timely service, cancellations, forced retirements for pilots, etc, the trends are not looking good for the airline consumer.
maybe a secondary system that could raise alarms since in this case, the weight calculated must have been outside of the standard deviance of the weights recorded over time?
> "The bug was identified quickly in part because some flight crews noticed the weights didn’t seem right and asked for manual validation of the figures....The Alaska captain said that, as for many things in aviation, pilots routinely use an acronym when they do the pre-takeoff “sanity check”: TLAR, which means “That Looks About Right.”"
Keeping well-trained humans in the loop in all partially or highly automated systems is important for this reason - and although corporate bean-counters in the overly finacialized US economy, whose prime motivation is to maximize shareholder profits by cutting labor costs, may not like to hear this, that mentality is the central reason behind the Boeing MAX debacle.
Notice that they are not using it as the only means for verifying system. They are using it as an additional check that supplements the extensive regression test suite. The test suite failed in this case because the bug occurred only under high load (after this, they added a new test case that simulates high load). The human verification that was in place in production as an additional safety check operated as intended and detected the bug.
Indeed. Designing software so that it helps humans to eyeball stuff where humans eyeballing stuff might help is really good.
For example in that incident in Ireland where crew erroneously told their jet plane it was very, very cold, and then (cold air is dense, less power needed) they barely had enough thrust to take off before they ran entirely out of runway - that was partly caused (Swiss cheese model) by the pilots being issued with software that didn't make it easy for them to cross check the facts they can eyeball. Is it certain they'd have realised if their software did show this information - like the software issued to similar crews with other outfits? No, but that's one more opportunity for everybody to not die, and the whole point of the Swiss cheese model is that lots of things have to go wrong or else you're fine.
One of the other potential mitigations for that incident illustrates where human eyeballs don't help. In principle a human pilot could realise they aren't accelerating enough. In practice humans are lousy at estimating acceleration to the tolerance needed so this isn't effective unless you use the same strip so often that your brain learns exactly how things ought to look.
However thanks to GPS a machine can measure acceleration very well, and it can know how long your chosen runway is and what its take off speed is - so it can estimate if you're accelerating enough and the latest models can announce "Caution: Acceleration" early in the take off roll, before V1, which means the crew can abort and check why they didn't have the intended acceleration - regardless of the cause.
That has, historically been my issue with LGTM. It is ultimately a meaningless rubber stamp. Many people get a large PR and give it a quick scroll, type LGTM and move on. If I had it my way, every PR with over 100 LOC diff should require two PR code comments for a reviewer to approve it.
I see it more as “this result is commensurate with my experience,” the problem with it, of course, being reliant on the pilot’s experience. But otherwise we’re just talking about a “sniff test” on a number. LGTM sucks in software because software inherently is a lot more complex than a number.
It's not a standard economics answer, even economists (normal ones anyway) aren't going to fly on planes entirely unregulated in a purely free market.
Gradient descent and reinforcement learning work eventually, but they take a lot of iteration and spectacular failures. Different exploration/exploitation tradeoffs are needed in different environments, the market signal isn't really strong enough to differentiate and the incentives don't actually line up perfectly.
The standard economics answer is more like, the market (i.e. regs) evolved to address coordination problems and align the market incentives better with the true long-term welfare function.
The trouble with this answer, and the people who accept, it is that they assume the market is fair or at least pretend so they can ignore the inherent problems.
At this point it should be apparent to anyone that markets are rigged, and anyone sticking to the idea of self regulating markets should not be trusted.
The trouble with this answer is that it makes two incorrect assumptions:
Companies regularly attempt to interfere with or harm any means of regulation, governmental or otherwise. Companies stifle competition that might regulate their behavior both legally and illegally without fear of consequence.
* Even if your answer held true, which is absolutely does not, the inherit problem with it is that there's an acceptable rate of failure before correction takes hold and that the losses are acceptable.
There is irreparable harm that occurs in many industries and the morally acceptable limit of zero is well below the market's acceptable limit.
Theory =/= Practice. We've seen time and time again that simply isn't how things play out consistently. It's largely why we have regulations/oversight.
The reason the "well-trained humans" are in the loop is often due to legally required oversight and safety nets. Plenty of companies do not (or would not if they could get away with it) operate with safeties they should otherwise have in place because the money lost does not outweigh the money gained and no one is forcing their hand. They can putter along and kill or hurt people for an indeterminate amount of time, so long as the consequences stay below a certain threshold to not invite public or governmental scrutiny.
maybe in one of those simple textbook economics where everyone has perfect information and the economy only produces one good from two different companies.
"The market fixes everything" is absolutely not _standard_ economics; it's a particularly fringe maximalist libertarian view which even most libertarians would not entirely subscribe to.
Moreover, it’s generally accepted as a standard part of economics that the market only ever begins to fix things (be efficient) when property rights are enforced, barriers to entry are few, and transaction costs are low.
It’s not clear that “aviation” manages more than about one of those items.
My ECO101 lectures (I don't know if that's "standard economics") included government regulations to handle externalities (which can usually be summed up as "deaths")
Even libertarian economists don’t make this argument, pretty much everyone wants dangerous spaces where the public can’t be expected to technically understand the risks and trade offs involved, standard economics says regulate.
How much should you pay the well-trained human in the loop, that’s believed best to go unregulated.
