The related [0] NYTimes article mentions one pattern for fixes, which was finding gaps between design and execution:
"For the screws that popped off during the shake test, it turned out that the engineering drawings did not specify how much torque needed to be applied. That was left to the contractor, Northrop Grumman, to decide, and they were not tight enough.
“You should have a specification to make sure it’s right,” Mr. Robinson said.
The review board released its report, noting a series of issues, and made 32 recommendations. NASA followed all of them, Mr. Robinson said.
One of the recommendations was performing an audit of the entire spacecraft to identify “embedded problems” — mistakes that occurred without anyone noticing.
The engineers checked the drawings and specifications. They looked at the purchase requests to make sure that what was ordered matched the specifications and that the suppliers provided the correct items.
“There were multiple teams set up, led by the most experienced people,” Mr. Robinson said. “They really dug into the paperwork.”"
I have found it's very easy to scapegoat the "engineers" for situations like this, but in the long term not the cause of the problems nor the actual solution. For example, screws have standard torques published, and for small common diameters is left implicit. The matter is, Northrop's blue collar assemblers didn't particularly care and just installed the screw however, and now the contractor is looking to shift blame. If the drawing did actually say "12 inch pounds" or however, it's still a coin flip if the screw was actually that tight, and the end result of the shake test would be identical.
For an analogy, say the website designers just specified "add a button" and you went and made the button fluorescent yellow with white text. Then the design fails the color contrast test. Will you say it's entirely the designer's fault for not specifying the button color?
Digging into the paperwork to the extent described may have the highest assurance level, but it is extremely expensive. This spacecraft in particular was quite over budget and most likely only able to be carried over the finish line by pure government inertia.
... they are not factory assembly line workers, right?
> over budget
this is the classic problem of too small overall budget. too few launches, not enough money, everything has to be perfect, because everyone wants their pet science/dream project to succeed. so overall efficiency is abysmal.
but the data gained for/through the scientific mission is nice.
> this is the classic problem of too small overall budget. too few launches, not enough money, everything has to be perfect, because everyone wants their pet science/dream project to succeed. so overall efficiency is abysmal. but the data gained for/through the scientific mission is nice.
You've got it 100% correct.
Satellite constellations are much cheaper on a per unit basis than are the bespoke single unit buy. Now, I get above are much harder when NASA is doing a research satellite (low volume or volume of 1), but it is definitely possible for Operational satellites (e.g. what NOAA flies for operational weather requirements and they need to keep running to feed the US meteorological capability).
Delinking satellite sensors from the specific satellite so you always have an excess of sensors you can use, and integrate as needed would also be a huge win for cost avoidance.
When I worked in aerospace using properly calibrated torque tools at the fastener's torque spec is drilled into our heads. How does an oversight like that happen?
We have a fascination to find a hero or villain as a singular explanation. It makes for a a good article or Michael Lewis book. The JWST might have also succeeded because of the failures learned from the Hubble telescope.
It sounds like he was/is a personable engineering manager that paid attention to detail. Making sure parts match the drawings, bringing in outsiders to have a second look at things, etc.
Basically this, and it’s a bigger deal than it may sound. Competent technical management is a scarce commodity worth its weight in gold around NASA Goddard. A lot of (both frontline and upper) management has no technical background and it shows.
JWST was a shambles when I interned there in 2014. Eternally delayed and broken due to nonsensical decisions by management with absolutely zero engineering knowledge. Sometime in the 90s they had decided to save money on devs by “auto-generating” the code from UML diagrams using this thing called Rational Rose. In the mid-2010s they were still trying to fix the code. I never quite figured out if it was an urban legend, but there was also a story around Goddard that when JSWT did its acoustic testing (to simulate launch conditions), a bunch of screws fell off—and when they put them back, they were left with several extra screws that nobody ever figured out the place for. So that’s the state the program was in.
Greg Robinson seems to be an engineering manager who’s actually highly competent at both the engineering and the management, and that’s exactly what the program needed to turn around. Honestly, when I was there I thought there was very little chance this thing would actually work, if it ever launched at all, so major props to him and the team.
I saw this in way more programs than just JWST. Word would get around among us contractors that X program had real technical management and Y didn’t, and we’d all try to get onto X. If you got stuck on Y, you’d spend most of your time doing paperwork and PowerPoint instead of coding.
It happened in the civil service too. Most of the branches in the software division had non-technical leadership and seemed to do little of note, but the few with technical leaders did incredible work. I remember one time a previously-good branch had its head depart (promoted?) and the new head wasn’t technical at all, and over a few months most of the good civil servants transferred somewhere else. At one point it seemed like most of the progress in the software division was being driven by one rather obscure-sounding branch, simply because that was the one with sound technical leadership and thus all the good engineers.
Why are there so many non technical managers/directors in charge of engineering or IT in Government type jobs? My wife sells to state and local governments. 90% of the time the people in charge have no idea about IT, SecOps, Software Development etc. Their only qualification is they have engineering or security in their title.
Promotions on the government and contractor sides are about appearances, not accomplishment. Strong technical people tend to like focus versus meetings and the ladder climb. Less visibility means less mobility. Giving an unpopular though factual technical analysis is career limiting.
To rise means to manage people and budget. Higher the better. Kelly Johnson's 14th rule is never followed:
"Because only a few people will be used in engineering and most other areas, ways must be provided to reward good performance by pay, not based on the number of personnel supervised."
The pay is awful and the benefits aren’t great. You also have people constantly telling you that you must be bad at your job since you work for the government and not the private sector. It’s also not a lot of fun when half of Congress repeats the lie that what you do is worthless or even worse that you’re part of some nebulous “deep state.”
If you don’t mind all of that then you should apply. There’s always a ton of openings for technical managers.
> Why are there so many non technical managers/directors in charge of engineering or IT in Government type jobs?
due to product/program/project managers making a case for themselves to lead these teams. it might work in most of the cases, but clearly does not in engineering intensive projects - because the lens required to look at and deliver these projects is more than input/output, deadlines/deliverables.
understanding the subject is necessary to make smart tradeoffs (common scenario in projects that are as large scale as this one), but few of these folks do.
engineering enterprises that have often become successful owe it in their early days to majority of management having technical expertise. this is not a need once the companies become large, but such attitude does show up as inefficiency holes, if one knows where to look.
There is less pressure to perform well in government, so it becomes less prioritized than in organizations that may cease to exist if they perform badly.
>Sometime in the 90s they had decided to save money on devs by “auto-generating” the code from UML diagrams using this thing called Rational Rose.
Mother of god I remember having to do a module/class on UML using that software for part of my CS course around 2008. It was supposed to be the future of software dev or something. I felt like it just massively overcomplicated things.
I never seen or heard of anything like it after that year.
It's simple, man, all you have to do to build quality software is just collect user stories, convert them into requirements, iterate the design until your design elements capture all the requirements, something-something sequence diagrams, implement the design (cough/snort/choke/laugh), get final user approval and you're done! /s
> Competent technical management is a scarce commodity worth its weight in gold around NASA Goddard. A lot of (both frontline and upper) management has no technical background and it shows.
Almost all management at GSFC is technical management - engineers, scientists, for the engineering, science, and flight project directorates, and other other directorates (codes) are led by management in their respective fields.
> JWST was a shambles when I interned there in 2014. Eternally delayed and broken due to nonsensical decisions by management with absolutely zero engineering knowledge. Sometime in the 90s they had decided to save money on devs by “auto-generating” the code from UML diagrams using this thing called Rational Rose.
Most NASA missions are actually shoestring operations in my opinion. I've seen really well funded things from NOAA (GOES-R for example, before their budget cuts to help out JPSS), and NASA's Human Space, but almost all NASA missions outside Human Space (HEO) are poorly funded. To quote someone, "Almost all US Space systems are sold to Congress at unrealistic budgets".
I'm familiar with some of these issues and attended some of the JWST design reviews. Most missions with novel sensors, or upsized sensors, or complex fabrications are regularly delayed for a number of reasons, one of which is actually technical complexity. Another is because NASA GSFC and NASA JPL are regularly forced to push fabrication of unique items to commercial industry even when commercial industry has no experience in creating that exotic widget type. This is regularly impacting NASA missions. I am amazed at how much GSFC is able to keep operational despite low OPEX budgets.
> Greg Robinson seems to be an engineering manager who’s actually highly competent at both the engineering and the management, and that’s exactly what the program needed to turn around. Honestly, when I was there I thought there was very little chance this thing would actually work, if it ever launched at all, so major props to him and the team.
Everyone at GSFC knew JWST would work, we just thought it would take awhile. I personally consider JWST much more successful than many other missions.
However, a single System Program Director is not the cause for success or failure of a mission. I did think he was more successful though around providing top cover for JWST downtown at SMD HQ and with NASA's overall mgmt.
> It happened in the civil service too. Most of the branches in the software division had non-technical leadership and seemed to do little of note, but the few with technical leaders did incredible work.
This is incorrect. Just because you don't see a person doing technical work doesn't mean they are not technical. Unfortunately, the demands of bureaucracy require EATD/Code 500, to be doing much leadership and not technical work.
> I remember one time a previously-good branch had its head depart (promoted?) and the new head wasn’t technical at all, and over a few months most of the good civil servants transferred somewhere else. At one point it seemed like most of the progress in the software division was being driven by one rather obscure-sounding branch, simply because that was the one with sound technical leadership and thus all the good engineers.
This does happen unfortunately, where a hero is replaced by a zero.
I'm impressed by the meticulous detail here, but I still have to believe that if the original goal was to build 10 of these, (one every 2 years) and accept a 60% fail rate for each, we'd have gotten more telescopes for the same money in the end.
I don't believe Science Mission Directorate (SMD), which funds science related missions that JPL and GSFC compete for, has ever been so flush with cash as to be able to develop a JWST constellation of that size.
We'd prolly need to raid the HEO budget and kill off some too-big-to-fail-missions in Texas to get the money for that....
