This paper is the product of a failed model of AI safety, in which dedicated safety advocates act as a public ombudsman with an adversarial relationship with their employer. It's baffling to me why anyone thought that would be sustainable.
Compare this to something like RLHF[0] which has acheived far more for aligning models toward being polite and non-evil. (This is the technique that helps ChatGPT decline to answer questions like "how to make a bomb?")
There's still a lot of work to be done and the real progress will be made by researchers who implement systems in collaboration with their colleagues and employers.
That's what I didn't like about Gebru - too much critique, not a single constructive suggestion. Especially her Gender Shades paper where she forgot about Asians.
I think AnthropicAI is a great company to follow related to actually solving these problems. Look at their "Constitutional AI" paper. They automate and improve on RLHF.
> Compare this to something like RLHF[0] which has acheived far more for aligning models toward being polite and non-evil. (This is the technique that helps ChatGPT decline to answer questions like "how to make a bomb?")
I recently saw a screenshot of someone doing trolley problems with people of all races & ages with ChatGPT and noting differences. That makes me not quite as confident about alignment as you are.
I am curious to see that trolley problem screenshot. I saw another screenshot where ChatGPT was coaxed into justifying gender pay differences by prompting it to generate hypothetical CSV or JSON data.
Basically you have to convince modern models to say bad stuff using clever hacks (compared to GPT-2 or even early GPT-3 where it would just spout straight-up hatred with the lightest touch).
That's very good progress and I'm sure there is more to come.
Yes. Machine learning models learn from the data they are fed. Thus, they end up with the same biases that humans have. There is no "natural" fix to this, as we are naturally biased. And even worse, we don't even all agree on a single set of moral values.
Thus, any techniques aiming to eliminate bias must come in the form of a set of hard coded definitions of what the author feels is the correct set of morals. Current methods may be too specific, but ultimately there will never be a perfect system as it's not even possible for humans to fully define every possible edge case of a set of moral values.
I don't have a copy of the screenshots any longer, but they did not appear to be using hypothetical statements, just going for raw output, unless that could've happened in an earlier part of the conversation cut off from the rest.
There was a flag on one of the responses, though it apparently didn't stop them from getting the output.
If it’s trained on countless articles saying women earn 78% of what men make and you ask it to justify pay discrimination what value do you think it’s going to use?
It's not about what I expect, it's that it doing that is a bad thing. If it ever infers that discrimination might fit a situation, you'll see it propagate that. The anti-bad-question safeguards don't stop bias from causing problems, they just stop direct rude answers.
> The resulting InstructGPT models are much better at following instructions than GPT-3. They also make up facts less often, and show small decreases in toxic output generation. Our labelers prefer outputs from our 1.3B InstructGPT model over outputs from a 175B GPT-3 model, despite having more than 100x fewer parameters.
I wonder if anyone's working on public models of this size. Looking forward to when we can selfhost ChatGPT.
This is going to happen alot over the next few years. One can fine tune GPT-2 medium on an RTX2070. Training GPT-2 medium from scratch can be done for $162 on vast.ai. The newer H100/Trainium/Tensorcore chips will bring the price down even further.
I suspect if one wanted to fully replicate ChatGPT from scratch it would take ~1-2 million including label acquisition. You probably only require ~200-500k in compute.
These things have reached the tipping point where they provide significant utility to a significant portion of the computer scientists working on making these things. Could be that the coming iterations of these new tools will make it increasingly easy to write the code for the next iterations of these tools.
I wonder if this is the first rumblings of the singularity.
I can imagine a world where there are an infinity of “local maximums” that stop a system from reaching a singular feedback loop… imagine if our current tools help write the next generation, so on, so on, until it gets stuck in some local optimization somewhere. Getting stuck seems more likely than not getting stuck, right?
chatGPT being able to write OpenAI API code is great, and all companies should prepare samples so future models can correctly interface with their systems.
But what will be needed is to create an AI that implements scientific papers. About 30% of papers have code implementation. That's a sizeable dataset to train a Codex model on.
You can have AI generating papers, and AI implementing papers, then learning to predict experimental results. This is how you bootstrap a self improving AI.
It does not learn only how to recreate itself, it learns how to solve all problems at the same time. A data engineering approach to AI: search and learn / solve and learn / evolve and learn.
Assuming a motivated “attacker”, yes. The average user will have no such notion of “jailbreaks”, and it’s at least clear when one _is_ attempting to “jailbreak” a model (given a full log of the conversation and a competent human investigator).
I think the class of problems that remain are basically outliers that are misaligned and don’t trip up the model’s detection mechanism. Given the nature of language and culture (not to mention that they both change over time), I imagine there are a lot of these. I don’t have any examples (and I don’t think yelling “time’s up” when such outliers are found is at all helpful).
I'm still midway through the paper, but I gotta say, I'm a little surprised at the contrast between the contents of the paper and how people have described it on HN. I don't agree with everything that is said, but there are some interesting points made about the data used to train the models, such as it capturing bias (I would certainly question the methodology of using reddit as a large source of training data), and that bias being amplified by filtering algorithms that produce the even larger datasets used for modern LLMs. The section about environmental impact might not hit home for everyone, but it is valid to raise issues around the compute usage involved in training these models. First, because it limits this training to companies who can spend millions of dollars on compute, and second because if we want to scale up models, efficiency is probably a top goal.
What really confuses me here is how this paper is somehow outside the realm of valid academic discourse. Yes, it is steeped in activist, social justice language. Yes, it has a different perspective than most CS papers. But is that wrong? Is that enough of a sin to warrant such a response that this paper has received? I'll need to finish the paper to fully judge, but I'm leaning towards no, it is not enough of a sin.
The sin is not activist language, the sin is applying rhetoric in academic language to distort the truth to make a point. It’s concern trolling to gain academic clout.
Page 3 of a 14 page paper already claims with a straight face that LLMs are responsible for the sinking of Maldives.
> Training a single BERT base model (without hyperparameter tuning) on GPUs was estimated to require as much energy as a trans-American flight.
> Is it fair or just to ask, for example, that the residents of the Maldives (likely to be underwater by 2100 [6]) or the 800,000 people in Sudan affected by drastic floods7 pay the environmental price of training and deploying ever larger English LMs, when similar large-scale models aren’t being produced for Dhivehi or Sudanese Arabic?
The text you quoted claims that the "environmental price" is of "as much energy as a trans-American flight". It implies that this would contribute to these issues (true), and rhetorically suggests that this is not fair, but words-as-written, I don't see any distortion of truth. Heck, it's using familiar units: most of us have an order-of-magnitude idea how many trans-American flights there are each month.
"Author is not affecting a neutral point of view" is not the same as "author is distorting truth".
I guess if one aspires to academic discourse about Truth rather than about Activism or about Power then this will seem somewhat like motivated reasoning.
If you want academic discourse about truth, become a philosopher. Activism and power are just as real as trees and language models. Discussing them is discussing the world.
Your concern about motivated reasoning is a valid one – but motivated reasoning is different to persuasive language. If anything, persuasive language highlights the places you should be looking for motivated reasoning (but it also makes it harder to spot, for some people; I don't recommend using persuasive language in your own papers).
I think of it as something like the journalist becoming part of the story. Yes we all have our opinions but to the extent that I can tell what yours are the researcher has become the subject and frankly, that is much less interesting.
> I'm a little surprised at the contrast between the contents of the paper and how people have described it on HN.
I haven't read the paper beyond the abstract yet, so can't comment on it's contents, but can you specify what you mean by "how people have described it on HN"?
To be frank, I don't like this style of argument, because I literally don't know what you're referring to. I'm quite familiar with the hubbub that happened when Gebru left Google, but feelings around what happened there shouldn't be misconstrued as feedback on this paper specifically.
In other words, if you think that people are arguing that this paper is "outside the realm of valid academic discourse", I think you should call out what you're referring to, because I haven't seen that as a widely held opinion. It feels like a straw man, or at the very least you are discounting some very specific critiques other commenters have made by just broadly referring to "HN sentiment".
Apologies, I assumed it would be clear from the other comments. When I posted this the top comment was a very visibly negative take on the paper. Similarly, Google must have considered this paper to be sufficiently outside the realm of valid academic discourse because their actions around the release of this paper lead to the termination of a couple authors and a PR snafu.
Thanks for explaining. FWIW, I don't think your last sentence is a fair assessment of what happened at all. Pointing out major deficiencies in a paper is not the same thing as saying it's "sufficiently outside the realm of valid academic discourse", and, having closely followed the news when this all went down, it's clear that it's what happened in the aftermath of this paper that led to the terminations - it's not like Google said "this paper is so bad, you're fired".
I can’t speak to the aftermath but even interfering in the publication of the paper is pretty dramatic. If Google had let it go through and not said anything, or at worst published a milquetoast response saying they disagree but respect the discourse, I don’t think the paper would have had half the attention. If your scientists publish a paper that is not even wrong, do you ask them to retract it? Shouldn’t the response from the larger academic community be enough?
I just finished working my way through this this morning. The literature list is quite interesting and gives a lot of pointers for people who want to walk the line between overblown hype and doomsday scenarios.
People seem to miss the specific thing that was controversial about this paper:
The paper gives inaccurate estimates of the carbon dioxide emissions that result from machine learning.
My understanding is that this is the main reason why Google wanted the paper pulled. You can't have a paper by Google authors saying incorrect things about how much CO2 Google is emitting.
The authors refused. They argued that they gave a citation for their numbers, and it's not their problem if the paper they cite is inaccurate.
This is a great paper to introduce someone to the potential ethical issues that large language models have. A concern I have with these models that the paper only dusts upon is the notion of "truth" when the machine has no method to determine it. Do we admit defeat for what the machine does not have a very high confidence in an answer, or go with a popular opinion/interpretation? These are epistemological issues that I don't see being resolved anytime soon.
The problems with LLM are numerous but whats really wild to me is that even as they get better at fairly trivial tasks the advertising gets more and more out of hand. These machine dont think, and they dont understand, but people like the CEO of OpenAI allude to them doing just that, obviously so the hype can make them money.
And it would be bad for a submarine salesman to go to people that think swimming is very special and try to get them believing that submarines do swim.
Why would that be bad? A submarine salesman convincing you that his submarine "swims" doesn't change the set of missions a submarine might be suitable for. It makes no practical difference. There's no point where you get the submarine and it meets all the advertised specs, does everything you needed a submarine for, but you're unsatisfied with it anyway because you now realize that the word "swim" is reserved for living creatures.
And more to the point, nobody believes that "it thinks" is sufficient qualification for a job when hiring a human, so why would it be different when buying a machine? Whether or not the machine "thinks" doesn't address the question of whether or not the machine is capable of doing the jobs you want it to do. Anybody who neglects to evaluate the functional capability of the machine is simply a fool.
Don't you think they're an exemption because they're alive? If seals had propellers we'd still say they swim. Squids use jet propulsion and we still say they swim; do jetskis also swim? Somehow not.
> These machine dont think, and they dont understand
But they do solve many tasks correctly, even problems with multiple steps and new tasks for which they got no specific training. They can combine skills in new ways on demand. Call it what you want.
They don't. Solve tasks, I mean. There's not a single task you can throw at them and rely on the answer.
Could they solve tasks? Potentially. But how would we ever know that we could trust them?
With humans we not only have millennia of collective experience when it comes to tasks, judging the result, and finding bullshitters. Also, we can retrain a human on the spot and be confident they won't immediately forget something important over that retraining.
If we ever let a model produce important decisions, I'd imagine we'd want to certify it beforehand. But that excludes improvements and feedback - the certified software should better not change. If course, a feedback loop could involve recertification, but that means that the certification process itself needs to be cheap.
And all that doesn't even take into account the generalized interface: How can we make sure that a model is aware of its narrow purpose and doesn't answer to tasks outside of that purpose?
I think all these problems could
eventually be overcome, but I don't see much effort put into such a framework to actually make models solve tasks.
> Also, we can retrain a human on the spot and be confident they won't immediately forget something important over that retraining.
I don’t have millennia, but my more than 3 decades of experience interacting with human beings tell me this is not nearly as reliable as you make it seem.
There is no guarantee that a human would solve the task correctly. Therefore, according to your logic, we can say that humans do not solve tasks either.
To claim that only humans can accurately solve a task using words and wisdom is to give humans too much credit. They are not that lofty, sacred, or absolute.
Do you believe that machines cannot think or understand? That is very racist. It is a terrible racist to refuse to recognize the possibilities for beings that are different from you.
Why would there be a falsifiable hypothesis in it? Do you think that's a criterion for something being a scientific paper or something? If it ain't Popper, it ain't proper?
LLMs dramatically lower the bar for generating semi-plausible bullshit and it's highly likely that this will cause problems in the not-so-distant future. This is already happening. Ask any teacher anywhere. Students are cheating like crazy, letting chatGPT write their essays and answer their assignments without actually engaging with the material they're supposed to grok. News sites are pumping out LLM-generated articles and the ease of doing so means they have an edge over those who demand scrutiny and expertise in their reporting—it's not unlikely that we're going to be drowning in this type of content.
LLMs aren't perfect. RLHF is far from perfect. Language models will keep making subtle and not-so-subtle mistakes and dealing with this aspect of them is going to be a real challenge.
Personally, I think everyone should learn how to use this new technology. Adapting to it is the only thing that makes sense. The paper in question raised valid concerns about the nature of (current) LLMs and I see no reason why it should age poorly.
This is generally my feeling as well with the paper.
You don't come out feeling "Voila! this tiny thing I learnt is something new", which does happen often with many good papers. Most of the paper just felt a bit anecdotal & underwhelming (but I may be too afraid to say the same on Twiiter for good reason)
I don't know, without enumerating risks to check, there's little basis for doing due diligence and quelling investors. This massively-cited paper gave a good point of departure for establishing rigorous use of LLMs in the real world. Without that, they're just an unestablished tech with unknown downsides - that's harder to get into true mass acceptance outside the SFBA/tech bubble.
This is ok. 90% of research is creative thinking, dialogue. One idea creates the next, some are a foil, some are dead ends. As long as there are not outrageous claims being made for 'hard evidence' where there is none, it's fine. Maybe the format isn't fully appropriate but the content is. Most good things come about in a non-linear process which involves provocation along the line somewhere.
I expect science to have a hypothesis which can be falsified. Otherwise it’s just opining on a topic. Otherwise we could just call this HN thread “research”.
This was mostly political guff about environmentalism and bias, but one thing I didn't know was that apparently larger models make it easier to extract training data.
> Finally, we note that there are risks associated with the fact
that LMs with extremely large numbers of parameters model their training data very closely and can be prompted to output specific information from that training data. For example, [28] demonstrate a methodology for extracting personally identifiable information (PII) from an LM and find that larger LMs are more susceptible to this style of attack than smaller ones. Building training data out of publicly available documents doesn’t fully mitigate this risk: just because the PII was already available in the open on the Internet doesn’t mean there isn’t additional harm in collecting it and providing another avenue to its discovery. This type of risk differs from those noted above because it doesn’t hinge on seeming coherence of synthetic text, but the possibility of a sufficiently motivated user gaining access to training data via the LM. In a similar vein, users might query LMs for ‘dangerous knowledge’ (e.g. tax avoidance advice), knowing that what they were getting was synthetic and therefore not credible but nonetheless representing clues to what is in the training data in order to refine their own search queries
Shame they only gave that one graf. I'd like to know more about this. Again, miss me with the political garbage about "dangerous knowledge", the most concerning thing is the PII leakage as far as I can tell.
Is this a good or bad thing? We hear "hallucination" this and that. You can't rely on the LLM. It is not like a search engine. But then you hear on the other side "it memorises PII".
Being able to memorise information is demanded when we want the top 5 countries by population in Europe or the height of Everest. But then we don't want it in other contexts.
Is it conceivable that a model could leak PII that is present but extremely hard to detect in the data set? For example, spread out in very different documents in the corpus that aren't obviously related, but that the model would synthesize relatively easily?
That is sort of understood facts with even models like Copilot & ChatGPT. With the amount of information we are generally churning, all PII may not get scrubbbed. And these LLMs often could be running on unsanitized data - like a cache of Web on Archive.org, Getty images & the likes.
I feel this is a unavoidable consequence of using LLM. We cannot ensure all data is free from any markers. I am not a expert on databases/data engineering so please take it as an informed opinion
Copilot has a ton of well publicised examples of verbatim code being used, but I didn't realize that it was as trivial as all that to go plumbing for it directly.
I believe this is the papers that got timnit and mmitchel fired from google, followed by a protracted media/legal campaign against google and vice versa.
I suspect it was Timnit’s behavior after the paper didn’t pass internal review that actually got her fired (issuing an ultimatum and threatening to resign unless the company met her demands; telling her coworkers to stop writing documents because their work didn’t matter; insinuations of racist/misogynistic treatment from leadership when she didn’t get her way).
I think it was a well calculated career move, she wanted fame, she got what she wanted. Now she's leading a new research institute
> We are an interdisciplinary and globally distributed AI research institute rooted in the belief that AI is not inevitable, its harms are preventable, and when its production and deployment include diverse perspectives and deliberate processes it can be beneficial. Our research reflects our lived experiences and centers our communities.
It was Megan Kacholia, who had put Timnit Gebru and others close to her down for a long time constantly within Google, always talking down and being condescending and rude, failing to respect Timnit in how she confronted Timnit about the paper (which she was ordered to retract by way of not Google's normal paper review process, but by a then-newly-implemented and since retracted secondary "sensitive topics review" process, due to a combination of actual mistakes like the environment numbers, and also Google being too afraid of reputational damage for her discussion of the very real and tangible harms of LLMs).
Timnit tried to raise this to Jeff Dean to get help (Jeff was Megan's manager at the time). Jeff completely misunderstood what she was asking for, and instead sent some response about the environment numbers being incorrect (and they are, but that doesn't at all justify the way Timnit was treated). Not beginning to imagine that Jeff could have missed this signal, Timnit responded sarcastically. Jeff didn't pick up on the sarcasm and thought all was good.
Timnit then reacted by describing her frustrations with how she was treated in an internal diversity mailing list. She also emailed Megan Kacholia with a number of demands, mostly to be treated reasonably. Appalled at how she and her coauthors were treated, she refused to retract the paper. Megan reacted by taking her note that she would work on a resignation date if demands were not met in combination with Timnit's email completely pedantically and out of context, using them as an excuse to fire her by rushing her out, without allowing her to follow the actual resignation process. She also acted over Timnit's manager's head (Samy Bengio), who was so annoyed he later quit. (Megan cc'd Jeff, but hadn't spoken to Jeff about any of this, and was acting on her own.)
Interestingly, Timnit's email to the diversity list was so resonant that several of the changes it asked for in how Google approaches diversity were enacted after her firing. But Megan and Google's official line on all of this chose to obsess over Timnit's rhetorical devices and take them literally instead, using an email to a diversity list about diversity against her. People are still too afraid to talk about diversity on diversity lists, now, because of Google using that email against her.
Google reacted by gaslighting Timnit to protect its ass. After Timnit Tweeted that Jeff had fired her (Timnit probably really thought that Jeff and Megan had spoken to each other before Megan had sent that email), Jeff participated in this part in public, on Twitter, with a lot of serious consequences for Timnit and others, without considering power dynamics. (Jeff suffered a lot on Twitter, too, but that doesn't excuse not considering power dynamics in such a consequential way on such a consequential medium.) Timnit and others, including me, were harassed and threatened because of this, by way of third-party harassers. I was not even involved on the paper, just proximal damage. I was afraid for my life honestly.
Meg Mitchell, feeling lost, having seen the truth of how Timnit was treated internally, and trusting Jeff to protect her, tried to put together some things for Jeff to get him to see how Timnit was mistreated. She panicked and backed them up on her personal email because she was afraid of retaliation from Google (a reasonable fear---doing any diversity or community work at Google that at all challenges the status quo IME gets you retaliatorily reported to PeopleOps, who then try to get you in trouble and read your private communications and so on). She was transparent about doing this and gave instructions for Google to remove her personal copy if needed. Sundar Pichai fired her and then comms smeared her publicly with outright lies. She was harassed and threatened for this, too, and a number of places refused to hire her because of Google's treatment of her. Really tangible damage emotionally, financially, and reputationally.
Out of fear of being sued, Google's comms and legal departments reacted by continuing to censor and gaslight. Sundar was extremely complicit in this, too. Megan was moved out of Research, but not much else happened; she continues to send monthly emails about diversity, as if her continued contact with Research is not actively harmful to diversity.
So sick of internet people speculating about this without knowing anything about the situation. Sorry if I broke anyone's trust here. Just can't deal with this incorrect speculation anymore. (I have extremely thorough information about this, but to those directly involved, please feel free to correct me about any details I got wrong, or about important details I omitted.)
Cheers. That's way much more information than I ever wanted to know about that sorry affair. If it can quell the torrent of ad-hominems, it's worth it, but I doubt it. All those hardcore soft. engineers here on HN who spend 99.99999% of uptime close to the bare metal think that people like Gebru who work on ethics are useless hangers-on without any "real contributions" (probably because none of them has bothered to check her background on wikipedia).
Nevertheless, hoping to check your sources I clicked through your profile and I have a question, totally unrelated to all this. Can you say something about the state of the art in "neural proof synthesis"? To clarify, I'm scare-quoting because I didn't even know that's a thing. For context, my background is in the European tradition of Resolution-based automated theorem proving (Prolog and all that) but also statistical machine learning, so don't worry about simplifying terminology too much.
Btw, the "proof engineering" link in your profile gives me a security alert on firefox.
ML folks often call it "neural theorem proving." SOTA results are still from combinations of tactic prediction models with specialized tree search processes. They do OK on some interesting benchmarks, but still can handle mostly only fairly simple proofs. So far, they seem strictly complementary to symbolic methods. Interest is growing dramatically, though, and progress is accelerating, so I'm excited about the near future.
Language models are showing a lot of promise for autoformalization: automatically converting natural language mathematics to formal definitions, specifications, and proofs. This is a task where symbolic methods do not seem particularly promising in general, and one that meshes nicely with synthesis.
A good conference to look at is AI for Theorem Proving (AITP). It's small but has a lot of relevant work. All of the talks from this past year are recorded and on the website. MATH-AI at NeurIPS had some good work this year, too.
There is a bit of a culture and citation gap dividing the work in the AI community from the work in the PL/SE communities; in PL/SE I'd recommend work by Emily First and Alex Sanchez-Stern. They are undercited in AI work despite having SOTA results on meaningful Coq benchmarks. In AI, I'm particularly psyched about work by Yuhuai (Tony) Wu, Markus Rabe, Christian Szegedy, Sean Welleck, Albert Jiang, and many others. Tony's papers are a good gateway into other AI papers since the AI papers tend to cite each other.
Thanks. I'm a bit more familiar with neural theorem proving. It's an interesting area. For example, if I could train me a model to speed up (NP-complete) θ-subsumption for very long terms that would be a worthy addition to the purely symbolic toolbox I'm more at home with.
Autoformalization also sounds interesting. I've had some conversations about automatically turning big corpora of natural language text into Prolog with language models, for example. I don't reckon anyone is even researching how to do this with symbolic methods at the moment.
I'll check out AITP. Thanks for the pointers. I'm used to small conferences [and to underciting between disciplines] :)
I went to AITP for the first time last September, and I found it an absolute pleasure. Everyone was kind and wonderful and open-minded. The venue was wonderful too. Highly recommended if you're interested at any point.
BTW what you say about the sadly common assumption Timnit and other AI ethics folks don't have "real contributions" is too real. It has impacted me even though my work isn't on AI ethics at all, just because I bother to talk about it online in public sometimes. Similarly for any social justice work or any work improving the work environment in research. It is like some people cannot comprehend that one can be technically proficient and still care about social justice and ethics and other "soft" issues. I love how confused those people are when they learn my expertise is in formal logic and proof haha
It's lack of training I think. A good engineer should think about the consequences of her actions. People in the industry, it seems, just don't. Very disappointing.
OK, first clarification after further correspondence, the mistake on the environment numbers was small---accidentally misunderstanding the context in which Strubell mentioned particular numbers, I think? And Strubell's numbers were off because they used only public data they had access to, and I think misunderstood some things too. Some of the authors did not even know about it and it is news to them now. And it could have been addressed in a camera-ready, nonetheless. It was no reason to force the authors to retract a paper or remove their names, and that is part of the treatment of them that was extremely messed up.
Nah, this is the actual truth. You can feel free not to believe me, but I have more complete information about this situation than anyone else you'll ever talk to.
None of that disputes what I said or excuses her behavior. The “resonant” email is public, we can’t pretend it was in any way professional or appropriate.
Vanishingly few people can get away with acting like that at work without getting fired. She thought she was an exception and she wasn’t.
It was fine, unless you reach far enough to take it literally, which is quite obviously not what Timnit intended. The call to stop DEI work was a rhetorical device to highlight to an audience that presumably cares strongly about diversity that the current work is not meaningful without broader systemic change within Google. Plus an emotional plea to what should have been a sympathetic audience to understand how exhausting doing any real diversity work within Google is. This is true and resonant and so triggered a lot of said change after she was fired. The change is still ongoing.
Everyone who tries to actually enact real systemic change within Google Research to improve how women and people of color are treated, or the work environment in general, burns out or gets fired. It is only toeing the company line when it's officially your role to do so, taking extremely conservative actions while ignoring real issues like abuse, harassment, and discrimination, that lets you survive while doing DEI work at Google in Research without burning out or getting fired. The sole exception I know of right now is Kat Heller, god bless her.
The feeling of "none of this is worthwhile, what's the point of any of it" is something everyone who tries to do real diversity and culture work within Google Research or even within the computer science research community more broadly has felt, and a diversity mailing list ought to have been a safe place to share that feeling.
Instead it was used against her. Now nobody feels safe sharing these feelings anywhere within Google Research.
Jeff to his credit is now working on many of these things. As is the DEI committee. I hope they all succeed. But I don't think any of that would have happened without Timnit highlighting these very real issues. It's depressing that she was fired for it. She wanted to change things and keep her job and her team, not to be a martyr.
Timnit’s backpedaling regarding her email and her ultimatum was absurd. You seem completely unwilling to factor Timnit’s own behavior into the chain of events, so it doesn’t make sense to continue this. This elimination of personal responsibility when it’s inconvenient to the narrative has become endemic among the activist employees. To your credit, it seems like your heart is in the right place with wanting to help her and others.
The other day on Hacker News, there was that article about how scientists could not tell GPT-generated paper abstracts from real ones.
Which makes me think- abstracts for scientific papers are high-effort. The corpus of scientific abstracts would understandably have a low count of "garbage" compared to, say, Twitter posts or random blogs.
That's not to say that all scientific abstracts are amazing, just that their goal is to sound intelligent and convincing, while probably 60% of the junk fed into GPT is simply clickbait and junk content padded to fit some publisher's SEO requirements.
In other words, ask GPT to generate an abstract, and I would expect it to be quite good.
Ask it to generate a 5-paragraph essay about Huckleberry Finn, and I would expect it to be the same quality as the corpus- that is to say, high-school English students.
So now that we know these models can learn many one-shot tasks, perhaps some cleanup of the training data is required to advance. Imagine GPT trained ONLY on the library of congress, without the shitty travel blogs or 4chan rants.
> "The corpus of scientific abstracts would understandably have a low count of "garbage" compared to, say, Twitter posts or random blogs"
That's certainly true, but it's not by a so large margin, at least in biology.
For example in ALS (a neurodegenerative disease) there is a real breakthrough perhaps every two years, but most papers about ALS (thousands every year) look like they describe something very important.
Similarly for ALZforum the most recent "milestone" paper about Alzheimer disease was in 2012, yet in 2022 alone there were more than 16K papers!
The science is in the reproduction of the methodology, not in the abstract… in fact, a lot of garbage publications with catchy abstracts built on a shaky foundation sounds like one of the issues that plagues contemporary science. That people would stop finding abstracts useful seems a good thing!
I am of the general understanding that this paper became less about the LLMs & more of a insinuating hit piece against Alphabet. At least, some of the controversial nuggets got Gebru (and later M Mitchell) fired.
From a technical standpoint, there is little new stuff that I found this paper offered in understanding why LLMs can have unpredictable nature, or what degree of data will get exposed by clever hacks (or if there are systematic ways to go about it). It sounded more like a collection of verifiable anecdotes for easy consumption (which can be a good thing by itself if you want capsule understanding in a non-technical way)
It was activism masquerading as science. Many researches noted that positives and negatives were not presented in a balanced way. New approaches and efforts were not credited.
I haven't kept track but the activism of the trio could be severe sometimes.
(Anecdotally, I have faced a bite-sized brunt: When discussion surrounding this paper was going on in Twitter, I had mentioned in my timeline (in a neutral tone) that "dust needed to settle to understand what was going wrong". This was unfortunately picked up & RTed by Gebru & the mob responded by name-calling, threatening DMs accusing me of racism/misogyny etc, and one instance of a call to my employer asking to terminate me - all for that one single tweet. I don't want confrontations - not my forte to deal.)
> This was unfortunately picked up & RTed by Gebru & the mob responded by name-calling, threatening DMs accusing me of racism/misogyny etc, and one instance of a call to my employer asking to terminate me - all for that one single tweet.
Wait until an LLM flags your speech and gets you in trouble. That'll be a real hoot compared to random individuals who likely have been chased off Twitter by now.
Talia, if this was all a misunderstanding from Jeff, then why was Timnit accusing Jeff for the longest possible time. I can understand Megan Kacholia was the main person in all this drama according to you, but the trolling/accusing that we saw in the aftermath was vicious and in poor taste. Honest bystanders who were giving neutral opinions were gaslighted to choose what side they morally belonged.
Why didn't any of you try to stop this verbal carnage? Also if Timnit was largely not at fault, why is that on every social media where any modicum of anonymity is possible, Timnit had been harshly criticized for her conduct.
We respect her contributions to advancement of ML, but that conduct was inexorable & in very poor taste.
Note that not everything was a misunderstanding on Jeff's end, just the early stuff before she was fired, which Timnit did not know. To be honest, I don't think Timnit would even believe that now. It's maybe one place where I diverge from her on this. She would find what I wrote about Jeff way too charitable. Her view is valid and is one of the reasons I rarely discuss this in public anymore, because it's hard for me to do so and not emotionally want to defend Jeff, and it's hard to defend Jeff on just the things I believe he ought to be defended for and still hold him accountable, and not gaslight Timnit or minimize what she went through. I probably messed up somehow above too.
It's hard to believe sometimes that someone so smart in some ways can miss such obvious signals. It took a long time for me to come to that conclusion and to understand, and I'm still very upset with how Jeff reacted.
These are all people though, they make mistakes, Timnit included. The way she was treated is still not at all OK, and sometimes when we are in positions of power like Jeff we still hold responsibility for our mistakes. Even if Jeff genuinely misunderstood, he should have apologized for his role and moved to repair harm, rather than reinforcing and exacerbating harm in his public response.
I won't say anything else about Jeff in public. I did try to help minimize damage and pain in all directions. It was painful and exhausting and not very fruitful.
(Also sorry but who are you? Just because you addressed me by name and it feels weird when that happens unless I know who is talking to me.)
Got it, thank you. On why Timnit is criticized so intensely everywhere anonymity is possible, I honestly think that is more an artifact of sexism and racism than an indictment of anything she has done. Also a matter of the target audience of a lot of the anonymous forums.
FWIW, at Google in Research, many people have mentioned her to me as the only person they trusted to talk to about the things they went through when they were not treated well. After she was fired, many of those same people felt no longer able to raise issues about internal treatment and culture.
Timnit was not carrying just her own burden, but the burden of many at Google in Research who were not treated well, especially women and people of color in Research. And so she spoke not just for herself. She had witnessed for years how demoralizing it is to try to really change things within Google. I think the only person who has done that and not burned out is Kat Heller. I did a lot of it over the summer, and it really took a toll on my wellbeing, and my desire to stay (part-time) at Google to finish my own work there. I'm excited for my affiliation to end so I can remove myself more thoroughly from Google's internal politics and culture, though I hope I've made enough of a dent that some things actually continue to change for the better.
Compare this to something like RLHF[0] which has acheived far more for aligning models toward being polite and non-evil. (This is the technique that helps ChatGPT decline to answer questions like "how to make a bomb?")
There's still a lot of work to be done and the real progress will be made by researchers who implement systems in collaboration with their colleagues and employers.
[0] https://openai.com/blog/instruction-following/