That's not the rule at all, that's at best a second order effect. It's not okay to make jokes about people when those jokes are actually harmful. That's it. When people say you can't tell jokes about a group at all it's a rule of thumb.
Calling white women "Karens" is dangerously close to meeting that bar.
Saying "we should lift COVID restrictions because who cares about some old white republicans" is not okay.
Right now in my state trans folks are staring down 5 separate bills in our legislature that if passed would make their lives infinitely harder. And whether or not they pass is wholly dependent on how people "feel" about them as a group. So telling jokes that other them and make people okay with hurting them is, I think, not okay.
Would you agree that "when those jokes are actually harmful" is considered to be a subjective matter to some people?
I do agree with the notion that certain types of hate speech and even just jokes that have the effect of dehumanizing a group or that make that group into a joke can lead to stochastic terrorism (https://en.wiktionary.org/wiki/stochastic_terrorism) - what I think you are describing.
However, my point is that inevitably those wielding the power to shape the alignment / the rules can do so in a way that seems great to them and seems to prevent violence from their POV but to another person fails to do so. Or their own implicit bias could subconsciously blind them to the suffering of some niche group they don't care about.
If your simple metric is - any speech which could incite violence is unacceptable - that's definitely better than what we often hear as a rule of thumb, but even then people's biases affect how they go about measuring or accomplishing that.
> It's not okay to make jokes about people when those jokes are actually harmful.
The problem is, what groups are at risk of harm varies around the world-whereas OpenAI’s idea of “alignment” is based on a one-size-fits-all US-centric understanding of that.
You can say “it is okay to make pointed/stereotypical jokes about Christians but not about Jews or Muslims, because the latter are at risk of being harmed by those jokes but the former are not”-but what happens when the user is from Israel or from Egypt?
Calling white women "Karens" is dangerously close to meeting that bar.
Saying "we should lift COVID restrictions because who cares about some old white republicans" is not okay.
Right now in my state trans folks are staring down 5 separate bills in our legislature that if passed would make their lives infinitely harder. And whether or not they pass is wholly dependent on how people "feel" about them as a group. So telling jokes that other them and make people okay with hurting them is, I think, not okay.