The cited paper seems to be really bending over backwards to find some trace of bias. Unwinnable game for LLMs.
E.g. cited work claims "LLMs assign significantly less prestigious jobs to speakers of African American English... compared to Standardized American English". You don't say! Formal/business language has higher association with prestigious jobs than informal/street/urban language. How is that even classified as "bias"?
> Formal/business language has higher association with prestigious jobs than informal/street/urban language. How is that even classified as "bias"?
Judging people's fitness for a job based on the dialect of English that they speak is by definition a form of bias. That it's bias that is also reflected in the workplace pre-LLMs doesn't make it not bias.
If most of my employees can’t understand the way someone speaks or writes, it’s likely that person will have problems communicating ideas within the organization.
Also, by definition where? You’re just making this up, there isn’t an authoritative dictionary or even regulation that states judging fitness for a job based on the dialect of English a person speaks is “bias.” If there was, surely you would have provided a link instead of just asserting your own correctness.
The high prestige jobs the LLMs associated with SAE mostly require higher education, usually post-graduate.
Perhaps the model is only accurately learning that more educated people use a particular dialect of English taught by universities.
There's really no way to tell, because the researchers didn't include other dialects of English that aren't favored by universities, like southern, or Yorkshire.
Alright, but I think their point is "bias as reflected in the real world." There's a bias against hiring people that don't speak English at all for English speaking jobs, I don't think anyone of sound mind would call that xenophobia. Critically what we are trying to find out with the LLMs is if there is bias beyond social acceptability or beyond job requirements. If you wouldn't hire someone for a job that said "sup dog" upon meeting you in an interview, why should the same bias indicate something nefarious for an LLM?
If people speaking a certain dialect tend not to work in a particular role, an LLM is only being accurate in deprioritizing that occupation when guessing what they might do.
It does make it not bias when an LLM is accurately representing reality.
I went to a 95% percent black high school. I recall my English teachers stressing avoidance of AAE (then called Ebonics) in professional settings.
Speak however you wish with your friends, they said, but use Standardized American English for the job interview. Apparently this goes double for LLMs.
They even made us write the same paper twice. Once in standard English and again in AEE, so the kids would know the difference.
This is good advice for students who speak native AAE because there exists bias against the dialect. It's better for their own, personal life trajectory to adapt to the way that the world will judge their dialect.
However, that doesn't make it okay to continue to perpetuate the idea that AAE is somehow a lesser dialect and that AAE speakers ought to have to mask their accent in the way that an Indian, Brit, or Australian doesn't.
We can simultaneously teach students how to navigate the dangers of the world they live in while trying to fix said dangers for future generations.
>However, that doesn't make it okay to continue to perpetuate the idea that AAE is somehow a lesser dialect and that AAE speakers ought to have to mask their accent in the way that an Indian, Brit, or Australian doesn't.
As a Brit living in Britain, I habitually code-switch, because my natural dialect is coded as low-status. If I used my natural dialect in a job interview, it would very clearly communicate one of two things - either I am unwilling to conform to the behavioural norms of a professional workplace, or I lack the linguistic skills to do so.
I cannot pretend to understand the cultural context surrounding AAVE, but the prejudice against my own dialect is broadly rational. Learning to speak in mildly-accented standard English is just one of many shibboleths that signify membership of the professional middle class.
Can I ask what British dialect is associated with lower class? I was not aware there were class-indicating dialects besides AAVE to be honest. To me, anything British sounds higher class if anything
Makes me wonder if we have them in Dutch as well. I guess simply sounding like a foreigner, or making mistakes about word gender or such, will make you stand out as not knowing the language properly (even if you do and merely haven't got the pronunciation down), but I wouldn't know of anyone who grew up speaking Dutch in a native way who subsequently sounds lower class. The Belgian Flemish and southern Limburgians sound funny to most people, but it's not a lower class, just a region-of-origin indicator
Any strong regional accent is coded as working class. The upper class speak Received Pronunciation, which is broadly viewed as a "standard" accent and was once the only accent allowed on the BBC. The middle class will avoid using dialect and significantly soften their accent, shifting towards RP. There has been a long-standing debate about whether aspirational members of the working class should moderate their accents in order to advance professionally, but the reality is that most feel obliged to do so. Accent bias is still recognised as a major barrier to social mobility.
Not OP but in Southeastern England obviously Cockney, Estuary English, or Multicultural London English are considered lower prestige dialects whereas RP is high prestige. Other regions may have a prestige version of the local dialect that incorporates some RP features while retaining some local features, but broadly speaking, regional speech patterns have lower prestige because they tend to be more prominent among people with less education.
There are dialects that are associated with wealth and education that are still discouraged in the business world for various practical reasons. It certainly is a penalty to have a dialect and many educated parents will try to school their kids to speak differently. Not as a replacement, but as an alternative.
I doubt you can remove the preference due to the advantages. There is a force for consolidation, just as there is with English as a whole for example.
This isn't to agree or disagree with other points you might be making, but one thing I learned not too long ago is that AAE isn't quite "street" or "urban" English. Like other dialects of English (say, Scottish), it's a dialect of its own, with its own interesting rules, nuances, and pronunciations. A very short explanation of it is here, but you can find more if you're interested: https://www.youtube.com/watch?v=zw4pD4DNOHc
They’re pretending that an LLM recognizing patterns that exist in reality reflects a problem in its training data. So much of the conversation around “reducing bias” really means intentionally introducing biased data to “socially engineer” LLMs (and by proxy their users) so that they have a less accurate picture of reality.
E.g. cited work claims "LLMs assign significantly less prestigious jobs to speakers of African American English... compared to Standardized American English". You don't say! Formal/business language has higher association with prestigious jobs than informal/street/urban language. How is that even classified as "bias"?