Just did exactly that with Avant. Could see where I was initially approved but then their manager said I was "still on contract" and quashed my application.
If you make a machine learning model explain itself to humans, you open the possibility to challenge the model's conclusion. It's not just the question of whether the model does its math right - it also means we, as society, can say, "I see your math and it works out, but because $POLICY, you're not allowed to take these inputs into account". It means adding control points to a previously opaque process.
you are right, and i absolutely agree. basing any decisions that substantially affect someone on machine learning alone should be illegal (even if it is the right decision)
I was denied, but now I know why instead of just guessing. I could see that my "willingness to pay" (unsure what that means) etc and income were all fantastic, their manager just didn't like that I was on contract instead of a permanent employee.
so arguably, in this case the machine learning algorithm was possibly right, and the human was wrong. on the face of it, not using machine learning here would not have changed the outcome.
what i am wondering is though if this can't be used to everyones benefit.
apart from neding a human to verify a decision, how about, if the human and machine learning decision disagree, then the consumer gets the right to appeal for a second human review?
Funny thing is that I specifically opted out of automated decision making, having been declined before. It wasn't a big deal, really. EU lenders are far warier of contract jobs than American ones.
oh there was an opt-out. that's interesting. how does that work? you obviously opted out before, but could you also opt out after they tell you what the automated decision would be? if so that would effectively be that kind of appeal that i suggested.
I applied for a loan a year ago, didn't opt out got denied, and wondered why. Then I applied again ~4 months ago, opted out of automated decision making, and the subject access request showed the conversation.
