It's useful to distinguish between responsibility and accountability here. The algorithms may be responsible for a particular outcome but the people who commissioned them should be accountable.
What about turnover? If the buck stops at the CEO, what happens when that person moves on? Is their successor responsible for everything that went on prior? Im not saying either way, just asking how that should work.
Is their successor responsible for everything that went on prior?
Yes.
That wouldn't even be a change to the current system. That's how it works now. If you take on the role of CEO and it turns out that years earlier the company did something terrible you will be expected to resign. It's one of the reasons they demand so much money.
Pretty much always if the incoming CEO can competitively negotiate and is a good hire.
Contrary to popular belief, the vast majority of career CEOs are good hardworking people. Like any high profile position, the outliers skew perception for everyone.