> CEO's at least in the USA have multiple legal obligations under federal law.

Lots of people have legal obligations.

In this case, I assume that in this case you're referring to a fiduciary duty (i.e. to act in the best interests of the company), which is typically held not by the CEO, but but by the directors.

Ultimately the responsibility to assign daily operation of the company rests with the board, both legally and practically, as does the decision to use a human or AI CEO.

Under Delaware law (where most U.S. public companies incorporate), directors' fiduciary duties are non-delegable. The actual exercise of judgment must be performed by the director, who must be a "natural person". Other jurisdictions might offer grey areas, but good luck finding one that meaningfully changes this.

More practically, legal accountability would be placed in the individuals approving LLMs actions and/or the entity providing the LLM service. The latter aspect being why many AI vendor deals fall through. Because everything is awesome until the contract comes and the vendor wants to take no responsibility for anything that results from their product.