Right, but that doesn't compromise the security of the service necessarily.
Users can catch a malicious server injecting incorrect keys by looking at security notifications and comparing security codes. This is part of the Signal protocol.
This may be tedious but only needs to be done in the event of phone keys getting reset (a once in a year event?), as all companion device keys are automatically verified with signatures provided from an account owner's primary (phone) device
Which means the server can just substitute keys in.