You may get way more accurate results from relatively small models as well as logits for each class if you ask one question per class instead.

Likely not, embeddings are very crude. Embeddings of a text is just an average of "meanings" of words.

As is embeddings lack a lot of tricks that made transformers so efficient.

This is very cool.