That's not entirely true in the case of base64 because of how statistical patterns within natural languages work. For example, you can use frequency analysis to decrypt a monoalphabetic substitution cipher on pretty much any language if you have a frequency table for character n-grams of the language, even with small numbers for n. This is a much more shallow statistical processing than what's going on within an LLM so I don't think many were surprised that a transformer stack and attention heads could decode base64. Especially if there were also examples of base64-encoding in the training data (even without parallel corpora for their encodings).

It doesn't explain higher level generalizations like being a transpiler between different programming languages that didn't have any side-by-side examples in the training data. Or giving an answer in the voice of some celebrity. Or being able to find entire rhyming word sequences across languages. These are probably more like the kind of unexplainable generalizations that you were referring to.

I think it may be better to frame it in terms of accuracy vs precision. Many people can explain accurately what an LLM is doing under all those matrix multiplies, both during training and inference. But, precisely why an input leads to the resulting output is not explainable. Being able to do that would involve "seeing" the shape of the hypersurface of the entire language model, which as sibling commenters have mentioned is quite difficult even when aided by probing tools.