Google will happily permit you to download trained models of <100MB to your phone via Google Translate, which permit not only offline vocabulary recognition of a remarkably wide range, but will also then translate that input into another language (also offline).

I believe the restrictions imposed by keyword-spotting have more to do with the always-listening and/or power-efficient nature of the task, rather than the restrictions on the legibility of offline speech recognition.