FWIW, wake words are a stopgap; if we want to have a Star Trek level voice interfaces, where the computer responds only when you actually meant to call it, as opposed to using the wake word as a normal word in the conversation, the computer needs to be constantly listening.

A good analogy here is to think of the computer (assistant) as another person in the room, busy with their own stuff but paying attention to the conversations happening around them, in case someone suddenly requests their assistance.

This, of course, could be handled by a more lightweight LLM running locally and listening for explicit mentions/addressing the computer/assistant, as opposed to some context-free wake words.

Home Assistant is much nearer to this than other solutions.

You have a wake word, but it can also speak to you based on automations. You come home and it could tell you that the milk is empty, but with a holiday coming up you probably should go shopping.

I want that for privacy reasons and for resource reasons.

And having this as a small hardware device should not add relevant latency to it.

Privacy isn't a concern when everything is local

Yes it is.

Malware, bugs etc can happen.

And I also might not want to disable it for every guest either.

If the AI is local, it doesn't need to be on an internet connected device. At that point, malware and bugs in that stack don't add extra privacy risks* — but malware and bugs in all your other devices with microphones etc. remain a risk, even if the LLM is absolutely perfect by whatever standard that means for you.

* unless you put the AI on a robot body, but that's then your own new and exciting problem.

There is no privacy difference between a local LLM listening versus a local wake word model listening.

That would be quite easy to integrate. RealtimeSTT already has wakeword support for both pvporcupine and openwakewords.

Modify it with an ultra light LLM agent that always listens that uses a wake word to agentically call the paid API?

You could use open wake word. Which Home Assistant developed for its own Voice Assistant

It was developed by David Scripka: https://github.com/dscripka/openWakeWord