Video calls are a big missing feature, that's one of the only reasons I have an alternative installed alongside it. From what I've heard, the devs have said that the reason why native audio/video calls aren't planned literally just comes down to complexity and the maintenance cost.
Matrix has been focusing more recently on trying to break some core functionality out into modules that can be imported -- encryption being the biggest example of this. Movement in that direction for video/audio would be a big deal. It's not just that Fluffychat needs help getting video working -- that would only leave them with another chunk of code to maintain. Ideally, they should be able to import a common library for this, but as far as I know, none exists that's usable for their needs.
Matrix video calls are a good example of this kind of ecosystem need in general -- Element had video calls before the spec actually supported video... because it used a Jitsi plugin. And that 3rd-party offering bought time for the spec authors to come up with more robust native support.
The problem is that 3rd-party clients offering the same support is still wildly complicated. There's a lot of room for improvement there.
