Thanks for the ideas and references!
I gotta say, though, that I will be pretty sad if having to write a custom protocol turns out to be the final solution. So much more convenient to use OTP (especially now that we finally have an infinitely extensible serialization library for it; Zerl). I'm shocked such an oversight would exist in a real commercial solution which is the BEAM.
The original application of dist clustering was dual computers in a single telecom switch. There's not really a need for a security barrier in that case; anyone with access to one computer would be expected to have access to the other.
Additional applications for dist have been explored over the years, but most of them involve clustering servers; where a security barrier isn't necessary; although it might be desirable --- I've used dist clusters where some people had access to only certain types of nodes; bypassing access control using dist clustering was certainly a possibility. Bolting security onto something designed without it often is pretty challenging. Especially if you want to keep all the existing applications working.
As another commenter said, OTP messages are meant to be between processes in the same privilege zone. That said, using a custom protcol via a good library can actually bring benefits relative to core OTP stuff.
For example, several of the gRPC libs I've used for Erlang/Elixir are pretty low-cognitive-overhead to use, and they come with all the added gRPC goodies: RPC semantics are described in one place rather than ad-hoc throughout code, protobufs have at least a documented (if not actually good) process for upgrades and backwards compatibility, multilanguage gets easier (even if your second language is just a tiny sliver of "dump protobufs into a database/Jupyter notebook/Rust program occasionally for offline reporting").
To be clear, this isn't a paean to gRPC; most of those features are table stakes for an IDL-driven protocol definition. Just saying that you do get some things in return for giving up the convenience of OTP, if you pick the right tools.
