Hi I like that there is a competitor to Kafka in this space and also the build in capability to do transformations. I got a few questions though which I could not find in your docs:
(1) Over at the Apache Arrow FAQ I read that the overhead of serialization in analytical frameworks can be around 80 to 90 % of total compute costs (r_1). While having no concrete numbers on this, from using Kafka together with Kafka Streams I can at least confirm that the overhead of serialization is (very) significant. My question therefore is: Does your WASM engine avoid (de-)serialization between your storage/stream layer and the engine and if not are there plans for this?
(2) Are supported WASM transformations stateless (i.e. single message) only or can they be stateful (i.e. window-ing and stream-stream/table join functionality)
(3) I could not find any reference to the WASM inline lambdas at all in the docs actually, am I missing something?
(1) arrow is great! currently, it does not, but yes it will when we move out of nodejs impl into our own v8 isolates inside an alien thread (seastar concept)
(2) stateful but only for a single partition
(3) will be released in the next week or so. If you look in the github repo you can look into `coproc`
Though the explanation went a bit over my head just one (two) follow up question(s): How did you end up with WASM for the inline lambdas? Did you have some discussion on alternatives like Lua? I am curious about insights on choosing scripting engines/implementations, hence why I am asking.
A different question for this could also be if alternatives to V8 where considered I guess as I believe there are quite a few pure WASM engine implementations out there (unless I did not get some feature that requires you to use V8 and rules out pure WASM engines).
1) multi language support. Ppl love go and rust. So some way to open it up to that community.
2) I like Lua a lot. Such a good tool. It just didn't have the target compatibilities of wasm + the security guarantees of wasm
Afaik, there are just a couple vms with superb performance. V8 and anotherone targeting x86 only. We want to support the AWS graviton instances, so only one choice
(1) Over at the Apache Arrow FAQ I read that the overhead of serialization in analytical frameworks can be around 80 to 90 % of total compute costs (r_1). While having no concrete numbers on this, from using Kafka together with Kafka Streams I can at least confirm that the overhead of serialization is (very) significant. My question therefore is: Does your WASM engine avoid (de-)serialization between your storage/stream layer and the engine and if not are there plans for this?
(2) Are supported WASM transformations stateless (i.e. single message) only or can they be stateful (i.e. window-ing and stream-stream/table join functionality)
(3) I could not find any reference to the WASM inline lambdas at all in the docs actually, am I missing something?
r_1: https://arrow.apache.org/faq/