Apparently, we don't have enough wasted cycles in modern software, just add a bunch more in a fundamental IPC infrastructure.

There are a lot of problems with D-Bus, but that it wasn't using JSON wasn't one of them.

The tooling for D-Bus is terrible. It has composite types but they are very primitive, e.g. you can't name fields in a structure without extensions (there's like three different extensions for this.) A lot of the code generators can't actually handle real-world schemas.

Now I understand that D-Bus itself is also a problem (as in, the protocol design beyond serialization) but varlink looks like a very solid step backwards. In any single individual use case JSON serialization is unlikely to be a huge bottleneck, but in practice if this will be used for all kinds of things it will add up.

I really wish the Linux desktop would coalesce around something like Cap'n'proto for serialization.

P.S.: It'd probably be possible to shoehorn in multiple formats but I think that's a mistake too. Something like capnp will have a strong impedance mismatch with JSON; mixing formats with different degrees of self-describability is unwise. But then formats like MsgPack and BSON which are direct JSON analogues miss out on the best benefits of using binary serialization formats, which makes them almost the worst of both worlds, and you'd probably be better off with using a highly-optimized JSON library like simdjson.

I've suggested it before and the systemd folks didn't seem completely opposed to it. Also because cbor parser is already in the dependencies due to FIDO2 disk unlocking

It gives you a lot of decent protocol wire design and then flatly ignores everything we've learned about these types of designs in the last 3 decades. Be on the lookout for client libraries to slowly add in all of these checks as the vulnerabilities are discovered in them.

[1]: https://seriot.ch/projects/parsing_json.html

CBOR is not. No full message length. Only primitive type values. Compound type values require length _in items_ (not bytes) but indefinite encoding is possible if allowed ("deterministic" constraint to message forbids it).

So technically this doesnʼt differ from e.g. JSON where you have to wait until final "}" or "]" for a long compound item, and have to install an artificial limit.

> Add in an "indefinite length" option and you've got potentially unbounded client memory usage to watch out for. > As if that wasn't enough you get extensible tags so the meaning of any message is entirely dependent on the context it was sent in.

So does any such a protocol. JSON, XML, any ASN.1, CBOR, whatever. If there are no extensible tags, there are field names - or they will be added by customer using what is allowed. Iʼve seen this in ASN.1 sequence-of pairs of name+value, directly emulating JSON-like dictionary. To limit it is not how security issues are handled.

I was thinking about this more and I think FTP actually had a lot of the right ideas here. You want two channels. One for fast interactive command messaging and a second one coordinate especially just for bulk transfers arranged over the command channel. The design flaw in FTP of putting these on two separate ports put it immediately in conflict with firewall best practices so I think it went mostly unnoticed that it fundamentally is a good arrangement.

What you want is one channel, sequenced packets, with a defined maximum message length that is negotiated at startup and never changes during the life of the channel. This should probably never be more than 65k. There should be a known length packet type, and an unknown length packet type, with any attempt to send more than the negotiated maximum triggering an error and disconnect.

If you do need to send more than the negotiated amount you should open a new connection, in a bulk transfer mode, that after the initial handshake, has no protocol and is merely an associated stream of bytes that are wholly uninterpreted by the middleware layer other than to help you associate it with the sequenced packet that requested its initiation.

You'd actually be able to use TCP (with DSCP even), mostly avoid head of line blocking and multiplexer latencies, and have a reasonable security guarantee in the sequenced packet mode, and never have a protocol which pretends that 4GB strings in the middle of a packet are necessary or even a good idea to "support."

The downfall of this is that it would be much harder to implement it on a serverless architecture and would be nearly as complicated as a protocol as WebSocket ends up being. It might be worth playing with as a concept anyways.

You would need to tightly control and multiplex the messages yourself, needing to do this in one connection or using something like TCP priority flag.

Personally I just think that TCP is a shitty protocol for building applications. In almost every use case either UDP or SCTP are a better choice. With raw UDP datagrams you aren’t guaranteed delivery but it’s great for the kind of telemetry where the last message in is what matters. You can also build quite flexible stuff on top of it.

SCTP gives you in sequence reliable datagrams and congestion control. This means you don’t have to devise message length communication into your application layer protocol and can rely on your transport. It also has multiplexing built right in. If it had a built in checksum it would truly be ideal. Not sure if something like secure communication really belongs at this level but if it had that I double we would ever use anything else. From SCTP’s Wikipedia page:

> SCTP applications submit data for transmission in messages (groups of bytes) to the SCTP transport layer. SCTP places messages and control information into separate chunks (data chunks and control chunks), each identified by a chunk header. The protocol can fragment a message into multiple data chunks, but each data chunk contains data from only one user message. SCTP bundles the chunks into SCTP packets. The SCTP packet, which is submitted to the Internet Protocol, consists of a packet header, SCTP control chunks (when necessary), followed by SCTP data chunks (when available).

> It also has multiplexing built right in.

It is not. You canʼt stop, for example, receiving from stream 0 during getting high-priority data from stream 9. No userlevel API allows to specify "now receive a message exactly from stream 9". Data packet TSNs are the same number sequence for all streams. You canʼt normally ignore a single data TSN for a stream payload: you have to issue SACK and brook constant retransmissions from other side. You canʼt specify receive window separately for each stream, to calm the sender down while the streamʼs receive buffer is full.

I donʼt know who and why spreads the myth it is really multiplexing, but now these stream numbers are merely another type of per-message external tag. All other is merely scam. It seems that initial design was moving toward this possibility but then something unfortunate happened.

> This means you don’t have to devise message length communication into your application layer protocol and can rely on your transport.

This is, well, tasty from it.

For a localhost/UNIX domain protocol, it might work since most of these considerations are significantly reduced.

As to DSCP, it might be useful across controlled networks but in general, in my experience it's not really ever honored broadly.

Back on the original topic, when I was building this for my Linux distribution I ended up just using basic JSON with TLS (for client certificate authentication) though since having authentication for remote management was a goal, and once I was already having to perform a TLS negotiation then the PDU consideration for performance wasn't something to really spend too much time on.

how?

man 3 cmsg

For a glimpse into the darker corners of hell. At least you get to grip it there. In GCed languages you have to hope the library designer gave you reasonable controls for timeouts, aborting large data transfers, and releasing buffers efficiently in a bursty network environment.

See all the controls any HTTP server gives you for this. "Maximum header size", "Maximum header(s) size", "Maximum Body Size", "Request Header Timeout", "Total Request Timeout."

None of those were added by default or by tasteful library authors. They were all added to answer to specific emergent security vulnerabilities that were discovered and then were demanded by the users.

Which fraction of CPU will JSON ser/des will take to justify using CBOR?

So I'd argue it's not an unreasonable question (although I lean closer in: better types then JSON is the problem, since a decent text serialization format is always going to be needed for debugging and development).

Any binary format can be dumped to a text when viewed for debugging. That’s really a nonissue. Have the tools dump to JSON if you like.

    [1.2,1.3,1.4]

OTOH this is pretty cherry picked. It is questionable if you really need floating point numbers if your numbers are all this small and low precision, but in actuality unless this comprises your entire message, you'll probably still lose to JSON overhead eventually anyways, whereas non-self-describing serialization like capnp has effectively no overhead by default. The advantage of encoding as doubles is that it is predictable and won't explode into taking massively more, its always the same size per number. If you want it to be smaller for some reason, compression is always an option, though I suspect for IPC it's the wrong thing to be concerned about.

Hexadecimal, if so. Nearly any decent modern language and runtime can it.

    $ python
    >>> import math
    >>> math.pi.hex()
    '0x1.921fb54442d18p+1'

(I donʼt count decimal floating-point as it is quite rare and its application is faint. In overwhelming most of cases, fixed point is better.)

Variable length binary number encodings are also an option.

The dynamic array overhead in a language is immaterial when discussion serialization formats.

Most task done over it seems to either fall into the "from time to time but not really performance sensitive at all" category or the "a mass of system events which can become huge" category.

> miss out on the best benefits the best benefits of using binary serialization format

I disagree (wrt. MsgPack), the biggest benefit is of binary formats is that you can ship binary blobs without any encoding (mainly space) overhead while for JSON this can easily be ~33%+ space overhead. And even for just e.g. numbers the space overhead can be bad. I'm not sure what you see as the biggest benefit but not having a minimalist structure isn't a property specific to binary formats (through more common there, and MsgPack doesn't require field names to be included). And bit fiddling micro space optimizations aren't that grate/as much a benefit as drawback. And as a system bus handles (emits) potentially a _ton_ of events saving 33% can matter I think.

What does matter though are roundtrips. In Varlink there are much fewer required for typical ops than there are in D-Bus. That's because D-Bus implies a broker (which doubles the number of roundtrips), but also because D-Bus forces you into a model of sending smaller "summary" messages when enumerating plus querying "details" for each listed objects, because it enforces transfer rate limits on everything (if you hit them, you are kicked off the bus), which means you have to refrain from streaming too large data.

Or in other words: marshalling is quite an irrelevant minor detail when it comes to performance, you must look at roundtrips instead and the context switches it effects, instead.

Using JSON for this has two major benefits: the whole world speaks JSON, and modern programming languages typically pretty natively. And it's directly readable in tools such as strace. With a simple "strace" I can now reasonably trace my programs, which a binary serialization will never allow you. And if you tell me that that doesn't matter, then you apparently live in an entirely different world than I do, because in mine debuggability does matter. A lot. Probably more than most other things.

Lennart

This is from someone who initially seemed to have a very similar perspective to you.

See the JSON section of the post: https://raphlinus.github.io/xi/2020/06/27/xi-retrospective.h...

IMO CBOR would be a better choice, you aren't limited to IEEE 754 floats for your numeric types. Yeah, some (de/en)coders can handle integer types, but many won't, it's strictly out of spec. I don't think building something as fundamental to an OS as relying on out-of-spec behavior is a great idea. It will result in confusion and many wasted hours sooner or later.

The other side of this coin, of course, is that now you have to support those other numeric types :) My usual languages of choice somehow don't support "negative integers in the range -2^64..-1 inclusive".

Also, there have clearly also been several attempts over the years to make a faster D-Bus implementation (kdbus, BUS1), which were never accepted into the kernel. It makes a lot of sense to instead design a simpler protocol.

There is clearly also a cautionary take about how microbenchmarks (here, for serialisation) can mask systemic flaws (lots of context switches with D-Bus, especially once polkit had to be involved).

A 100 digit number cannot be encoded losslessly in D-Bus btw, nor in the far majority of IPC marshallings on this word.

Having done systems-level OS development since 25y or so I never felt the burning urge to send a 100 digit number over local IPC.

Not that 100 digit numbers aren't useful, even in IPC, but typically, that's a cryptography thing, and they generally use their own serializations anyway.

Actually, not new. Earliest CVE I found was from 2017, which feels a decade later than it should be. I guess no one thought of pushing JSON over trusted interfaces, and probably for good reason.

I think the concern is that large numbers can in fact be encoded in JSON, but there is no guarantee that they will be decoded correctly by a receiver as the format is underspecified. So you have to cater for the ill defined common denominator.

I know you know JSON is nominally only 53-bit safe, because JS numbers are doubles. But in practice I'd wager most JSON libraries can handle 64-bit integers.

It would add a little complexity to the server, but then clients can choose if they want to use a human readable format that has more available libraries or a binary format.

As for strace, tooling could probably be added to automatically decode cbor to json, either as part of strace, or in a wrapper.

There could also be a varlink proxy (similar to varlink bridge) that could log or otherwise capture requests in a human readable format.

Doesn't systemd use binary logging?

(Yes, we then store the structure log data on disk in a binary format. Lookup indexes are just nasty in text based formats).

Hence, not sure what the Journal has to do with Varlink, but any IPC that the journal does is text-based, and very nicely strace'able in fact, I do that all the time.

[Maybe, when trying to be a smartass, try to be "smart", and not just an "ass"?]

Although I use strace all the time, I hardly ever look at the payload of read and write calls, although I could see why it would be useful. But given a binary protocol it wouldn't be terribly hard to build a tool that parses the output of strace.

> [Maybe, when trying to be a smartass, try to be "smart", and not just an "ass"?]

thanks for the kind words and elevating the tone of the discussion.

Nah, it'd be a rage-inducing nightmare.

Doesn't strace already come with desetializers for many common data structures?

I’ve worked with profiling code where the marshaling cost for JSON was the biggest cost. Namely it involved a heap allocation and copying a ton more data than was actually needed, and I ended up fixing it by turning the JSON into a static string and dropping the values in manually.

The systemd maintainers have probably done their due diligence and concluded that it isn’t an issue for their forseeable use cases, but it does lock everything in to doing string processing when interfacing with systemd, which is probably unnecessary. And you can’t trivially swap systemd out for something else.

systemd is so pervasive that it would be fine to add a binary-format-to-JSON translation ability into strace. That shifts the cost of debugging to the debug tools, rather than slowing down production code.

Doing any string processing tends to require a lot of branching, and branch mispredictions are most likely to slow down code. It also turns every 1-cyle load/store instruction into N-cycles.

String processing in C, which is what systemd and a lot of system tools are written, is pretty abysmal.

systemd is also non-optional, so if it turns out that it’s causing cache thrashing by dint of something generating a lot of small events, it’s not something you can do something about without digging into the details of your lowlevel system software or getting rid of systemd.

And it’s potentially just that much more waste on old or low-power hardware. Sure, it’s probably “negligible”, but the effort required to do anything more efficient is probably trivial compared to the aggregate cost.

And yeah, it may be better than D-Bus, but “it’s not as bad as the thing that it replaced” is pretty much the bare minimum expectation for such a change. I mean, if you’re swapping out things for something that’s even worse, what are you even doing?

I see there’s a TCP sidechannel, but why increase the complexity of the overall system by having two different channels when you could use one?

Dunno. This isn’t really an area that I work in, so I can’t say for sure it was the wrong decision, but the arguments I hear being made for it don’t seem great. For something fundamental like systemd, I’d expect it to use a serialization format that prioritizes being efficient and strongly-typed with minimal dependencies, rather than interoperability within the application layer with weakly-typed interpreted languages. This feels like a case of people choosing something they’re more personally familiar with than what’s actually optimal (and again, the reason I’d consider being optimal in this case being worth it is because this is a mandatory part of so many devices).

EDIT: Also, the reason that binary serialization is more efficient is because it’s simpler - for machines. JSON looks simpler to humans, but it’s actually a lot more complex under the hood, and for something fundamental having something simple tends to be better. Just because there’s an RFC out there that answers every question you could possibly have about JSON still doesn’t mean it’s as good as something for which the spec is much, much smaller.

JSON’s deceptive simplicity also results in people trying to handroll their own parsing or serialization, which then breaks in edge cases or doesn’t quite 100% follow the spec.

And Just because you’re using JSON doesn’t force C/++ developers to validate it, someone can still use an atoi() on an incoming string because “we only need one thing and it avoids pulling in an extra dependency for a proper json parser”, then breaks when a subsequent version of systemd changes the message. Etc. If the goal is to avoid memory safety issues in C/++, using more strings is not the answer.

It's a command that prints the system calls (done via library). So you see write(55, "[1,2,3,4]\n")

  write(55, "[1,2,3,4]\n", 10)

  {
    "syscall": "write",
    "params": {
      "fd": 55,
      "buf": "[1,2,3,4]\n",
      "len": 10
    }
  }

It can be parsed by any modern language, and deserialization is cheap.

Can we please rewrite strace with this in mind? Preferably in Rust.

You'd do that because everything under the sun knows how to generate and parse json. Performance looks like a total non-issue for this use case. The messages are tiny, local and infrequent. You wouldn't want to do this on some kind of a high performance application on the data plane, but on the control plane it's totally fine.

Even if you expect that all production use cases will use some kind of higher level library that does schema validation, it can still be quite nice during debugging to a) be able to inspect the messages on the wire when debugging, b) to be able to hand-write and inject messages.

The presentation brings up performance as one of the problems with the D-Bus though, so it seems like it is an issue.

it's hard to say, for most D-Bus taks perf. is an absolute non issues, but still you have bus-1 and all kind of tries to make D-Bus perform faster, reason is there are edge cases where it matters

and even if the marshaling speed might not matter the size can, like if something goes really wrong and tens of thousand of system events get spammed the difference between the memory usage of in-flight messages of a compact format vs. json can make the difference between memory pressure effectively killing your server and you recovering reliable

though what is an open question is how relevant that is in pracive

and that is quite hard to say without inside information for companies running systems where such a thing could happen on a scale where such a thing does happen (e.g. google)

https://raphlinus.github.io/xi/2020/06/27/xi-retrospective.h...

Given how central this is going to be why would you skip the tiny bit of extra work needed to use a binary format?

Even if they eventually come to the conclusion they need something more performant in the future, it's probably still a net win to make the switch today. It's also possible it turns out fine for their use case.

I personally would think they would want to benefit from an IPC system that includes provisions for sending file descriptors.

from the guy who brought you binary logfiles!

Per https://systemd.io/JOURNAL_FILE_FORMAT/, the benefits of the binary format are:

The systemd journal stores log data in a binary format with several features:

Fully indexed by all fields

Can store binary data, up to 2^64-1 in size

Seekable

Primarily append-based, hence robust to corruption

Support for in-line compression

Support for in-line Forward Secure Sealing

As a user system-administrator, I see the cryptographic checksum as a benefit of being able to show tampering evidence of on-system log files.

It's so robust, it doesn't even let you modify the journal if you want to (e.g. https://github.com/systemd/systemd/issues/20673).

> Support for in-line compression

Mind that journald only supports compressing single lines, but not the whole journal (https://github.com/systemd/systemd/issues/31358), which is pretty limiting.

In what situations is it a harder problem than this?

It's a tradeoff, if you do full compression clearly it won't be fast.

You're free to compress it again before archiving it.

Text logs are append-based and seekable as well.

> Support for in-line Forward Secure Sealing

I once typed an SSH password in the username field, and the only way to erase that was to erase all the logs. So this has some significant downsides.

Also, I am tired of waiting minutes for a journalctl query to load.

I hope this was a personal system. Changing logs in this manner would have almost certainly led to your dismissal anywhere I ever worked. This anecdote just re-enforces the need for Forward Secure Sealing.

Varlink is designed for IPC, so the expected data rate is maybe hundreds of messages per second in the worst case.

JSON can be parsed at 3 Gigabytes/second [0]. Even unoptimized stdlib parsers in scripting languages get 50 MB/sec.

That's more than enough, standard library support is much more important for a project like this.

And if there is ever a reason to send tar archive or video data, there is always "upgrade" option to switch to raw TCP sockets, with no JSON overhead at all.

[0] https://news.ycombinator.com/item?id=19214387

[1] https://github.com/TeskaLabs/cysimdjson

Of all the things they could do, they decided to choose one of the worst possible formats for IPC data[1], maybe to pander to the web crowd.

[1] Maybe XML would be a close competitor.

I'm begging you to read the format it's replacing. https://dbus.freedesktop.org/doc/dbus-specification.html Ctrl+f Type System. JSON is beautiful, elegant, and minimal by comparison. yyyyuua(yv) It's like the printf message protocol but worse.

it isn't, fundamentally can't be as it has no support for binary blobs

and base64 encoded strings are a pretty terrible solution for this

and the space overhead of many small number can also be pretty bad

and while a lot of IPC use cases are so little performance sensitive that JSON is more then good enough and large blobs are normally not send over messages (instead you e.g. send a fd) there are still some use-cases where you use IPC to listen to system events and you have a situation where a pretty insane amount of them is emitted, in which case JSON might come back and bite you (less due to marshaling performance and more due to the space overhead which depending on the kind of events can easily be 33%+).

But what I do not know is how relevant such edge cases are.

Probably the biggest issue might be IPC system mainly used on 64bit systems not being able to properly represent all 64 bit integers .... but probably that is fine too.

For example, the metadata of the blob could be returned via JSON, then the request is “upgraded” to a pure binary pipe and the results read as-is.

binary blobs is just the biggest example and was only mentioned in relation to the "lingua franca" argument, many other common things are also larger in JSON. Only if you have many larger not escaped utf-8 strings does this overheads amortize. E.g. uuids are something not uncommonly send around and it's 17 bytes in msgpack as a bin value and 38 bytes in json (not inlcuding `:` and `,` ). That 211% the storage cost. Multiply it with something going on and producing endless amounts of events (e.g. some unmount/mount loop) and that difference can matter.

Through yes for most use cases this will never matter.

Yes, there’s a more optimal format than hex-encoding UUID values. However it simply does not matter for any use case this targets.

16 bytes vs 38 bytes is completely meaningless in the context of a local process sending a request to a local daemon. It’s meaningless when making a HTTP request as well, unfortunately.

I’d have loved Arrow to be the format chosen, but that’s not lowering the barrier to entry much.

sure most people don't write code for such edge cases, like ever

but e.g. systemd does sometimes because they reliable appear when running things at e.g. googles scale

True, but binary formats like CBOR or MessagePack are also that and efficient and have support for binary strings. This JSON is human readable thing is the nerverending argument of high level programmers. Anyone can read hex given enough time.

Peep the Python interface parser: https://github.com/varlink/python/blob/master/varlink/scanne.... This is the quality you can expect for an ad-hoc format, which is to say, pretty bad. Making the hardest part farming out to a parsing library you already have and is a built-in in your language is a no-brainer choice.

In it OP says it’s not slow, and the “hundreds of messages per second” you are referring to is not regarding the performance but the expected rate.

It’s possible for something to be fast, but only happen infrequently.

[0]: https://www.youtube.com/watch?v=emaVH2oWs2Y&list=PLWYdJViL9E...

Debugability is important but the answer is to build debugging tools, not to lobotomise the protocol for the vanishingly tiny fraction of packets that are ultimately subject to debugging.

Christ almighty, this is a terrible proposal.

I’ll go even stronger than that: IPC and RPC should prefer plaintext-representable forms by default and only abandon them for binary once the real world costs of the textually-representable protocol are found to be unacceptable and unmitigateable.

The benefit of being able to use pre-existing introspection tools that were not designed with your protocol in mind—-and use those tools without extensive configuration—-is huge.

I think the existence of bad textual formats (e.g. JSON) and the presence of largely-orthogonal-to-binaryness useful affordances in popular binary formats (e.g. schema validation and strong typing in Protobuf) muddies the underlying truth: textual-representability-by-default is rarely costly and often a huge boon to protocol implementors and protocol debuggers.

https://news.ycombinator.com/item?id=41694711

BigInt is a new concept and not technically supported. So whether it works in your random library of choice is probably a potshoot. "Use within JSON: Using JSON.stringify() with any BigInt value will raise a TypeError, as BigInt values aren't serialized in JSON by default. " https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe...

(which expands on the slightly fluffier text in https://datatracker.ietf.org/doc/html/rfc7159#section-6 which broadly says the same thing... and https://datatracker.ietf.org/doc/html/rfc8259#section-6)

If you look at a parser matrix, flouting this rule is asking for trouble. Which is why the recommendations on MDN for bigint recommend an explicit parsing from a string as needed.

(I'm genuinely curious why this is being downvoted. Does someone out there feel so strongly that because an arbitrary string of characters technically can be parsed as any infinitely large number, that it's wise to ignore RFCs and dozens of existing implementations when making use of JSON as a common interchange format?)

JSON recommends (for interoperability) that numbers be limited to those representable as IEEE 64-bit binary floats-but does not require that. A document which ignores that recommendation is still officially valid JSON. By contrast, I-JSON, as a subset of JSON, upgrades that recommendation to a requirement, so documents which ignore it are not valid I-JSON

The others do not forbid larger numbers but note that doing that will make for bad times, and if you look at JSON parsing matrices (say, the "Parsing JSON is a Minefield" one), you can see that the majority of implementations do not do allow larger. So. float64 is a defacto standard and I-JSON is simply clarifying this. Given the main purpose of JSON is an interoperable data exchange format it would be a very bad idea to do otherwise.

https://en.wikipedia.org/wiki/JSON?useskin=vector#Interopera... (essentially same point made here)

And why should Qt be concerned with what jq can or can't do? If you're writing and reading back from Qt, the limitations of other implementations don't apply.

The whole point of JSON is almost every programming language can read and write it - and if you want this to be the case, stringify anything unusual, like large integers.

That urge that forces every developer to reinvent RPC every other week.

Binder solves a real-world problem — priority inversion. It is also the only IPC on Linux that allows to perform a blocking call from process A to process B (and have process B synchronously call back to process A if needed).

D-Bus, Varlink and other "IPC solutions" on top of sockets can not do those things. Android uses synchronous Binder calls for 90% of it's application APIs, so there is clearly a real use case for it.

Incidentally, Binder itself does not specify message format. A message is a simple memory buffer. Android runtime uses a custom binary serialization format, but you can use JSON or whatever. Binder is an alternative to sockets/pipes, not D-Bus or other wrappers on top of them.

Binder also specifies a message format -- even if not fully, since the kernel is going to peek into your message for things like FDs, binder objects, etc. Or your userspace is going to need to "special treat" these fields somehow.

It competes in the same space as D-Bus no doubt. If the story of Android had been different, with the companies spinning off some years later, it may have very well ended up using Luna RPC (https://www.webosbrew.org/pages/luna-service-bus.html ) (which is practically indistinguishable from D-Bus, bar the JSON) or anything else, really.

> D-Bus, Varlink and other "IPC solutions" on top of sockets can not do those things.

WTF? Sure you can do on top of even pipes. Even XDR could...

"Synchronous" is a very loose word here if you mean to be interrupted in the middle of a RPC call, anyway.

Of course, you "can". Implement message-based communication on top of streaming. Emulate blocking calls on top of non-blocking socket API. Implement authentication via ancillary data (try not to throw up in process). Use some clever tricks to solve priority inversion. Somehow distinguish your fds from all others file descriptors to ensure that no confused deputy problems arise.

Congratulations! You have reached feature parity with Binder.

> Binder was not even designed for Linux.

Neither are Berkeley sockets.

What is your point here? You claimed that "other IPC solutions" _could not_ do this on top of sockets, but as I've shown and now you admit, you definitely _can_. Obviously I'm not suggesting you roll this by hand, nor claiming that a plain UNIX pipe is a fully featured RPC system; just that there's a million RPC systems out there that do it and don't need anything else than sockets.

Then again, Dbus is not really what I would call an incredible piece of software - we are very much in the adequate territory if even - so anything shaking up the status quo can’t be too bad.

I don't think the serialization/deserialisation overhead matters much when also taking into account the bus connection overhead. I would've preferred a better data exchange format but out of all problems I have with Linux IPC, the performance of JSON is the least of my worries.

First of all, in systemd, which is a heavy D-Bus user, we effectively IRL only send integers > 2^53 when we use UINT64_MAX as a special "niche" marker meaning "unlimited"/"unset"/"undefined". But the thing is that JSON has a proper concept for this: the "null" value (or simply not specifying a specific JSON field). Hence, in reasonably clean APIs this is mostly a non-issue.

That said, sd-varlink/sd-json (i.e. systemd's implementation of it), of course is 64bit signed and unsigned integer clean when it processes JSON. More-over it automatically handles if you do what the various specs on the internet suggest you do if you have an integer > 2^53: you encode it as decimal value as a string.

Would it be better if JSON would have been more precise on this, yes. Is it a big issue? No, not at all.

Hexadecimal is better - more compact, much less encoding/decoding effort. Alternatively, array of two integers does as well.

Text forms for any error, not NaN, are also useful.

But issues with non-UTF-8 encodings - as in CJK world - may spoil the picture pretty more.

It's a small format, it has limited pitfalls, it's fast to marshal and unmarshal (and with simd gets even more fast), it's deflate-friendly, would performance be an issue they can pack that through CBOR or msgpack or what-have-you.

JSON is so prevalent I'm pretty sure it's a lot of people's mental model of what non relational structured data looks like, if it's not just a file.

It's nice from a dev perspective to always work directly with the same high level concepts that you use when thinking about this stuff.

Other data formats are often advertised in terms of JSON, either subsets or supersets, or equivalent representations.

I'd use something else for high performance stuff.... But for everything else it's pretty awesome.

As for JSON, please don't even start me on that. It is one of the worst serialisation decisions we ever made as a society. Poorly defined and unreliable primitive types, terrible schema support and expensive to parse. In a kernel, it does not belong! Give it a few months and varlink will be YAML over the top of that.

D-Bus is just a bus so it can solve a bunch of different IPC problems at the same time. e.g. D-Bus can handle ACLs/permissions on RPC methods, D-Bus can handle multiple producers and multiple consumers, and so on. I think ultimately all that's really needed is IPC, and having a unified bus is just to allow for some additional use cases that are harder if you're only using UNIX domain sockets directly.

If there are going to be systemd components that have IPC, then I'd argue they should probably use something standard rather than something bespoke. It's good to not re-invent the wheel.

Not that I think Varlink is any better. It seems like at best it's probably a lateral move. I hope this does not go forward.

This is my point.

My favourite DBus situation a number of years ago was a CentOS 7 box that reboot stopped working on with a cryptic DBus error that no one has ever seen before. I had to sync it, power cycle the node from the ILO card and cross my fingers.

I really don't give a shit about this. I just wanted to run my jobs on the node, not turn into a sysadmin due to someone else's dubious architectural decisions.

It would be quite straightforward to map a capability-like UNIX socket into each service’s filesystem and give it a private view of the world. But instead…

> Public varlink interfaces are registered system-wide by their well-known address, by default /run/org.varlink.resolver. The resolver translates a given varlink interface to the service address which provides this interface.

…we have well known names, and sandboxing, or replacing a service for just one client, remains a mess. Sigh.

And OK fine. It's not that bad, most C ABIs are able to work around this reasonably OK (not all of them but sure, let's just call it a skill issue.) But then what do you do when you want anything more complicated than a completely fixed-size type? Like for example... a string. Or an array. Now we can't just use a struct, a single request will need to be split into multiple different structures at the bare minimum.

And plus, there's no real reason to limit this all to the same machine. Tunneling UNIX domain sockets over the network is perfectly reasonable behavior and most* SSH implementations these days support this. So I think scoping the interoperability to "same machine" is unnecessarily limiting, especially when it's not actually hard to write consistent de/serialization in any language.

* At least the ones I can think of, like OpenSSH[1], Go's x/crypto/ssh[2], and libssh2[3].

[1]: https://www.openssh.com/txt/release-6.7

[2]: https://pkg.go.dev/golang.org/x/crypto/ssh#Client.ListenUnix

[3]: https://github.com/libssh2/libssh2/pull/945

In Lustre RPC _control_ messages go over one channel and they're all a C structure(s) with sender encoding hints so the receiver can make it right, and any variable-length payloads go in separate chunks trailing the C structures.

Whereas bulk _data_ is done with RDMA, and there's no C structures in sight for that.

Capnp sounds about right for encoding rules. The way I'd do it:

  - target 64-bit architectures
    (32-bit senders have to do work
     to encode, but 64-bit senders
     don't)
  - assume C-style struct packing
    rules in the host language
    (but not #pragma packed)
  - use an arena allocator
  - transmit {archflags, base pointer, data}
  - receiver makes right:
     - swab if necessary
     - fix interior pointers
     - fail if there are pointers
       to anything outside the
       received data
     - convert to 32-bit if the
       receiver is 32-bit

As for syntax, I'd build an "ASN.2" that has a syntax that's parseable with LALR(1), dammit, and which is more like what today's devs are used to, but which is otherwise 100% equivalent to ASN.1.

Having actual interior pointers means not having to deal with pointers as offsets when using these objects. Now the programming language could hide those details, but that means knowing or keeping track of the root object whenever traversing those interior pointers, which could be annoying, or else encoding an offset to the root and an offset to the pointed-to-item, which would be ok, and then the programming language can totally hide the fact that interior pointers are offset pairs.

I've a feeling that fixing up pointers is the more interoperable approach, but it's true that it does more memory writes. In any case all interior pointers have to be validated on receiving -- I don't see how to avoid that (bummer).

What a fun sub-thread.

You are adding more problems that don't exist to the specification.

As for strings, just shove a char[4096] in there. Use a bit of memory to save a lot of parsing.

D-Bus does in fact already have support for remoting, and like I said, you can tunnel it today. I'm only suggesting it because I have in fact tunneled D-Bus over the network to call into systemd specifically, already!

> As for strings, just shove a char[4096] in there. Use a bit of memory to save a lot of parsing.

OK. So... waste an entire page of memory for each string. And then we avoid all of that parsing, but the resulting code is horribly error-prone. And then it still doesn't work if you actually want really large strings, and it also doesn't do much to answer arrays of other things like structures.

Can you maybe see why this is compelling to virtually nobody?

For the love of God, use a proper slice/fat pointer, please.

Switching over to slices eliminates 90%+ of the issues with using C. Carrying around the base and the length eliminates a huge number of the overrun issues (especially if you don't store them consecutively).

Splitting the base and the offset gives a huge amount of semantic information and makes serialization vastly easier.

Correct as you may be though the argument here is that you should just write raw structs into Unix sockets. In this case you can't really use pointers. So, realistically, no slices either. In this context a fixed-size buffer is quite literally the only sensible thing you can do, but also, I think it's a great demonstration of why you absolutely shouldn't do this.

That said, if we're willing to get rid of the constraint of using only one plain C struct, you could use offsets instead of pointers. Allocate some contiguous chunk of memory for your entire request, place struct/strings/etc. in it, and use relative offsets. Then on the receiver side you just need some fairly basic validation checks to ensure none of the offsets go out of bounds. But at that point, you've basically invented part of Cap'n'proto, which begs the question... Why not just use something like that instead. It's pretty much the entire reason they were invented.

Oh well. Unfortunately the unforced errors of D-Bus seem like they will lead to an overcorrection in the other direction, turning the core of our operating system into something that I suspect nobody will love in the long term.

The only problem I have with Cap'n Proto is that the description is external to the serialization. Ideally I'd like the binary format to have a small description of what it is at the message head so that people can process messages from future versions.

ie. Something like: "Hmm, I recognize your MSG1 hash/UUID/descriptor so I can do a fast path and just map you directly into memory and grab field FD. Erm, I don't recognize MSG2, so I need to read the description and figure out if it even has field FD and then where FD is in the message."

It's better to design APIs to be extensible in ways that doesn't require dynamic introspection. It's always possible to have a "generic" header message that contains a more specific message inside of it, so that some consumers of an API can operate on messages even when they contain some data that they don't understand, but I think this still warrants some care to make sure it's definitely the right API design. Maybe in the future you'll come to the conclusion it would actually be better if consumers don't even try to process things they're not aware of as the semantics they implement may some day be wrong for a new type of message.

Versioning, at the least, is extremely difficult without this.

Look at the Vulkan API for an example of what they have to do in C to manage this. They have both an sType tag and a pNext extension pointer in order for past APIs to be able to consume future versions.

In general there's very little you can really do with a dynamic schema. Perhaps you can convert the message to JSON or something. But if your code doesn't understand the schema it received, then it can't possibly understand what the fields mean...

Because traditional POSIX IPC mechanisms are absolute unworkable dogshit.

> It is one of the worst serialisation decisions we ever made as a society.

There isn't really any alternative. It's either JSON or "JSON but in binary". (Like CBOR.) Anything else is not interoperable.

- Whatever: people in more niche languages are pretty used to needing to do FFI for things like this anyhow.

- Many of them already have a better ecosystem than D-Bus. e.g. interoperability between Protobuf and Cap'n'proto implementations is good. Protobuf in most (all?) runtimes supports dynamically reading a schema and parsing binary wire format with it, as well as code generation. You can also maintain backwards compatibility in these formats by following relatively simple rules that can be statically-enforced.

- JSON and JSON-but-binary have some annoying downsides. I really don't think field names of composite types belong as part of the ABI. JSON-like formats also often have to try to deal with the fact that JSON doesn't strictly define all semantics. Some of them differ from JSON is subtle ways, so supporting both JSON and sorta-JSON can lead to nasty side-effects.

Maybe most importantly, since we're not writing software that's speaking to web browsers, JSON isn't even particularly convenient to begin with. A lot of the software will be in C and Rust most likely. It helps a bit for scripting languages like Python, but I'm not convinced it's worth the downsides.

I pray to God nothing meaningful is actually doing what you are insinuating in any serious environment.

As for JSON, are you really that short sighted that it's the only method of encoding something? Is "oh well it doesn't fit the primitive types, so just shove it in a string and add another layer of parsing" acceptable? Hell no.

Edit: hell even XML is better than this!

Asn.1 is both quite complicated and not very efficient.

They could certainly have gone with protobufs or another binary serialisation format but would it really be better than the current choice?

I don’t think the issue they are trying to solve is related to serialisation anyway. Seems to me they are unhappy about the bus part not the message format part.

ASN.1 is not that complicated. Pity that fools who thought ASN.1 was complicated re-invented the wheel quite poorly (Protocol Buffers I'm looking at you).

That's very wrong. ASN.1 is complicated because it's quite complete by comparison to other syntaxes, but it's absolutely not inefficient unless you mean BER/DER/CER, but those are just _some_ of the encoding rules available for use with ASN.1.

To give just one example of "complicated", ASN.1 lets you specify default values for optional members (fields) of SEQUENCEs and SETs (structures), whereas Protocol Buffers and XDR (to give some examples) only let you specify optional fields but not default values.

Another example of "complicated" is that ASN.1 has extensibility rules because the whole "oh TLV encodings are inherently extensible" thing turned out to be a Bad Idea (tm) when people decided that TLV encodings were unnecessarily inefficient (true!) so they designed efficient, non-TLV encodings. Well guess what: Protocol Buffers suffers from extensibility issues that ASN.1 does not, and that is a serious problem.

Basically, with a subset of ASN.1 you can do everything that you can do with MSFT RPC's IDL, with XDR, with Protocol Buffers, etc. But if you stick to a simple subset of ASN.1, or to any of those other IDLs, then you end up having to write _normative_ natural language text (typically English) in specifications to cover all the things not stated in the IDL part of the spec. The problem with that is that it's easy to miss things or get them wrong, or to be ambiguous. ASN.1 in its full flower of "complexity" (all of X.680 plus all of X.681, X.682, and X.683) lets you express much more of your protocols in a _formal_ language.

I maintain an ASN.1 compiler. I've implemented parts of X.681, X.682, and X.683 so that I could have the compiler generate code for the sorts of typed holes you see in PKI -all the extensions, all the SANs, and so on- so that the programmer can do much less of the work of having to invoke a codec for each of those extensions.

A lot of the complexity in ASN.1 is optional, but it's very much worth at least knowing about it. Certainly it's worth not repeating mistakes of the past. Protocol Buffers is infuriating. Not only is PB a TLV encoding (why? probably because "extensibility is easy with TLV!!1!, but that's not quite true), but the IDL requires manual assignment of tag values, which makes uses of the PB IDL very ugly. ASN.1 originally also had the manual assignment of tags problem, but eventually ASN.1 was extended to not require that anymore.

Cavalier attitudes like "ASN.1 is too complicated" lead to bad results.

So, it's quite complicated. Yes, what I have been saying from the start. If you start the conversation by "you can define a small subset of this terrible piece of technology which is bearable", it's going to be hard convincing people it's a good idea.

> Cavalier attitudes like "ASN.1 is too complicated" lead to bad results.

I merely say quite complicated not too complicated.

Still, ASN.1 is a telco protocol through and through. It shows everywhere: syntax, tooling. Honestly, I don't see any point in using it unless it's required by law or by contract (I had to, I will never again).

> but it's absolutely not inefficient unless you mean BER/DER/CER, but those are just _some_ of the encoding rules available for use with ASN.1.

Sorry, I'm glade to learn you can make ASN.1 efficient if you are a specialist and now what you are doing with the myriad available encodings. It's only inefficient in the way everyone use it.

Subsets of ASN.1 that match the functionality of Protocol Buffers are not "quite complicated" -- they are no more complicated than PB.

> Still, ASN.1 is a telco protocol through and through.

Not really. The ITU-T developed it, so it gets used a lot in telco protocols, but the IETF also makes a lot of use of it. It's just a syntax and set of encoding rules.

And so what if it were "a telco protocol through and through" anyways? Where's the problem?

> It shows everywhere: syntax, tooling.

The syntax is very much a 1980s syntax. It is ugly syntax, and it is hard to write a parser for using LALR(1) because there are cases where the same definition means different things depending on what kinds of things are used in the definition. But this can be fixed by using an alternate syntax, or by not using LALR(1), or by hacking it.

The tooling? There's open source tooling that generates code like any XDR tooling and like PB tooling and like MSFT RPC tooling.

> Sorry, I'm glade to learn you can make ASN.1 efficient if you are a specialist and now what you are doing with the myriad available encodings. It's only inefficient in the way everyone use it.

No, you don't have to be a specialist. The complaint about inefficiency is about the choice of encoding rules made by whatever protocol spec you're targeting. E.g., PKI uses DER, so a TLV encoding, thus it's inefficient. Ditto Kerberos. These choices are hard to change ex-post, so they don't change.

"[T]he way everyone use it" is the way the application protocol specs say you have to. But that's not ASN.1 -- that's the application protocol.

There is no open source tooling that combines really used scheme understanding - in 5G this includes parameterized specifications by X.683 - and decoder able to show partially decoded message before an error, with per-bit explanation of rules led to its encoding.

> E.g., PKI uses DER, so a TLV encoding, thus it's inefficient.

When it is used, ~5% space economy is never worth people efforts to diagnose any problem. I strictly vote for this "inefficiency".

Uh, no, structs, records, whatever you want to call them, are in many, if not most programming languages. "Structs" is not just "C structs" -- it's just shorthand for "structured data types" (same as in C!).

> Asn.1 is both quite complicated and not very efficient.

Every rich encoding system is complicated. As for efficiency, ASN.1 has many encoding rules, some of which are quite bad (BER/DER/CER, which are the first family of ERs, and so many thing ASN.1 == BER/DER/CER, but that's not the case), and some of which are very efficient (PER, OER). Heck, you can use XML and JSON as ASN.1 encoding rules (XER, JER).

> They could certainly have gone with protobufs or another binary serialisation format but would it really be better than the current choice?

Protocol buffers is a tag-length-value (TLV) encoding, same as BER/DER/CER. Having to have a tag and length for every value encoded is very inefficient, both in terms of encoding size as well as in terms of computation.

The better ASN.1 ERs -PER and OER- are much more akin to XDR and Flat buffers than to protobufs.

> I don’t think the issue they are trying to solve is related to serialisation anyway. Seems to me they are unhappy about the bus part not the message format part.

This definitely seems to be be the case.

I have had a misfortune to work for 5G which is full of PER-encoded protocols. Dealing with discrepancies in them - incompatible changes in 3GPP standard versions, different vendorsʼ errors, combined with usually low level of developers and managers in a typical corporation - was an utter nightmare.

IETF, in general, provides a good policy combining truly fixed binary protocols when they are unavoidable (IP/TCP/UDP levels) and flexible, often text, protocols where there is no substantial overhead from their use. Their early moves, well, suffered from over-grammaticalization (as RFC822). CBOR is nice here because it combines tagness and compactness. 3-bit basic tag combined with value (if fit) or length, it is commensurable with OER in efficiency but is decodable without scheme - and it is extremely useful in practice.

It's plenty clear from discussion context that OP is talking about C struct but yes, replace C with any languages which suit you. It will still be part of the language semantic and not an IPC specification.

The point is you can't generally use memory layout as an IPC protocol because you generally have no guarantee that it will be the same for all architectures.

Certainly but that’s hardly structs anymore. You are implicitly defining a binary format which is aligned on the sender memory layout then.

Also, there are benefits to the format being human readable.

(also, unix uses files for everything, including /proc talking to drivers!)

Someone took a look at D-Bus and decided "You know what? It just isn't shitty enough. It's 2024 and D-Bus runs halfway decently on the average laptop. Let's fix that."

I wouldn't be surprised if the Scooby Gang pulled the mask off the head dev, revealing ThePrimeagen, in a scheme of long-form comedy that's a cross between one of his "let's write a network protocol" streams and his new enterprise development satire series.

More or less all the display specific stuff is an extension on top of the core design.

https://wayland-book.com/protocol-design.html

OTOH, it’s too complex for simple protocols with a few messages.

IIRC, Pipewire uses a variation of the Wayland protocol.

D-Bus evolved from merging their needs into a single approach.

https://media.ccc.de/v/all-systems-go-2024-276-varlink-now-

Why was it designed that way in the first place if all we need (supposedly) is a socket and a simple request/response protocol? If it wasn't needed, why does D-Bus have it? And if it is needed, what will replace it when systemd switches to Varlink?

Why do people keep replicating this terrible design decision? It puts the lowest-entropy part of the name at the beginning, so that you have to parse (or type) the maximum number of characters before you get a unique specification. Try tab-completing a "flatpak run" command sometime.

To mitigate senseless tree top listing, IDEs propose a bunch of means like context-related hints or unpacking forms like "o.e.t" to "org.example.test".

AllSystemsGo: Varlink Now [video] - https://news.ycombinator.com/item?id=41670431 - Sept 2024 (2 comments)

Varlink: Interface description format and protocol for humans and machines - https://news.ycombinator.com/item?id=25621936 - Jan 2021 (5 comments)

Varlink – A plain-text, type-safe, discoverable, self-documenting interface - https://news.ycombinator.com/item?id=20950146 - Sept 2019 (10 comments)

https://www.phoronix.com/news/Systemd-Varlink-D-Bus-Future

Anything that add new features (early boot IPC in this case) is going to require "more code, more bloat, more complexity, and more attack surface area" unless you rip other features out at the same time.

Lots of discussion here has been around the choice of JSON as a format, with some valid points being brought up about specific limitations and possible footguns.

But these miss the point: this massively moves the needle with regard to lowering the barrier to entry when writing services. I’m confident I can write a simple Varlink service in about 20 lines of Python, with no dependencies after reading the linked docs. You cannot in good faith say the same thing about dbus.

That’s nuts. Yeah - if my service is for some reason sending/receiving high-precision 128 bit numbers or huge binary files then it’s not perfect.

But nothing really does that. It’s not worth making the entire stack more complex and less performant for a possible use case that doesn’t really fit.

I can't wait to have apps and services crash and burn left and right due to accidentally (or maliciously) created non-UTF8 filenames in world readable directories.

That is hardly the only problem with such a serialization format, although it is one of them.

> I can't wait to have apps and services crash and burn left and right due to accidentally (or maliciously) created non-UTF8 filenames in world readable directories.

It is not only accidentally or maliciously. Sometimes it may also be done deliberately because it is a character set other than Unicode, without the intention to be malicious.

(A user might also deliberately want to display file names using a different character set, perhaps in order to ensure that all characters can be displayed unambiguously. Of course, this is not necessarily what everyone intends, but some people will.)

[1] https://ruudvanasseldonk.com/2023/01/11/the-yaml-document-fr...

Red Hat FOSS system software is imposing foss-ified client work to the general public. The cloud industry benefited greatly from their work, the other users are involved in a multi-decade flamewar.

How would you address the issue of dbus unavailability in early boot listed in the presentation via extension?

It is a common occurrence that whatever Lennart comes up with fulfills several, erm, novel requirements while not fulfilling some old ones that aren't even mentioned. The parallels to binary logging are there. "Fancy" features that no one asked for - check. Simplicity and thoughtful use of existing functionality (the "full exploitation" principle of the Unix design philosophy) - nope.

Are these required or useful in this context?

>Dbus doesn't do something? Extend dbus.

How are you so certain that this is the best use of resources, do you have personal XP in the problem domain?

> Don't make some other random and worse things.

How did you determine that these changes are "worse" or "random?"

At least 64-bit numbers are crucial in any system context.

I would encourage you to work with either data at scale or embedded hardware for a fresh perspective.

Seems you have pretty misread me. Well, "any system" (where systemd is useful in principle) "context" and not "any" "system context", if I correctly guessed where is your point.

As the whole protocol shall be supporting Linux, in a fully universal manner, support of 64-bit values for pointers (even in crash reports), file offsets (including 32-bit systems), etc. is a must.

> I would encourage you to work with either data at scale or embedded hardware for a fresh perspective.

Thanks, I got used to, different platforms (like ARM) including 32- and 64-bit ones. Also ancient beasts like 6502, PDP-11, and even S/360 (Soviet clones) at good old school times. Beg your pardon, no experience with 8051, 8042, ESP8266 and so on, but they arenʼt in question here. Again, the very topic is full-scale Unix context - more so, closer to desktop/laptop/server, which are nearly always 64-bit now (mainstream distribution vendors deliberately abandoned 32-bit versions for these domains a few years ago), so I donʼt expect much digression.

[1] https://www.crockford.com/dec64.html

I just happened to read Lennart Pottering's original post on sd-dbus yesterday actually. It gives a good introduction to D-Bus for those not up to date.

https://0pointer.net/blog/the-new-sd-bus-api-of-systemd.html

Nanomsg ng and cap'n proto already exist.