Hacker News new | past | comments | ask | show | jobs | submit login

I’ve seen some multicast market data protocols with remarkably poor ability to detect or recover from drops. And they are very much not of the form where a newer datagram supersedes the older one.



Yes, although people have been doing marketdata networks the way I said above using IP multicast for at least 20 years now, so in general they choose protocols and network architectures carefully to minimise problems. You do see problems from time to time but they are somewhat rare. Some of the restrictions are interesting. For example IP multicast was basically completely banned on the trading floor where I worked except for marketdata, because of an IP multicast snafu from some random application that took out the whole network once.

One thing to realise about marketdata specifically is it's really different from other low-latency situations most people are familiar with (netcode in a game for example). As I mentioned before, it's not that big a deal generally to miss a few packets - the thing that is a big deal is to make decisions based on stale data. So you're not generally trying to reconstruct the full state after a drop- you just want the freshest current packet as fast as possible. If/when you need to reconstruct state you can make specific requests if needed.


Out of curiosity, what specific protocol are you taking about? I’m very familiar with a couple of these protocols, and the issue has nothing to do with wanting to know the history of the data — the issue is that they use what is, in effect, an ad-hoc delta encoding, and you can’t reliably reconstruct the complete current state if you’re missing a packet. On top of this, sometimes the packet sequence numbering is designed creatively, to be polite about it, so you don’t necessarily find out that you missed a packet as soon as you would like to be able to.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: