In my experience teaching undergrads they mostly get this stuff already. Their CompArch class has taught them the basics of branch prediction, cache coherence, and instruction caches; the trivial elements of performance.
I'm somewhat surprised the piece doesn't deal at all with a classic performance killer, false sharing, although it seems mostly concerned with single-threaded latency. The total lack of "free" optimization tricks like fat LTO, PGO, or even the standardized hinting attributes ([[likely]], [[unlikely]]) for optimizing icache layout was also surprising.
Neither this piece, nor my undergraduates, deal with the more nitty-gritty elements of performance. These mostly get into the usage specifics of particular IO APIs, synchronization primitives, IPC mechanisms, and some of the more esoteric compiler builtins.
Besides all that, what the nascent low-latency programmer almost always lacks, and the hardest thing to instill in them, is a certain paranoia. A genuine fear, hate, and anger, towards unnecessary allocations, copies, and other performance killers. A creeping feeling that causes them to compulsively run the benchmarks through callgrind looking for calls into the object cache that miss and go to an allocator in the middle of the hot loop.
I think a formative moment for me was when I was writing a low-latency server and I realized that constructing a vector I/O operation ended up being overall slower than just copying the small objects I was dealing with into a contiguous buffer and performing a single write. There's no such thing as a free copy, and that includes fat pointers.
> Fairly trivial base introduction to the subject.
Might be, but low-latency C++, in spite of being a field on its own, is a desert of information.
The best resources available at the moment on low-latency C++ are a hand full of lectures from C++ conferences which left much to be desired.
Putting aside the temptation to grandstand, this document is an outstanding contribution to the field and perhaps the first authoritative reference on the subject. Vague claims that you can piece together similar info from other courses does not count as a contribution, and helps no one.
> this document is an outstanding contribution to the field and perhaps the first authoritative reference on the subject.
I don't know how you arrive at this conclusion. The document really is an introduction to the same basic performance techniques that have been covered over and over. Loop unrolling, inlining, and the other techniques have appeared in countless textbooks and blog posts already.
I was disappointed to read the paper because they spent so much time covering really basic micro techniques but then didn't cover any of the more complicated issues mentioned in the parent comment.
I don't understand why you'd think this is an "outstanding contribution to the field" when it's basically a recap of simple techniques that have been covered countless times in textbooks and other works already. This paper may seem profound if someone has never, ever read anything about performance optimization before, but it's likely mundane to anyone who has worked on performance before or even wondered what inlining or -Funroll-loops does while reading some other code.
I am well aware of this fact because I've researched the topic and I can state it without any degree if uncertainty. The only and resources there are scattered loose notes and presentations in conferences such as Timur Doulmer's work for C++ On Sea, and even so he's clear on how his work is mainly focused on real-time audio processing, which has different requirements than say HFT.
> The document really is an introduction to the same basic performance techniques that have been covered over and over. Loop unrolling, inlining, and the other techniques have appeared in countless textbooks and blog posts already.
Go ahead and cite the absolute best example you can come up from this incredible list of yours. The very top of your list will suffice to evaluate your whole list. I'll wait.
> Go ahead and cite the absolute best example you can come up from this incredible list of yours. The very top of your list will suffice to evaluate your whole list. I'll wait.
This could really do without the "incredible list of yours" and "I'll wait" snark. You don't need to be so condescending, but for the sake of helping others who are actually curious about the subject:
It has significantly more breadth than the small collection of techniques in the paper linked in this post. It doesn't have the "HFT" buzzword attached, but then again the techniques in the linked paper above aren't unique to HFT either.
There are also several books on the subject, but given that the above link is a good place to start and the resources are freely available, that's where I'd recommend you look.
I don't think so. For instance, one basic technique in low-latency C++ is running specific code periodically even with no-op parameters just to keep it warm in the cache.
It's one thing to claim that you can cherry pick some directly or indirectly pieces of information from an unrelated source. It's another entirely different thing to claim that unrelated source is an authoritative reference in an unrelated field, specially when no one recognizes it as such.
> A creeping feeling that causes them to compulsively run the benchmarks through callgrind
I'm happy I don't deal with such things these days, but I feel where the real paranoia always lies is the Heisenberg feeling of not even being able to even trust these things, the sneaky suspicion that the program is doing something different when I'm not measuring it.
On the software side I don't think HFT is as special a space as this paper makes it out to be.[1] Each year at cppcon there's another half-dozen talks going in depth on different elements of performance that cover more ground collectively than any single paper will.
Similarly, there's an immense amount of formal literature and textbooks out of the game development space that can be very useful to newcomers looking for structural approaches to high performance compute and IO loops. Games care a lot about local and network latency, the problem spaces aren't that far apart (and writing games is a very fun way to learn).
I don't have specific recommendations for holistic introductions to the field. I learn new techniques primarily through building things, watching conference talks, reading source code of other low latency projects, and discussion with coworkers.
[1]: HFT is quite special on the hardware side, which is discussed in the paper. The NICs, network stacks, and extensive integration of FPGAs do heavily differentiate the industry and I don't want to insinuate otherwise.
You will not find a lot of SystemVerilog programmers at a typical video game studio.
As someone who does quant trading professionally and game development as a hobby, they both are performance sensitive, but they emphasize different kinds of performance. Trading is about minimizing latency while video games are about maximizing bandwidth.
Video games try to cram as much work as possible within about 16 milliseconds whereas for most trading algorithms 16 milliseconds is too slow to do anything, you want to process and produce a response to input within the span of microseconds, which is 3 orders of magnitude faster than a single frame in a video game.
The problem spaces really are quite distinct in this respect and a lot of techniques from one space don't really carry over to the other.
I'm curious how HFT relates to pro audio programming. The timescale is close to gaming (usually <10ms for desktop PC work), but you really care about the tail of the distribution. All the audio processing happens in a high-priority callback that's called for every audio frame and needs complete within the time budget, so you typically never allocate, do IO, or use any blocking synchronization primitives [1].
It's not hard real-time like you're going to crash your car, but if you miss your deadline it causes an unacceptable and audible glitch.
I've always been a bit surprised that Jane Street uses OCaml. I know they've put a lot of attention into the GC, but it still seems fundamentally indeterminate in a way that would make most audio devs nervous.
Audio has a lot of buffering behaviour that you wouldn't generally see in event-reactive HFT. Think of all the plugins that you know of that have non-zero latency, compressors with 'lookahead' etc. There are maybe some similarities where the logic is more complex (loop unrolling, SIMD and so on) but I feel like plugins are generally optimizing for throughput (CPU usage) and quality (oversampling etc) rather than purely latency in most cases.
I guess the difference I'm interested in is whether HFT tends to be "realtime", in the sense that there's a hard deadline that you need to hit every time.
Put another way, audio is operating on a much longer timescale, but cares a lot about worst-case latency (as well as throughput and quality). Is that true of HFT also, or are they more concerned with average latency? If computing a trade takes too long, can they just discard it, or is it catastrophic to miss that deadline?
I can't go into too much detail (I currently work at an HFT, but not on the HFT bits), but some of the FPGAs make decisions without even seeing a entire packet coming in on the wire. Latencies are more often measured in nanos than micros from what I've seen lately.
Tangentially, how do you like working for an HFT? I'm a low level software / FPGA developer and have thought about going into the HFT space. Is the work life balance as terrible as people say?
It's really going to come down to which firm you're at and who your manager is. I'm at a smaller shop currently, and it's great. I basically work 8-5 most days, sometimes a little later if processes don't come up cleanly for market open at 5P (CME basically operates 23 hours a day with different trading sessions).
There are firms that are more like sweatshops. I spent roughly a decade at Citadel; when I started, average tenure was thought to be about 18 months. Hours were brutal, but it largely depended on your manager. I had one (he was fired 6 months after I started) that insisted on 80 hour weeks, ass in seat, no work from home, not even on the weekends. The kicker was, my team didn't even have the work at the time to justify all of us pulling 80 hour weeks. I also spent months at times pulling 100 hour weeks. That is, if you slept, it was at your desk (there were cots/sleeping bags available, but I never saw anyone bother). Maybe go home every 2nd or 3rd day for a change of clothes (you could shower at the office). Then, I had other managers where it was more 8-5. But, I was always on call 24/7, even while on vacation. That was the truly rough part. I would say on a whole, the reputation of working for Citadel is worse than it deserves, but it really matters who your manager is. That said, not sure I know anyone left at Citadel anymore other than Ken G. (I've been gone over a decade now).
My best advice when you're interviewing people, ask pointed questions about work-life balance. Avoid asking overly broad questions like "How's the work/life balance?". Instead prefer: "When was the last time you took a 2 or 3 week vacation?"
edit:
At my current role, I do sometimes work outside of 8-5 and on weekends, but its because I choose to, because I'm working on something interesting that excites me. I'm not expected to, and no one gives me a hard time if I don't.
Not OP but it varies wildly between companies. JS and Citadel are both top tier trading shops and they could not be more different when it come to wlb.
It's not hard to sniff out during the process though.
Yes, there is a lot of effort to understand, for example, the 99th percentile tick-to-trade latency as well as the average latency. For a lot of those worst-case timings there are things that are occasionally out of your control like the kernel, the network, the hardware. Then there are things always in your control like how you implement your software.
Most HFT algos are busy-spinning. You'll see a core pinned at 100% as it checks for input availability over and over. The vast majority of the time it is actually doing nothing, from an external point of view. When that tick appears that it needs to react to, it springs into action. It might react to some input in the range of a few hundred nano seconds but for literally billions of nanoseconds it's doing nothing at all.
Audio is processing a buffer of data (usually whatever your audio interface buffer is set to), then waits patiently until the next buffer arrives. There's a regular pulse to it. If your CPU isn't hurting, there's way more time between those buffers than are required to process them. The order of magnitude between work and 'not work' for HFT is huge compared to audio generally. HFT logic is simple, small and fast. Audio is hopefully complex if you paid good money for it. The best way to go fast is to do almost nothing.
In terms of the impact of not meeting a particular deadline, it's an opportunity cost in HFT. In audio, you get audio dropouts if you can't process it fast enough. In HFT, if your reaction time is slow, you're not at the front of the queue for that juicy trading opportunity you and everyone else just spotted, then you miss it entirely and someone else gets the prize. HFT is all about making thousands of small statistically favourable bets and being able to execute on them fast enough to realise the opportunity before the next fastest guy.
In audio, you can usually tolerate a few milliseconds of latency (more when mixing, less when recording). This means you can buffer your audio stream, for example in blocks of 64 or 128 samples.
As far as I know, these millisecond latencies are orders of magnitude higher than what would be tolerable in HFT. There the units are microseconds and nanoseconds.
As somebody who has done both, there's certainly a lot of similarities in how you write software.
I'm talking about real-time audio, though. There's also just audio playback (Pro Tools, e.g.) which tolerates much higher latency, but enables higher audio fidelity due to larger buffer sizes (e.g. in compressors to enable look-ahead) as well as more clever algorithms (which are often prone to latency spikes, such as a "standard" FFT, if you want a very simple example).
Both, finance and real-time audio, differ in their time scales, but more in the breadth of their time scales: real-time audio is often between 1–50 ms (yeah, we try to keep it below 10 ms, but that's tough to do in a complex setup. I've seen artists often compromise and increase buffer sizes (== latency) to accommodate for their complex virtual instruments and filters), finance in ns–ms (it's not HFT anymore at a certain point, sure, but those HFT architectures also have a layered response, with FPGAs being quick but simple and more complex strategies having higher latency; also, if you're e.g. asked for a quote on a bond, it's not a shame to take 500 ms to think about a price, because those trades are bigger but slower).
You also mentioned the consequences of missing a deadline. Well, indeed, I've seen this variance to be less of an issue in finance (we're only caring about xx µs, so I hesitate to call myself a HFT guy) than in real-time audio. A click in a live set is a no-go. A crash of your DAW is fatal, that's it with your live set. Nobody dies, but people will be very upset. I've seen real-time audio being obsessed with being reliable and keeping the variance in latency really small.
In finance, I've seen it more that people kind of accepted that variance was huge because there's some hidden allocations in a system. Hoping that those stop once the system is warmed up or that they rarely occur on a critical trade.
As a side note, in finance I've even heard this well-known HFT firm: "Well, it's not the end of the world if your process crashes. If you can restart quick enough."
It's also that in finance, your data is more heterogenous: Real-time audio has audio (isochronous, homogenous buffer of floats), MIDI (asynchronous, formerly small events), parameter changes (asynchronous, often just a float) and config changes (changes the topology of your audio processing graph; enjoy to transition from A to B without causing an audible glitch). Finance has processing graphs that only process asynchronous data (heterogenous market data, telling you what's on the order book of an exchange; trade data for which trades you performed with your orders and quotes; symbol data when new securities appear or change (historically only with the start of the trading day); pre-computed steering data for algorithms and/or data from third-party data sources).
Thus, the processing graphs of both differ in the data that's transferred as well as in the structure and domain. In finance, there is no regular tick-based processing. You have to accommodate a very varying amount of data per time period, whereas in audio, it's a very regular pace and the amount of data is often the same (± activations of voices in instruments).
Real-time audio has plugins, though. And those, from experience, do whatever you can imagine. Plugins are great, from a creative perspective, but the craftsmanship is of very varying degree. I bet I'll even find a plugin that draws the UI from the real-time audio thread.
Of course, time spans are different. But read the paper, it's dealing with incredibly basic techniques. When we're speaking on the level of "data should be in cache" and "cold code should be far away from the hot path" the fields are practically identical.
You're entirely correct that quant work is dealing with pure latency in situations where game engines might amortize throughput over multiple frames. But a question about "how do I get started in performance/latency work?" isn't usefully answered by "get a Hudson River Trading internship".
"which is 3 orders of magnitude faster than a single frame in a video game."
You're right on the money. I worked in HFT for half a decade.
Back in the late 2000s you were on the cutting edge if you were writing really good C++ and had overclocked some CPUs to hell and back and then shoved them in a rack in a datacenter in new jersey. "To hell and back" means "they only crash every hour or two" (because while they're up, they're faster than your competition).
Around the mid 2010's it had moved entirely to FPGAs. Do something smart in software, then give the FPGA something dumb to wait for (e.g. use software to produce a model and wait for some kind of alpha to develop, then tell the FPGA "ok wait for X > Y and fire the order" was basically the name of the game. And it was often better to be faster than it was to be smarter, so really you just kept the FPGA as simple and fast as possible.
At that point, your hard realtime constraints for the language doing the smart stuff really dissolve. Compared to an FPGA getting an order out the door in some fraction of a mic, a CPU doing anything is slow. So you might as well use python, yknow?
People also play all kinds of different speed games. There's latency races around NJ and chicago where microseconds matter around price movements and C++ isn't really part of the picture in a competitive way anymore, but that's not to say someone isn't ekeing out a niche where they're doing something vaguely smarter faster than opponents. But these things, at scale, tend to be questions of - either your alpha is very short-lived (microseconds, and if it isn't organically microseconds, it will be competed until it is), or it is fundamentally longer-lived (seconds to minutes?) and you might as well use python and develop 100x faster.
The silly thing is, the really good trades that focus on short-term alphas (ms's and less) are generally obviously good trades and so you don't have to be super smart to realize it's a really fucking good trade, you had better be fast and go get it. So there's also this kind of built-in bias for being fast because if a trade is so marginally good that you needed a crazy smart and slow model to tell it apart from a bad trade, it's probably still a relatively crummy trade and all your smarts let you do is pick up a few extra pennies for your trouble.
I'll close by saying don't take my words as representative of the industry - the trades my firm specialized in was literally a dying breed and my understanding is that most of the big crazy-crazy-fast-microwave-network people have moved to doing order execution anyway, because most of the money in the crazy-crazy-fast game dried up as more firms switched to either DIYing the order execution or paying a former-HFT to do it for them.
Even if the low-latency logic moves to the FPGA, you still need your slow path to be reasonably fast, say about 100us, and absolutely never more than 1ms.
To my knowledge Python is not suitable, though I know some players embed scripting languages that are.
It depends entirely on what you've got on the FPGA and what's slow-path. My firm had order entry done exclusively by the FPGA and the non-FPGA code would have orders in the FPGA for sometimes tens of seconds before they would fire. The non-FPGA code was still C++, but the kind of everyday C++ where nobody has been wringing all the water out of it for speed gains. If we rewrote it all, it'd probably be python. But that's just us. Every firm has their own architecture and spin on how this all works. I'm sure somebody out there has C++ code that still has to be fast to talk to an FPGA. We didn't - you were either FPGA-fast or we didn't care.
edit: also, fwiw, "say about 100us, and absolutely never more than 1ms." things measured in that many microseconds are effectively not what my firm considered "low-latency". But like I said, there are lots of different speed games out there, so there's no one-size-fits-all here.
Your hardware is configured by your software. You pre-arm transactions to be released when a trigger occurs.
Releasing the transaction when the trigger occurs is the fast path, and what the hardware does.
The slow path is what actually decides what should be armed and with what trigger.
If your software is not keeping up with information coming in, either you let that happen and trigger on stale data, or you automatically invalidate stale triggers and therefore never trigger.
100us is short enough to keep up with most markets, though obviously not fast enough to be competitive in the trigger to release loop.
But it's mostly an interesting magnitude to consider when comparing technologies; Python can't really do that scale. In practice a well-designed system in C++ doesn't struggle at all to stay well below that figure.
The idea for aggressive orders in HFT is to buy before the market goes up, and sell before the market goes down.
HFT signals are about detecting those patterns right before they occur, but as early as the information is available globally. For example someone just bought huge amounts of X, driving the price up, and you know Y is positively correlated to X, so Y will go up as well, so you buy it before it does.
X and Y might be fungible, correlated mechanically or statistically.
You can also do it on Y=X as well; then what you need is the ability to statistically tell whether a trade will initiate/continue a trend or revert. You only need to be right most of the time.
On one hand you have quantitatively driven strategies that try to predict either a price or direction based on various inputs. Here you’re mostly focused on predictive accuracy, and the challenge is in exiting the trade at the right time. This is where a lot of the speed comes into play (what is your predictive horizon, and can you act fast enough to take advantage of the current market prices?).
The other mode of trading tends to focus on structural mispricing in the market. An easy to understand example is an intermarket arbitrage trade where one market’s buyer or seller crosses prices with the opposite side of the market on another exchange. These events permit a trader to swoop in a capture the delta between the two order prices (provided they can get to both markets in time).
As easy opportunity has dried up (markets have grown more efficient as systems have gotten faster, and parties understanding of the market structure has improved) you see some blending of the two styles (this is where another commenter was talking about mixing a traditionally computed alpha with some hardware solution to generate the order), but both come with different technical challenges and performance requirements.
Isn't there challenges with slippage and managing the volumes while exiting? And isn't speed also about processing the data feed as fast as possible to time the exit decisions accurately?
Absolutely to both questions, with different answers depending on what style of strategy you’re running.
The market mechanics trades tend to have no recoverability if you miss the opportunity, so you’re often trading out in error and it’s a matter of trying to stem the loss on a position that you do not have an opinionated value signal on.
And there’s definitely an angle to inbound processing speed for both styles of trading, with differing levels of sensitivity depending on the time horizons you are attempting to predict or execute against. Using the example above again, detecting the arb opportunity and firing quickly is obviously paramount, but if you’re running a strategy where you have a 1 minute predictive time horizon sure, there’s some loss that can be associated with inefficiency if you aren’t moving quickly and someone else is firing at a similar signal, but generally speaking there’s enough differentiation in underlying alpha between you and any competitors that the sensitivity to absolute speed isn’t as prevalent as most people expect.
Basically it boils down to going fast enough to beat the competition, and if there isn’t any you have all the time in the world to make decisions and act on them.
This is true -- in the early 00's there was hardly any competition and we could take out both sides of a price cross with 100ms latency. Even after colocation we could still be competitive with over 4ms latency (plus the network). Trading technology has come a long way in 20 years.
If every other exchange is selling $AAPL at $100 and suddenly the top level of one exchange drops to $99, then if you just take out that order you basically gain a free dollar. Do this very fast and have pricing the product accurately and you will print tons of money.
Yeah I gather that is the expectation, but if you are the first to execute an order you will sell that order at the old 100 price before it lowers. You are fighting for making an order before the information spreads to the other bots. (Right?!)
Latency isn't even as important in HFT as people claim. What's most important is deterministically staying into a reasonable enveloppe (even at the 99.9 percentile) to satisfy real-time requirements and not fall behind.
When it's really important, it's implemented in FPGA or with an ASIC.
This is close to the question I asked above, comparing to audio. If you're processing 32 samples at a time at 48kHz, you need to process a frame every 667us. If you only hit the deadline 99.9% of the time, that's a glitch more than once per second. You'd need a bunch more nines for an acceptable commercial product, so most audio devs just treat it as a hard realtime constraint.
I think the main split is whether you care more about average latency or worst-case latency (or some # of nines percentile).
In HFT you typically only measure latency for events that you did react on, not all events (at least, in the systems that I've built).
That's arguably a bit problematic as events you don't react to can cause you to fall behind, so it's useful to also track how far behind you are from the point where you start processing a packet (which requires hardware timestamping, unfortunately not present on many general-purpose NICs like what's in the cloud).
For a given market, you may have less than 1M samples a day (I've even seen some so-called HFT strategies or sub-strategies that only had 300 samples per day).
So overall, you'd have fewer events than 44.1kHz, and there are typically expected outliers at the beginning of the data before the system gets warmed up or recovers from bad initial parameters (which I suppose you could just ignore from your distribution, or you could try not to require a warmup).
But you're right, you probably want to look at quantiles closer to 1 as well. You're also looking at the max regardless.
Whatever is relevant to the domain you're tackling. How you structurally approach latency in a web server is different than a 3D renderer, which is different than an audio stack, etc. All of those domains have great open source or source available code to learn from, but it wouldn't be useful for me to say "go read HAProxy" if you're never going to touch an HTTP packet. When I'm tackling a problem the first thing I do is research everything everyone else has done on the problem and read their code, benchmark it if possible, and steal all the good ideas.
The basic principles never change. Avoid copies, never allocate, keep the hot path local, and really, good codebases should be doing these things anyway. I don't code any differently when I'm writing a command line argument parser vs low-latency RPC servers. It's just a matter of how long I spend tweaking and trying to improve a specific section of code, how willing I am to throw out an entire abstraction I've been working on for perf.
In the domain of web stuff, effectively all the major load balancers are good to study. HAProxy, nginx, Envoy. Also anything antirez has ever touched.
Application servers are also interesting to study because there's many tricks to learn from a piece of software that needs to interface with something very slow but otherwise wants to be completely transparent in the call graph. FastWSGI is a good example.
Very anecdotal but of the people I know in game studios who are tasked with engine work, and people who make a killing doing FPGA work for HFT firms, both camps shook their head at Casey’s HMH thing. Uniformly I do not know of a single professional developer of this sort of caliber who looked at HMH and thought it looks great. Quite the opposite. I think they found his approach and justifications unsound as it would instil awful practices of premature unfounded optimization and a disdain for normal library code in favour of hand-rolling your own half-baked implementations based on outdated trivia. I agree with them on the basis that HMH exposed an unprepared and inexperienced audience to something that has to be regarded with utmost care. For this, I refer to Jonathan Blow’s presentation of “a list is probably good enough” as an antidote. I think JB’s recommendations are more in line with actual practices, whereas Casey just raised red flags uniformly from here-and-now engine devs shipping multi platform games.
I have not watched Handmade Hero, so I can't offer any comments any their comments based on factual claims about the software. A few things though:
I am not linking to handmade hero, I'm linking to a separate project of his (his performance aware programming course) that is actually aimed at being an educational piece.
I lied, I will comment on one factual piece. "normal library code in favour of hand-rolling your own half-baked implementations based on outdated trivia." Yes, that is the whole point of the series (not the characterization as half-based and outdated trivia). The point was to show how to build a game (and its engine) from scratch to as big of a degree as possible. The avowing of library code is the point, to show what it takes to build engines rather than call a library so that the industry has more people who would even attempt doing such a thing.
Equally anecdotally, based on available online information, he worked for a long time on core technologies at RAD Game tools, a company which essentially every gamer, expect maybe pure mobile gamers, has purchased a game that used their technology. It may be possible that he acts (or acted in HMH) based on outdated trivia and favoured premature unfounded optimization, but I find it hard to believe based on the content of his I've engaged with and his track record.
Casey's a fundamentalist, in the religious extremist sense. Just as a theologian might have something to learn from the most extremely devout, so it is that there is valuable information in Casey's proselytizing.
But you should no more follow Casey in form and function than you would any other fundamentalist.
While gesturing at religion and saying he's a fundamentalist is an easy rhetorical technique, it doesn't add much. Yes good resources do exist, but they are few and far between. The number of new undergraduates from cs or software engineering who learned to think in the ways needed to make low latency applications are a rounding error based on my experience with them. Most forgot it, weren't taught it well, or never saw it at all. They are more worried about learning the syntax of the current hot framework rather than on anything even resembling "Avoid copies, never allocate, keep the hot path local".
The religious comparison is also a telling one given the state of the industry; for we aren't the theologian, we're the common folk looking for someone or something to follow in order to write better code.
Who are Casey's alternatives? Gesturing the cppcon as a learning resource has "read research papers to learn about a field" vibes. They can be highly informative and worth the effort, but not for beginners.
Who are Casey's contemporaries? If he's a fundamentalist then the atheists are nowhere to be seen by the beginners. Instead we have agile ceremony shamans, clean code bible missionaries, tdd doomsday cults, oop/rust/haskell/etc. zealots, a thriving megachurch industry of Lambda schools, and the mother of all cargo cults masquerading as web dev.
Agree there are not enough performance zealots. Modern architectures are almost magically fast if you get all of the tiny invisible things that kill your performance sorted out correctly. The fact that the knowledge of how to achieve this is closer to black magic lore than foundational educational knowledge is sad.
Performance is surprisingly often a feature. When things become instantly fast, instead of horribly slow, new opportunities are suddenly feasible for a lot of things.
> The fact that the knowledge of how to achieve this is closer to black magic lore than foundational educational knowledge is sad.
It is sad, doubly so since the things you need to get within an order of magnitude of what your hardware can do aren't the arcane assembly and "premature optimization" boogeymen students picture. Forget the 10x engineer, going from the 0.0001x engineer to the 0.1x would be a massive improvement and it's low hanging fruit.
They're simple things: have your code do less, learn what good performance should be, understand the hardware and internalize that the point is to program it, and use/build better tools (e.g. perhaps your programming model is fundamentally flawed if it essentially incentivizes things like the N+1 Selects Problem).
> Performance is surprisingly often a feature.
Performance is, unsurprisingly, often a missing feature in most software. Every day I need to boot up Teams I feel we stray further from Moore's light.
The whole point of Handmade Hero is to make everything from scratch as a learning exercise, so using third-party libraries would be counter-productive.
How I might approach it. Interested in feedback from people closer to the space.
First, split the load in to simple asset-specific data streams with a front-end FPGA for raw speed. Resist the temptation to actually execute here as the friction is too high for iteration, people, supply chain, etc. Input may be a FIX stream or similar, output is a series of asset-specific binary event streams along low-latency buses, split in to asset-specific segments of a scalable cluster of low-end MCUs. Second, get rid of the general purpose operating system assumption on your asset-specific MCU-based execution platform to enable faster turnaround using low-level code you can actually find people to write on hardware you can actually purchase. Third, profit? In such a setup you'd need to monitor the overall state with a general purpose OS based governor which could pause or change strategies by reprogramming the individual elements as required.
Just how low are the latencies involved? At a certain point you're better off paying to get the hardware closer to the core than bothering with engineering, right? I guess that heavily depends on the rules and available DCs / link infrastructure offered by the exchanges or pools in question. I would guess a number of profitable operations probably don't disclose which pools they connect to and make a business of front-running, regulations or terms of service be damned. In such cases, the relative network geographic latency between two points of execution is more powerful than the absolute latency to one.
The work I do is all in the single-to-low-double-digit microsecond range to give you an idea of timing constraints. I'm peripheral to HFT as a field though.
> First, split the load in to simple asset-specific data streams with a front-end FPGA for raw speed. Resist the temptation to actually execute here as the friction is too high for iteration, people, supply chain, etc.
This is largely incorrect, or more generously out-of-date, and it influences everything downstream of your explanation. Think of FPGAs as far more flexible GPUs and you're in the right arena. Input parsing and filtering are the obvious applications, but this is by no means the end state.
A wide variety of sanity checks and monitoring features are pushed to the FPGAs, fixed calculation tasks, and output generation. It is possible for the entire stack for some (or most, or all) transactions to be implemented at the FPGA layer. For such transactions the time magnitudes are mid-to-high triple digit nanoseconds. The stacks I've seen with my own two eyeballs talked to supervision algorithms over PCIe (which themselves must be not-slow, but not in the same realm as <10us work), but otherwise nothing crazy fancy. This is well covered in the older academic work on the subject [1], which is why I'm fairly certain its long out of date by now.
HRT has some public information on the pipeline they use for testing and verifying trading components implemented in HDL.[2] With the modern tooling, namely Verilator, development isn't significantly different than modern software development. If anything, SystemVerilog components are much easier to unit test than typical C++ code.
Beyond that it gets way too firm-specific to really comment on anything, and I'm certainly not the one to comment. There's maybe three dozen HFT firms in the entire United States? It's not a huge field with widely acknowledged industry norms.
> optimization tricks like fat LTO, PGO, or even the standardized hinting attributes ([[likely]], [[unlikely]]) for optimizing icache layout
If you do PGO, aren't hinting attributes counter-productive?
In fact, the common wisdom I mostly see compiler people express is that most of the time they're counter-productive even without PGO, and modern compilers trust their own analysis passes more than they trust these hints and will usually ignore them.
FWIW, the only times I've seen these hints in the wild were in places where the compiler could easily insert them, eg the null check after a malloc call.
I said "or even", if you're regularly using PGO they're irrelevant, but not everyone regularly uses PGO in a way that covers all their workloads.
The hinting attributes are exceptional for lone conditionals (not if/else trees) without obvious context to the compiler if it will frequently follow or skip the branch. Compilers are frequently conservative with such things and keep the code in the hot path.
The [[likely]] attribute then doesn't matter so much, but [[unlikely]] is absolutely respected and gets the code out of the hot path, especially with inlined into a large section. Godbolt is useful to verify this but obviously there's no substitute for benchmarking the performance impact.
In my experience teaching undergrads they mostly get this stuff already. Their CompArch class has taught them the basics of branch prediction, cache coherence, and instruction caches; the trivial elements of performance.
I'm somewhat surprised the piece doesn't deal at all with a classic performance killer, false sharing, although it seems mostly concerned with single-threaded latency. The total lack of "free" optimization tricks like fat LTO, PGO, or even the standardized hinting attributes ([[likely]], [[unlikely]]) for optimizing icache layout was also surprising.
Neither this piece, nor my undergraduates, deal with the more nitty-gritty elements of performance. These mostly get into the usage specifics of particular IO APIs, synchronization primitives, IPC mechanisms, and some of the more esoteric compiler builtins.
Besides all that, what the nascent low-latency programmer almost always lacks, and the hardest thing to instill in them, is a certain paranoia. A genuine fear, hate, and anger, towards unnecessary allocations, copies, and other performance killers. A creeping feeling that causes them to compulsively run the benchmarks through callgrind looking for calls into the object cache that miss and go to an allocator in the middle of the hot loop.
I think a formative moment for me was when I was writing a low-latency server and I realized that constructing a vector I/O operation ended up being overall slower than just copying the small objects I was dealing with into a contiguous buffer and performing a single write. There's no such thing as a free copy, and that includes fat pointers.