After having spent an insane amount of time in late 2017/2018 building an HFT bot for Binance I can say this is a pretty solid article.
In our case we were doing triangle trading between BTC/ETH/USDT pairs and had our buys/sell delay down to 3-7ms. At one point moving 0.3-0.7% of Binance’s daily volume.
Few notes:
* Finding an objective point of truth for value when all of the currencies are floating is hard but vital to success. This was the hardest problem we encountered. We tried taking the realtime average of BTC and ETH across all exchanges, we tried tying it to the shortest route to USD, and several other routes... but ultimately this is where we ended up “losing” most of our alpha.
* Order books are seemingly simple but the devil is in the details. This especially matters for paper trading.
* Efficiently using API limits at exchanges is an optimization problem in and of itself.
* Our model was relatively simple but we focused on speed and edge cases. For instance Binance would rotate IPs on their load balancers and we’d constantly check the latency between each open SSL connection and use the fastest. Further we wouldn’t decode the buy response to plaintext we’d just read the raw stream.
After several epic months our entire project fell apart after a cryptic phone call about “institutional access” that didn’t follow the 1s websocket update. The access was quiet expensive and we said no to it and shortly after all of our strategies went to crap.
Best we could tell someone was front running us due to an artificial delay for our account (delay between trades went to ~20ms up from our prior steady speed of 3-7ms) and/or a bunch of the trades in the orderbook were bogus.
Frustrated we tried our strategy on another account and the delay dropped again to our normal range and was profitable again (the orderbooks were slightly different between bots!).
It was in that moment we realized playing in unregulated markets is not fun or something we wanted to continue to do. Intermediary risk was something we didn’t account for.
Further we realized that there will always been a better resourced or more dedicated team willing to fight you for your alpha.
After months of effort and a ton of fun we decided it was best we went back and focused on a problem where we could build a long term competitive advantage.
Even in regulated markets there are problems. I have a friend who built an HFT algorithm that he was using for trading stocks. He had some really good results with early testing. But at some point he was confused why many of his buy/sell orders weren't being executed despite being open for several minutes (hours?) his algo would make bids far away from the current spread, anticipating movement. He finally concluded that some institutional traders must have access to sub-penny ordering, despite it being against regulation.
I didn't believe him at first, since, the more likely problem was elsewhere, but then a few months later sub-penny trading in dark pools was all over the news. This was like 5 or 6 years ago. He's since moved on to other things having come to similar conclusions, that trying to play such a rigged game was futile.
> But at some point he was confused why many of his buy/sell orders weren't being executed despite being open for several minutes (hours?) his algo would make bids far away from the current spread, anticipating movement
* Did his levels actually get crossed? It could be that he never had the opportunity to get filled at the price he hoped for.
* What was his queue position? Stock exchanges tend to be price time ordered. If others had submitted orders at the same price before he had, they would have priority when getting filled.
* Apropos the prior point, was his broker actually submitting orders when he sent them? Some brokers may avoid showing deep out of the money orders, which could have affected his queue position.
> He finally concluded that some institutional traders must have access to sub-penny ordering, despite it being against regulation.
Really interesting, thanks!
Could you share bit more about the 'institutional access' call? Does it mean that selected few have access to real-time book updates and all the rest is @100ms delayed? So much of level playing field :) Is it common for asian crypto exchanges?
> the orderbooks were slightly different between bots
That sounds like a big deal. If this is repeatable, you should document it better. Unregulated doesn't mean a license to do blatantly illegal things. Crypto exchanges certainly get taken to court.
Sure it wasnt your book building algo and a snapshot retrieval race?
There's a much less nefarious explanation. I'm not saying they didn't manipulate the order book feeds. I work in the regulated trading space. I've written market data decoders. One is in production inside of a bigger trading system I wrote at a customer as I type this.
The less nefarious explanation is simple. You decide how likely it is: there is no mechanism making sure that IP packets sent to all data feed subscribers arrive at the same time. Exchanges of consequence distribute market data via UDP multicast, over physical links that are as identical as possible (think identical lengths of fibre).
Now if you're receiving JSON via Websocket and parsing it using an allocating parser and your NIC driver is in kernel space and you use a GC'd language and if the exchange loops through a list of TCP connections to send a message to them one at a time and there is jitter in packet delivery time in upstream hosts (and other internet weather) and and and ... you simply cannot expect identical order books at the submillisecond timescale.
A fortiori, throw half of that crap away and suppose they were using userspace NICs and no-alloc single threaded C++ that chills entirely in L1 cache. Still consuming TCP over the public internet.
Over the public internet its likely going to be a TCP feed. I would expect OP was capturing that feed and doing the analysis offline. Most firms do this, even with the collocated UDP setups.
After 2 weeks of digging through our code I’m 100% confident it wasn’t our code. Was able to hook up to a total of 3 additional accounts via api, all 3 books matched, and were started at different times.
We wrote support and they told us they would investigate. Never heard back.
Sure. That's not the point. The point is corruption increases risk of doing business regardless of whether somebody gets caught. People dying because of investigating corruption only makes the point stronger.
Ran an ML model years ago had a number of great months then out of no where no trade I or the ML would make, would work. Looked like someone was front running my orders and messing with my trades. Weird delays, trades would take to long to go through all, and all sorts of odd events on Level 2. Ended up shutting it down took a good 2 months before my manual trades started going through at a normal rate again.
This is why the whole $0 trading fee and robinhood concern me. I'm paying for the trades and someone is still messing with me.
I think, though I'm not sure, that RH makes their money by investing in treasury bills with the cash balances of people's portfolios. You're (most likely) not getting secretly screwed.
Contrary to popular belief robinhood makes a good bit of money from selling order data, more than most brokerages. This is due to the fact that most robinhood users are inexperienced, and thus, their trades are less likely to indicate big movements of institutional money. I.e. HFTs pay more for data that is less risky.
Edit: Fun fact, the founders of Robinhood worked at an HFT before starting robinhood, which used to be shown on their LinkedIn until several years ago. Bad optics I suppose.
What's the value of HFT? If exchanges were required to add a random delay to very trade to work against high frequency traders, would anything of value be lost?
If exchanges added a random delay, that would mean traders have to plan for a longer time horizon, which means they need to offer more conservative prices. That means when you're selling, you get less money, and when you're buying, you get less stuff.
Some people have this notion that if their trade got matched against a HFT that they have somehow lost out. The exact opposite is true. If the party on the other side of your trade was a HFT, then that implies that all other parties were offering worse prices than the HFT was. If the HFT had not been there, you would have got a worse price (whether on the buy side or the sell side). The presence of high-frequency traders (or any traders, for that matter) reduces spreads and increases liquidity.
The only people who get worse deals as a result of high-frequency trading are people who would sell you the same stuff for more money.
The substance of your post is describing price discovery - price discovery isn't the issue here. The issue is that the price discovery is being done in milliseconds.
The substance of the complaint is the speed of the auction that discovers the price. The faster price discovery is being done, the fewer traders can participate. If a few milliseconds can change the value of a financial instrument then we have to accept that waiting a second or two will allow a lot more traders to reevaluate and offer a fair price.
The HF in HFT looks like a play to reduce the number of traders who can act on information, which means the buyer/seller is probably getting scalped. HFT traders are making money arbitraging the speed of information dissemination, which indicates that other traders would offer different (/better) prices if the market waited a half-second or so to let everyone gather all the relevant data.
> The substance of the complaint is the speed of the auction that discovers the price. The faster price discovery is being done, the fewer traders can participate.
The faster the price discovery, the shorter the validity time. This means a more accurate price.
> If a few milliseconds can change the value of a financial instrument then we have to accept that waiting a second or two will allow a lot more traders to reevaluate and offer a fair price.
You're making the mistake of assuming the price is static. If I have to quote you a price on an instrument that's valid for the next 5s, I have to be more conservative than if I'm quoting for the next 5ms. Which means you get a worse price.
> The HF in HFT looks like a play to reduce the number of traders who can act on information, which means the buyer/seller is probably getting scalped. HFT traders are making money arbitraging the speed of information dissemination, which indicates that other traders would offer different (/better) prices if the market waited a half-second or so to let everyone gather all the relevant data.
There's literally nothing stopping or limiting the numbers of players operating at this speed. What the race actually results in is all market participants getting smaller spreads and better prices.
"The faster the price discovery, the shorter the validity time. This means a more accurate price."
A continuous market price is an illusion; let's not forget that. Every market is made up of trades, which are discrete, and every price has an unstated amount of uncertainty, which may be large by any standards. You can't know from first principles whether a change in quoted price is even a change in the market, because it could be within the +/- range that's implicit. Apple is quoted to the nearest penny, at least, but the idea that the current price is accurate to within 0.004% is ridiculous.
Edit:
"Fischer Black famously defined an efficient market as “one in which price is within a factor of 2 of value, i.e., the price is more than half of value and less than twice value,”
...I hadn't heard this before Matt Levine mentioned it, but I was like "yeah, obviously, why haven't most people gotten the message?"
But:
"BNP Paribas...said [Saudi] Aramco was worth exactly $1.424394 trillion."
I wonder which particular millisecond that held for.
Edit Edit:
The market has a whole (mostly) unseen dimension other than uncertainty, which is depth. You can only buy or sell so many shares at the instantaneous market price. Further away, there may be orders, but the price of the whole company is going to be way outside of that. The shorter the timescale, the shallower the "market" so you can't just say we're making progress by doing things faster. It's like when research lasers are said to make unbelievable power, but it's like for a femtosecond or something. Liquidity, in my mind, requires depth, just as with water.
I don't believe liquidity necessarily has anything to do with efficiency, but more liquidity can never hurt. Even if the price of every stock right now is 100x what it should be, I still want lower spreads.
And to address your concerns about depth, I believe that's where market makers (which are related to HFT) come in.
Everything is about costs and benefits. It doesn't make sense to say "I want an improvement in X, no matter how small" without regard to cost, because as X becomes tiny, a very small cost could mean a net loss to society, and multiplied by lots of people, is a large loss.
Look at gas stations. If people are spending $1 driving around to save $0.10, then that's not good and at least public policy shouldn't encourage it.
> If a few milliseconds can change the value of a financial instrument then we have to accept that waiting a second or two will allow a lot more traders to reevaluate and offer a fair price.
I'm not quite getting this point. If I have a sell order at price x and it's filled by a HFT 1 second before someone with a slower algorithm, how does that result in a more "fair" price for me? If the HFT instead posts a buy order at an unfair price x-1, there's nothing stopping the slower traders from taking my sell order at x one second later.
That would be totally boring trading and unremarkable, but that isn't what HFT firms are doing. If that were the strategy then exchanges would presumably just sell a skip-the-queue service when two traders offer the same price and there wouldn't be much point worrying about how far machines are from the exchange.
HFT trading isn't about executing the same trade as someone else but a tiny margin faster. That wouldn't have any special impact on market spreads or liquidity, for example.
There aren't any complaints against the T in HFT; as traders they are helpful. The value questions are about the HF and whether it is a desirable part of the market or an unhelpful arbitrage opportunity created only by implementation details of the exchange.
Many HFT systems provide liquidity. If there is no liquidity, retail investors like you or me cannot buy or sell.
Just imagine you want to exchange a currency because you go traveling and the exchange tells you "Sorry, nothing available right now, gotta come back in a few weeks". That's what would happen if there is no liquidity.
I always had questions about the liquidity on offer though seeing participants flee the market in black swan events. HFT liquidity could be illusionary - its only there when its not quite required similar to how the bank "only offers you a loan when you don't need one". Of course this is exactly the point when liquidity is required; normally there's sufficient liquidity in normal times from market participants. Its easy to offer liquidity in normal times; harder to do so when no one else wants to offer it.
That's not how things work. You would see much less demand/supply for a particular stock at every price level, which means you'd pay much more to buy less and would have to pay much more to sell less also. There are no upsides to having fewer market makers.
Why? If the HFT firm was willing to offer me $X 2 milliseconds ago they are probably still willing to offer $X now. It isn't like there has been time for anything to change; there are going to be short periods of time where there is literally no new information.
And they are just as likely to be offering me more now than less as conditions change.
We obviously can have liquidity on in-between timescales, because we had it before modern HFT trading, right?
The average person may be overly paranoid about HFT, but it doesn't make sense to say they benefit from it, because they are not going to be in a position where they benefit from an execution in a fraction of a second.
Price improvement of fractions of a penny has gotten silly too. It's easy to think of it as more significant than it is, until you figure it as a percentage (or the spread for that matter).
It's kind of like how ultra-sensitive people are to gas prices...
Yes, adding delay generally lowers the economic value of markets. In addition to improving liquidity, as mentioned by others here, HFT also improves the relative pricing of related instruments. There is an enormous number of financial instruments traded for the purpose of prioritizing resources, transforming assets, and offsetting risk. Equities and cryptocurrencies are a small part of this connected financial universe. There are many trading entities with unique information and/or understanding about different parts of this constantly changing universe of prices. None of them has a complete view, but by communicating with each other through trading, their expertise is pooled. HFT speeds this communication network, which improves the dynamic consensus.
As sibling said, it's liquidity. Worked at an HFT for a while, and they once got in a snit with an exchange over some policy change. They turned off their trading there one morning, and within 20 minutes, the exchange was on the phone begging them to come back.
Liquidity, which honestly has very little value to a traditional stock market. HFT and day trading both just exist as rent seeking behaviors on company valuation growth.
When You do this kind of (HFT) trading, how does ones tax returns look like? E.g. I assume you still have to report every trade with its gain/loss to a tax authority?
Depends on the nature of the trades. If you are hedging and maintaining a delta-neutral position (very common situation for market makers and arbitrageurs) then technically your trades are part of a straddle, which is reported separately from capital gains. (in the U.S., at least)
I learned this because I had a six-figure capital loss on my spot crypto trades, and a slightly larger gain on unregulated futures which do not fall under capital gains. Capital losses can only offset up to $3k in other income, so I was terrified that I’d have to pay taxes on the massive “gain” without being able to count the losses against it. Fortunately, the straddle is the correct way to report this, allowing gains and losses to be matched across instruments that may otherwise fall into different income categories.
> It was in that moment we realized playing in unregulated markets is not fun or something we wanted to continue to do. Intermediary risk was something we didn’t account for.
Indeed. This is the really difficult thing about the crypto space: winnings, if you can keep them. And you can't if the house is just going to front-run all your orders.
"we realized that there will always been a better resourced or more dedicated team willing to fight you for your alpha"
- that's part of what makes financial markets such a fun and interesting challenge but agreed, intermediary risk at the timescales you were operating at in this kind of unregulated market is real and not fun
I'd love to hear what problem you moved to that you believe you can build a long term competitive advantage on if you can talk about it?
My partner and I had built broadbandnow.com and were looking for a new interesting problem to solve to get us fire up again about fun tech problems. This just scratched the itch. The team was running day to day of the other business while we worked on the HFT bot.
Shortly after we realized we either should exit or hire an exec team to run the business. After a few months of executive search we found an offer we liked and took it.
These days I’m interested in using some heuristics based on public data to help consumers make informed decisions about nursing homes and in home health care. Still early on this project but lots of data and not many people looking to do good in that market. Seems like a great place I can add value.
> Best we could tell someone was front running us due to an artificial delay for our account (delay between trades went to ~20ms up from our prior steady speed of 3-7ms) and/or a bunch of the trades in the orderbook were bogus.
I wouldn't be surprised, given that traditional HFT companies are building cryptotrading desks and they have a lot more capital to play with too.
You can't simply front-run someone else by having deep pockets.
The only way that is possible is if you gain privileged information about that person's orders before they actually hit the book. The most practical way to do this is to be the exchange.
In late 2017/early 2018 when their websocket was new I believe you had to be authenticated for either the initial order book query or the websocket subscription. Either way since we always locked our api keys to our IPs it wouldn't have mattered.
Thanks for sharing. Do you think building the strategy on another exchange such as Coinbase Pro or pursuing a strategy that wasn't as latency-sensitive might've yielded more success?
I ran a triangle trading arbitrage bot on Coinbase that was profitable at the end of 2017. But a few weeks later, it was losing money on hard to diagnose slippage. The number of “opportunities” also collapsed, which indicates that other bots were doing the same or that coinbase was doing this internally.
I pulled the plug. Tried to run it on Binance but the websocket only updated once a second, so there was way too much risk.
I spent the last year working fulltime on a system similar to the one described here.
I trade the top ~20 cryptos on binance. I use deep learning models (combination of temporal, causal convnets and RNNs) with heavy data augmentation. I built my own tooling for data collection, training, backtesting and live deployment. Having a data engineer background coming into this was hugely helpful: most of my time was spent manipulating data in some way (and not playing around with the models).
One of the most demanding parts was estimating spread/slippage costs and including it into the loss function.
Most of what the author talked about, I learned the hard way.
I'm now at the point where I ran some tests (trading small amounts) live on binance and the results are positive: I do manage to make small profits, but more importantly, the recorded live trades reflect very closely the backtest trades (for a given period). I'm currently scaling up my model and adding better monitoring / reporting / CI.
I'd be happy to chat with anyone having done similar projects or willing to exchange ideas.
I'd love to hear more about what kind of data augmentation you're doing. A friend of mine recently got a GAN to work for timeseries which is really interesting.
I've done a lot of work in the space and would love to chat - just emailed you :)
I use a supervised learning setup (although with a custom loss function).
The kind of data augmentation I do is adding different candles sizes. I validate with 5m candles, but I train with 2,3,4,5,6,7m ones.
I also sample more frequently more recent data. I train jointly with ~22 symbols, but in each X with those symbols, I randomly set some to 0, some I invert their price, some I invert time-wise. This helps generalization for some reason. I tried many kinds of noise, but what I described above is what I found to work best in my case.
I have a more ambitious idea to generate synthetic data using self play: have a bunch of agents trading one against another. This create new price data I can train the agents with, and repeat (this self-play training scheme would be similar to what DeepMind did with AlphaGo/AlphaZero). The issue with it is the need to tune the parameters exactly so that the resulting synthetic data is realistic enough that I can tranfer the agents to real data.
For example, during self-play, should you have only trading agents or should you add "retail traders" that buy during bubbles, "normal buyers" that buy only below, sell above certain prices, institutional buyers that randomly move the price a lot in a given direction. This is a lot of parameters to get right, and it's an optitization problem on it own. You could treat this a as two-fold optimization problem such as in this paper: https://arxiv.org/pdf/1810.02513.pdf, but it gets tricky very fast.
What lowers my concerns is that such issues might come once the bot is profitable, and by then I might have found other ways to raise capital (I'm doing a trading bot as a way to raise capital to create an AI research lab). Being able to setup a profitable crypto trader is a good thing to add on a resume or for personal branding even if I shut it down at some point. I'll be in a very different position by this point so it's a bit premature to be concerned about that now, although it's still somewhat of a concern.
On the positive side, I hope this scares the competition away.
This post is an exemplar of the crucial relationship between domain-specific knowledge and ML competency in the ML space. The bulk of the post is detailing the tricky ins and outs of trading, and overall the author gives the impression that they're broadly knowledgeable about stock markets.
Contrast this post with those you see with ML hobbyists who delve into medicine or fake-news and produce useless results testament to their lack of domain-specific competency.
Bingo. I've made this comment several times on Hacker News in the past as well and in my opinion it's the number one reason I've seen ML projects fail to have impact at companies it's deployed: the operators (typically higher level Math/CS types) simply don't understand the domains well enough and so frequently end up making absurd recommendations/suggestions (often to the detriment of other business areas).
The successful application of ML requires a deep understanding of the domain it's being applied in.
Domain knowledge is essential to almost any project that aims for eventual commercial success, it is quite rare than an outsider will come into a field, apply some ML and make a killing.
That is not what Simons did to make Renaissance Technologies successful. Simons cultivated domain knowledge long before he started a hedge fund, because he had an interest in trading and gambling even as a professor. He also hired people with financial experience.
The historical record overlooks the people he hired who knew a thing or two about trading, while fixating on the team of NLP scientists he hired from IBM. Likewise Simons wasn't initially successful in the very, very early years. It wasn't until the late 80s that the Medallion firm really came into its own.
Yes, I understand that the "financial naiveness" so to speak of RenTec is overplayed in the media, but my point is that superior domain knowledge wasn't what enables Medallion to win. And to be fair, most of the early years were before systematized quant trading.
Funnily enough I think the ML hobbyist problem is most pervasive in the "predict the stock market" domain. There was a post on HN a few days ago [1] that was overfitting the validation set and hand-waving away fees and spreads. The author concluded that "there was no subtle underlying pattern" because they failed to find one.
Great post! Very refreshing to hear about a) the honest level of effort involved in this type of endeavor, b) the amount of nonsense trading advice out there.
Maybe in a future post you could discuss the security and banking side of this in more detail? In the 6ish years I’ve played around with crypto trading (and I really mean play, nothing close to your level), I’ve had 2 exchanges hacked and lose all customer funds, another 2 had major security breaches causing days of downtime but recovered, and one site seized by the FBI.
Then there are the horror stories of banks freezing your account when you move funds in and out of exchanges. Luckily That hasn’t happened to me.
I bet you have some good stories and perspective on that side of it, I would love to hear it.
Author here. Honestly, I don't have a good answer. I spread my capital across enough exchanges so that if one runs away with it gets hacked it doesn't ruin me. It's just a risk I'm taking.
I'm also not trading much capital. Because the system is more on the HFT side, the actively traded capital isn't that high, and I don't care about losing it. Any profit I try to get out of the exchanges regularly. I wouldn't feel comfortable leaving large sums on those exchanges.
The banking side is becoming more mature, I think, as many exchanges like Coinbase provide custodial cold-storage options for institutional clients.
Counter-party risk always exist.
> Then there are the horror stories of banks freezing your account when you move funds in and out of exchanges.
Depends on the country. What happened to me is that a bank did not freeze my account. Instead, they simply reported it to the government, and asked AML questions regarding the transfer. The government, on the other hand, wanted me to provide bookkeeping records. Otherwise, they were going to assume that every transfer coming back from cryptocurrency exchange was pure profit.
Basically, I was not raided, my accounts were not frozen, but the government knows my wallet addresses (and I had to pay back 4 years worth of cryptocurrency trading profits with interest applied, which also left me realize how little I had made profit in the end).
Extra warning: ensure your country allows individual cryptocurrency investors to reduct losses from winnings. Without such law, if you win 100 dollars and then lose 100 dollars, you would still owe the government taxes while you are at 0. This is the case in surprisingly many countries.
There was a "fun" post on reddit with someone that made a fortune out of bitcoins a few years ago when there was that big surge, who then managed to lose it all via some poorly thought out cryptocurrency trading. He'd found himself with an enormous capital gains tax bill he couldn't pay off. It hadn't occurred to him at all that the profits on bitcoin trading would be taxable.
The only caution I took away from that post is that it's very easy to make mistakes applying ML to financial markets if you don't know what you're doing.
There looks like a lot of overfitting the validation set going on in that post.
It's also a mistake to conclude that "there was no subtle underlying pattern" just because the author couldn't find one.
Throwing XGBoost at a bunch of technical indicators isn't gonna cut it but I have had some solid real-world success (as have several people I know) applying ensembles of deep learning models (with regime switching based on model residuals) to profit from "subtle underlying patterns".
I’m impressed with this system, but I’m even more impressed with the author’s writing style. I’d love to see more technical posts written with this level of clarity.
Claiming a 4000% return while staying market neutral seems a little too good to be true.
First: those levels are insanely high, so the algo must be taking some absurd risks and have the worst sharpe ratio, or getting pretty close to being 100% accurate.
Second: if you can scale this across markets, and assuming the same return, that investment will turn into 12 billions in 4 years. I doubt that you'd write a blog post about it if you had found such a gold mine.
Naively using linear scaling on financial models provides zero guidance to how the model would actually perform... Scaling financial models is an extremely hard problem.
See RenTech limiting the size of their Medallion fund because it was getting too large to scale....
Is't it so that many of these strategies don't scale well? When you are in low volume trading you are collecting all the best trades but as soon as you go 10x you are affecting way too much
Back in 2017/2018, not at all surprising. I don’t see that kind of opportunity in the crypto markets anymore though, they’ve gotten a lot more efficient. Used to be able to scalp 1% on big price moves in altcoin futures at least once a week, now the prices move in lockstep with spot.
> Second: if you can scale this across markets, and assuming the same return, that investment will turn into 12 billions in 4 years.
Scalability and profitability are orthogonal. If it could scale indefinitely, you'd be right. But no trading strategy can scale indefinitely.
That doesn't say anything about whether or it "works", and it's not a reason to be suspicious of the results, in of itself. All successful trading strategies are capacity constrained.
I would love to know how this fared recently in the large sell-off.
What he says about some markets possibly being predicable rings true to me. But the article was far from convincing that the BTC market is actually predicable.
The natural assumption should be that the author was in the right place at the right time. Although he went through great lengths, I'm not convinced this is anything other than luck.
Author here. Let's assume you cannot got short (which you can't in most crypto markets) and can only make money when the market goes up. The best you can do is avoid losing money when the market goes down.
But there are no pure market downturns. On a daily scale the market may be down, but that does not mean that on a millisecond or second-scale you will only see downward movement. There is just an overall downtrend, but there is still almost the same upward movement to make money. For HFT systems, it really doesn't matter if the market, on a daily scale, goes up or down. There is no difference.
In fact, on many markets the system does better in downward trends. Probably because there is more liquidity on that side of the book, a bias that may come from certain market participants.
How do you maintain a market neutral portfolio without the ability to go short? Also I would be interested in hearing about your Sharpe ratio. Thanks! Great article!
You can't go short but you can still hedge your exposure by going on short on something very correlated - like Bitcoin futures. I do this manually depending on how much capital I'm trading.
I actually have no idea what my Sharpe ratio is, sorry. With the system constantly changing, data formats changing, exchange balance apis changing, and accurate monitoring already being a challenge in itself, it's very difficult to keep track of exact returns and historical data. I could probably calculate it if I spent several days (weeks?) trying to process all my historical trade data, but that would be a waste of time for me personally.
If you don’t know your Sharpe, how do you know you are actually making alpha? Without knowing your return and volatility, you can’t really evaluate the performance of your algo or know if changes are helping.
Ever try trading futures basis and spreads? I was killing it in mid-2018 in that space, although I got cocky and used my winnings to make foolish leveraged bets that did not end well.
Through hypothesis testing you can estimate the probability that this was due to luck is very low. Assuming that a monkey would have a 50% chance of profiting on a day, the chance of going a month without a losing day is less than 1 in a billion.
You have to make some basic assumptions to do any statistical inference, because if you don't then literally anything can be explained by luck. For example, even if the author did a follow-up post (which I'd love to see as well!) every year for 30 years and made money every time, it could still be "selection bias".
The number of monkeys required to match the author's results over a 12-month period is well over the number of atoms in the universe.
Depends on which null hypothesis you're testing. He told us that he rarely had a down day, and based on Bitcoin's volatility I'd say it's reasonable to assume the probability of it going up or down on any given day is a coin toss (regardless of the longer-term trend and independent of other days). Thus the probability that he's a day-trading monkey is 0.5^n, where n is the number of up days he had in a row. (since he had a few down days it's actually a binomial coefficient but the value is < 1e-80 anyway, far less than a conventional p-value of 0.05)
On the other hand, if we wanted to test his 3900% yearly return, we might assume that monkey returns are equal in distribution to Bitcoin's price and then test the hypothesis that he's a monkey via something like a paired t-test. The problem here is that we only have one data point so p-value is undefined, and due to high variance it would probably take about n=10 points to get something significant. The upside of this approach is that you can get a confidence interval for how much better he is than a monkey, instead of just a yes/no answer.
In any case, since the author has at least 365 data points, he probably has an extremely good idea of both a) whether he's a monkey, and b) how much better he is than a monkey.
IANAquant, but he said in the intro that he used a market neutral strategy, so he _should_ make money both when the broader market is going up or down.
It would be interesting to know though!
> The biggest edge probably comes from the effort put into building the infrastructure.
I feel like this should be in bold, but either way, I love reading that in these posts. In every way, from research to confirm your models are correct, to be able to trust real time trades, you need a solid architecture. This thought isn't only for trading remember, where it's the same in tons of solutions to problems. If comment readers have other examples, I'd love to hear them in responses.
I've played with writing bots before and this post hits on so many of the edge cases I personally ran into. I have never heard it this well explained before. Brilliant.
It's interesting they suggest the higher the timeframe, the noisier the time series, when to my understanding the opposite is typically found -- the lower timeframes exhibit a more random walk and the higher timeframes exhibit trending behavior.
Can someone comment on how taxes are handled when automated trades are made like this? It's something that seems wholly absent from the cost calculations.
Author here. Personally, I just don't. I tried doing it but it was too complicated. So I end up just hiring a tax accountant specializing in crypto, send them all the data I have, and pay a few $k. In case something goes wrong, it's their fault and they take the risk.
It is a big loss of your time and nobody will give you back that one. I suggest you to use your time in non zero-sum games, something that can create value for you and society. Now that you have some saving you can definitely afford it.
The next best thing of not doing it is to quit doing it now.
Disclaimer: I built a similar system in the past, took some gains and then realised the above. I then quitted to build a company.
I have been working on such a system and it is NOT a huge loss of your time. I've learned so much things in the past year, in market microstructure, in networking (infrastructure, protocols), cloud computing, cloud management (docker swarm, kubernetes), linux kernel bypass, distributed systems, data base, and I've read hundreds of papers on neural networks, gaussian processes, etc... If you are wondering if you should get into this, it is one of the best learning experience you will ever have.
It must be said that it is a lot easier to make money in a stock market that has had low volatility and no significant, prolonged dip in the last 5+ years. My own long-term investments have earned 20% return over the last 5 years with zero trading. I realize that this article is specifically about crypto, but trends in all markets is generally up across the board.
As someone who just finished a BSc in computer science and started a MSc in Financial Technology and Computing this post is really interesting to me, keep ‘em coming :D!
Author here. Perhaps if you already have existing HFT infrastructure and connections to efficiently trade on such exchanges. But such infra costs millions. If you don't have this, you're probably at too large of a disadvantage to find any alpha.
At least that's my understanding based on conversations I've had, I've never traded equities.
To be competitive in US equities HFT, you need an FGPA with 40GbE ports hosted in a server (which needs to power and cool the FPGA, and deal with the less latency-sensitive bits of your system). You'll need some storage as well.
That server needs to be co-located with your target exchange(s) matching engines, and connected via 40GbE. You might additionally want remote market data via mm-wave microwave.
You can probably put together a basic but competitive hardware setup for $70k or so, if you ignore redundancy, and you only need to trade a single market. More realistically, you'll need at least two, plus shared storage, and probably more depending on what markets you intend to trade on.
Then you have monthly costs: colocation for the server(s) ($5k-ish+), port fees for the order entry ($500-ish), port fees for market data ($500-ish), physical connectivity fees ($20k-ish) , cross connect fees for the connectivity ($500-ish), wireless connectivity fees, you might need roof access (more fees), market data fees (per exchange), memberships, and trading costs.
I haven't done this for a while, but it easily adds up to $100k per month or more.
So you need to be making quite a bit to pay off your infra, before you start thinking about profit. And your model will age pretty fast, so you'll want to be working on a few possible replacements concurrently.
Crypto is a relatively inefficient market. Equity markets are too efficient: there are too many smart players and alpha is very hard to find. Crypto is a little easier, though that is changing fast.
Low trading fees is what originally caused me to change gears from etf programming trading to btc programming trading in the early days. I imagine it is the same for most.
Author here. Actually, the system is mostly taking liquidity from the market so it's not even doing that :) Perhaps, "opportunity for others to create more liquidity"
Jokes aside, it's actually something I am thinking about a lot. Such systems don't create value, but they extremely intellectually interesting and I've learned a lot. You can say the same for many other projects, for example academic research in many fields. Most of it is just noise to promote the author and does not create value in the world. But it's intellectually interesting, so people work on it.
Other people write compilers for fun to learn something new without creating value. I don't think this is fundamentally different.
I've thought a lot about this too (I used to work in HFT). Here's what I think:
- The only part I didn't like in your article was how you described creating indicators as exploitation. The limit order book is public by design so all traders can look at it. People have the free choice to trade on a centralized exchange or not. This is a trade-off between revealing information and being able to trade quickly without calling all your friends asking if they want to buy some Bitcoin.
- I'm guessing you used data from other exchanges outside the one you were trading as indicators too. That's unquestionably good since your trading helped information propagate faster or more accurately than it would have otherwise.
- Markets are only zero-sum in isolation. Most participants derive utility from things outside short-term profit and loss. Maybe they trade to manage risk, to hedge, to gamble, have a longer time horizon than you, whatever. They just want to trade and get back to their lives. They don't want to waste time squeezing the last fraction of a basis point out of their fills. It's hard to believe, but they actually enjoy getting picked off, run over, paying too much spread, whatever things make you feel bad or indifferent about the service you provide.
I used to get filled making markets on Nasdaq (which pays resting orders a rebate, and charges crossers) when BX (which pays crossers a rebate) was at the same price, and could lay off the trade for an instant profit. The people who traded with me paid for the luxury of saying "fuck it, send it to good ol' Nasdaq." I used to think it was stupid of them, and from the perspective of a prop trader, it was mind numbingly stupid, but they probably had more productive things to do than read every exchange fee schedule or hook up to every small exchange.
- Providing liquidity has nothing to do with resting limit orders vs. crossing the spread. Providing liquidity is about taking risk off the hands of people that don't want it, and moving it across time to someone else who does. If you're market neutral, trade many round trips every day, and end relatively flat, you've played that intermediary role as a liquidity provider regardless of what order types you use.
- Crossing against mispriced orders is doing the world a favor. You're not the bad guy picking them off. If anything, they're the bad guy for holding the market at an incorrect price.
So maybe think of yourself as more of a service provider. Not only will you feel better, but viewing trading through that lens tends to make you a better trader. Strategies truly built around an exploitation mindset are fundamentally unsustainable, since you run out of people to exploit. Providing a service works forever.
FWIW, the rest of what you wrote is almost exactly how the pros do things. If you built this system yourself, you could make far more than 200k at a prop firm. If you're interested, reply with a throwaway and I can refer you to a friend who's still in the business.
Ultimately all investors from the hedge funds down to the long sucker, it is all skimming for returns. HFT just skims from other skimmers returns. Yes those investments may have created value, but ultimately people invest because they want returns.
The money made will be used for more skimming but also some for investment in other possibly beneficial projects.
Extracting financial value via investment (skimming) may allow someone to start a company, support a community or family and have valuable time to make those things better. Just as an investment in a company/idea creates value in that company, the value gained from trading where the first step really is skimming, can be used to create value in the real world or even just time which may lead to more real actual value created.
But you're not discovering anything valuable because there may not be a person on the other end of a trade. You're just learning about the chaotic boundaries of the trading algorithm.
No, your interpretation is incorrect, and that's not how this works at all.
> There may not be a person on the other end of the trade
uh... what? If the executed order fills, then there was someone on the other side of the trade.
Anyways, by providing liquidity in any market as a market maker, you are effectively aiding in the overall process of price discovery. HFT shops are generally market makers, although other strategies are also possible depending on how the operation intends to generate alpha. For HFT MMs, they pretty much only make money by clipping spread, i.e. submitting dual buy and sell orders at the midprice with the expectation that they will make the (ask-bid)/2 on average. They then cancel these orders as the orderbook's structure changes and as prices and markets move, resubmitting at "better" (more favorable) levels.
It doesn't matter if people are buying or selling or both. HFT MMs provide a valuable service to financial market participants - if participating counterparties submit orders and cross a HFTs latest uncancelled order, it will fill, allowing market participants to quickly gain or lose exposure to their security or instrument of choice.
There's no magic here and you seem really confused about the underlying dynamics of trading and market microstructure.
Exactly. Regular speculators at least have the (hazy) claim that they're reducing the spread for people trading markets with actual utility. But given BTC's lack of economic meaning, that doesn't work here.
If anything, providing BTC liquidity might just be increasing net societal harm.
I feel like commercial activity finds value and removes it from the system.
To create value, you need to give someone else money. If I wanted to build a better society, I'd make sure everyone had the best possible education. Paying for this would bankrupt me, maybe even bankrupt the entire country. But with every single person in the country walking around with a deep understanding of music, art, mathematics, engineering, and science... as a society, I'm sure we'd do great things. Value would be created in the very long term, but not for me, the potential investor.
Then on the other hand, we have things like automated trading. That boils down to asking a bunch of people "will you pay $5 for this $4 bitcoin?" Anyone that says "sure!" just gave you a dollar. Do that millions of times per second, and you remove as much value from the system as possible.
The reason why you feel this way is because there is a large difference between value† as defined by markets (which I'm writing with a dagger †) and value as defined by everyone else (which I'm writing without a dagger). Value† is weighted by wealth. Value is not. Feeding a starving African village generates value, but feeding a starving African village does not generate value†, because the village has no money. Figuring out how to monopolize the telecom industry generates value†, because it benefits heavily-weighted interests while only penalizing lightly-weighted interests, but it does not generate value.
Markets promote value† creation, but they don't promote value creation, except insofar as the two concepts coincide. It is instructive to think about the circumstances under which the two concepts coincide, the extent to which those circumstances hold, and the extent to which we are moving towards or away from those circumstances.
For what it's worth, I use "makes money" for creating value†, and "creates value" for the common sense.
For those interested in the topic, I've found the Lean Manufacturing literature helpful. They're usefully obsessed with creating customer value, meaning doing or making something valuable for their customer. Which has some of the problems you mentioned, in that traditional commerce mainly serves people in proportion to the money they have. But you can pretty easily extend the Lean approach to value to non-commercial situations.
In our case we were doing triangle trading between BTC/ETH/USDT pairs and had our buys/sell delay down to 3-7ms. At one point moving 0.3-0.7% of Binance’s daily volume.
Few notes:
* Finding an objective point of truth for value when all of the currencies are floating is hard but vital to success. This was the hardest problem we encountered. We tried taking the realtime average of BTC and ETH across all exchanges, we tried tying it to the shortest route to USD, and several other routes... but ultimately this is where we ended up “losing” most of our alpha.
* Order books are seemingly simple but the devil is in the details. This especially matters for paper trading.
* Efficiently using API limits at exchanges is an optimization problem in and of itself.
* Our model was relatively simple but we focused on speed and edge cases. For instance Binance would rotate IPs on their load balancers and we’d constantly check the latency between each open SSL connection and use the fastest. Further we wouldn’t decode the buy response to plaintext we’d just read the raw stream.
After several epic months our entire project fell apart after a cryptic phone call about “institutional access” that didn’t follow the 1s websocket update. The access was quiet expensive and we said no to it and shortly after all of our strategies went to crap.
Best we could tell someone was front running us due to an artificial delay for our account (delay between trades went to ~20ms up from our prior steady speed of 3-7ms) and/or a bunch of the trades in the orderbook were bogus.
Frustrated we tried our strategy on another account and the delay dropped again to our normal range and was profitable again (the orderbooks were slightly different between bots!).
It was in that moment we realized playing in unregulated markets is not fun or something we wanted to continue to do. Intermediary risk was something we didn’t account for.
Further we realized that there will always been a better resourced or more dedicated team willing to fight you for your alpha.
After months of effort and a ton of fun we decided it was best we went back and focused on a problem where we could build a long term competitive advantage.
Edit: typos and formatting