It's an interesting idea, but most successful retail algo traders probably won't trust their bread-and-butter algos to a cloud-type solution. I've been algorithmically trading for a few years now, and I've invested money into Ninjatrader, where I program my algos and run things from my own computer.
If anyone had their own algos, they would probably be too paranoid that Quantopian would run backtesting on every single algo, and cherry pick the best ones for themselves. Whether or not it's true doesn't matter, it's most likely the common thought process that any successful algo trader would have.
That would leave only the inexperienced and beginning traders that would be more apt to fail, since algo trading is very, very, very hard.
We also believe setting our own incentives to match our members' needs is key to sustainable trust. That's why Quantopian doesn't invest its own capital - we want to be purely focused on and motivated by serving quants.
Being a successful quant takes many things - talent, mentorship, access to data, great systems, discipline. We started with backtesting because many aspiring quants end up skipping rigorous backtesting because of the time necessary to develop the test harness. We focused on community from the start to help connect new talent with mentors.
If you're not ready to trust us with your algorithms, I hope you'll trust us with a bit of your time. Come share some of what you've learned over the years with our community.
Keep in mind that Getco is one of their investors. That is serious conflict of interest! These unsophisticated retail users are going to be eaten alive.
Getco is, by all accounts, a money machine. I don't think they're gonna throw it all away by doing something pointlessly illegal like front-running hobbyists. Or maybe I don't know what you're getting at.
I saw this as a way to dig into the order flow business, but it seems they don't have their ducks in a row -- I would have waited for the production trading setup first (which I presume GETCO would provide via OCTEG LLC) before getting PR coverage
I'd argue that a lot of the hard parts of algo writing are solved by Quantopian. Hard:
* Data. You need to test your idea. Most historical stock data (like Yahoo) excludes companies that went bankrupt or were bought or otherwise disappeared. That's called survivorship bias. If you run a backtest on the finance industry and you don't include things like Lehman, you're going to get the wrong answer. Add in things like Hurricane Sandy, MLK Day, 9/11, mergers, acquisitions, stock splits, etc. and data is very painful to put together.
* A backtester. Once you have you data, what do you put it into? How do you calculate commissions? How do you calculate slippage (your order affects the price, remember)? How do you avoid look-ahead-bias and other bugs that plague backtesters?
Coming up with an idea to trade is hard, but it's only a part of the problem. I'd say it's the most fun part of the problem, but it's only a part. Quantopian is trying to remove all of the hard parts and let you do the easy parts. We have tens of thousands of lines of code (backtester, IDE, etc.) and we're leaving the most exciting 100 lines of code to our members.
I would describe data collection/mangling/processing as more annoying and tedious than "hard". I'd seen the Quantopian github account before, and appreciate what ya'll have done. I didn't know there was a business model behind it, but I get it now.
You said it solves "the hard parts of algo writing". I guess that's true if speaking of algos in a general sense, it makes it easier to get started. But if the goal is to write profitable algorithms, then its much less true. I just looked again at the API, and using it to create a profitable strategy would actually be more difficult (and less profitable) than it would be to trade elsewhere.
The main reason is because the function to make a trade will only place market orders. Its well known (in the financial literature) that an algorithm which trades with limit orders will almost always outperform one which trades with market orders (pretty much obviously true since the limit order is the better price). That's because where a passive trading (market making) algorithm is earning the spread, with market orders you have to beat the spread just to break even.
Of course, that's why many brokers only allow market orders (they collect the spread as their trading fee).
Also, I don't see how the backtester avoids look-ahead bias. Backtesting like that just encourages Data Dredging[1] an algorithm overfitted to the test set. It would be better to backtest on random subsets of the data (cross-validation and all that). But I can understand why that isn't done (would be harder for users to create should-be-profitable algorithms and start trading with them).
The main reason is because the function to make a trade will only place market orders.
We (I work for Quantopian) will surely support other order types when we support live trading.
Also, I don't see how the backtester avoids look-ahead bias.
We provide over ten years of historical minute bar data for U.S. equities, with no survivorship bias. This means two things:
1. The amount of data we provide is sufficiently large that if you test your algorithm against a bunch of stocks over that entire period of time and it performs reasonably well, it's unlikely that it's overfitted to the data in a way that is going to bite you on the ass in live trading.
2. But just to be even more paranoid, the smart algorithm writer will do just what you describe -- divide the available data into lots of subsets, randomly pick which subsets of the data to test again each time you backtest, and don't start live trading an algorithm until you've confirmed that it performs well on random subsets of data that you haven't previously tested it on. Right now on Quantopian you'd have to do all that data segmentation and selection by hand, but I suspect that we will eventually add features to make it easier to do automatically.
> We (I work for Quantopian) will surely support other order types when we support live trading.
I suspect they won't be true limit orders, but stop orders (true limit orders wouldn't slip). Also, for a passive trading algorithm its important to have level II data (the order book), or the algorithm is flying blind.
Re 1, its not the size of the data set which prevents overfitting (larger data sets actually make it more likely), but the use of it when developing/training the algorithm. Repeated testing and tweaking the algorithm will overfit, unless one is careful to maintain the complexity of the algorithm.
Anyway, best of luck to your users! But they should be warned that active trading on signals (price prediction) is the hard way to algorithmic profit. That's why market making and arbitrage is the bread and butter of the pros, not signal trading.[1]
In that video the quotes to which I guess you are referring are "[investing] is almost entirely a non-professional role" (08:40), "[Market-making] is a highly professionalized role" (09:13), and "[an arbitrageur is a] highly professional role" (11:22).
I thought the video had something in support of your "price prediction is the hard way to algorithmic profit" statement, but I can't find anything very explicitly in support of that (what I thought was your implication). He kind of implies something like that at 15:00 when he says "we have no idea how [price prediction] works [at longer then a few days in the future", but that's not really very strong.
When you said "active trading on signals [...] is the hard way", did you mean active trading at minute-to-day or greater holding periods? That seems a) right :); but b) slightly at odds with your "it's important to have level II data", since I would have thought that is less important at a hourly-to-day or greater holding period.
Backtesting is only 1/2 to 1/3 of the actual task of algorithmic trading. When it comes to actually trading money, if you're not careful and cover every corner case, you really could lose your shirt. I can't believe that 100 lines of code is all that you would need to cover all the edge and failure cases that could lose yourself a lot of money.
What happens if the network drops while you're in a trade, or in the middle of executing an order? What if you have a partial fill, and you have a partial buy order hanging around and a full sell order out there? What if you have a stop limit order to exit a trade, and it blows through your limit? Do you have a backup stop order just in case? There are a lot of issues that you can't backtest that can only be learned once you start trading real money. I've had situations where I ran my algos on the DAX overnight, and I woke up to find that the exit order never executed, and I was 1000 euros away from where I should have gotten out. Luckily, this trade was in my favor, but it scared the shit out of me because it could have easily gone the other way. I've also had my internet connection drop overnight, and I had to scramble to figure out how to get out of a trade I was in.
The other issue is interpreting backtesting data, and knowing the difference between over-optimized data (ie. curve fitting) vs something with an actual edge. You can make almost any algo profitable if you curve-fit, even a simple MA cross-over can show extremely profitable results, if you over-optimize the data. But it won't work in real life. So being able to sift between falsely good algos and actually profitable algos is very, very hard, and takes experience. This is the biggest problem with trying to find an actual algo with an edge, it's very very hard.
I wrote my own backtester and I download my own data nightly, and that definitely takes a lot of time and effort, but the hardest part is the actual trading, by far. The psychology involved with trading is an order of magnitude harder than coding, and the hardest thing I've ever done in my life. I blew through a shitload of money, just to learn the ropes.
In terms of algo trading books, I really don't have any recommendations. I found most of the algo trading books are similar, telling you to watch out for curve fitting, etc. Where they lack is helping you come up with actual trades. My recommendation is a book called "Mastering the Trade" by John Carter. He gives out trade setups that he actually used. They may no longer be profitable, but it's the closest thing you can get to actually learning various day trading techniques, and you can implement those to get an idea, and then work your way from there.
I'm not hating on Quantopian, it looks pretty nicely done, and hopefully it takes off. But from experience, I know that algorithmic trading is by far the hardest thing I've done. If inexperienced traders jump in, they'll spend a lot of money on "tuition" for sure.
Try the paper "The Winners and Losers of the Zero-Sum Game" by Lawrence Harris. I found it most enlightening. I'm paying it forward, since it was here that I saw it recommended (hat tip to tptacek iirc).
What makes a zero-sum game so hard? Competition. Read Nate Silver's book chapter on Poker for a great explanation of that. In summary: he won a lot of money playing online poker when the services were growing and there was a large supply of fishy players (losers). When the growth of new poker players stopped, and the fish had mostly left (after losing their money), then he started losing.
2) If you're talking about chargebacks, it would be YOU that the buyers would be charging back the money from, not the warehouse, since you were the merchant that accepted the payment.
And am I right in that you are basically saying that the buyers take on all risk of storage at the warehouse facility? If so, you should probably outline that as well, including what the insurance policy of the warehouse, covers, etc.
I should have been more accurate - the bank would be liable if the customer charged back and couldn't reclaim the money from Vinetrade (e.g. we'd ceased trading).
There isn't huge risk for the buyers - all wine in the warehouses is insured at full replacement value. It's more of a hypothetical "what if" or worst case scenario.
This is an extremely important message. Fraud is very easily the thing that can sink many small companies that aren't used to the levels of fraud that you will see on the Internet. And because there are so many ways to defraud a merchant that isn't obvious, merchants need to be extremely careful and educated on the process. And handling chargebacks will become a full-time job for someone in your company, once your company becomes big enough. I work at a payments company, and fraud is something that ends up being a headache, although we are at lower-than average industry norms.
It seems relatively easy for someone to turn credit card payments directly into cash, by making a trade and then extracting cash through the bank account. And the gap between your withdrawal policies and the chargeback policies of credit cards are pretty wide. I would be very careful about this. Forcing users to enter bank accounts make it a bit harder for average people to defraud, but not a motivated fraudster. Also, allowing people to take delivery of wine through the purchase of a credit card, and then being subject to a chargeback will be extremely painful, although I'm not sure how easily chargebacks are fought in Europe. If you do suffer a lot of chargebacks though, even if you successfully fight them all, be prepared to be dropped by your credit card processor.
I think your business is actually very interesting, but I'm surprised that credit card companies would allow you to accept credit cards at all, since I would think they would lazily lump you in as a form of brokerage site. Not that I believe this, but it seems like an easy translation to stock brokerages, which don't accept credit card payments, but force you to transfer real cash into your accounts before you trade. Then once you have money in a brokerage, they can extend margin, but they maintain their own risk by monitoring the margin levels, etc.
Good luck though, I read through your site and I think it's a very neat idea!
Please don't try to twist your dumb rant into some sort of free speech issue.
The multiple people telling you to shut up isn't because they are trying to squelch discussion on the topic. They are telling you to shut up because you, specifically, bring negative value to the discussion and you are responsible for decreasing the signal to noise ratio.
In your example about the dumb, drunken football fan yelling at the TV, the people in the bar are now starting to tell him to STFU because they want to watch the game and not have to listen to the useless, drunken ranting of someone with zero credibility.
What? I don't agree at all. I wrote something in my personal blog on the Internet. That's the bar. If you want to tweet, or write something in on your blog, that's also the bar.
But this is a kind of club where I didn't post those writings. And when I have discussed them, I've been civil as is our rule here.
We have two ways of saying "This is noise, not signal." One is to downvote. The other, flagging, is reserved for special cases."Shut up!" is not one of those ways. And if you count, my words here on HN have over 100 votes.
If you don't like what I have to say, argue with it and/or vote on it. That's what we do here. This isn't Reddit.
The core problem is that 100 more people on Hacker News think that your top comment is valuable, and further that you take this as justification that your viewpoint is supported by the community. As there is no way for folks to only participate in the HN that disagrees with your rants, folks are being negative. That's the only tool they have.
When you sell books on how CoffeeScript is great and then rant that people aren't solving difficult medical device issues, I wish that there was a version of Hacker News that did not include your comments. Maybe someone will make that.
You're being incredibly rude and unfair. The guy wrote a blog post on his personal blog. It gets posted to HN and gets massive criticism. And now you're complaining that he posted a response to that in the comments? If you don't like the voting behavior, add something valuable to the discussion rather than slinging empty complaints and repeating ad hominems. e.g. http://news.ycombinator.com/item?id=5090255 and http://news.ycombinator.com/item?id=5089296
Rude? Maybe. Unfair? Hardly. The OP completely mischaracterizes what the best groups in our industry do, then laments that people aren't helping his friend, and this lamentation is the extent of his contribution to his friend's plight.
Ad hominem means that I disagree with him because of his person, which is not true. I disagree with him because of specific hypocrisy and finger pointing. I am further disappointed that this story is so popular on Hacker News, because I read Hacker News for insightful posts, and this is just demagoguery - popular because it's what people want to hear. That doesn't make him more wrong, but it does make me more likely to say something about it.
pg is good about handling people with kid gloves. He addresses the OPs points in a rational way that is insightful and without reproach. But I don't think that pg speaks to the problem, which is that a popular, long time contributor to the community is apparently upset that people work to make money and that some things are difficult for non-technical reasons. Bad apples ruin a community, and if the community isn't tended to it devolves into blithering noise.
I also did not comment on his original post, but on a comment thread that began with:
That being said, sometimes a man in a saloon has a few drinks and yells at the television, telling the coach of some football team what to do next. Just because he's drunk and in a saloon doesn't mean he's wrong, just boorish.
And any drunkard yelling at a television is wrong. His non-participation means that he doesn't actually understand the problem, and without that he cannot be anything other than wrong.
"disagree with him because of hypocrisy and finger-pointing"
Neither of those are ways to disagree with someone's words. That's the Ad Hominem fallacy. What you may mean is that you dislike the fact that I'm saying them, or the way that I'm saying them, or the color of my tie, or something along those lines. Which is a perfectly valid feeling to have, but it isn't disagreement and Hacker News has specific guidelines discouraging incivility as well as a long-standing antipathy to Ad Hominem abuse.
This kind of response further devolves the discussion. I said nothing so flippant, and suggesting so is disrespectful to me.
Neither of those are ways to disagree with someone's words.
Hypocrisy shows that someone either doesn't understand a problem, or that they are not honest in arguing against it. Finger pointing shows that they want someone else to solve this problem for them, which suggests that they feel superior to the issue.
Hacker News has specific guidelines discouraging incivility as well as a long-standing antipathy to Ad Hominem abuse.
To paraphrase your original article, Why the fuck do people on Hacker News always cry "ad hominem" and point to the guidelines so much? Ad hominem attacks are far more common in real life, when people actually know each other. I am only aware of you insomuch as I have read "your words", so why are you so quick to assume that I disagree with you personally?
Your original post is either naive, flippant, or blinded by emotion. It isn't constructive, it denigrates the work of others, and childishly demands that the world be a better place.
Don't lean on ad hominem, that only serves to mask that there are far bigger problems on the table.
> "Hypocrisy" is one of those empty criticisms, like "Unprofessional." If someone says to you, "smoking is bad," it doesn't matter whether he smokes. Maybe, his advice is actually more relevant if he's an older fellow who smoked and now regrets not making a different choice when he was your age.
It's not an empty criticism. It speaks to your credibility. You are admonishing Google and Facebook for working on "stupid" ideas, when you yourself are working on a presumably stupid idea, since you're not saving anyone's life but instead working on a way so that you can make money. How credible can your rant be if you aren't following your own advice?
I would rather hear the same rant from someone in the field who is frustrated because they are "fighting the good fight", not from someone who is engaging in the exact same "stupid" behavior that they are ranting about.
No, he's 100% right. This irrelevant "hypocrisy" charge scares people into pointing things out that should be pointed out. So what if someone admonishes Google/Facebook for working on "stupid" ideas while working on an even stupider one? Wouldn't that put said person in a better position to know about working on stupid ideas?
Crying "hypocrisy" is just the "cry wolf" fallacy.
There is a perception difference, for sure, but an argument's validity is not reduced by the person saying it. Granted, here it's much more opinion than fact, so...
No, but the likelihood of the argument having any value is greatly reduced if the person themself doesn't believe in it enough to actually follow the argument.
I doubt there is a high likelihood of a meaningful conversation from someone who is screaming "Why doesn't Google stop working on stupid problems and fix this, while I continue working on my own stupid problems".
>No, but the likelihood of the argument having any value is greatly reduced if the person themself doesn't believe in it enough to actually follow the argument.
Total nonsense. So a drunk who's destroyed their own life telling some teenagers about the dangers of alcohol greatly reduces the likelihood of his argument being valid since he still drinks? Ridiculous. Crying "hypocrit" is just ad hominem. Nothing more noble than that.
It isn't the argument's validity, but the source's trustworthiness. These are related but not the same; the source's trustworthiness is a contributing factor to whether or not I want to spend (waste?) the time verifying the validity.
Oh, nonsense. It's entirely about the argument's validity, and this bit about the source's trustworthiness is a red herring. Is this "source" claiming to reveal secret information about Google's being a big ad business we can't verify for ourselves and that would change our minds if we decide that he can be trusted?
Nothing of the sort. He's expressing an opinion about things that nobody disputes. Calling him a hypocrite means you don't like his opinion and think that calling him names is a good way to weaken his argument. Of course it's meaningless and defending such silliness on the grounds that this is all about the credibility of an information source is beyond silly.
I agree with mikeash here. If a hellbanned person still wants to remain part of the HN community, it's really not that hard to create a new account. Sure, sometimes they get it wrong, but the vast majority of the case, they get it right.
That means, that by you turning on dead posts, you are subjecting yourself to the 80% case where they get it right. If one of them is spamming with huge blocks of text, etc... well, you signed up for it in the low-probability case that one of the dead posts actually contained valuable information.
Keep it simple. Don't start adding multiple layers of hellbanning protection just to fix a low-probability edge case.
A very good open source package for handling date-time is ICU (http://site.icu-project.org/) which handles a lot of globalization/internationalization issues.
I think HN has the highest quality comments of any tech-related forum site. There are consistently tons and tons of great comments from people that are very well-informed.
Reddit sometimes have a lot of very great comments, because of their reach they can get comments from a lot of experts on a wide variety of topics. But it also has a lot more noise, and a lot of "joke" posts, something that HN actively discourages, and which keeps the signal-to-noise ratio high.
EVERY forum is going to be filled with trolls and assholes. HN certainly has its share, but I would say for the most part, the signal-to-noise ratio is the highest. This is the reason why I left Slashdot after being a member for so long.
I only knew Aaron Swartz from the things written about him on the Internet. But it really feels like he had a lot more to contribute to this world, not just technologically, but socially and philisophically as well. His death to me feels much like when Kurt Cobain died, back in 1994. Both were artists who had much more to contribute, but their lives were cut much too short by mental illness/depression. It's sad.
Kurt Cobain died just a few years before the Internet really took off. I always imagine that he would have embraced this, and would have done things like released his music for free, and would have radically changed the way music was distributed. In a few years, there will be even more technology invented, and it could very well be that had Aaron Swartz stayed alive, he could have found other ways to contribute new things as well.
Whenever I've been in NYC, I've never had a problem getting cabs. Going to Newark airport however cost me something like $110 if I remember correctly, because the cab driver gave me these special "fees" at the end, it felt like highway robbery.
Now in SF, it's the worst cab situation I've ever seen. I've had cabs that I ordered via the phone never bother to show up, and once waited 90 mins in front of a hotel, as the hotel employee tried to flag taxis as they passed by the line. I don't bother taking the cab anymore. If I need to be somewhere for a day like New Year's Eve, I'll park downtown earlier that afternoon and then take the Muni there.
I just might try out Uber this New Year's Eve and see how that goes.
> "Going to Newark airport however cost me something like $110 if I remember correctly, because the cab driver gave me these special "fees" at the end, it felt like highway robbery."
It is. There is no flat rate to Newark airport, so you can either take the metered rate or negotiate your own flat fare. Bonus of the custom-negotiated fare is also that customarily you don't tip on it.
There's no good reason why going to EWR from Manhattan should cost $110 (short of complete and utter gridlock). Insist on negotiating a fare, or just call a car service - their vehicles will be nicer, the driver will be less surly, and the price will be substantially lower.
If anyone had their own algos, they would probably be too paranoid that Quantopian would run backtesting on every single algo, and cherry pick the best ones for themselves. Whether or not it's true doesn't matter, it's most likely the common thought process that any successful algo trader would have.
That would leave only the inexperienced and beginning traders that would be more apt to fail, since algo trading is very, very, very hard.