you take the data out of the hose and run a bunch of "machine learning" over it to figure out what twitterverse is saying about apple. or rather, apple stock.
if you can guess the opening direction the next day better than 50% of the time i'd say you would have some money on your hands.
we're actually doing something similar to this, but have been tracking a number of stocks as well as other things.
twitter + sentiment analysis = prediction market; but in most observable cases, we're seeing twitter lag behind the market, not actually predict it. oh well!
Our internal data is kinda one of the things we curate and I can't really give that away, but I do have access to an academic corpus that might be redistributable. I'll have a look. :)
Lets assume Information propagates according to some diffusion model, forward only in time, and after some delay affects the price of a given stock. Twitter is the perfect feed to match against high frequency stock data to measure each stocks sensitivity to particular kinds of news (location, keyword)
So you build a weighted model of what kind of news affects which stocks, and how strongly.. and do algorithmic trading on that statistical arbitrage, and exploit the market not being perfect [thus improving market efficiency].
The engine you get is quite similar to one which acts as a real-time agent looking for twitter activity on topics of interest to you...
But, if an algorithm can find this mispriced asset, and trade to exploit it before a human can, then that should improve the markets efficiency and reduce the arbitrage.. or so it goes.
What is the lag of twitter compared to paid news services? Probably enough to make investing in building such system totally unattractive compared to aforementioned paid feeds.
So you'll have to make your trading decisions based on sentiment or even insider info... Very risky and not very profitable proposition...
Not that twitter is totally useless for trading, but not very useful for HFT uses you describe.
you take the data out of the hose and run a bunch of "machine learning" over it to figure out what twitterverse is saying about apple. or rather, apple stock.
if you can guess the opening direction the next day better than 50% of the time i'd say you would have some money on your hands.
i kid, i kid...