People who do hard work tend to bite off more than they can chew, especially at MIT, and there is obviously a limit to how much you can handle. What MIT does right when it comes to stress relief is its housing system. You get to pick where you want to live, and you mostly get your choice (or you can move eventually). That way you can live within a supportive community. It really makes a difference during those hell weeks.
+1. Columbia is the same way; you're surrounded by talented people who don't need to be stressed, but they choose to be. A lot of people try to do everything (which many are used to from high school) and most find that it's too much too late. After a while, though, you find about the right balance. For most people, this happens second semester senior year (:
It is online learning, but as far as I understand, there is another side to it. Right now, if you get a new data stream, you have to play around to build the right models for it (mostly manually, with trial and error) before you do online learning. As the number of data streams grows, this doesn't scale well. Their algorithm adapts online to the data stream without you having to design any model.
You bring up a great point. To classify something as a trend or not a trend, we have to use this artificial black box to supply ourselves with examples of what's a trend and what isn't. The nice thing, IMO (and this is something I admittedly gloss over at the very end) is that doing prediction/forecasting with this method is almost the same as doing classification, even though you don't have any labeled examples when doing prediction. To do classification, we compare current activity to past examples of activity, and decide if it looks like the positive examples or the negative examples. For prediction, we compare current activity to past activity, and see how similar-looking past activity continued to evolve over time.
But I think what you are asking is how such a method would come up with its own trends, given just a stream of tweets. This is a supervised approach (http://en.wikipedia.org/wiki/Supervised_learning), so for now, you would need to train it (possibly online) by giving it examples of what should be a trend and what shouldn't. It would be interesting to make it semi-supervised (http://en.wikipedia.org/wiki/Semi-supervised_learning) so that you would only need to provide a small number of labels.
It sort of comes down to the question of what's really being learned here? Are they modeling some inherent process of topics becoming popular (or memes spreading in a population) that could be used in other situations, or are they just modeling some arbitrary algorithm that twitter uses to mark some topics as "trends"? If they're just modeling twitter's existing algorithm, then it's less interesting because that algorithm already exists. Since they're able to detect the trend before twitter does (well, before twitter announces it anyway), then it seems like they're probably onto something more fundamental.
It sort of comes down to the question of what's really being learned here?
That's a great question. We are learning to recognize trends and non-trends based on previous examples. Since the Twitter trends algorithm gives us such examples, you could say we are learning to replicate the outputs of an arbitrary algorithm --- and you'd be right. But learning from examples is a very general thing, so the method has applications beyond detecting trending topics.
Are they modeling some inherent process of topics becoming popular
No, we don't model the process of something becoming popular. (To do this, one might suppose people spread popular topics in X way and unpopular topics in Y way, and try to estimate from the data whether the topic is popular or unpopular.) The beauty of this is that we never have to build a model, because we rely directly on the data. As a corollary, this approach is applicable out of the box for any domain with time-varying data (though I suppose you might have to take care to measure the right kind of time varying data).
I agree with you --- this seems like a perfectly good way to tackle the problem of trend prediction directly. What we had in mind was something that would be more generally applicable to any kind of time series data, and we figured it would be interesting to test it on Twitter trends.
Thanks for the interest! We collected tweets by sampling a small percentage of all tweets in a time window, to emulate what one might get from the streaming API (I did this as part of the VI-A http://vi-a.mit.edu/ masters program at MIT, and was an employee of Twitter). We did pick a fixed set of topics to track, but those were randomly sampled (though we did get rid of topics that trended multiple times in a large time window, like the name of a football player who scored in multiple matches, in which case we don't know which event we are trying to detect). One thing the algorithm doesn't do at the moment is come up with its own trending topics. It just tests prediction of trending/non-trending on a hold-out set taken from the original set of topics.
My problem is often in the other extreme. I tell myself "I don't want to bother so and so, I still haven't tried x." But there's a point at which you should ask for help if you're no longer making progress. Especially if your question is expected to have a relatively quick answer ("take a look at y and do z") that you just don't happen to know yet.
