For me it's the jarring noise. A coffeeshop tends to have consistent levels of muddied noise, whereas the office might be quiet and then suddenly a clear, loud voice starts, and another couple of clear, loud voices arrive - jarring. Also, office lights are too bright, they switch me off somehow. Then the erratic temperature - freezing for a couple of hours, then tropical for a few, then back to icy. The office is, ironically, the worst possible environment for getting any work done.