Except that it's not that easy. Your body will conserve energy if you're hungry, making you feel colder and more lethargic. If you go without a sweater, you'll burn more calories and end up eating more.