Like others have said i think this might already been explored. However, I like the view that you have to let the machine to be able to 'do nothing'. Which is what resnet was first for. But i'm willing to bet this so called wisdom can still be squeezed for performance, because people tend to not think about it.