Well, that is cool and not what I would expect. That said, for practical applications make training easier using leaky-relu, tanh, etc. activation functions.