Read this article and slowly re-implement the accompanying code: https://karpathy.github.io/2015/05/21/rnn-effectiveness/ It's difficult and tedious but also very useful to implement back-propagation by hand.