I work on neural nets back in the mid 1980s, so I was somewhat familiar with their structure, backpropagation, squashing functions, etc. I wanted to get back into them, so I started by rederiving all backprop formulas by hand here https://github.com/tbensky/NeuralNetworks/blob/main/Backprop....

Then I wrote up a NN from scratch in Python to train some simple vectors and even got it to train some MNIST characters https://github.com/tbensky/NeuralNetworks/blob/main/ANN/ann.... (but it was slow).

With that basic refresher, I got into PyTorch and worked on training a PiNN: https://github.com/tbensky/PiNN_Projectile (neat video of it training: https://youtu.be/0wlHa1-M7kw).

Now I'm working on understanding kernels with CNNs (this is my question, I'm making good progress on answering): https://ai.stackexchange.com/questions/46180/kernels-on-a-tr....

Having loads of fun!