It's refreshing to see a page with Last-Modified actually showing the date the document was last modified. Nowadays, too many websites report the current date even when no single byte of the content is changed since publication.
I have completed this tutorial by Micheal Nielsen and it has enormously supplemented my understanding of the subject. The good part is that it just had enough mathematical rigour the subject demands.
I didn't get lost in the details and was able to see the bigger picture after going through his work. It has brought in enormous change to my career and life. I can't thank him enough.
Agreed - it gives a very solid overview of the field. Truth be told, it was a bit too mathematically rigorous for my taste, so I had trouble producing code for backpropagation. However with the second link (backpropagation step-by-step, with numbers so you can check progress) and then TensorFlow Jupyter notebooks I think I cracked it. Now I just an excuse to use ANN at work... ;)
This is neither basic nor an introduction, and has several inaccuracies. My guess is the info is from the 90s era of neural networks.
Reply to below: it uses pretty complicated language and doesn't explain many terms (e.g. activation functions, supervised learning, etc. are not explained). Errors: the statements the article makes about the error surface are really misleading and in some places wrong. Same for the statement about sigmoids. Some of their definitions are also wrong today, such as "epoch." The list goes on.
What do you think is wrong or outdated with the article? It's not that far off how ANN are described in the beginning of current scientific articles. Backpropagation is definitely still relevant even though changes can be made to it and the networks have grown in size and changed in layout. The basics are more or less the same and from my quick scan they seem accurately described here. What is it you expect from a simple intro?
While I don't share his total disdain for the article, he got the date spot on, and I'd say that
- no mention of non-sigmoid activation functions
- calling overfitting 'grandmothering' and
- diving comparatively deep into gradient descent (which while absolutely the most common, is not the only way to train a neural network), while being an inch deep on practically every other aspect.
Its a bit of a confused intro that simultaneously tries to teach you the basics while throwing jargon at you ('delta-rule', 'beta-coefficient', 'hyperparaboloid', etc.).
What actually is new since the 90s, besides convnets?
Is ML becoming more prevalent because of theoretical breakthroughs, or is it because of hardware improvements, or perhaps because there is more training data available now?
What enabled the ML revolution we're witnessing now is mostly the advent of powerful and low-cost GPGPUs and the abundance of training data, as you mentioned. Of course, there are theoretical breakthroughs and interesting new applications [1], but it wasn't the deciding factor. Even ConvNets aren't new. For example, AlexNet is basically a much larger reimplementation of the original Lenet idea [2] from 1998.
This is not to discredit any new work that's being done. I think it's awesome to see so much progress in a field that's certainly on its way of defining an era. I'm just pointing out that we have known about most of the fundamentals for a long time.
I have to do some research. I'm primarily commenting on this to keep track of it.
Ultimately I want technology to be my better half. Fill in my gaps if you will. I want my vitals linked to a server, a "CRON" job monitoring my sleep/wake cycles. I've been doing some scraping. I'd like to develop my own thing runs "autonomously and grows"
Gotta read, between this and machine learning. Gotta focus though. Not sure when I'll come back to this. Thanks for posting this.
After reading that article and what I just described, it does not seem like neural networks is what I'm after. Still it is something to use.
I wanted to write a "quick search" scrape pages related to a search (though last time I checked, the Google API wasn't available anymore where you could search Google back end.)
Anyway. Maybe a source like Wikipedia. Still parsing words and assigning them values...
That part about using ANNs for finding regularities in patterns. That sounds interesting. It seems many successes came by determining the next step in some form of evolution whether it was a product or service. Blockbuster to Netflix, Blackberry's to iPhones, etc... Maybe look at something like that.
At any rate, gotta read. That was quite informative. The whole thing of "not knowing what it actually does" is pretty nuts too. Give it data and watch it go!
- http://neuralnetworksanddeeplearning.com/
- https://mattmazur.com/2015/03/17/a-step-by-step-backpropagat...