You need an error signal - a target value is compared with the networks prediction. That error is carefully assigned proportionally to the network weights that contributed to it and the weights adjusted a small amount in that direction. This is repeated many times.

Backpropagation suffers from vanishing gradients on very deep neural nets.

Recurrent Neural Nets can be very deep in time.

Or the weights could be evolved using Genetic Programming.

> Backpropagation suffers from vanishing gradients on very deep neural nets.

Especially when using saturating functions (tanh/sigmoid)

> Or the weights could be evolved using Genetic Programming

GA, not GP http://en.wikipedia.org/wiki/Genetic_algorithm

> Or the weights could be evolved using Genetic Programming.

Some algorithms, such as NEAT[0], use a genetic algorithm to describe not only the weights on edges in the network, but also the shape of the network itself - e.g., instead of every node of one layer connected to every node of the next, only certain connections are made.

0. http://en.wikipedia.org/wiki/Neuroevolution_of_augmenting_to...

It would be interesting to occasionally train the generated C against a compiler.

In "Learning to Execute" by Zaremba & Sutskever http://arxiv.org/abs/1410.4615 An RNN learns snippets of python

Their next paper is "Reinforcement Learning Neural Turing Machines" http://arxiv.org/abs/1505.00521 based on Graves "Neural Turing Machines" http://arxiv.org/abs/1410.5401, which attempts to infer algorithms from the result.

In a lost BBC interview from 1951 Turing reputedly spoke of evolving cpu bitmasks for computation.

It would also be ideal to use a higher level interpreted language, and have it try to generate one page scripts rather than giant mega projects like linux.

there are various ways, but one is to unroll the network about some timestep and treat it as a regular NN. You might find this helpful:

http://minds.jacobs-university.de/sites/default/files/upload...

That sounds rather absurdly computationally expensive.

Thanks for the link, I'll take a look.