Depends on the method. Evolutionary methods can absolutely find structure that we missed, and they often go hand in hand with learning. Like AlphaGo move 37.

AlphaGo had a lot of driver code involved to make it tick, it wasn't just a big network deciding what to do. You would need something similar here, without someone figuring out that driver code you aren't revolutionizing anything with todays neural networks.

AlphaGo without the search algorithm, just the neural net, operates at a professional (but not superhuman) level.

Later refinements with AlphaZero and MuZero also reduced the driver size.

The most naive form is just Monte Carlo tree search using the neural net as the bias/eval function. A basic MCTS is just 100 lines of code or so.

Yes, since Go is a very simple game. Making a proper driver for much more complex domains like engineering blueprints is not something we know how to do today.

Edit: Also you are missing the Go engine in that comment, it can't train without a Go engine to train against that evaluates the results of each move. That Go engine is a part of the training algorithm and thus is also a part of the driver code, you would need to produce something similar to train a similar AI for other domains. We don't know how to write similar blueprint engines or text evaluation engines, so we can't expect such AI models to produce similar results.

GPT3 can do some basic arithmetic.

It makes errors doing basic arithmetic's, so it just memorized some symbolic transitions and didn't figure out the underlying math.

Agreed, it's just rote learning.