There is a great paper, Weight Agnostic Neural Networks [0], that explores this topic. They experiment with using a single shared weight for a network while using an evolutionary algorithm to find architectures that are themselves biased towards being effective on specific problems.
The upshot is that once you've found an architecture that is already biased towards solving a specific problem, then the training of the weights is faster and results in better performance.
From the abstract, "...In this work, we question to what extent neural network architectures alone, without learning any weight parameters, can encode solutions for a given task.... We demonstrate that our method can find minimal neural network architectures that can perform several reinforcement learning tasks without weight training. On a supervised learning domain, we find network architectures that achieve much higher than chance accuracy on MNIST using random weights."
The upshot is that once you've found an architecture that is already biased towards solving a specific problem, then the training of the weights is faster and results in better performance.
From the abstract, "...In this work, we question to what extent neural network architectures alone, without learning any weight parameters, can encode solutions for a given task.... We demonstrate that our method can find minimal neural network architectures that can perform several reinforcement learning tasks without weight training. On a supervised learning domain, we find network architectures that achieve much higher than chance accuracy on MNIST using random weights."
[0] https://arxiv.org/abs/1906.04358