I had a college professor who was really big on letting your subconscious mind work on tasks to intuit answers to difficult questions in this same way. Contemplate a problem consciously, then go on a walk or some other calming exercise and have the answer pop into your head. Your mind has been working on it in the background and foregrounds the task once it's done.
People often experience something like this while driving, when you just get an instinct that a kid is going to jump out, or that other car isn't going to stop, without having consciously paid attention to it. You slam on the brakes and are left with that feeling that you just KNEW that was going to happen. A part of you did know it, and foregrounded that prediction when enough variables lined up, overriding whatever you were previously thinking about.
Perhaps this effect of leaving a problem for a while and then returning also causes you to approach it with a different part of your brain. Like picking a direction at a crossroad and then being unable to go a different direction.
> Contemplate a problem consciously, then go on a walk or some other calming exercise and have the answer pop into your head. Your mind has been working on it in the background and foregrounds the task once it's done.
My current role is that of a principal engineer or quasi-architect and I do exactly this when I need to make a decision regarding which direction we're going to go with something.
An environment that is conducive to your subconscious performing well really helps.
This also works for unconscious problem contemplation, but you have to be ready for it when it happens.
In my younger days, I'd run for 30mins outside often times without any music. During those runs, I'd let my thoughts bubble up. It was a welcome chance to let my brain tell me what was bothering it. Sometimes these would be technical problems du jour, but after handling those, nearly always there would be a social problem, usually interpersonal, that would bubble up. It was a very good technique to identify those issues organically.
Something about the kinematic state also made it easier to accept, process, and set aside the idea than it would be if I was sitting in a room silently.
Nowadays, any time I'm doing anything involving semi-repetitious kinematic activity, I find myself making a choice between listening to something, music or a podcast, or simply nothing and using the time to sort through what ails me. There is a certain mental healing that comes with certain physical activities.
My bedroom theory is that all thinking layers are built on top of the mechanical/somatic processing one. And that thinking is basically virtual hunting to put it grossly. We walk in circles when searching solutions. Moving physically might stimulate and bubble up to the abstract layers.
I swim laps at the Y. The meditative state my mind enters about 10 laps into a 32-lap mile is amazing. The subconscious bubbles up, as a sibling commenter pointed out, and begins actively problem solving at the surface when your conscious mind subsides. Sibling commenter experienced this while running for thirty minutes.
I have a very peaceful home. It's quiet, safe, a little remote, about an hour or two northwest of Austin in the Hill Country. I have a library in my house where I do most of my work, and sometimes I get on my motorcycle and ride for a bit into the middle of nowhere, which becomes meditative, much like swimming, allowing my mind to work through problems on its own in a very beautiful environment.
I can work (whether it's actual coding or sitting quietly in thought) for 12-14 hours a day, max. Then I must stop and get 7-8 hours of sleep. That balance of good sleep and hard work is key to maximizing productivity and the ability to solve problems during available work hours.
A healthy diet free from high-glycemic foods and foods with a high inflammation index is also very important.
People often experience something like this while driving, when you just get an instinct that a kid is going to jump out, or that other car isn't going to stop, without having consciously paid attention to it. You slam on the brakes and are left with that feeling that you just KNEW that was going to happen. A part of you did know it, and foregrounded that prediction when enough variables lined up, overriding whatever you were previously thinking about.