Take the 2-8-0, which https://en.wikipedia.org/wiki/2-8-0 comments:

> From its introduction in 1866 and well into the early 20th century, the 2-8-0 design was considered to be the ultimate heavy-freight locomotive. The 2-8-0's forte was starting and moving "impressive loads at unimpressive speeds" and its versatility gave the type its longevity. The practical limit of the design was reached in 1915, when it was realised that no further development was possible with a locomotive of this wheel arrangement. ...

> During World War II, 14 of these locomotives were equipped with superheaters, which raised their tractive effort from 28,777 lbf (128.0 kN) to 33,557 lbf (149.3 kN)

Then compare to https://en.wikipedia.org/wiki/EMC_E3 , a diesel from 1938 with "Tractive effort Starting: 56,500 lbf".

That's from time when people became convinced that diesel was better than steam. Why? Quoting https://www.american-rails.com/diesel.html

> This changed after General Motors successfully demonstrated the diesel's viability during testing of its FT freight design in 1939.

> The demonstrator set toured the country, convincing skeptic after skeptic that diesels were not only efficient and reliable but could also outperform the iron horse.

Not simply because it was dirt cheap.

Stats at https://en.wikipedia.org/wiki/EMD_FT . The top speed of 95 mph is 15 mph higher than the Union Pacific Big Boy.

Note also "Steam locomotives could be built with fewer precious materials, and with less conflict with military needs.", which is yet another factor that OP's thesis didn't consider.

Did you catch things like "no further development was possible with a locomotive of this wheel arrangement" and from earlier "the six cylinders could easily consume more steam than the boiler could produce". These designs were pushing the limit of what steam could do, and diesel looked like it could pass those limits. And did.