A 4-layer in that form factor should be pretty doable with no fancy features like blind vias. The RP2040 and W5500 are the same size, and ethernet PHYs can be found in about 3x3mm or even smaller. There should be about 20x25mm of usable space in that module form factor (even conservatively, like 18x23).
I don't have the time to give it a shot myself, but I could try to help if needed.
The issue is more the space needed by all the passives, the crystal, the massive flash chip. I can just about make it fit but now I have the issue that the phy needs some vias to the center pad for gnd but that's always right at the point where my ethernet jack is on the other side.
I don't have the time to give it a shot myself, but I could try to help if needed.