The larger sample sizes are a ... sort-of "proxy" ... to overwhelm underlying latent correlations.

The whole thing is actually a subtle sort-of generalization of the "Prosecutor's Fallacy".

So skepticism with the small sample sizes is absolutely warranted, unless some strong evidence is shown indicating mechanism-based independence.