I think the real argument is that the screen size has dramatically increased, making it way more uncomfortable to use compared to an SE (I had/have a 5, 6s and 13 mini so I've experienced this myself). The physical size difference of the object isn't that substantial IMO, but I understand your point of view.
Yeah I have to agree here. The first gen SE size was good for me to use for one handed control which was very useful where the device acts as a remote control for a few devices in my home. The screen for the 2nd gen SE is an inch taller and half an inch wider which makes it impractical for me due to RSI. I imagine the screen size for the iphone mini is much bigger than even the new iphone SE so the only choice is to use my iphone se home remote until every app stops working.
The fact is the iphone mini wasn't popular despite some demand for a small phone, phones are heavier due to the evolving chip architectures plus the necessity to stretch every millimetre of screen size. Likewise for laptops (Like many I would pay over the odds for a modern Thinkpad that felt just like a x220) but in reverse, laptops get lighter and thinner with worse keyboards whilst phones get bigger and heavier.