If you're reading "just tables" without, or with simple relations and conditions, it's ok (because, as you pointed out, that's basically SELECT with)
The problem is when for example you have inheritance, or moderately complex relations.
Then you'll see Django spamming your DB with requests and the DB may be fast but the operation will be slow because of the sheer number of requests.
So yeah, in a sense you can optimize for it, or use SqlAlchemy (which is good, but it has a different philosophy than the Django ORM in the way it's used)
Inheritance is a different beast that ... well, let's just say that it may creates and endless debate whether one should or should not have inheritance in the model...
My experience is heavily in Java (JPA2 and Hibernate) and less of Rails (simple/moderate apps). Typically an ORM provides a way to handle custom queries but still help maps the result to the Java object model (which helps a lot vs writing getter/setter :)).
I don't know how people down south in Silicon Valley design their model (especially young scrappy startup) but I have a slightest doubt that they go that far (advanced/complex). It's not like startup start with "analyzing requirements" and move on to "modelling session" :) => they just go ahead and write code.
So in theory, the data model shouldn't be complex enough to warrant edge-case.
If you're reading "just tables" without, or with simple relations and conditions, it's ok (because, as you pointed out, that's basically SELECT with)
The problem is when for example you have inheritance, or moderately complex relations.
Then you'll see Django spamming your DB with requests and the DB may be fast but the operation will be slow because of the sheer number of requests.
So yeah, in a sense you can optimize for it, or use SqlAlchemy (which is good, but it has a different philosophy than the Django ORM in the way it's used)