> Why can't the process itself decide how to handle what should be a null pointer return from malloc or mmap et al?
At the point where the kernel knows that it should be returning a null pointer, you're probably already operating in a severely degraded state. You will be scanning the page table constantly, looking for pages to reclaim, wasting tons of CPU. You will have reclaimed all of the reclaimable pages you can reclaim, so io utilisation will be through the roof - calling a random function will cause a disk read to page the executable into memory. Your system will not be functional. And that's ignoring the reality that modern MMUs don't actually let the kernel know when all the memory is in use.
If you want to handle memory pressure better for a given application, shove the application in a cgroup and use a user space oom killer. It's not possible for a program to react gracefully to a system OOM.
At the point where the kernel knows that it should be returning a null pointer, you're probably already operating in a severely degraded state
That's only currently the case. It doesn't need to be. As per your example, you could just as easily provide an upper bound to the page table scan, and return NULL when enough pages can't be found in 100us.
It's not possible for a program to react gracefully to a system OOM.
A program should never receive a system OOM, the fact that Linux' memory accounting is so bad the system itself can run OOM is the problem. It is perfectly possible for a system kernel to execute in bounded memory, and never let programs claim parts of that. Linux just isn't designed that way.
You could def provide an upper bound, though I imagine that'd just cause more people to be annoyed on this thread, complaining that linux doesn't give them memory when there's tons available. And to be honest, if you want this behavior you can basically get it already; check your mem.pressure and react to that. Linux isn't going to give you a null pointer, but you can def react to memory pressure.
> It is perfectly possible for a system kernel to execute in bounded memory
Is it possible while maintaining anything like the feature set that linux provides currently?
The only other option is failing mmap and malloc as well as disabling overcommit. And even that doesn't prevent death by swapping.
There's no real API to inform applications about low memory conditions.
At the point where the kernel knows that it should be returning a null pointer, you're probably already operating in a severely degraded state. You will be scanning the page table constantly, looking for pages to reclaim, wasting tons of CPU. You will have reclaimed all of the reclaimable pages you can reclaim, so io utilisation will be through the roof - calling a random function will cause a disk read to page the executable into memory. Your system will not be functional. And that's ignoring the reality that modern MMUs don't actually let the kernel know when all the memory is in use.
If you want to handle memory pressure better for a given application, shove the application in a cgroup and use a user space oom killer. It's not possible for a program to react gracefully to a system OOM.