You want to enable thrashing prevention: https://docs.kernel.org/next/admin-guide/mm/multigen_lru.htm...

Make sure you use zram (or zswap) as well. May also consider enabling userspace oom handler like others already suggested, but it's less important than the rest.