Invalid pointer dereferencing isn't guaranteed to cause segfaults, you might just get garbage memory or nasal demons from LLVM handling undefined behavior.
That's the key advantage of using integers (or a GC'd language) - it's memory safe.
But if you use integers, and delete an array element and reuse it for something else, but somewhere there is a use-after-free integer, you also get garbage memory
Or worse, actual data belonging to someone else. If the integer is a user id, and you delete an user, reuse it for another user, the former user might see data for another use. That is a big security issue