Plus, of course, a lot of modern systems use memory overcommit --- if you ask the OS for memory, it gives you uninitialised address space, and it only allocates pages of physical RAM the first time you touch each page of address space.
This has two effects:
(a) malloc() never returns NULL. It always returns a valid address, even though your system may have out of memory.
(b) by the time the kernel finds out that it's run out of physical pages, your process is already trying to use that memory... which means there's no way for your process to cope gracefully. You have to trust the kernel to do the right thing (either to scavenge a page from elsewhere, or to kill a process to make space). If you're very lucky, it'll send you a SIGSEGV...
This is not always true. For example, you can use ulimit to set an upper bound on the amount of address space the process can consume. Also, on a 32-bit system it is easy to imagine a process trying to allocate over 4G of memory and running out of address space.
With overcommit you will only get NULL from malloc() when you run out of the address space (or from the subset of address space that is used by malloc). When you overrun the address space ulimit, you are killed by some signal.
however, ulimit will only affect your process; if other processes consume enough RAM, you can still run out of pages while under the influence of ulimit.
OK, mmap() will fail and thus malloc() will return NULL. The point is that you cannot rely on that. For example try what will happen with recursive functions that call alloca(1024).
Sorry how is this relevant to heap allocation? alloca just allocates memory off the stack - it's literally just bumping the stack register by the number of bytes you requested (and hence it's freed when the stack pointer is reset on returning from the function call). Alloca should fail when your current thread runs out of stack space - which will generally be only a few megabytes. The total stack for a new thread is allocated from the heap at its creation, not during its execution and certainly not as part of a call to alloca.
Stop moving the goalposts. Besides, alloca() tends to crash your program a lot sooner as stack allocation is usually very limited, especially in a shared-memory multithreading context. Hardly anyone uses alloca().
As other people are commenting, malloc can still return NULL, but the critical point here is that even if you handle the NULL cases, you will still miss the other situations where your process runs out of memory. In short, if the OS over-commits RAM, it is impossible for a program to safely handle out of memory situations.
One reason why overcommit is popular is because of fork(). Imagine a process that does malloc(lots of memory), then forks and execs a tiny program (/bin/true or something like it). If the fork() call succeeds, this means the OS has guaranteed that all the memory in the child process is available to be written over. i.e. it has had to allocate 'lots of memory' x 2 in total, even if only 'lots of memory' x 1 will actually be used.
Without overcommit, fork() can fail even if the system won't ever use anywhere close to the limit of RAM+swap space.
I with you on the no-point-checking-for-malloc-fail issue. It has to be done in every place, or it rarely is effective. My old colleage Mike Rowe used to call it "Putting an altimeter on your car. So if you drive over a cliff, you can see how far it is to the ground."
Definitely. In practice, it's impossible to do correctly. Even if your own code gets every malloc() correct (and also properly checks the return code of every syscall and library call that might fail due to lack of memory), you still have to trust that every library you use is also perfectly written and handles its memory failures just as perfectly. It'll never happen.
There's just one place that it makes sense: for very large buffers. If they fail, it doesn't mean the system is doomed. So it doesn't hurt to check that video-frame buffer alloc, or that file-decompression buffer. But for any of the small change (anything within orders of magnitude of the mean allocation size) its pointless.
> In short, if the OS over-commits RAM, it is impossible for a program to safely handle out of memory situations.
That's a non-sequitur.
If you don't check for NULL return, you have undefined behaviour--that is: your program might end up doing just about anything. If the OOM killer kills you, all visible effects of your program will still be perfectly consistent with the semantics of your program. And you have to be able to safely deal with that scenario anyhow, as much more fundamental resources such as electricity might run out at any time as well.
Imagine your program is processing a folder of emails. If the program is halted part-way through (or if you yank the power cable out) then the mail folder will be left in an inconsistent state.
If the program runs in an OS that does not overcommit memory, it is possible to write code that checks every malloc() and, if it hits a memory limit, you could shut down gracefully, fixing the mail folder so that its state is correct.
If the OS overcommits, then even if you checked every malloc() call, your program might die at any time because of a SIGSEGV or the OOM killer nuking it. There's no way to tidy up an incomplete run.
It's nothing to do with undefined behaviour, or dereferencing NULL pointers.
If you leave something in a corrupted state when execution is aborted at some random point, that means that your code (and possibly your on-disk data structure) is defective. It is perfectly possible to handle that case safely (using transactions, journaling, rollback, atomic replace, ...).
However, if you don't check for allocation failures, you get undefined behaviour, which you indeed cannot handle safely ... other than by avoiding the undefined behaviour in the first place by checking for allocation failures.
Also, if you check for allocation failures, you won't get a SIGSEGV. SIGSEGV is for invalid virtual addresses, not for lack of resources.
You can get memory errors outside of malloc() on an OS that overcommits RAM. Whether that is a SIGSEGV or some other signal, or just the heavy boots of the OOM killer nuking your process, doesn't really matter. The key point is that your program cannot handle them. It is impossible to handle every case.
Allocation failures can and do occur outside of malloc() when overcommit is the memory policy. Malloc might return memory that it believes is valid, but when you try to write to it later on, the OS discovers that it has no free space in the swap and no spare pages to evict. Result: process death (SIGSEGV? SIGBUS? not sure, but it doesn't matter)
> You can get memory errors outside of malloc() on an OS that overcommits RAM.
Did anyone claim otherwise?
> Whether that is a SIGSEGV or some other signal, or just the heavy boots of the OOM killer nuking your process, doesn't really matter.
It does, because you can catch SIGSEGV.
> The key point is that your program cannot handle them. It is impossible to handle every case.
It can, and it has to, if you don't want it to be defective. (Where "handling" does not mean "continue execution", but "don't corrupt persistent state"--you cannot continue execution with insufficient resources anyway, there is no way around that).
> Result: process death (SIGSEGV? SIGBUS? not sure, but it doesn't matter)
Have a system call that allocates memory to a process or process group, but doesn't assign it. This call will fail if overall memory usage is too high.
When the process tries to get memory, it will only tap into that reserved allocation if the system is out of memory.
The OOM killer will never kill processes that have reserved spare memory.
The result? Long-running daemons with stable memory usage can be relatively easily protected against the OOM killer. Ideally the OOM killer never runs amok. But in the real world, it would be nice to be able to limit the damage if it has, and guarantee a usable shell for a superuser while it is going on.
The thing is: There isn't really any such thing as "the memory of a process". What about a page of a shared library that hasn't been loaded yet, but is mapped in more than one process, for example?
Also, if memory is guaranteed to be available at a later point, it cannot really be used for anything, as there is nowhere to put the contents of that memory the moment the process it's reserved for wants to use it.
But there are two APIs that can be used to implement the same goal: Linux has a procfs setting "oom_score_adj" that can be used to decrease the risk of a particular process being killed (commonly used for sshd for obvious reasons) and there are the mlock()/mlockall() syscalls that you can use to make sure some address space is backed by actual RAM rather than potentially swap.
It is true that the reserved memory could not be used for anything. This is a feature. The memory has been reserved for emergencies, and will keep part of the system usable when everything else goes belly up.
The two alternate methods that you specify are not as useful.
I cannot with oom_score_adj have a shell that can be logged in to and be entirely usable during a fork bomb. Yes, you can ssh in. But good luck running arbitrary commands.
With mlock()/mlockall() I can guarantee that a process is responsive during heavy memory pressure, but good luck if it needs to allocate more memory.
And neither system makes it particularly easy to set things up so that the OOM killer will avoid killing all daemons whose memory usage remained stable during memory pressure. (And that is the biggest problem with the OOM killer, that it killed random things you'd have preferred stayed up.)
> I cannot with oom_score_adj have a shell that can be logged in to and be entirely usable during a fork bomb. Yes, you can ssh in. But good luck running arbitrary commands.
But that's due to PID exhaustion, not due to memory exhaustion.
> With mlock()/mlockall() I can guarantee that a process is responsive during heavy memory pressure, but good luck if it needs to allocate more memory.
Well, good luck if the process needs more memory than it had reserved in your scheme. Obviously, you have to mlock() sufficient memory for peak need.
> And neither system makes it particularly easy to set things up so that the OOM killer will avoid killing all daemons whose memory usage remained stable during memory pressure. (And that is the biggest problem with the OOM killer, that it killed random things you'd have preferred stayed up.)
No, it doesn't kill random things, it kills things that are easy to reconstruct (little CPU use so far) and that have lots of memory allocated, and where the oom_score_adj doesn't tell it to spare the process for other reasons.
I have definitely been on a system with plenty of pids available, that was completely unusable due to memory exhaustion.
As for the OOM killer, its logic has changed over time. All that I definitely know is that if a slow leak in application code causes Apache processes to grow too quickly over time, it is usually a good idea to reboot the machine because you never know what random daemon got killed before the actual culprit.
Of course you also need to fix application code...
The logic that is applied probably works well in the desktop/developer case where what went wrong probably went wrong recently and there is a person who can notice. That isn't the context where I've usually encountered it.
You might be technically right, but I guarantee you that by your definition the vast majority of software out there is defective. The solutions you mention are non-trivial. It is really easy to make a mistake. While it may not be the operating systems fault at the end of the day, designing a system like this is setting up the overall experience for failure. A well designed system should make it easy to get right and not the other way around.
> You might be technically right, but I guarantee you that by your definition the vast majority of software out there is defective.
Your point being? That it's untechnically correct?
> The solutions you mention are non-trivial. It is really easy to make a mistake. While it may not be the operating systems fault at the end of the day, designing a system like this is setting up the overall experience for failure. A well designed system should make it easy to get right and not the other way around.
Which is all kindof true, but doesn't change that those APIs are the way they are, for historical reasons. Just because some API is broken, doesn't mean your code will work correctly when you pretend it's not broken.
edit: just to be sure: yes, the special-casing of zero-sized allocations is somewhat of a bug, which we still have to live with. Other than that, the API is actuall perfectly fine--if you don't check for allocation failures and you can't handle it if your program gets interrupted at any point, that's just a bug in your code, there is no useful general way to abstract those problems away.
I think the point is that it's easier to use techniques that keep the mail folder always in a consistent or recoverable state, such as journaling or copy-modify-move, than it is to fully handle out-of-memory conditions. Since the former technique is both easier and can handle problems that the latter can't, it's always preferable.
> If the fork() call succeeds, this means the OS has guaranteed that all the memory in the child process is available to be written over. i.e. it has had to allocate 'lots of memory' x 2 in total, even if only 'lots of memory' x 1 will actually be used.
That isn't the case anymore, at least on Linux and BSD. fork() uses a copy-on-write scheme so the OS only allocates/copies the parent memory space if the child attempts to write to it.
You jut repeated the parent's point while thinking you disagreed with it.
The point was that after you fork() in a copy-on-write scheme, the OS now has promised that more memory is available to write on than may actually exist. If the OS avoided overallocating, it would have to right there and then reserve lots of memory (without necessarily writing to it) just to be sure that you wouldn't run out at a later date.
> (a) malloc() never returns NULL. It always returns a valid address, even though your system may have out of memory.
This is only true on Linux, for example, FreeBSD by default has partial overcommit and WILL start returning null after overcommitting some percentage of memory (I think 30% or so by default)
Most major linux distros default to "heuristic overcommit", aka vm.overcommit_memory=0. The intent being to refuse obviously excessive allocations without sticking to a strict limit.
You can also enable behaviour like you describe by setting overcommit_memory=2 and some ratio overcommit_ratio (default 50) or exact number of kbytes overcommit_kbytes
It should be noted though that (a) this behaviour is configurable, and (b) it's not actually true. If you malloc() so much memory that it cannot possibly be provided, Linux will return a failure, even in overcommit mode.
Linux can easily be configured not to do that. Then malloc returns null, when the underlying mmap nicely fails, and this can happen when the address space is nowhere near exhausted.
It is a well known design feature (I would call it a flaw) of Unix/Linux which makes forking processes with copy-on-write memory possible.
You can run out of both physical memory and swap space. Or your system is swapping is so heavily that, for all practical purposes, the system becomes unusable.
Probably one of the best examples of my claim that UNIX has bad architecture. A well-designed system will either prevent or catch this sort of thing with apps able to detect and/or recover from it without geniuses writing them. Then, there's UNIX/Linux...
I should probably add this to my list of non-security reasons (or an angle at least) for using Nizza-style architecture [1] where critical functionality is taken out of Linux part to run on microkernel or small runtime. Maybe even potential to go further and have such apps that monitor for this sort of thing with a signal to apps that works across platforms. Not just this design flaw, but potentially others where app doesn't have necessary visibility. What you think?
Coming from a Windows viewpoint, I would have thought the same thing. AFAIK, under Windows you just end up hitting the system swap file. Of course, you can start getting into a situation where physical memory is so low that everything grinds to a (ahem, virtual) halt....
Yes, that's the first response. "running out of physical pages" here in the GP really means "running out of free or reclaimable pages and running out of swap".
erm... allocating zero was never a good idea. it had varying behaviours on varying platforms for as long as i could remember and was one of those special cases to avoid.
the allocators didn't necessary even meet the alignment specification either... even if the documents said they did. i remember reading that malloc on windows would return things on 8-byte boundaries only to find pointers ending in 4 (rather than 0 or 8) coming out of it...
and as many have pointed out returning null may or may not happen in a number of situations. i've seen it happen enough times when i ask for too much memory to believe that it is a useful fail case to check for...
erm... allocating zero was never a good idea. it had varying behaviours on varying platforms for as long as i could remember and was one of those special cases to avoid.
No, allocating zero bytes is a perfectly reasonable thing to do, as long as you remember that NULL is not necessarily an allocation failure.
thats not even close to reasoning imo, its just an assertion.
... but i agree that the code is safer if you check for bad cases.
i can't see why allocating zero ever is more sensible than not doing it at all (and therefore avoiding even having to know about this problem, much less deal with it).
That requirement was present in C89 and removed in C99, which also added that a 0-sized allocation doesn't have to return a null pointer (whether done by malloc, calloc or realloc)
Dang, is that right? See, I checked C99 when I posted the comment. I don't know where my old copy of C90 is, but I was mildly surprised. I had that WTF feeling "I could have sworn it was supposed to work that way!
In any case, if I'm compiling with -ansi or -std=c90 on a GCC-based platform that is hosted, the library had better behave in the C90 conforming manner. Only if if -std=c99 is used should it take the liberty with realloc(ptr, 0) being like malloc(0).
Note how we still carefully implement a parallel requirement to the one in ISO C. Namely that my_realloc(NULL, size) calls my_malloc(size), regardless of size, just like realloc(NULL, size) is required to be equivalent to malloc(size). We want all the routines in this allocator to be drop-in replacements such that any code which is written to the ISO C allocator spec can be blindly retargetted to use them.
Pretty much any serious C program, if indeed it doesn't have an entire allocator of its own, at least wraps the standard one, for the sake of more uniform behaviors, as well as easy retargettability to embedded scenarios. (Not only embedded machines, but say, embedding in a larger application, where you are told "you must use this table of funtion pointers as your allocator" and it doesn't quite look like malloc: zero allocations are not allowed, realloc is missing, ...)
In this particular program, realloc is wrapped under a function called resize; resize had to be patched not to rely on realloc(nonnull, 0), but users of resize don't have to change. The programmer doesn't have to look for fifty uses of resize to fix them. However, it's worth it to wrap the functions at a lower level anyway, with an identical API.
The C library is far from perfect. Do not expect consistency, completeness, beauty, symmetry and so on.
There are worse things in it than realloc(ptr, 0) not behaving in the neat way that you would like. Oh such as:
isalpha(str[42]); // str is char array
being undefined behavior if char happens to be signed, and str[42] is negative, because isalpha takes an int argument which is expected to hold a positive byte value [0, UCHAR_MAX), and not char value. Whoooops!
The fact when you reallocate you mean to trow the data away does not make sense... In general the article seems honestly a bit confused in what it states.
> Unfortunately, some people are not sane, and have decided that it's equally valid for realloc(p, 0) to return NULL meaning-failure and not free p.
What you want is the following semantics:
char *my_old_vector = ... ; /* We created / allocated in some way. */
add_important_stuff_to(my_old_vector);
char *new_vector = realloc(my_old_vector,size+100);
if (new_vector == NULL) {
/* I still have the old vector and can recover. */
}
What I don't understand is why some programmers invest so much energy in using realloc. When you know that for most cases realloc actually does a new alloc and copy of the existing content, but has the corners of undefined or unwanted behavior, why not doing it in your own function in a way that you fully (for your needs) control?
Looking who wrote the text, I also respect cperciva and believe he must have some good reasons and I'd be glad read which use cases he had, more than just "what's wrong with different reallocs." Because I'm not surprised that the corner cases aren't to everybody's (or even anybody's) satisfaction. It's C. Less is more and all that.
Yes. If the allocation block is 16 bytes for example, the string growing from 10 to 11, 12, 13 etc could still be on the same place.
But in which use cases are frequent reallocs actually needed, so much that you can recognize the performance impact? I'd really like to know, as I personally never had such problems. When the single allocations were too expensive I've just used some kind of memory pool. For small stuff realloc is still more expensive than just a few instructions on average when some pool is used.
The classic example of frequent reallocs is this perl code:
$x .= $_ while (<>);
I believe this has been fixed now, but perl used to realloc for each append operation, which resulted in O(N^2) time complexity if realloc didn't operate in-place.
I'd expect that such growth seldom allows the realloc to remain in-place (unless it's the last thing allocated before and already in a big preallocated chunk of the allocator)? Have you observed what you then get in your program? Which allocator is used underneath?
You certainly could wrap realloc(), but besides the bizarre/unnecessary behavior with realloc(ptr, 0), it behaves exactly as you would expect: on failure it returns NULL and doesn't touch your pointer, on success it returns the new pointer. It doesn't seem like it requires any more "energy" than using malloc() and programming in C to begin with.
Because it's nice to be able to use standard library C functions. It's so rare that you can actually use something in the stdlib rather than in an external library, and people are generally eager not to rewrite functionality which ostensibly should be provided by the standard library.
Well, it shouldn't have corners of undefined or unwanted behaviour. That's kind of the point.
Obviously though, there is a huge advantage to having a single layer of heap management and letting the heap management algorithm have the best insight in to how memory is being used and needed. Rolling your own realloc on top of the heap manager is as likely to create new inefficiencies as remove them.
You don't have to "roll your own" for the cases that are always defined. That means, when NULLs and 0 aren't involved, realloc works, call it from your "myrealloc" function. For 0 and NULL, make sure that your "myrealloc" function handles them exactly as you'd like. Then call your function everywhere in your code. I believed it's "Programming 101."
> it shouldn't have corners
It's not how C traditionally worked. Almost every function was knowingly made to be not "too good." I see modern programmers expect something else, what even 100 times slower languages not always give. Special cases have different possible treatments, and as soon as the exist nobody can make something that would please everyone.
> You don't have to "roll your own" for the cases that are always defined. That means, when NULLs and 0 aren't involved, realloc works, call it from your "myrealloc" function. For 0 and NULL, make sure that your "myrealloc" function handles them exactly as you'd like.
Ah, I misunderstood what you meant by "doing it in your own function". Yes, wrapping the semantics will work, though doing it right and without imposing some overhead takes a bit of work.
> It's not how C traditionally worked. Almost every function was knowingly made to be not "too good." I see modern programmers expect something else, what even 100 times slower languages not always give. Special cases have different possible treatments, and as soon as the exist nobody can make something that would please everyone.
As the original post pointed out, traditionally it didn't work like this. This is how it has evolved.
I did use poor semantics though, as "undefined behaviour" has a specific meaning in C that is of course how the language was defined. What I meant was "ambiguous or unwanted behaviour". Having a clear behaviour for a particular case, even if it is just returning an error code, is fine by me.
To me it seems more reasonable to draw the insanity line at returning NULL for a successful zero byte allocation.
(While it's true that dereferencing a pointer to a zero byte allocation should probably trigger undefined behavior that doesn't mean that it shouldn't be possible to do the != NULL test to see if the allocation was successful.)
That's because you think of it as a special case, which it isn't. If you want to store a string that's n bytes long, you need to allocate n bytes, so a natural thing to write is, for example dst.len=strlen(src);dst.buf=malloc(dst.len); --it's insanity if you need to add a special case for strings of a specific length.
You don't have to call malloc directly in the second assignment. If you depend on special handling of null without the "if" on invocation you can have your own function for that.
This is way too hyperbolic. I don't know what specific "once upon a time" period is being described. However malloc(0), realloc(NULL, x), and realloc(y, 0) has been a portability headache for as long as I have been writing C. None of this is all that surprising.
The behavior of realloc(NULL, x) is always equivalent to malloc(x) according to the C standard. But yes, I never remember a time when malloc(0) or realloc(y, 0) played nice.
> The behavior of realloc(NULL, x) is always equivalent to malloc(x) according to the C standard.
AFAIK if you try to do this with even a fairly recent Microsoft libc it won't work. I haven't checked the standards and history on this one, but keep in mind it's been only a recent development that MS gives a crap about C99.
The idea that C89 did not mandate this makes sense to me because I remember now-obsolete Unixes not liking this either.
[Edit: The MS documentation claims that it supports realloc(NULL, x) going back quite a while. I know I've been bitten by it not doing that in the current century, however...]
> So what happens if realloc(p, 0) fails? It's going to return NULL, since that's what realloc does if it fails; but should it also free the allocation p? The sane answer is yes
If realloc(p, 0) is a synonym for free(p), what does it mean for it to "also free the allocation p" in the event of failure? Unpacking it, the statement seems to be saying the same thing as, "if it fails, then what it should do is to not fail".
Freeing a memory allocation is guaranteed to never fail. Allocating zero bytes can fail (with dumb C libraries). The question is "if realloc fails to allocate zero bytes, should it still free the original allocation".
Or, more likely, it just encourages customers to abandon your alternative C library in favor of glibc to avoid breaking configure scripts (which is what forced musl's hand). I definitely think you're right in principle, but I can't blame musl's author for bowing to market pressure any more than I can blame Intel for sticking to x86 backwards compatibility.
I'm still not clear on this. What does "allocating zero bytes" mean, if not "free the original allocation"? Keep the malloc metadata structures around but update their size to 0?
Some systems handle a zero-byte allocation request by allocating a unique non-NULL pointer which cannot be dereferenced. If they run out of unique pointers, they can fail to allocate zero bytes.
The change appeared in C99 apparently, in my ISO/IEC 9899:1999 PDF the part about a 0-sized realloc being equivalent to a free has already been removed, and that a 0-sized allocation can return either a NULL or a non-null pointer has appeared under implementation-defined behaviours (j.3.12)
I don't get it. For me it's always a simple logic: if you didn't have anything (0 bytes) to allocate/reallocate you simply don't do that. Period. It's like division by zero -- you always have to check if divisor is not equal to zero; if it was -- skip division altogether.
Except the semantics of a zero-length buffer are perfectly well-defined, while the semantics of division by zero are not defined at all in the types commonly used in computing. A proper comparison therefore would be addition, maybe, which is also perfectly well-defined for zero operands. Do you special-case "zero addition" in your code? if(a==0){if(b==0){x=0;}else{x=b;}}else{if(b==0){x=a;}else{x=a+b;}} ?
I'd argue that zero-length buffer is far from "perfectly well-defined". Quoting C99:
" If the size of the space requested is zero, the behavior is implementation- defined: either a null pointer is returned, or the behavior is as if the size were some nonzero value, except that the returned pointer shall not be used to access an object."
To maintain my sanity as a C programmer, I steer away from "implementation specific behavior" as much as possible, thus I prefer checking if buffer size is non-zero before allocating.
I am not talking about C, but about the abstract concept. C's implementation is obviously kindof broken. Just as C would be broken if the behaviour of "zero additions" was somehow "implementation defined". That doesn't mean that something is fundamentally wrong with adding zero, it just means that you have to work around a broken language.
The topic is about C and I never said that such behavior is normal or anticipated. It's one of many cases where C standard lacks concreteness and we can only workaround such issues. In a perfect world, many things would be different.
Well, then I might have misread your comment. I thought you meant that "it's just illogical to allocate 0 bytes anyway, just as it doesn't make sense to divide by 0".
Zero-length objects in general may behave quite differently, because there may or may not be a guarantee that pointers to two different zero-length objects compare as unequal. In particular, this is a point where GCC's C and C++ implementations differ!
Except that the interactions with the rest of the standard become very bad. E.g. memcpy to or from NULL, even for zero bytes, is undefined and that will be taken advantage of by your compiler in ways you really didn't expect.
Because checking for this case actually makes the logic in every piece of code more difficult, since you now have to account for the edge case. This is superfluous, because in the edge case nothing bad was going to happen, anyway; a correct program wouldn't ever dereference the return value, and an incorrect program is incorrect regardless of the behavior of malloc here.
I don't know if the distinction between which behaviors are "sane" and "insane" is a good one here. Evidently the only insane choice is to rebuff the standard in your implementation of the core C memory management functions.
The REAL problem here is C's garbage way of dealing with errors. NULL is overloaded both to be a valid pointer and an error signal. In the author's ideal world, how would a malloc failure be communicated?
Null is not a valid pointer, though. It's a valid pointer value, but it's undefined to dereference such a pointer. If the purpose of malloc is to return an address to usable memory, then null it outside of that range. Values outside of the legal range of a function are fine for error conditions. Because of that, null seems like a reasonable return value for error on malloc: I could not allocate memory for you, which I indicate by returning an address that you are not supposed to dereference.
However, I do agree it would be better if there was some other kind of error condition, preferably something like the option type from various other languages, or just an easy way to return multiple values like in Go. But C is not that language.
There are embedded systems out there (that have C compilers) that have addresses at 0 actually. 8051 C programmers know what I'm talking about.
IIRC, 0 is totally an address when inside the Linux kernel as well. What else do you call the first block of memory on a system?
Just because in userland the Linux Kernel pages memory to a very high-numbered region does not mean that under all circumstances "0" is an invalid address. The entire concept of "null == 0 == invalid" is an innate falsehood, an abstraction brought about by the malloc function (that a lot of other libraries have decided to copy).
But really, what do YOU call the first physical byte of RAM? Most systems I know of... from 8051 all the way to even Linux Kernel... calls it 0.
Nobody says that NULL has to be 0. C's NULL is just a pointer value that can be compared for equality but can not be used for pointer arithmetic or be dereferenced. Implementations commonly reserve some address (often 0) to represent that pointer value, as that is just the most efficient way to do it (rather than making pointers larger than the normal word size to be able to represent the sentinel value)--also, object sizes are limited to one less than the number of possible addresses anyway, simply because you can have objects that are 0-sized.
You're confusing a "null constant" and a null pointer. An integral 0 being converted to a pointer at runtime is not required to be a null pointer, only an integral 0 converted at compile time is.
Well, yes, but from the view of C, pointer values are arbitrary anyway. There is no guarantee that a particular pointer's value has any particular relationship to which RAM cell of your hardware you are accessing.
Dereferencing null is undefined behavior in a C program - which is another way of saying it is considered an error, and there are no guarantees about what will happen as a result of it.
From the standard (http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1570.pdf), "If an invalid value has been assigned to the pointer, the behavior of the unary * operator is undefined." In a footnote, it goes on to define what invalid values are. "Among the invalid values for dereferencing a pointer by the unary * operator are a null pointer, an address inappropriately aligned for the type of object pointed to, and the address of an object after the end of its lifetime."
And null is indeed defined to be 0. Also from the standard: "An integer constant expression with the value 0, or such an expression cast to type void *, is called a null pointer constant. If a null pointer constant is converted to a pointer type, the resulting pointer, called a null pointer, is guaranteed to compare unequal to a pointer to any object or function."
> Dereferencing null is undefined behavior in a C program - which is another way of saying it is considered an error, and there are no guarantees about what will happen as a result of it.
Indeed. It is undefined within C. It is an abstraction that C builds on top of. Dereferencing 0 on the vast majority of embedded systems (or kernel-level code) is very well defined.
This is just one of the areas where C has a "leaky abstraction".
It's an abstraction taken from C++ (and virally transmitted back to C99). ANSI NULL can be anything the implementer wants it to be, not just (void *)0.
Frankly it's the dumbest misfeature from C++ that should have had nullptr from day one. What's the point of constructing a rigorous type safety system if you're going to go and break it with magic integers that can act like pointers.
This has two effects:
(a) malloc() never returns NULL. It always returns a valid address, even though your system may have out of memory.
(b) by the time the kernel finds out that it's run out of physical pages, your process is already trying to use that memory... which means there's no way for your process to cope gracefully. You have to trust the kernel to do the right thing (either to scavenge a page from elsewhere, or to kill a process to make space). If you're very lucky, it'll send you a SIGSEGV...