"It's a small toolchain that we can keep in our heads and make arbitrary changes to, quickly and easily. Honestly, if we'd built on GCC or LLVM, we'd be moving so slowly I'd probably have left the project years ago."
"For example, no standard ABIs and toolchains supported segmented stacks; we had to build that, so it was going to be incompatible from day one. If step one had been "learn the GCC or LLVM toolchains well enough to add segmented stacks", I'm not sure we'd have gotten to step two."
Their own explanation for wasting hundreds of thousands of man-hours on a "quirky and flawed" separate compiler, linker, assembler, runtime, and tools is because they absolutely needed an implementation detail that is completely invisible to programs and which they are now replacing because it wasn't a good idea in the first place (segmented stacks). And it's worth writing out a 1000 word rationalization that doesn't bother even mention the reason that implementation was necessary in the first place, to better run on 32-bit machines. In 2010.
Or they say that they had to reinvent the entire wheel, axle, cart, and horse so that five years later they could start working on a decent garbage collector. Never mind that five years later other people did the 'too hard and too slow' work on LLVM that a decent garbage collector needs. What foresight, that.
That's not sense, that's people rationalizing away wasting years of their time doing something foolish and unnecessary.
The replacement to segmented stacks is copying stacks, which as far as my knowledge of LLVM takes me, would be very difficult to add. You need a stack map of pointers to successfully move pointed-to objects on the stack from the old region to the new.
There is a great deal of work going on in LLVM on this issue for precise GC of other languages, and (from the outside) it looks like more hours have been spent on it than on the entire Go toolchain. As Go developers don't have the resources or expertise to make such wide-ranging changes to LLVM, it would have blocked Go development.
GCC is similar. Those working on gccgo are trying to work out how to add precise GC and copying stacks. It is much more complex than it was on the gc toolchain.
There is great value in having a simple toolchain that is completely understood by the developers working on it. In fact, that very idea, that code you depend on should be readable and widely understandable, is one of the goals of Go. Applying the goal to the toolchain is a case of eating our own ideological dogfood.
> The replacement to segmented stacks is copying stacks ...
Which again is not an answer. Why are segmented stacks necessary? Why are copying stacks necessary?
This reasoning, which is their best apparently, amounts to saying that they had to implement their own compiler, linker, assembler, and runtime because they decided they had to implement their own compiler, linker, assembler, and runtime.
Lightweight goroutines depend on small stacks. General API design in Go depends on lightweight goroutines.
In particular, Go style is to never write asynchronous APIs. Always write synchronous blocking code, and when you need to work concurrently, create a goroutine.
You cannot do this in C with pthread, because OS threads are too heavyweight. So you end up in callback-based APIs that are harder to use and harder to debug (no useful stack traces).
This small feature has a surprisingly wide-ranging effect on the use of the language. It is a very big deal.
Go is very much about reinventing these low-level things.
Threads and stacks are orthogonal. You can have coroutines with contiguous stacks, and threads with non-contiguous stacks.
Furthermore it's extremely easy to use non-contiguous stacks in C just by knowing the stack amount used by functions, which the compiler already knows.
This is a totally absurd reason to reimplement an entire toolchain.
I'm not sure what about my comment is worth downvoting, but to try one more time:
If you came to me tomorrow and said "I want to build a language just like C but with non-contiguous stacks" I agree, I would use LLVM or GCC. But that's not what happened.
The history is three engineers decided to see if they could do better than C++ for what they did every day. That meant trying lots of things. One of the many was goroutines, but they needed a flexible platform on which to try lots of ideas that didn't make the final cut.
It just so happens, two of them had worked on a toolchain before. Ken's from Plan 9. (Which long predates the existence of LLVM.) And as he knew his compiler well, it was very easy to modify it to try these experiments.
In the end the language stabilized with several unusual features, several of which would be difficult to add to other compiler toolchains they were not familiar with. Is that the point they should switch to using LLVM?
Building on a toolchain you know that lets you experiment makes a lot of sense. Knowing a toolchain means you get to work quickly.
The end result still has useful features that LLVM does not. For example, running ./all.bash does three complete builds and runs all the tests. It takes about 60 seconds on my desktop. Last time I tried LLVM, it took minutes. Go programmers love fast compilers.
> If you came to me tomorrow and said "I want to build a language just like C but with non-contiguous stacks" I agree, I would use LLVM or GCC. But that's not what happened.
Except that is exactly what happened. Russ says: "segmented stacks; we had to build that, so it was going to be incompatible from day one."
That's the rationalization though. It wasn't about features that you can all but do in plain ANSI C being 'too hard'. We all know what really happened is that they were comfortable with their Plan 9 toolchain and made a demo using it... which is fine. Then they continued to develop their demo for 5 years instead of throwing it out and doing it right, and now they are stuck having to make excuses for why their compiler and assembler and linker and runtime and tools are sub-par.
LLVM is a C++ monstrosity that takes hours to compile. Other programming language projects have to maintain a "temporary" fork of LLVM to achieve their goals: https://github.com/rust-lang/llvm/tree/master
To quote rsc from https://news.ycombinator.com/item?id=8817990:
"It's a small toolchain that we can keep in our heads and make arbitrary changes to, quickly and easily. Honestly, if we'd built on GCC or LLVM, we'd be moving so slowly I'd probably have left the project years ago."
"For example, no standard ABIs and toolchains supported segmented stacks; we had to build that, so it was going to be incompatible from day one. If step one had been "learn the GCC or LLVM toolchains well enough to add segmented stacks", I'm not sure we'd have gotten to step two."