Yes, I suppose my larger point was more to illustrate that nothing is free. GHC is software like any other piece of software and it has a lot of competing constraints we must balance.
If the cost of supporting reusable IRs like you want is high in terms of LOC and needed changes, but the amount of people who will leverage that work is incredibly small, it's honestly not clear to me if making that change and maintaining it is sensible or worthwhile. It might be worth it purely from a code cleanliness standpoint if it did a lot of cleanup, but that's not immediately clear either. We have to maintain everything people submit to us, presumably forever - the barrier to entry for large changes should be high, and well-motivated.
It is similar with external core; it was several extra thousands of lines of code throughout the compiler that was rarely - if ever - used by anyone. Did a few people use it to do cool work - for example, by writing the Intel Research Haskell compiler? Absolutely, and Intel's results were incredible - but that does not mean it's worth keeping that ball of code around for that one use case.
I'd actually argue keeping it for that one case would be a terrible decision for everyone, and when I deleted that part of the compiler, Intel's compiler lingered in my mind, as it was a user. But the fact is nobody was maintaining, submitting patches or using it in a modern GHC, including them - so I rm -rf'd it.
GHC is a research compiler, but it is also a production compiler for many users, and a project several dozen of us work on.
So while I'm sure your work is incredibly interesting (most of our large features come from people doing interesting research, so I understand your plight!), this really doesn't mean large, sweeping changes on our part - for your one case - are very worthwhile at all for us. But maybe you don't need sweeping changes, either!
(Of course, it's impossible to say how many people did not use GHC because of these constraints that otherwise were unsuitable for their projects - but we have to draw a line somewhere and balance our own needs vs those of others.)
> Just a well-documented API would be nice.
To be clear - what API are you exactly envisioning? An API that makes it easy to construct Core or STG programs using GHC's ASTs, and then feed them through the rest of the compiler? Or an API to do things like construct and manipulate the same core representation GHC uses, to later do whatever you want with perhaps with a separate tool?
It sounds like this is something you could tackle with a library that wrapped the GHC API. The problem is the Core representation in GHC is very prone to change, so I'm not sure there's any sane way to really maintain things like API stability between versions. But you also might not need that - just a simple API around "GHC Core version X.Y.Z" to build programs may be enough.
We could distribute such a library with GHC, but this sounds like something easily doable out-of-band first, at least.
Either way, something like thisI can surely see the need for. Where it should live is a different question.
> I also agree that typing is extremely useful. But for small research projects, it can get in the way of the actual goal.
Yes, but again, GHC's internal needs and developer needs outweigh those of random small research projects that might use it - the fact that it makes some kinds of work slightly harder is unfortunate, but in the grand scheme of things, this isn't really a concern at all for any of us, and likely never will be. There's no free lunch, I'm afraid.
I would be mostly interested in reading the Core or STG representation, so I can apply transformations to them. Of course, it would also be nice to generate intermediate code from scratch, but that is something which is not forming an obstruction at this point (as it would be easy to just generate Haskell code instead).
I'm not looking for high-performance solutions, just some tools to make researching new ideas a little simpler.
To give you some idea about the things I'd like to do:
* Transforming programs into "incremental programs", so they run faster on changing inputs.
* Automatically translating functional data-structures to other languages (C++)
* Sending code over the network
I tried using the GHC parser directly, but that route turned out to be too much work (too many cases to handle). Then I looked at Core, but the documentation was lacking or not up to date.
So, from where I stand, the internal documentation is the biggest problem, and, according to good software engineering practices, there can be no excuse for having no good internal documentation :)
Besides just plain documentation, a tutorial with examples would be nice, but I can understand if that is not a priority.
Anyway, keep up the good work :) I'm happy with GHC as it is. I just wished it was a little more "open".
If the cost of supporting reusable IRs like you want is high in terms of LOC and needed changes, but the amount of people who will leverage that work is incredibly small, it's honestly not clear to me if making that change and maintaining it is sensible or worthwhile. It might be worth it purely from a code cleanliness standpoint if it did a lot of cleanup, but that's not immediately clear either. We have to maintain everything people submit to us, presumably forever - the barrier to entry for large changes should be high, and well-motivated.
It is similar with external core; it was several extra thousands of lines of code throughout the compiler that was rarely - if ever - used by anyone. Did a few people use it to do cool work - for example, by writing the Intel Research Haskell compiler? Absolutely, and Intel's results were incredible - but that does not mean it's worth keeping that ball of code around for that one use case.
I'd actually argue keeping it for that one case would be a terrible decision for everyone, and when I deleted that part of the compiler, Intel's compiler lingered in my mind, as it was a user. But the fact is nobody was maintaining, submitting patches or using it in a modern GHC, including them - so I rm -rf'd it.
GHC is a research compiler, but it is also a production compiler for many users, and a project several dozen of us work on.
So while I'm sure your work is incredibly interesting (most of our large features come from people doing interesting research, so I understand your plight!), this really doesn't mean large, sweeping changes on our part - for your one case - are very worthwhile at all for us. But maybe you don't need sweeping changes, either!
(Of course, it's impossible to say how many people did not use GHC because of these constraints that otherwise were unsuitable for their projects - but we have to draw a line somewhere and balance our own needs vs those of others.)
> Just a well-documented API would be nice.
To be clear - what API are you exactly envisioning? An API that makes it easy to construct Core or STG programs using GHC's ASTs, and then feed them through the rest of the compiler? Or an API to do things like construct and manipulate the same core representation GHC uses, to later do whatever you want with perhaps with a separate tool?
It sounds like this is something you could tackle with a library that wrapped the GHC API. The problem is the Core representation in GHC is very prone to change, so I'm not sure there's any sane way to really maintain things like API stability between versions. But you also might not need that - just a simple API around "GHC Core version X.Y.Z" to build programs may be enough.
We could distribute such a library with GHC, but this sounds like something easily doable out-of-band first, at least.
Either way, something like thisI can surely see the need for. Where it should live is a different question.
> I also agree that typing is extremely useful. But for small research projects, it can get in the way of the actual goal.
Yes, but again, GHC's internal needs and developer needs outweigh those of random small research projects that might use it - the fact that it makes some kinds of work slightly harder is unfortunate, but in the grand scheme of things, this isn't really a concern at all for any of us, and likely never will be. There's no free lunch, I'm afraid.