The slides are a bit tricky to read without the rest of the accompanying talk. I couldn’t find a video with the brief search I had.
One thing the talk touches on is that writing a Lisp (cross-)compiler in Lisp is hard. The slides suggest a few reasons and here are some more.
In CL much of the compilation process requires evaluating CL code. It seems that you are therefore fortunate in using CL: you can just take the code and eval it. But this doesn’t work because (wanting to be portable and supportive of optimising compilers) many objects in CL can’t be inspected. For example the following is valid code:
(funcall #.(lambda (x) (1+ x)) 5)
In the code above one constructs a closure at read time, puts it into the ast and the compiler must take this call to an opaque host-platform closure (which evaluates to itself) and somehow marshal it into something for the target platform.
So to write a CL cross-compiler one must first implement a CL interpreter and this interpreter must have objects with enough useful state that one can correctly marshal closures from the interpreter into compiled objects in the target system, even when those closures close over shared values.
One way to reduce this pain is to restrict the compiler to only compiling a subset of the language, and then requiring that the compiler is written in that subset. One may only compile more complicated programs when the host and target are the same instance of Lisp.
However the “easily compiled portable subset” is probably too small and you will end up with either a long bootstrapping process of increasingly useful compilers or having to implement a lot of emulation of your target platform. An example in the slides is that the result of (byte ...) is implementation defined so you can’t just use the value from the host implementation. Another example is floats. In CL there are 4 float types which are allowed to be equal to one another in certain ways (typical modern implementations have 2 32-bit and 2 64-bit; others might have some being decimal or packed into 63 bits), so one cannot rely on the host’s floats behaving a certain way, so one instead has to emulate the target platform’s float implementation to get reliably portable results. Then again, maybe it is possible (but perhaps a bit painful) to write the compiler without using any floats.
Another bootstrapping difficultly in CL is it’s object system which just makes everything harder, especially if one wants to use objects for lots of the implementation-specific types.
SBCL goes for writing the compiler in a subset of CL and the result is indeed sanely bootstrapable. Other implementations typically require eval and some runtime written in eg C and a slow process of evaluating the improving compiler on itself to bootstrap the compiler. This is easier at first but can lead to difficulties (some of which are described in the talk).
> the compiler must take this call to an opaque host-platform closure (which evaluates to itself) and somehow marshal it into something for the target platform.
If we're cross-compiling, we are almost certainly doing file compilation whereby we have to externalize the compilation product to be transported to the target where it is loaded.
If we use ANSI Lisp file compilation as our source of requirements, then we only have to handle, as source code literals, objects which are "externalizable". Closures are not.
See 3.2.4 Literal Objects in Compiled Files
Of course, you can adopt externalizable closures as a requirement in your own compiler project, if you like.
Not sure what you intend to mean, but for a CL file compiler, that's not valid code. No CL file compiler needs to be able to externalize a function object.
One thing the talk touches on is that writing a Lisp (cross-)compiler in Lisp is hard. The slides suggest a few reasons and here are some more.
In CL much of the compilation process requires evaluating CL code. It seems that you are therefore fortunate in using CL: you can just take the code and eval it. But this doesn’t work because (wanting to be portable and supportive of optimising compilers) many objects in CL can’t be inspected. For example the following is valid code:
In the code above one constructs a closure at read time, puts it into the ast and the compiler must take this call to an opaque host-platform closure (which evaluates to itself) and somehow marshal it into something for the target platform.Similarly one can do the following:
So to write a CL cross-compiler one must first implement a CL interpreter and this interpreter must have objects with enough useful state that one can correctly marshal closures from the interpreter into compiled objects in the target system, even when those closures close over shared values.One way to reduce this pain is to restrict the compiler to only compiling a subset of the language, and then requiring that the compiler is written in that subset. One may only compile more complicated programs when the host and target are the same instance of Lisp.
However the “easily compiled portable subset” is probably too small and you will end up with either a long bootstrapping process of increasingly useful compilers or having to implement a lot of emulation of your target platform. An example in the slides is that the result of (byte ...) is implementation defined so you can’t just use the value from the host implementation. Another example is floats. In CL there are 4 float types which are allowed to be equal to one another in certain ways (typical modern implementations have 2 32-bit and 2 64-bit; others might have some being decimal or packed into 63 bits), so one cannot rely on the host’s floats behaving a certain way, so one instead has to emulate the target platform’s float implementation to get reliably portable results. Then again, maybe it is possible (but perhaps a bit painful) to write the compiler without using any floats.
Another bootstrapping difficultly in CL is it’s object system which just makes everything harder, especially if one wants to use objects for lots of the implementation-specific types.
SBCL goes for writing the compiler in a subset of CL and the result is indeed sanely bootstrapable. Other implementations typically require eval and some runtime written in eg C and a slow process of evaluating the improving compiler on itself to bootstrap the compiler. This is easier at first but can lead to difficulties (some of which are described in the talk).