My impression is that LuaJIT is too perfect/complex to the GitHub collaborative model work, where developers uses the software, identify a bug/missing feature for his use case, dig in code for a day/week, and do a push. In LuaJIT the stakes are much higher, and a small change can have big repercussions. Relying on occasional open-source contributors seems like a direct path to abandonware.
Fear not. First of all, occasional contributors are perfectly compatible with LuaJIT. Take a look at the Git log. Many of the commits are bugfixes that don't impact the overall "perfection". It's not a crystalline structure:
Also, I don't think CloudFlare have the intention of entirely relying on sporadic contributions. As they said earlier this month, "we will be investing significantly in its development and are interested in hearing from people who would like to work full time on LuaJIT as employees of CloudFlare." (I'd be curious to know how that's moving along!)
I disagree, it lowers the barrier of entry for people to document code, clean up documentation, or just review the code and report issues (e.g. security related). Even if LuaJIT doesn't see a lot of random contributors, it makes it easier to review commits and what not.
Good benchmarking systems are few and far between, and so many unexpected factors can cause noise, before you even write your first performance test. I'd love a Travis CI for performance tests, but it's a seriously hard problem.
Is it possible (as in, easy), to use something like gerrit/rietveld on top of (or next to) github?
I am imagining a bot what will take GH pull requests, submit the patch to gerrit for review, and answer to the user something like "thank you for your contribution, this project uses an external platform for code review, a discussion was automatically created to review your changes, please follow this link to participate".
Another option is to use a code review tool similar to Gerrit / Rietveld but built from the ground up for integration with GitHub. Recent entrants include https://review.ninja, https://gitcolony.com, and https://reviewable.io (disclosure: this last one's mine).
So true. Some (increasingly rare) code does not translate into modern trends. actually it's more like craftmanship and LuaJIT is a craft. Just like Gigablast open source search engine.
I don't know much about Lua and LuaJIT, so let me ask a naive question:
If you would start out with a JavaScript JIT (like V8) what things would you have to add (i.e. things that are not required to JIT JavaScript) besides the obvious modifications in the parser?
One point I can think of is support for efficient compilation of co-routines.
> If you would start out with a JavaScript JIT (like V8) what things would you have to add (i.e. things that are not required to JIT JavaScript) besides the obvious modifications in the parser?
This is such a tempting thought -- that JavaScript and Lua are similar enough languages that an engine for one could be retargeted to the other with some parser changes and a few new features.
In practice it doesn't work out that way. Here is the story of Tessel, who originally set out to do it the other way around (implement JavaScript on LuaJIT) but reversed course after two years: https://tessel.io/blog/112888410737/moving-faster-with-iojs
"We believed that over time we could cover all of the corner cases of JavaScript with our own runtime, but perhaps we should have taken a cue from the “Wat talk” that it was going to be an uphill battle. While the semantics between JavaScript and Lua are very similar, they are also just slightly different in many ways (For example, the comparisons between null vs undefined vs falsy), and capturing all of those idiosyncrasies has proved frustrating for both the users and developers of the Runtime. [...] I still do believe it’s possible to make a nearly compatible runtime, but it’s going to take much more work than we expected and that resource investment would be an unwise business decision."
Lua has goto (and JS does not), which means not all cfg's are well structured. That could impact the compiler in certain ways.
Lua also has finalizers, which effect the GC's design. The Lua/C api also makes it impossible for the GC to move objects, which means pretty much every current JS VM's GC is out.
Lua 5.3 has 64 bit integers, which would effect most JS VM's in a significant way (but to my knowledge LuaJIT doesn't support 5.3 so...)
There are other issues too, but these are just off the top of my head.
> Lua also has finalizers, which effect the GC's design.
V8 has a weak callback mechanism, which (while not exposed to JS) allows reentering JS from inside a weak callback - which means you can emulate Lua's __gc on top of this mechanism.
> The Lua/C api also makes it impossible for the GC to move objects, which means pretty much every current JS VM's GC is out.
If we disregard lua_topointer then Lua/C API only leaks internal pointers for strings (lua_tostring), userdata (lua_newuserdata, lua_touserdata) and threads (lua_newthread,lua_tothread) - everything else is manipulated using lua_State's stack.
This means VM only has to take care with regards to these objects. Userdata and threads can be just allocated outside of movable part of the heap and strings can be "externalized" (i.e. they payload relocated into the immovable space) on first access via lua_tostring. Coincidentally last thing is something that V8 supports[1] (though of course externalization is not a cheap operation as it requires copying).
> Lua 5.3 has 64 bit integers, which would effect most JS VM's in a significant way
Yeah, that's certainly a whole ton of work, but most of this work would be pretty technical.
JS engines might actually get int64/uint64 value types in the future (at some point there was an ES7 proposal - but currently it does not seem to be on track for inclusion).
LuaJIT does not support 64 bit integers. It uses double NaN tagging for storing object references. That's the basic principle of LuaJIT design and one of the most important source of its superior performance. In this matter, it is very similar to JS VMs.
Support for 64 bit integers would require to abandon this model and completely redesign the LuaJIT VM. Mike opinion about that was very negative.
I believe that this was one of the reasons he decided to abandon the project: he was disappointed by Lua creators decision to introduce 64 bit ints and the fact that LuaJIT can't be made Lua 5.3 compatible without rebuilding it from scratch (but that's only my personal impression).
The reason people use LuaJIT instead of v8 is because LuaJIT is faster than v8 (on some code) and is smaller and is more easily embeddable. Or at least that is my impression, I have no personal experience with it.
For me the main reason was the FFI, even if back in the day it did not support re-entrance - e.g. something in the "C" land has to call back "lua" land.
But the JIT in luajit is simply too impressive to skip it over. I was able to quickly prototype things with it, running almost at "C" speed, and some times even faster.
I'd think that the javascript JIT would need to be gutted and extensively re-engineered. There's a reason why each language has their own jit - in order to speed up specifics parts of the language.
While certain things are similar, the specific optimizations I'm sure follows the spec of the language so closely that it's not readily transferable to other languages.
The general optimization strategies, like tracing, etc are techniques that can be ported, but if you start with a highly optimized jit for javascript, you're gonna have to rewrite large portions - so much that it would as much works as rewriting luajit from scratch.
If you have an array in Lua, which is a table used as a contiguous list of items, this is how you get the number of items:
num_items = #my_array
Your qualm, apparently, is that this # is not built for all use cases of tables.
In C++, what operator tells you how many instance variables a class has? There is none, because that wouldn't make sense in many use cases. Lua tables, offering prototype-based inheritance, experience a similar conceptual ambiguity in the case where you want to take the length of an object.
Lua has made a design decision that the built-in length operator only works on arrays = tables-used-as-a-list and on strings -- cases without ambiguity. If you want the length operator to make sense for a hash map usage, it's a few extra lines of work for you. It's consistent with the langauge being small and flexible, a theme which Lua embraces with elegance.
>Your qualm, apparently, is that this # is not built for all use cases of tables.
My qualms are multiple:
(a) this is an operator when it doesn't need to -- a function would do --,
(b) this only works for contiguous list of items.
(c) Lua, like PHP and JS, has the same type for "contiguous list of items" and hash tables, not even providing a builtin just for the first and/or the latter (e.g. ES6 now does with "Map").
I can understand the powerful metaprogramming capabilities of tables. I also understand that they are not needed, and are even a hidrance, in the tons of cases where all you want is a simple hashmap or a simple vector that wont change under your feet.
>* In C++, what operator tells you how many instance variables a class has?*
I somewhat agree, though it hasn't stopped me from using it extensively. The example that you give isn't a very good one, it makes more and more sense as you get to understand the language.
However: 1-indexed arrays; and words as block delimiters ("end end end end...") are things that make me regularly take a step back and think, this would be a bit nicer if the language did things the normal way. Note that 1-indexed arrays are especially annoying in LuaJIT because the FFI naturally encourages you to manipulate C arrays, often resulting in a mix of 0-indexed and 1-indexed arrays. Ouch.
I would have titled that as "CloudFlare starts discussion about LuaJIT project governance". We don't have a solution or prescription. We're happy to help Mike in the transition in any way possible and are taking baby steps as we do so.
jgc is the author of the linked email, and this submission initially had a title that implied Cloudlfare was laying out plans for the future governance.
A comment like this is a bannable offence on HN, especially when the account has made many previous comments that were uncivil or unsubstantive, as this one has.
Your account has some great comments in its history too, though. You clearly know how to use HN as intended. Please just do that from now on.
What was the best case scenario (in your opinion) for this message?
My mission is to speak the truth, ONLY the truth.
I don't mind ruffling feathers along the way, especially in regards to JGC, who is actively taking steps to destroy the FOSS principles of the LuaJIT project.
I'd sooner see myself banned than retract or apologize for a single character of anything I've said thus far, or plan to say soon.
I'm not sure I understand your question, but there are lots of ways to speak the truth, and on HN you're expected to do so civilly and substantively. Personal attacks are especially not allowed.
Sometimes people justify their uncivil, unsubstantive comments by posing as warriors for the truth and so on, but this is silly, because the truth isn't served by such posturing at all, and rudeness makes a comment less persuasive.
All this is being driven by Mike Pall who created LuaJIT.
If you really think that I, personally, am "actively taking steps to destroy the FOSS principles of the LuaJIT project" then get involved on the Lua mailing list and ensure that whatever principles you want are maintained.
The biggest disconnect I have with you is that you are asserting that I am messing up the FOSS nature of LuaJIT but I don't see how moving it to Github (with Mike's help), getting people involved as widely as possible etc. is doing that. What, specifically, are you concerned about in what I/CloudFlare are doing?
> Absolutely no concrete assurance (from CF officially) that the LuaJIT project will be developed openly.
The project was never completely developed in the open, with the test suite and the interaction between sponsors and Mike being private.
> On top of all that, the murky posts on the actual list indicate no significant progress has been made on figuring out a new direction for the project, and little to no effort has been made to do more than announce to the community what's (allegedly) going on.
The progress is indeed slow, but it is not surprising given the short delay since the announcement and the complexity of the code.
LuaJIT has currently a bus factor of one. Some people are familiar with some aspects of the code but there are no guaranties that they have the time to take over the project.
From what I gather, ATM, Cloudflare doesn't have employees that could step up and replace Mike, which means it would be foolish of them to burn the bridges with the community.
From what I gather, ATM, Cloudflare doesn't have employees that could step up and replace Mike, which means it would be foolish of them to burn the bridges with the community.
This is true. There is not obvious replacement for Mike at CloudFlare or elsewhere. I've been hoping that discussion on the mailing list would lead to someone emerging or a consensus.
Also, Mike has said he wants the transition to be 'one to two months' and I have not personally been involved with the LuaJIT community so stepping as lightly as possible while trying to do what Mike wanted (which was have CloudFlare take the load off his shoulders and help the transition).
I very much doubt that you and I are going to come to agreement on this. Mike's been involved on the mailing list. Perhaps you should address this directly with him.
Absolutely no concrete assurance (from CF officially) that the LuaJIT project will be developed openly.
The qualms I have with this dialogue are the same as before, because CloudFlare has little to no idea how they are going to handle this. JGC tweeted me about how 'Oh, we get so much benefit from LuaJIT being FOSS" but here we have CloudFlare walling LuaJIT into it's own entity on GitHub where I predict commit bits will be few and far between.
More than that, I don't think there has been enough narrative between Mike and the 'new LuaJIT crew' (CF) to determine how the project should be structured. In this thread, agentzh had a fantastic idea to vet somebody through Mike, someone the community knows can be trusted and also is somewhat familiar with the LuaJIT internals, and that person could serve as a canary between the project and CF.
I write a fair bit of Lua for game scripting, and I've even made a few bucks here and there helping folks with their custom plugin ideas etc, but I've never touched C before. Well, when the previous announcement was made, I immediately Amazon'd some C books, which I plan to devour in my free time. At which point I'll be learning Rust, and reimplementing LuaJIT in Rust, and hopefully convince Mozilla to host the git, such that it will be protected from FOSS corruption.
My worst fear is CF taking this project into the shadows, developing it closed-source (which they absolutely have a right to do) and not sharing their insights with the community.
I think everybody with any kind of invested interest in LuaJIT needs to be gearing up right now, such that we can do our parts to keep this project alive.
JGC tweeted me about how 'Oh, we get so much benefit from LuaJIT being FOSS"
And we do. We paid Mike Pall to work on open source LuaJIT, we've contributed to NGINX, hired people to exclusively work on open source projects. Here's the harsh economic reality: it is simply better business for us to spend a relatively small amount of money on open source support to get what we need from fantastic projects like LuaJIT than to try to develop this stuff ourselves.
My worst fear is CF taking this project into the shadows, developing it closed-source (which they absolutely have a right to do) and not sharing their insights with the community.
How do we "have the right to do" that? Whatever makes you think us trying to closed source this would have any benefit to us? How is the Github account (of which Mike Pall is an owner) us walling it off?
More than that, I don't think there has been enough narrative between Mike and the 'new LuaJIT crew' (CF) to determine how the project should be structured.
I predict that if I hadn't sent an email to the list soliciting ideas and input and had announced a new structure you would have complained that everything had been done in the shadows.
Where is the proof that CF is doing anything remotely like what you say? All they've done is create a Github repo (which is many developers preferred platform) and say that they are moving to a more distributed governance model. All at the request of the original maintainer, Mike Pall.
They've even explicitly stated that CF will not be taking over the project; they're only helping move it to a new home and to find a group who can maintain it to replace Mike Pall. There's zero evidence that CF's intentions are to the contrary, so I do think what you are saying is completely unfounded. Please correct me if I'm wrong.
While logicrime's comments may be wrong and a bit over the top, I don't think this comment should have been flagged. We've seen a well-explained criticism of a potential, if unlikely, future. These comments are much less inflammatory than other comments that have been left alone on HN.
I still plan on embedding LuaJIT into my automation software, so I personally look forward to to seeing what CF and any LuaJIT successors produce. So let's take logicrime's fears into consideration instead of just shutting them down.
I'm not accusing anyone of incompetence. Only a fool would make the implication that you aren't a knowledgeable programmer. I'm accusing CF (not you specifically) of making the first moves to morph LuaJIT into a corporate tool.
> Well, when the previous announcement was made, I immediately Amazon'd some C books, which I plan to devour in my free time. At which point I'll be learning Rust, and reimplementing LuaJIT in Rust, and hopefully convince Mozilla to host the git, such that it will be protected from FOSS corruption.
The point isn't the LOC. If you've never even touched C, you're not just going to have to learn that, you're going to have to learn how to write an optimizing compiler (because frankly if you've never touched C I'm skeptical you have any experience in this). And not just that: you're going to have to learn how to write the world's most optimizing trace-based JIT compiler for a dynamic programming language.
Mike spent 10 years designing LuaJIT and it is in a league of its own, not paralleled by anything else. Do not expect this inane project of yours to be solved by looking at 25,000 lines of C code. Especially if, point of fact, you do not even know C. Expect it to be 'solved' after a decade of research and hard work at minimum.
I'm not sure what paranoid reality you live in where you think this is feasible, or even desireable given your original post (frankly even though a port isn't needed Rust would be an awful choice for a 'port' due to the fact it's simply not got as good availability, the compilers and tools are less mature), but when I said see you in 15 years, it wasn't a joke - it was a conservative estimate.
