I usually don't comment on these posts, but I felt the need to in this one. A couple of things stood out:
- Python 2.4? Really? In 2014?
- Yes, writing compatible code with a 10+ year old release of Python is going to be hard. Who uses 2.4 still? You have to use the lowest common denominator, I mean I don't think Python 2.4 even supports context managers!
- Yes, in 2014 you should be supporting Python 3, or at least have it on your roadmap.
- The ethernet address (MAC?) via uuid.get_node(). Or for something a bit more advanced you can use netifaces[1] (or even better psutil.net_if_addrs[3]). One of Pythons core strengths is the number and variety of the packages available. Took 2 seconds of googling to find a cross platform solution.
- Calling C from Python is really really simple. Use the built in ctypes library, or something better like cffi[2]
I'm obviously firmly in the Python camp so perhaps I'm a bit biased, but I don't see any clearcut reason to switch other than "the current code is not optimal, let's re-write it in X", where X could be anything. I would actually say Go would be a much better fit than Lua in this situation. I guess the memory reduction/CPU usage is a point, but really your program seems to be network orientated. What's wrong with asyncio? How is replacing everything with a C library actually better?
Pretty much this. Also US Govt and especially the millitary. When they buy "Linux" they buy RHEL. (Sometimes SUSE when they are working with Lockheed). Mostly because of support and RHEL5 and RHEL6 are certified with an alphabet soup of certifications (EAL4, FIPS-140-2, etc) and so that is what is used. Heck even RHEL7 doesn't have all those rubber stamp even though it is 2 years old almost now.
I don't do a lot of python anymore but we never depended on installed version. Everything was in an a virtual environment where we could control the libraries.
virtualenv doesn't manage different versions of the Python interpreter though. By default it uses the system Python. If you want another Python, then you need to install it yourself (somehow) and then point virtualenv at it. virtualenv is just a way to create separate bundles of libraries.
It's for both libraries and different versions of Python. This is common and trivial.
I install specific versions of Python using the Deadsnakes PPA since I can easily pull in any version going back to 2.3 if I need. Or you can compile and install from source.
Then create a folder for your project, 'virtualenv -p /usr/bin/python3.5 venv' to create the venv in that directory.
Activate/deactivate the environment as you wish when you want go between the system Python environment and the one in your folder.
I replied to @mrits because @mrits was presenting virtualenv as replacement for using pyinstaller to manage the installation of non-OS-managed Python versions (which @cowsandmilk was talking about).
I'm unsure how what I said:
> If you want another Python, then you need to install it yourself (somehow) and then point virtualenv at it.
You can include the python interpreter inside the virtualenv. It's not entirely painfree (python and any pip packages are normally dynamically linked), but it works reasonably enough, especially when deploying it on the same OS version.
By default virtualenv links to the Python interpreter that you point it at (when you create your virtualenv). I'm unsure what method you would use to include the Python interpreter install in the virtualenv directory (other than creating a normal virtualenv, and then copying the files in after the fact).
I sometimes wonder if it's possible to use a gobo-linux style system, whereby the package manager just downloads the package to a versioned folder, possibly expanding it out, to a a package cache - then the venv need only symlink to it. The same could be done for python installs. Then do the same for the system python (e.g /bin/python points to /someplace/pyjunk/system/python, which in turn is a symlink to a particular interpreter, e.g. /someplace/pyjunk/interpreters/py2/2.7/cython_2.7_blub-i386/python).
I never understood the fear of leaving the system's package manager's packages. Maybe I've been on Slackware too long, but compiling/installing packages from source is the norm. Also, package managers tend to ship with --enable-everything-everywhere-no-matter-what configure options. Compiling from source you can say yes, support libpng, no don't care about libjpeg etc and get a much leaner setup.
I know there's the argument that by straying you don't get security updates, but I can't really imagine Python 2.4 getting much security updates nowadays.
On the other hand, occasionally security updates will be surfaced as a patch file and you're waiting on the package managers. At least if you build it yourself you have the option of applying patches.
This is just an argument against bad package managers. Good ones will offer binary packages by default, but allow you to tweak (or just pass ./configure options to) an installation formula, at which point they'll switch over to pulling in the source package, applying your changes to it, building it, and installing the result, while still offering you full package-level management of the result (including having the package satisfy deps correctly.)
I never understood the fear of leaving the system's package manager's packages.
It's trivial when you have a handful of machines, all under your control. It's somewhat less trivial when you are dealing with 1000s of machines under the control of dozens of different people/departments.
Hi, I'm the Founder at Distelli and I wrote the original version of the Distelli agent in Python that Brian rebuilt using Lua. The main reason I picked Python 2.4 is because we have large enterprise customers that are / were running Python 2.4 on servers and I had to support Python 2.4 to serve their use case.
They were probably running RHEL5, but even in 2014 they would have been in the Production 3 phase which is really should only be used for legacy purposes, and that's definitely not the sort of system you push new software to. Red Hat even pushes those customers towards virtualization because a) hardware rarely lasts 7 years and b) they won't bother backporting drivers for new hardware to the old kernel at that point.
I still use a few RHEL5 machines but nobody would ever dream of pushing new software to them.
In the real world...if you have a large fleet of RHEL5 boxes/VM's, this a fact of life. We're slowly phasing out our CentOS 5 boxes, but you still need to be able to run new software on them.
This is something I don't understand. If someone is using Java, they'll install the version of Java they need. If they use Ruby, they'll install the version of Ruby they need. If they need Lua, they'll do the same thing.
Why everyone is relying on python that's installed there for RedHat's needs (e.g. yum).
Unlike other languages, Python is probably the best one that can have multiple versions coexist together.
A lot (all that I'm familiar with) of the enterprise software packages that I've worked with ship with their own, very specific (read: regression tested against) version of java. Enterprise software usually requires a ton of servers, so it's not like anything else on those servers will be using the jvm.
> Why everyone is relying on python that's installed there for RedHat's needs
Environments where the build of your server is strictly controlled and conservative...banks, mission critical apps, that kinda thing where stability from upstream is hugely important.
AFAICT if you want support from Red Hat, you better not mess with the system configuration. It's probably best not to mess with too much on your production boxes anyway (maybe that's why Docker is so popular)
How about running a (newer) local version of Python, without of course touching the system Python?
To my mind, it's usually a good idea not to use system Python, both packages and the interpreter, if you're not also controlling them. This makes building your deployables more involved, but makes the actual deployment more painless.
> I had to support Python 2.4... large enterprise customers that are / were running Python 2.4 on servers...
How did the switch to Lua not break this in the same way?
Sorry for the naive question, I'm just not following how it can be possible to change languages, but impossible to change versions. At first glance, the latter seems like a special case of the former.
Shipping it as a single download with zero dependencies meant that we would not have to worry in the future about what else was installed on the machine and the versions of system packaged the customers were running.
That's fair enough, but it sounds like you've constrained yourselves and your code for the sake of a few enterprise customers, and now you've had to clear your technical debt with a complete rewrite. Even if batbomb were right I would argue that any solution is better than supporting 2.4. The very minimum you would want is 2.5 at a stretch.
In that case, Lua makes a lot of sense, frankly. I can write 100 lines of code in perl, but trying to get that deployed on mac, windows and linux can be a nightmare, especially if its using a library external to perl.
I'm a Python devotee, but I appreciate the anecdote. I'm curious whether the switch 1) cost any features/algorithms and 2) whether the anticipated performance improvement really panned out since the switch, or if the gain wound up being simply architecture/deployment-related. Stepping back, did the whole re-write really balance out with all the work that had to be done, or could you have stuck with Python and simply solved the problems at hand?
1) All features/algorithms were ported, and we only added new functionality that never existed in the old Python code base. A few of the features that got added include: dispatching builds, getting live logs, re-encrypt keys, deploy encrypted artifacts, fix many bugs, support running multiple agents on a single computer, etc. I should also mention that we previously had two separate downloads: a CLI used for uploading releases, and the agent used for dispatching deployments. The functionality of both of these code bases where merged into our new Lua based "client" so customers only have to install a single binary that is less than 10MB (the binary size depends on your platform).
2) Deployments with the new agent were noticeably faster, (like 1 second deployments that used to take 20+ seconds). However, this speed up was largely due to the Python codes design which had a "continuation" loop which was polling based, but the Lua code used coroutines and simply continued the threads when the steps were complete.
Overall, I think the choice to use Lua (or more specifically luvi) was a great decision. Note that it did take a few iterations in order to come up with the optimal way of using Lua + libuv. Originally I was taking the Node.JS approach of using callbacks, but that approach has two problems:
[a] It is difficult to properly implement "back pressure" so that all queues between producer and consumer are bounded.
[b] It is difficult to handle errors properly.
Lua coroutines allowed me to write a simple "green thread" library that encapsulated these challenges.
Couldn't you have run two versions of Python at the same time on their systems? Would it have been possible to ship a Python version with your software?
"I had never seriously used Python before joining Distelli, so there was a bit of a learning curve"
That was your problem, you just made a huge mistake and depending on how big your python code base was, you have now alienated most of your developers by moving to a language they are not already comfortable with.
Do your customers already have lua installed on their servers or are you bundling it now? You could have just a easily bundled a newer python interpreter. Lua is a nice scripting language, if you want to embed it into another app or to create a plugin system. It's not a nice general purpose scripting language, mostly because of the vast pypi repo.
How is this alienating anyone? This is a command line tool. The end users are suddenly using a different language as a result. Their install process was streamlined and system dependencies were removed by including it in the install package.
The same thing could have been achieved within Python, but from an end user perspective I'm guessing this is an improvement for most of their customers.
It's alienating Distelli developers. The reason cited for the move is totally unacceptable in terms of the new skillset that their developers will need to acquire to continue development of the tool.
The cost of effort their developers will have to put in to be as effective as a LUA developer as they were as a python developer is really not worth this move. There are more to moving to a different platform/language than just the end users.
They only have a few developers, so it's really not a big deal. I think they only have 2 or 3, including the CEO who, obviously, has a lot of other things to do than write code.
Being as the size of the team is so dependent on a small number of high performers, I don't think it's unusual that they picked a language one of the main people in the company is extremely fluent in.
I've worked with Brian before and he is a beast. I don't know if 10x engineers exist, but if they do I would certainly consider him one.
Describing him as a 10x developer makes me even more skeptical of the decision to switch. Developers who deliver 10x as many features inherently do so at the cost of maintainability.
> Developers who deliver 10x as many features inherently do so at the cost of maintainability
I don't think what you say is completely true. People who are more experienced tend to be able to produce features much more quickly than those who are junior, just because they've already run into issues before so can anticipate them or solve them more quickly. If anything I would argue people like that actually write code more quickly that is more maintainable.
I don't know if he can ship 10x as many features; I didn't think term 10x engineer was used literally. I just know I've worked with quite a number of people and he is one of the best developers I know.
The end users are the ones that keep the lights on though. It sounds a lot like that Python was more and more of an anchor around the team, and the costs of migration were smaller than the costs of the other options. Yes, some developers were probably upset that they couldn't use their preferred language anymore. However, it sounds like because they're now using a language designed to be embedded, they can focus a lot more on solving problems than fighting with a language that has a lot more platform-specific behavior.
Are you in a position to speak for Distelli developers?
"The cost of effort their developers will have to put in to be as effective as a LUA developer as they were as a python developer is really not worth this move."
I suppose this could be true in a poisonous environment where scheduling does not accommodate the need to learn a new skillset.
Unless you are familiar with the employer this is an incredibly hostile presumption. I.E. I would not work for such an employer nor would I want to be their customer
What does accommodating for a new skillset make a difference to the cost?
There will still be a minimum effort (e.g. in hours) that cannot be reduced by the nature of the employer. A dev will still have to put in the hours to learn a new language.
It seems the cost being described here is cost to the employer, not to the developer.
I use Lua as a general use language and I'm incredibly satisfied. It's an amazing language.
I also need to mention that Lua is the easiest language I've ever encountered, so any competent developer can learn it quickly, it's not alienating anyone.
Given that, I agree that LuaRocks isn't so vast as other languages' repositories and this can be a drawback of using Lua. But why again are you judging a language by it's package manager? If it's a good language wouldn't it be better to suggest more people to write packages instead of suggesting people not to use it?
> "the current code is not optimal, let's re-write it in X"
I read it more as "relying on a runtime that must be installed on the user's OS instead of included in our executable is a nightmare". The dramatically reduced conceptual footprint of the language is a nice bonus though.
Honestly, every time I hear a story like the one in the OP, I'm tempted to find a way to make once-per-decade upgrades be so ruinously expensive that companies will stop opting for agreements structured around them.
Sure, maybe there would be a few "smart" companies that do a proper cost-benefit analysis and decide to budget for a faster upgrade cycle or that bigger chunk of change for decade upgrades... but most companies would file it as a problem "for somebody else" in a decade as most companies don't even seem to think past the next quarterly earnings report.
My experience of this is there are a few categories:
* Some companies have absolutely no in-house tech staff to speak of, and so they want to invest in technology exactly once and never have to deal with it again.
* Some companies have in-house tech people, but are entirely marketing/sales driven with little or no input from the tech staff. So they focus exclusively on writing new features and never clean up technical debt.
* Some companies choose a platform to bet the farm on, develop a huge amount of in-house code for it, and then realize that since they wrote it for specifically the exact version of the platform they first installed, any upgrade involves a complete refactor/rewrite, which they then dismiss as too expensive.
The first case can be solved somewhat by subscription-model managed services where the upgrades just happen at the discretion of whoever's managing the platform.
The other two are failures of management and planning, and absolutely should be laid to rest at the feet of the management staff involved. Preferably in an expensive way, pour encourager les autres.
Or the new version doesn't offer a compelling reason to upgrade, and would require extensive regression testing, potential downtime, and potential lost money for little to no benefit to the end user. Upgrading core software costs money, upgrading core software just because a new version comes out costs more often. Sure one should test to see if there is a compelling reason, but if there's not, why should one mess with something that already works well?
Refusing to "mess with something that already works" is a recipe for disaster.
What companies often learn too late is that there are basically two options:
1. Make maintenance -- both in terms of upgrading underlying software, and addressing accumulated technical debt -- a regular, required part of their process, or
2. Occasionally face a potential existential threat from the unbounded cost of staying fixed in perpetuity to a stack that "already worked" once upon a time but no longer does due to lack of upstream support and available developers to keep it limping along.
The genuine problem is that "what does this do for next quarter's results" is the only concern, which means long-term thinking is actively discouraged.
Installing from source is really not that terrible - we do so for one of our clients (running CentOS 6.5, IIRC) and it just took us a couple of hours to have a full build script.
uuid.get_node(): ...and which iface's address will I get if I'm on a multi-interface machine (not to mention that it was added in 2.5 version of Python)?
netifaces: ...would require me to build a C extension which would mean going down the rabbit hole of building and bundling python and this library.
As opposed to shipping LUA with C extensions? Also, there are things like shedskin: https://github.com/shedskin/shedskin - you don't even have to ship python. Or micropython: https://github.com/micropython/micropython with the code embedded. Then you can easily either shell out or include any C extensions you want.
The overall tone of the responses here are surprisingly negative and doing a lot of Monday morning quarterbacking, as if this was some kind of failure.
There are wild assumptions about their clientele and the desires of the company, completely disregarding the fact that the company did this to meet their own stated objectives.
They wanted something easy to deploy (single executable), easily portable, very small CPU footprint, and very small memory footprint.
Lua is well regarded for all these things, and the company succeeded in meeting all these objectives.
Congratulations Distelli. Nice job. And thanks for sharing.
It may be true that they meet their stated objectives, but some of those objectives could have been achieved without a rewrite, and some of those objectives don't make sense.
I've written plenty of single executables for multiple targets in Python. Figuring out how to do that would certainly have taken less time than the rewrite.
Small resource usage may make sense, but they didn't relate it to any actual customer need, and given it was already running on RHEL, I doubt this was a reasonable goal. Unless it also needed to run on pocket calculators, the resource needs of a Python program are well within reason for most target environments.
To be clear, I don't think Lua is a worse choice than Python for a new project; my objection is to the rewrite. There are problems with both languages. Sure, now packaging is easier, but when he runs into a hard problem that would have been solved by an existing Python library is he going to rewrite again?
It's a junior developer mistake to blame issues on your tools and rewrite instead of getting things to work in your existing environment. Mature tools like Python (or even to a lesser extent Lua) are rarely the problem.
There are certainly exceptions where there are serious problems such that changing tools makes sense. But none of the problems mentioned in the writeup are that.
The bad part of the rewrite is that there is no possible way to know what was lost. There are edge cases which were discovered and handled in the old product that will have to be rediscovered in the new one.
It sounds like they haven't lost any business over this, so it will be okay in the long run. But I suspect they could have done this a lot easier and cheaper. Just because it turned out okay doesn't mean this is an exemplary choice we should learn from.
One of my replies may be in your "negative" pile. It wasn't intended to be negative, just to point out that the cited reasons in the article for the switch weren't specific to python...just how they decided to use it. They may have been concerned about packaging python because of the size, but didn't call that out.
Had the title been "Why we Chose to use Lua" it probably would have garnered more positive comments.
If you rewrite your app you end up with something better, in any language. It's an uncontrolled experiment, and so the headline is very likely misleading - this isn't about Python and Lua. And many of us have been majorly inconvenienced when a manager read a similar headline and declared that we would rewrite our app in language y, regardless of whether that language is appropriate to our circumstances.
Something like http://roscidus.com/blog/blog/2013/06/09/choosing-a-python-r... where the author actually considers multiple candidates and performs a controlled comparison would be far more valuable. Given that the goals sound very similar I'd be very interested to see a comparison between Lua and OCaml.
> If you rewrite your app you end up with something better, in any language.
You end up with something that is superficially better, but misses a lot of edge cases that the ugly legacy application covered. Fortunately, you have usually moved on to save other projects, leaving lowly maintenance developers to fix the problems, thereby creating job security. It's a win win situation, really.
Working code is worth a lot. No amount of unit tests will make up for the real user testing done in the wild. I have had some horrendous pieces of code that I kept because I knew they worked.
I'm the founder at Distelli and I just want to add a quick bit of background around the original python decision (2.6 vs 2.4) as well as the decision to "rewrite in Lua".
1. I started the company as a single founder and wrote the first version of the agent. See this HN post that gave me the encouragement to keep going (thanks HN!) - https://news.ycombinator.com/item?id=6059481.
2. I wrote the original agent in python 2.6. It was great because I could use things like python-requests (which was awesome!). However my first encounter with an enterprise customer showed me that was not feasible because they were running only python 2.4 and would not be upgrading for another 2 years.
3. I had the choice of either not getting the customer or back porting the code to Python 2.4. I chose to back port the code to python 2.4 and get the customer. This was a lot of work but worth it because I was able to then raise a round with a16z. - https://www.distelli.com/blog/distelli-series-a-funding
4. When Brian joined Distelli, he actually worked on the Python code for many months and made many improvements and added many features. When it was time to add the streaming logs feature he built the prototype in Lua just to prove that the idea would work and then we talked extensively to figure the direction we should go in.
5. We decided to move to Lua because Brian was very familiar with it and it was a lightweight language and we could ship a single download with zero dependencies. We also looked at using Go. At the end of the day the decision was Brian's to make and own as the senior engineer on the project. The passage of time has convinced me that was the right decision.
I'm sure we could have stayed with Python with all the suggestions on this page, but we decided to go with Lua. It was a good fit for what we needed and honestly I felt that my original code was not in great shape and would need a lot of work to refactor and a rewrite felt like the right decision at the time. Looking back I still think it was the right decision and I'm very happy with the end result.
I have been programming Python by day and Lua by night my entire career for the past 7 years, and I find the main difference between the two is, Python has the overwhelming advantage in its package manager providing ease of access to thousands of packages[1], and Lua is faster and lighter by a constant factor. In your case, Python's package system is a disadvantage - you want to have as few dependencies as possible.
I think your company made a good decision. The only reason not to have done it was because your existing code was valuable, but as you've said it wasn't in great shape and required tons of work to refactor and rewriting anyway.
I've had that happen about as often as confusing Python dictionaries returned by different types of functions. (Which is - not often enough for me to remember the last time that happened)
- tables with "holes" (nils) in them,
When I'm building a vector and I want holes, I use false to fill the holes.
- multiple assignment with wrong number of vals or vars (mismatch silently ignored),
For me, when I'm thinking about multiple assignment to function returns, it does mean having to read documentation for a library function more carefully than if I was reading the same for another language. If it's as simple as "local a, b, c = 1, 2, 3", then it's no issue.
- variables are global by default.
If I'm using linting, when I can't have global variables implicitly, then it's like C or Java where I must declared all my variables and by default they are local to the scope.
Otherwise it's just like javascript where it's best practice to always declare variables using "var".
The described issues that prompted the switch don't really have much to do with Python. The issues map back to running on whatever installation of Python the customer might already have.
Lua, in and of itself, didn't solve the problems. Shipping their own, known, Lua interpreter + extensions solved them. (which they could have done with Python)
The reason why people don't want you to ship your version of Python is that it takes a lot of space. Lua makes that hurdle much easier.
For example my local Python venv is about 130 MB. It symlinks in another 52 MB of standard Python libraries. And I know it won't run without some unknown amount of additional dependencies in C-libraries. If I wanted to send that to someone I'm starting at 200 MB. After compression that might be 40 MB or so. You're going to have to work to strip that down.
By comparison their download including Lua, code, and libraries is under 6 MB.
I have a minimal running Python application that was started recently and has done nothing. It takes close to 90 MB of RAM just to start.
Theirs is generally under 10 MB while it is running.
So at every step Python requires 5-10 times as much data. If you're trying to get other people to put you in their containers, this can matter a lot.
PyInstaller will generate a ~minimal installation. I've written an app which has a full python runtime, and images, and dozens of python files and modules in 6MB.
Or it can not matter at all. To the kind of enterprise that is still running RHEL 6, maybe shipping a 200mb install on a CD will show them that you're a serious business.
There are use cases where 200mb of disk space matters, but for the overwhelming majority of applications, a 5% (say) increase in developer productivity would be well worth adding 200mb to the install.
The point of the Distelli Agent is to deploy software. Think of it as the "bootstrapper". Serious customers would rather use that 200MB for their own software, not for the "bootstrapper" :). Here at Distelli we optimize for the customer, not for our own internal developer productivity.
Good soundbite, but if using 200MB of extra space would let you lower your prices while offering the same service (by increasing developer productivity) then I bet many customers would take that.
And did you try spending a person-week or so of dev time trying to minimize the size of the Python version? That would be a much smaller one-off cost than the costs of migrating everything to Lua, and I could easily imagine you'd get it down by one or two orders of magnitude. (You could do the same for Lua too I'm sure, but if we're talking about say 5mb vs 200k then that's "who cares?" size for most customers).
Nowhere in this article does it really state that they're less productive in Lua, though. In fact, the main engineer on it is well versed in it. Your 5% may actually be negative in this case.
From where do you get this impression that Python is the only high productivity language in the world? There are plenty of languages that can compete with Python in terms of productivity and a lot of them offer better abstractions for it. Lua is known for being one of the best escape hatches for scripting.
If you're more productive in Lua than in Python then I support using Lua 100%. It's switching languages to gain a measly 200MB of disk space that I'm objecting to. For 99% of projects 200MB is just an irrelevant consideration compared to how much difference language productivity makes.
For general cases, I agree that 200 MBs isn't enough to consider very important nowadays, but I still don't get how we're automatically talking about language productivity being a trade-off. Is the assumption that Python is a more productive language regardless of what it stands against?
btilly was talking about the disk space as if it were important.
My opinion is that Python is more productive than Lua, but if you disagree by all means use Lua. But if you're talking about saving 200mb of disk space as though it makes the difference, then I think you're using the wrong criteria for choosing your language.
This is exactly right. From "Challenges with Python " in TFA:
> I ended up introducing incompatibilities with older versions of Python when I tried using a finally block with an exception block—that syntax was not available until version 2.6
Lua changes major parts of the language in minor revisions. 5.1, 5.2, and 5.3 are not compatible, and in much larger ways than Python 2.6 and 2.7. Lua 5.1 is currently the most recent version supported by LuaJIT which he specifically mentions. Lua 5.3 introduces integers!
> For example, I needed to add support for detecting the Ethernet addresses of the host computer, which can be done with the netifaces native library (or shell out to ifconfig and parse the results, which would not be compatible with Windows).
You're going to have exactly the same problem with Lua, except that that library won't exist so you'll have to write it.
Lua's great, but this isn't a case of Lua being better than Python for your use case. It's a case of you redoing your deployment in such a way that you happen to have more control.
Lua is famous for breaking compatibility between versions. And Lua users are quite aware of the situation. But outsiders tend to overblow the situation, making more noise than the people actually having to deal with it.
The Lua creators are very careful and deliberate in how they break compatibility.
One such thing they do is they avoid subtle changes that can lead to incorrect/different results, and prefer hard things that lead to obvious compile time errors.
But most of the time, the changes between minor revisions aren't a big deal. On the Lua side, authors have been pretty good about figuring out how to make their scripts run across all versions because the changes haven't been that big. On the C side, the API changes can be a little more dramatic, but authors have been good about handling the differences.
Most are stuck with Lua 5.1 anyway as it's the version supported by LuaJIT2. But 5.1 is very good. And the Lua community is very loosely connected and quite different due Lua's embedded nature.
I made much Lua code, and one thing that makes a major difference, is that Lua devs frequently discuss what they are planning to break.
For example I wrote many games in Lua 5.1, that actually work in Lua 5 if I wanted to, and also work in Lua 5.2, despite 5.2 not existing when I wrote them.
I don't checked yet the changes on 5.3, but I suspect my games might work on 5.3 without changes too.
You pretty much always ship Lua embedded in your application though, as it is just a C library. It is the expected use case, rather than using a system copy.
This article does not say Lua is "better" than Python, it simply describes why we choose Lua. Bundling Python and rebuilding for each platform was certainly an option. However, luvi has the advantage that it has many platform specific builds prebuilt and extending luvi with more CMake files + C libraries is easy.
Plus me (the author of the Lua agent) was already familiar with Lua, so the easiest way to get from A to B was for me to use a known-good tool.
Exactly. It wasn't about Lua being better than Python, just that "in this challenge, the solution was Lua". Another challenge will have another solution.
I've only ever done game development in Lua and have never considered it for anything else honestly. Very cool use case here.
I don't think they could have created a statically-linked Python binary that contained the entire Python installation and core libraries, and the entire application in less than 5 MB. The entire app in a self-contained executable. No futzing with PYTHONPATH, LD_LIBRARY_PATH, virtualenv, installation, conflicting with the system-installed Python, etc. That's what Lua let them do.
But don't take it from me. Here is Guido van Rossum last year:
> The final question was about what he hates in Python. "Anything to do with package distribution", he answered immediately. There are problems with version skew and dependencies that just make for an "endless mess". He dreads it when a colleague comes to him with a "simple Python question". Half the time it is some kind of import path problem and there is no easy solution to offer.
In cca 2005 timeframe we used to ship py2exe-d windows desktop application (using wxwidgets framework), with exactly zero issues. The target was a wide range of windows releases, from 95(!) to XP.
I haven't seen any of problems mentioned in the blog post.
This is really meant for monkmartinez, but you're right.
I'd like to add, that blog can be dismissed outright. It's also out of date in regards to PyInstaller. It does support Python3 now. 2.7;3.3-3.5. I've used PyInstaller for years and prefer it over the other packaging solutions, it's really good.
I wouldn't be digging up old blogs as a source for dirt and relying on any information within without verifying it was or still is true.
Exactly my thought, that whole operation seems like it could have been done in Python. This has nothing to do with the Lua being a 'special' or 'faster' language.
It isn't linked to in the article, but if you're interested in this please do check out luvit by tim caswell (https://luvit.io/). It's a very impressive system.
At a very high level luvit looks like node.js for Lua, with some extra goodies thrown in like being able to bake your apps into single binaries. This is awesome in itself and the article discusses these things.
That said, I think the major innovation over using javascript is that it allows the support for coroutine-based I/O. This means you can get the performance benefits of non-blocking node.js without the headache of callbacks/promises. I think the linked article only touches on coroutines briefly but they are a major piece of Lua's awesomeness that you shouldn't miss.
Though you aren't required to use coroutines, the package management server which hosts packages for luvit is called "lit" and it seems to demonstrate this ability nicely:
"Lit is written in lua and uses the same system platform as the luvit project, but is has a very different I/O style. Instead of using callbacks and event emitters, it uses coroutines so that code can be written with simple blocking I/O, but still maintain the abilities of a non-blocking event loop." (from the README on https://github.com/luvit/lit)
I also suggest looking openresty based frameworks, it is non blocking I/O, but you can write procedural code normally and Nginx will handle events internally. That means no callback hell
There's Lapis, by leafo http://leafo.net
And I'm also improving Nginx experience on the framework I develop, Sailor, because it was originally developed for Apache http://sailorproject.org
On the python side, this matches my experiences with python. Python is a great language, and I like it very, very much. However, packaging and deployment in python remains such an amazing mess, even after 20 years.
And outside of scientific computing, python doesn't offer any real edge, to me at least. Ruby has better tooling libraries, java/go/c++ have better performance, JS has more powerful website frameworks.
Why should I bother with that packaging mess after having it done once already?
> packaging and deployment in python remains such an amazing mess, even after 20 years.
I'm Lua fanboy, but I've to admit, that I'm missing clean deployments and upgrades with virtualenv and pypi available in Pyhton. I've never had any issue so far, am I being just lucky bastard? It has been like that for many years already.
> Ruby has better tooling libraries
On the other hand, I've honest to maintain one Redmine instance (which is written in Ruby) and this is real nightmare for me. Each time I've to upgrade, I've to drink a lot day or two in advance :-)
> "....Redmine instance (which is written in Ruby) and this is real nightmare for me. Each time I've to upgrade, I've to drink a lot day or two in advance...."
Well that is your problem right there - don't drink in advance, drink afterwards to try delete the traumatic experience from your memory!
those handle python deps ok, but not lib dependencies, at least last time I was forced to use it. If your company has a mix of older and newer projects, you'll end up with eg conflicting dependencies on libxml.
My solution was to docker all the things. It made my life so much better; you could simultaneously run multiple services on the same box without awful fragile hacks.
Docker is a great tool, but not all customers will have that available to them, and the docker experience isn't great for customers on Windows/Mac since they have to run Linux in a VM, which means the agent would have the footprint of an entire OS on those platforms.
> I'm Lua fanboy, but I've to admit, that I'm missing clean deployments and upgrades with virtualenv and pypi available in Pyhton. I've never had any issue so far, am I being just lucky bastard? It has been like that for many years already.
There's a few python libraries out there which just crash the entire stack, like Python-Mysql (or was it PyMysql, or one of the other very similarly named libraries?). And I might be biased, but talking to a mysql-db shouldn't be hard.
>On the other hand, I've honest to maintain one Redmine instance (which is written in Ruby) and this is real nightmare for me. Each time I've to upgrade, I've to drink a lot day or two in advance :-)
I rather mean tools like rake as a task-runner, guard, rubocop, thor as a templating system, rspec, chef as config management, kitchen, food-critic, sinatra for easy web services. It's a whole bunch of stable tools and tooling libraries (and they are integrating with each other).
Python is a great language, and I like it very, very much. However, packaging and deployment in python remains such an amazing mess
When are dynamic languages finally going to learn from these problems and fix them? Smalltalkers were making these exact complaints to me in person in the 90's and I know they predated my involvement by a lot.
What if someone created a Smalltalk-like uber-debugging-godlike-power-over-the-runtime alternative compiler target for Golang? Then you'd have all the good parts of dynamic languages with the dead-simple distribution model of Golang.
I have recently installed Go on my machine. Then I made a quick little server and put it on Digital Ocean. It was seriously the easiest thing I've done with "shipping" code in a long time. I love Python, it is my hammer and everything looks like a nail. However, Go makes it so easy to share/deploy all the things. I could see Go becoming my hammer.
Have you tried writing "a quick little server" in Python and deploying it?
Packaging and deployment aren't problems in Go OR Python if you're only solving trivial problems.
With a larger project, you might have an easier time packaging and deploying Go code, but that's largely because there are no libraries to package. Admittedly packaging can be a pain in Python, but that's usually because of poor choices in dependencies. Packaging Python with a few mature dependencies isn't hard in my experience. It's the projects where some idiot has pulled in every 0.x versioned library in pip that are hard to package. When Go has as wide a variety of libraries as Python people will run into the same problems in Go.
You could argue that at least for now Go doesn't allow you to shoot yourself in the foot that way, but I'd rather have the option.
I'm not defending Python in particular here. I'd say the same things I've said here about any mature language with extensive libraries.
If you're espousing Go because of easy packaging and deployment, I strongly suggest that you consider whether that's actually a feature Go will have long term, and whether you're currently paying for that feature by having no libraries available to you. The only lesson I would take away from Go's easy packaging is to only use mature dependencies that pull their weight.
Go and Python deployment aren't quite that equivalent. I've seen build and deployment problems over multiple years with widely-used Python libraries including Python-MySQLdb and Python Imaging Library (PIL). The Go equivalents for such essential libraries are typically in pure Go, so it's much easier to avoid C build and link errors when choosing project dependencies. The prevalence of pure-Go libraries is likely to remain high in the long term even as Go's base of libraries expands.
C build and link errors are only one of a myriad of issues that one can have with packaging. I've had far more problems with badly-managed packages in pure Python than with stuff like PIL. And one only has to look as far as JavaScript to see that pure native language dependencies won't save you.
I stand by what I said: give it a decade and Go packaging will be just as miserable as packaging in any other language.
Have a look at OCaml. For simple standalone binaries it's a real sweet spot: it's powerful and safe without being complex. It has the best module system going and a mature open-source ecosystem, and in the last few years it's converged on a good package/dependency manager. There's a great series of blog posts from someone who looked at replacements for Python and ultimately moved to OCaml, summarized at http://roscidus.com/blog/blog/2014/06/06/python-to-ocaml-ret...
i played with it last week but decided to play around with Rust instead since there was no debugger support and the official statement from the go team was that they have no intention of making integrations with gdb
Yeah, the only real advantage of lua is that you can link your code into the binary, and that it's super lightweight. But you can still deploy a python binary with your code, so the only real reason for switching is, "I like lua, and I felt the need to make my team switch."
Not that I don't lua. It's like the best parts of javascript, mixed with some AWK and scheme, with all the crap removed. OTOH, from-1 indexing is evil.
I find that the only reasonable complaint that can be made about Lua is the from-1 indexing. Like: "I found this really awesome language and it's ONLY wart is that it doesn't index arrays like every other programming language in the world".
Yet, I still admire the Lua creators for making this bold choice. Why should we use from-0 indexing for all eternity? Just because it's always been that way? This is a real question if you have the answer.
> Why should we use from-0 indexing for all eternity? Just because it's always been that way? This is a real question if you have the answer.
Well, we obviously can use 1-based indexing. Ada/VHDL allows you to define the ranges of your indicies even beyond 0 or 1 based (very useful in fixed point mathematics where you sometimes want to index your bits by a negative power of 2).
The big disadvantage is that you lose the algebraic properties of mapping to the ring of integers modulo N when your indicies are 1-based.
This leads to a bunch of off-by-one bugs in various places. One of the most pernicious is that while you can get to the next index with "next = prev % N + 1" you actually have to do "prev = (next - 1) % N". Note that the previous REQUIRES the parentheses due to precedence of operations.
0-based may be convention, but it isn't ill thought out. FORTRAN(1-based) and C(0-based) coexisted a LONG time. Pascal was(is?) 1-based. Most people had experience with both.
People who designed programming languages CHOSE the 0 convention. Overwhelmingly.
After seeing the light with Ada I never want to go back - indexing based on enums, modulos, etc. just makes so much more sense (as long as you're using something strongly typed)
> I find that the only reasonable complaint that can be made about Lua is the from-1 indexing.
Lack of arity checks is definitely much more problematic than this, but still... if you can count the number of tragic design flaws in your language on one hand, you're in great shape compared to the rest of what's out there!
No, we use from-zero indexing because, A) it's closer to the hardware, and B) because of mod(). There was an interesting comment thread on ESR's blog about this a while ago, but unfortunately, I can't find it.
Sure, technically you /can/ do it, but only in the same way that you can use negative indices in JavaScript arrays and objects. The 0-indexed element won't tie into length operators without additional manual bookkeeping with table.setn.
That's reason enough IMO to just suck it up and use 1-based indexing, even before you add munificent's (stronger) point about standard libraries and the rest of the ecosystem.
I agree that 1-based indexing isn't as much of a downside as people make it out to be, but saying you can ignore it and index from 0 "no worries" is just being silly.
In lua, iterating over arrays/tables with things like ipairs() is the idiom, so it's not so bad, but if you want to do anything involving mod, you have to add an extra plus 1. this is where bugs happen.
Except the standard libraries, all third party libraries, and every existing Lua user maintaining your code will assume 1-based indexing.
Cars don't require you to drive on the right side of the road either (or whatever side your nationality prefers), but you're going to have a bad time in traffic if you try to switch.
If you need 0-based arrays, use them. Its no big deal. Any Lua programmer worth their salt knows that Lua uses 1-arrays - so? Set up a 0-array and off you go.
In fact, I'm learning a lot from this thread, particularly about idiocy and sarcasm, although very few actual real arguments for why array[1] is 'inferior' to array[0], because in fact there is no such valid argument.
Lua standard libraries do not suffer this problem. If you have a legacy / cross-platform issue about it, well then .. its a good thing one is 'wise enough to know array[0] is a thing', but its hardly relevant to the question of whether this 'inferior accessing of arrays in Lua' is anything more than fallacy.
Idiomatic Lua involves intelligent use of pairs/ipairs/table index schema, and metatables if needed, to solve the data-access problem. And guess what? A competent Lua developer either has had zero problems with this issue (because they are competent and know how to wire up a C/C++ data structure to the Lua vm), or they have infinite, multiple "problems with Lua", of their own devising, nevertheless, but hey .. you can't "blame Lua" for that. Lua gives you the keys to the universe. If you crash into someones mailbox, well then ..
As a crazy exercise to learn both Python and Lua at the same time, I've been porting a small Numpy project to Torch. I will say that both the languages are equally nice, but Numpy is much cleaner compared to some of the seemingly hacky things I need to do in torch.
For example. In numpy:
x = numpy.array([1,2,3,4])
y = 1/x
In torch:
x = torch.Tensor({1,2,3,4})
y = x:apply(function(n) return 1/n end)
In any case, its been a fun and interesting exercise!
As ufo said, Lua could also support the 1/x syntax since it does have operator overloading. Additionally, there is nothing hacky about having an apply method taking a function.
Any rewrite is going to be superior to the old code (in any language).
It sounds like the author just wanted to move it to Lua to me. I would only go with Lua over Python if you needed something more embeddable. Like the game scripting usecase it's known for. The lost libraries and language features may make some things harder. I package up Python and my code, deploy it to other platforms all the time with under 5MB footprint in a single executable using PyInstaller.
That said, it'll work out fine with Lua. My question would be who will maintain something years down the road. There's far more Python devs than Lua. The blog said he joined in 2014 so it may end up being someone else.
Did you ever contemplate Golang as a possible solution? Seems to me like it would have provided a cross platform build environment superior to both Python and Lua in terms of standardization and ease.
yes we did consider Golang but Brian' familiarity with Lua was a major factor in the decision. Most importantly however, we foster a culture of ownership and it was Brian's decision to own and he owned it and it was the right decision.
Couldn't they have used virtualenv with Python? Also, Python packages have a way of declaring dependencies including specifying the Python version that your package requires. This should've guarded against any incompatibilities. I don't think any of this was specific to Lua.
I don't think virtualenv would solve their problems here; they are installing their agent on a customer's machine. I imagine they didn't want to ship the entire Python interpreter or alternatively install whatever version they want to use on the customer's machine.
But you're right, I don't think Lua is unique in being able to fix this issue. I think it's more that the author was already very good with it and liked it.
I really don't see the 'beautiful language' part. I tried LUA for the first time making a custom game mode for DOTA2 a while back (so admittedly not a huge amount of experience), and the language just seemed terrible to work with. Its focus on keeping the language simple meant that the standard library and language features are so bare-bones doing the most basic things was a huge chore.
...compare that to Perl where there is no "complete syntax" of Perl. This shows how simple it is and thus it is easier to write tools for and teach others how to use it.
[b] The complete default API consists of ~200 functions:
Simplicity is beauty IMHO. I would expect any competent developer to get up to speed on the Lua language within a week, it would then take them another couple of weeks to get up to speed on the libraries we use with Lua.
The real power in Lua comes when you couple it with competent APIs that do the "real" work. The Distelli Agent uses libuv for the OS interface, libcrypto (part of openssl) for the crypto primitives, zlib for compression, LPeg for parsing, sqlite for an embedded DB, etc.
The job of the Lua code is simply to "tie" these core components together.
I'm wondering what were the bigger pain points for you. I suspect that the bigger problem in your case is that you used Lua in an embedded setting with few libraries available (which is kinda intentional from the Dota devs)
From the article: "The Lua standard library uses the POSIX C library, making it portable; but that also means that you can’t use sockets, it doesn’t support signals or timers, and all OS calls block the thread. Using the POSIX C library is like going backwards in time."
I can't speak to the "Lua standard library" but I can say that I believe every single statement above about the POSIX C standard is incorrect as written.
Wow, the comments here are amazingly bad. A bunch of people sneering at the author for having to support older versions of Python that his customers use. Suggesting that he bundle hundreds of megabytes of dependencies and force the users to install them. Can you give this guy a break?
My current job is heavy in Python and a previous job was heavy in Lua. It's hard to tell from the article, but it seems like this is an environment that is embedded in another executable. In that context, Lua is certainly lighter weight and luvi sounds like a fantastic extension to the language (lua scripts are generally single threaded).
That said, anyone who has written Lua knows that scaling is difficult and relies on really high quality code. I'd be wary about replacing python scripts with Lua, especially if the problem with python was users wanted more advanced features. Still, my use case isn't yours and I'm glad the migration went well.
It's great to see people discussing the use of Lua for systems applications. It has a lot of advantages over many dynamic languages, as far as footprint, performance and deliverability, while offering a comparable development experience.
Full unicode support is rarely necessary in my experience. Lua "strings" are raw byte arrays. It is the job of the programmer to validate the byte arrays are in the character set of your choosing and doing appropriate conversions and validation when necessary.
Lua is great when you will write at most a few thousand lines of code, you can ship or control your interpreter binary and the number of developers who will ever deal with the code is single-digit. Beyond that it is miserable. Hope that Distelli Agent remains within those constraints.
i suspect it's more about lua being a really nice, embeddable scripting language. there's a reason game companies all* use it for content developer scripting languages.
language preferences aside, sometimes it's all about the right tool for the job.
* not literally _all_ of them, but lots and lots and lots.
- Python 2.4? Really? In 2014?
- Yes, writing compatible code with a 10+ year old release of Python is going to be hard. Who uses 2.4 still? You have to use the lowest common denominator, I mean I don't think Python 2.4 even supports context managers!
- Yes, in 2014 you should be supporting Python 3, or at least have it on your roadmap.
- The ethernet address (MAC?) via uuid.get_node(). Or for something a bit more advanced you can use netifaces[1] (or even better psutil.net_if_addrs[3]). One of Pythons core strengths is the number and variety of the packages available. Took 2 seconds of googling to find a cross platform solution.
- Calling C from Python is really really simple. Use the built in ctypes library, or something better like cffi[2]
I'm obviously firmly in the Python camp so perhaps I'm a bit biased, but I don't see any clearcut reason to switch other than "the current code is not optimal, let's re-write it in X", where X could be anything. I would actually say Go would be a much better fit than Lua in this situation. I guess the memory reduction/CPU usage is a point, but really your program seems to be network orientated. What's wrong with asyncio? How is replacing everything with a C library actually better?
1. https://pypi.python.org/pypi/netifaces
2. https://cffi.readthedocs.org/en/latest/overview.html#simple-...
3. https://pythonhosted.org/psutil/#psutil.net_if_addrs