I'd love to use Lua instead of Python for some tasks, given that LuaJIT kicks ass. But poor support for something as important as unicode is a deal breaker.

Am I just missing some information in this regard?

A major, and non-negotiable, design goal of Lua is that it targets the ANSI C environment to maximise portability. ANSI C does not define Unicode support.

Incidentally, those two libraries might be unchanged because they're stable. The rate of change in the main lua implementation is quite measured.

Besides, Lua strings are interned raw byte-arrays (like atoms in Erlang or Lisp atoms), and can have \0s and other arbitrary binary data. While the standard string library is ignorant of Unicode, there's nothing preventing extra libraries from providing Unicode-aware string operations.

Lua also left out native regular expression support, because it would be larger than the rest of the standard libraries put together. You can just load LPEG if it matters.

For roughly half the people using Lua, library choices have already been decided by the big C++ (or whatever) project they're adding Lua to, so the question never arises in the first place. That's the primary use case for Lua. It's a really nice standalone language, too, but it was designed for embedding. Many of its strengths (such as the small, orthogonal, clean design) come from that focus, but so do some quirks. If you use Lua as a standalone language, you'll occasionally need to do some digging for libraries, particularly if you're not comfortable with C. The situation has been improving significantly over the last year or two, though.

I don't have a specific Unicode library recommendation, but would suggest checking the mailing list archives (http://lua-users.org/lists/lua-l/).

I haven't had a need for Unicode in Lua, but when I stumbled across some information about it, the essential story was that there is nothing stopping you from writing code to manipulate Unicode data (obviously), but there is also nothing built in to help you. All the built in string stuff is ANSI/ASCII/whatever.

However, the language Io (which often gets mentioned in the same breath as Lua, though it's quite different in some ways) has a stronger Unicode story: http://www.iolanguage.com/scm/io/docs/IoGuide.html#Unicode

Lambda Lounge is the best software dev user group ever.

contact at puredanger.com

Also, Unicode is an odd thing to consider to be an essential part of a programming language. Usually when I hear it I think the person must think it's a part of accepting those that speak other languages. I would point out the fact that Ruby took much longer to get Unicode support than Python as a counterpoint to that, as Ruby's creator, Matz, is Japanese.

The fact that Ruby took so long to get real Unicode support is due to Japanese resisting Unicode in favor of their own encodings, EUC-JP and Shift JIS, just like Finnish used to cling to ISO-8859-1(5) for so long.

On this computer (OpenBSD/i386), icu has over 1 MB of libraries and a 15 MB data file. The whole Lua distribution fits in one 200k library. Bloating the core language with that seems impractical.

Imagine you need to go through a library every time your string includes or might include the letter "s" or "v". Basically you'd need to use this library for all your strings. But then you lose compatibility with 'normal' string type and need to be constantly aware of the difference. You might want to use some other library that doesn't support this Unicode library at all, etc. It quickly becomes a very painful world to live in.

As you might imagine, just having support for Unicode baked in the language is very nice. Defaulting to Unicode for all text is even better.

I can understand why Lua in particular doesn't come with Unicode support out of the box, being so small. My comment was written in response to the more general claim that Unicode support is a strange thing to consider essential in a programming language.

(According to http://www.bckelk.ukfsn.org/words/etaoin.html, s/ä and v/ö are comparable in frequency.)

All the language core needs for Unicode is reasonable support for tagging string literals (i.e., U"blah") and a binary-safe string type. It's best if there's either a standard or de facto community standard library for doing Unicode string ops, but it doesn't need core support anymore than the Linux kernel needs to know about parsing HTTP.

At the very least, there should be a first-class type that maps 1-to-1 with a Unicode CodePoint. Then there should be easy ways to do common operations on strings in terms of CodePoints. (Like string comparison, substring matching, concatenation.) Furthermore, the encodings should be handled in a transparent way.

If the goal is to keep Lua to a 200k core, then there should be a mechanism to add such functionality as if it's built in.

EDIT: "transparent" meaning, it looks like core functionality.

There is. See "metatables" - the behavior of tables (Lua dicts) and userdata (handles to C pointers, or raw C pointers) is intentionally left minimal but extensible, so that new first-class ("transparent") types can be added.

For example, Lua doesn't have full regexp support* , but there's LPEG (http://www.inf.puc-rio.br/~roberto/lpeg/lpeg.html), a library that adds a PEG-based matching/parsing engine (which is a superset of REs). It's no less usable for being in a library rather than the core.

* Though what it does have (http://www.lua.org/manual/5.1/manual.html#5.4) if often good enough - the main thing missing is groups, e.g. "a+(ab|bc)?d".

Moreover, I wasn't searching for the best example -- merely using something I have personal experience of.

It really depends on what the language is used for. Web-facing applications are, in general, going to received unicode input, so for them it matters. But for an embedded programmer, for example, it might be utterly irrelevant.

Horses for courses, really. It just so happens that Lua has other properties that are very attractive to web apps too, which the presentation outlined.

-- and yes, still alive and commenting :P

ICU4Lua hasn't been touched since April 2009, and is at 0.2B. [http://luaforge.net/projects/icu-lua/]

That sort of status just doesn't inspire confidence in a language. I understand you can use it for game programming, or that you can write your own libraries for it, but that sort of thing doesn't encourage adoption.

Hear hear. I too don't care much for the argument that Lua doesn't have unicode because C doesn't have unicode. For two reasons. First of all, Lua can be straightforwardly embedded into a host of other languages (for example, http://tinyurl.com/39qhxzw). Second, who in their right mind thought that a language designed for embedding into commercial applications shouldn't have good I18N? (And honestly, how in the world could someone from Brazil develop a programming language that doesn't even support their own spoken language? It's as if Ruby was developed by someone in Japan.)

A bit of history. From what I have gathered in its documentation, Lua steals quite unabashedly from NewtonScript. NewtonScript was a proto-style OO language developed in a hurry for the Newton when the Dylan language wasn't going to deliver. Like Lua, it's a language designed originally for an interpreter, which runs embedded in an outer C++ environment and must interoperate with it. But NewtonScript doesn't just use Unicode pervasively: it was the first major language to do so. And C++ interoperability with the language is just fine. So this Lua-needs-to-work-with-C-and-thus-can't-do-unicode thing is nonsense both in fact and precedent.

And while we're on the subject of NewtonScript: Lua's let's-almost-do-proto-style-OO-but-require-the-user-to-do-extra-work is really incredibly annoying. Lua should have had proto OO built into the language, like NewtonScript, and not just "available", in a hacked way, through its meta model. Lua wants to be more "general" than NewtonScript, but it just winds up being (IMO) rather less usable.

If you're putting Lua in a commercial application, the commercial application itself will provide the i18n, and Lua can use it with very little trouble. Lua's strings are raw byte arrays - you can load arbitrary binary data in them. A library that reads UTF-8 strings (say) can work with them just fine. (And while I don't speak Portuguese, there are examples in PiL that use it without problems.)

I disagree with you about whether metatables are annoying, but it's a matter of taste. I haven't ever used NewtonScript, but I do use metatables for quite a bit more than just prototype OO - it's easy to turn a table into a proxy + cache to a function, for example. Also, my Lua redis library (http://github.com/silentbicycle/sidereal) turns table reads and writes into syntactic sugar for redis db key, list, and set operations.

> I disagree with you about whether metatables are annoying, but it's a matter of taste.

I didn't say metatables are annoying in and of themselves. I said they're annoying as a hacked-together substitute for a true proto OO.

Lua has many good things. But 18N and a usable OO ain't among them.

...

> You'll need a lib to e.g. calculate UTF-8 lengths,

...

> There are no technical limitations there.

You cannot be seriously making this argument. Why not just code everything in assembly?

* Though there are recommendations at http://lua-users.org/wiki/LuaUnicode .

In practice, it could look like this:

   U = require "unicode"
   s = U"some unicode"      -- using a library sure is hard
   length = s:len()
   replaced = s:gsub("thing1", "thing2") -- global substitute
   etc.

Last updated June 21st 2010. Readme seems to indicate it supports all the important stuff like compare, concatenate, encode, lower<->upper, Regexp, url escaping.

Says beta and I haven't tried it yet tho : /

    $ time php -r 'echo "Hello, world!";' # smaller php binaries can be compiled
    real 0m0.555s
    $ time perl -e 'print "Hello, world!";'
    real 0m0.385s
    $ time ruby -e 'puts "Hello, world!"' # mri
    real 0m0.073s
    $ time python -c 'print("Hello, world!")'
    real 0m0.067s
    $ time lua -e 'print "Hello, world!"'
    real 0m0.013s
    $ time js -e 'print("Hello, world!")' # spidermonkey
    real 0m0.010s
    $ time awk 'BEGIN { print("Hello, world!") }'
    real 0m0.004s

E.g. perl(1) is faster than ruby(1) at printing "Hello, world!". But since you presumably had ruby hot in cache and not perl the latter seems to be almost 4 times as slow.

Here's a better benchmark, which runs each of these 500 times and takes the average: http://gist.github.com/630868

Which yields these results:

               Rate clojure   php emacs python  ruby   js perl  awk  lua shell     C
    clojure 0.844/s      --  -97%  -97%   -98%  -99% -99% -99% -99% -99%  -99% -100%
    php      24.6/s   2812%    --   -2%   -43%  -75% -76% -78% -82% -82%  -84%  -89%
    emacs    25.2/s   2882%    2%    --   -41%  -74% -75% -78% -81% -82%  -84%  -88%
    python   43.0/s   4995%   75%   71%     --  -56% -58% -62% -68% -69%  -72%  -80%
    ruby     96.7/s  11361%  294%  284%   125%    --  -5% -15% -28% -29%  -38%  -55%
    js        101/s  11919%  313%  303%   136%    5%   -- -11% -24% -26%  -35%  -53%
    perl      114/s  13397%  364%  353%   165%   18%  12%   -- -15% -17%  -27%  -47%
    awk       134/s  15743%  444%  431%   211%   38%  32%  17%   --  -2%  -14%  -38%
    lua       137/s  16134%  458%  444%   219%   42%  35%  20%   2%   --  -12%  -36%
    shell     156/s  18359%  534%  519%   262%   61%  54%  37%  17%  14%    --  -28%
    C         216/s  25440%  777%  756%   401%  123% 113%  89%  61%  57%   38%    --

           Rate python swipl ruby perl  lua luac bash ocaml  awk subshell ocamlopt    c py_pyc shell
    python   46.3/s     --  -17% -56% -70% -76% -78% -78%  -79% -83%     -87%     -88% -88%   -90%  -92%
    swipl    55.9/s    21%    -- -46% -63% -70% -73% -73%  -74% -80%     -84%     -85% -86%   -88%  -90%
    ruby      104/s   125%   86%   -- -31% -45% -49% -51%  -52% -62%     -71%     -72% -74%   -79%  -82%
    perl      152/s   229%  172%  46%   -- -20% -26% -28%  -30% -44%     -57%     -60% -61%   -69%  -74%
    lua       189/s   309%  239%  82%  25%   --  -8% -10%  -13% -31%     -47%     -50% -52%   -61%  -67%
    luac      206/s   345%  268%  98%  35%   9%   --  -2%   -6% -25%     -42%     -46% -48%   -58%  -64%
    bash      211/s   356%  277% 103%  39%  11%   3%   --   -3% -23%     -41%     -44% -46%   -57%  -63%
    ocaml     218/s   372%  290% 110%  44%  15%   6%   3%    -- -20%     -39%     -42% -45%   -55%  -62%
    awk       273/s   491%  389% 162%  80%  44%  33%  30%   25%   --     -23%     -28% -31%   -44%  -52%
    subshell  357/s   672%  539% 243% 135%  89%  74%  69%   64%  31%       --      -6%  -9%   -26%  -38%
    ocamlopt  379/s   719%  577% 264% 149% 100%  84%  80%   73%  39%       6%       --  -4%   -22%  -34%
    c         394/s   751%  604% 278% 159% 108%  91%  87%   80%  44%      10%       4%   --   -19%  -31%
    py_pyc    485/s   950%  768% 366% 219% 156% 136% 130%  122%  78%      36%      28%  23%     --  -16%
    shell     575/s  1143%  928% 452% 278% 203% 179% 172%  163% 110%      61%      52%  46%    18%    --

Edit: Since you can precompile Lua, I added that as 'luac'. It's not usually done, since Lua compiles VERY quickly, and the source is more portable. It's a data point, though. I also added a .pyc for Python - Python byte-compiles more slowly. There's little difference between lua and luac, but the difference between python and py_pyc is huge.

I also added shell with a new subshell, both sh and bash. And, source, as requested.

OCaml:

    let _ = print_string "Hello world.\n"

SWI Prolog:

    swipl => sub { system qq[swipl -g "write('Hello world.\n')." -t "halt."] },

    print "Hello, world."

    Lua 5.1.4: 34k allocated, all memory freed.
    Perl 5.10: 195k allocated (140k leaked!)
    Ruby 1.8.7: 670k allocated (665kb still reachable at exit)
    OpenJDK 6: 1.5M allocated (1M in use at exit).
    Python 2.6.5: 3M allocated (1.2MB in use at exit)

  $ time clojure -e '(println "Hello, world!")'
  real	0m0.781s

http://www.usa.canon.com/cusa/support/consumer/digital_camer...

For the sake of any readers here who don't read the blog post: I do NOT recommend this camera for video. I happened to have it sitting around for family snapshots and chose to experiment with it here.

The lua users wiki is full of interesting ideas for tackling the basics in ways that leverage lua's simple flexibility.

In my experience 1-based indexing is not a problem when you're just writing Lua code. I've sometimes been frustrated when creating Lua bindings to C-modules which involved some kind of array access though.

It just bugs me that such an obvious wart exists in an otherwise very clean language.

People who write off Lua because of indexing from 1, Python because of the significant whitespace, Lisp for its pares, etc. probably haven't gotten to the really interesting stuff yet.

("foreign" with regards to programming language.)

In C, it's an offset into a typed block of memory, so 0 indexing makes sense. But lua does not deal directly with memory.