Hacker News new | past | comments | ask | show | jobs | submit login
Erlang/OTP 21's new logger (ferd.ca)
193 points by signa11 on Nov 28, 2018 | hide | past | favorite | 55 comments



I'm really impressed with all that is happening in the Erlang/Elixir world nowadays. I didn't realize that this community was so actively introducing new features and improvements to what is already a rock solid toolset.


I agree. Just starting myself with Elixir, and boy do I love the pipe operator, pattern matching, and the wonderful OTP goodies. I've only practiced with Elixir for a couple weeks, but for work when I write PHP code, I feel sad, and look forward to work on the next Exercism task. Also joined the Elixir slack, never seen such an awesome community. I hope Erlang/Elixir continues to thrive


The worst part of Elixir is when you're working with other languages and you know it'd be better in Elixir.


It's not perfect. There's a few strange things like calling function variables with the dot notation, but it's there for a reason, and the rest of it worth the worst of the quirks.

Side note, I wish variable rebinding were off by default but allowed on names with a sigil (or a ! ).


> a few strange things like calling function variables with the dot notation, but it's there for a reason

I feel like anyone who has written a Yacc-based parser for programming-language syntax will appreciate Elixir's grammar even more for these "quirks." Elixir's syntax is as beautiful as it can be while also being explicitly unambiguous. Which means that the language syntax is very easy to change or extend—unlike other PLs I could name, where the set of parsing rules resembles a late-game Jenga tower.

Or, to put that another way: Elixir isn't just a joy to use; it's a joy to be a core maintainer of. (How often have you ever heard anyone consider that part of a programming language's UX?)

Oh, and this simple, explicit set of parse rules also results in a highly-predictable and regular AST, which means you can write an macro-based DSL library in Elixir—one that will actually works with any arbitrary Elixir code hanging out inside your macro body—in not-much code, and without worry that a minor language change in the future will break your lib.


> fork and base your own programming language upon. (How often have you ever heard anyone consider that part of a programming language's UX?)

Any time lisp macros come up?


The dominant relational database system (Ecto) very elegantly dsl's databases.


Specifically speaking the optionality (inexplicitness) of parens for function calls causes some parsibg difficulty. The compiler automatically assuming that a bare atom is a function call with () versus a variable, for example.

However, this is really minor in the grand scheme of things, it really is one of the most beautiful pls otherwise


Other language that have strong typing for instance or good performance?


I enjoy Elixir tremendously, but usually I’m forced to use another language (Python) due to popularity, not technical merits.

It’s hard to sell anything that’s not in the TIOBE top 10.

To answer your comment more directly though, for many systems the benefits of excellent concurrency, monitorability, and fault tolerance can outweigh those of a strong type system or raw performance. It really depends what you’re doing, and how your team likes to work. No such thing as a silver bullet and all that.


You don't have to let go of the FP goodies for a good type system or speed: take F#, or Haskell, or even Scala (with akka) or Kotlin. Ocaml or SML would do, too, but their concurrency story is not as good.


Same here. Yet, as I'm gonna be job hunting soon, I feel that investing in learning more in Elixir/OTP won't land me a job (let alone a remote one). Too few jobs asking it. It seems that the vast majority is all about microservices/java (still), k8s, nodejs and the like.


Actually, the worst part of Elixir is when you're realizing that something asynchronous, of all things, would be easier in JavaScript, of all languages.

Eg JavaScript lets you trivially memoize asynchronous requests, by caching Promise objects instead of responses. In Elixir you'll end up with enormous case expressions like this one: https://github.com/melpon/memoize/blob/master/lib/memoize/ca...


Elixir has a Task module that can be used like Promises.

A task is a process that calculates something and eventually returns a result, which you can get synchroneously with `Task.await/1`. There's also an equivalend for Promise.all, which is just `Enum.map(tasks, &Task.await/1)`.


You can't use those to cache requests.

You can store an Elixir task but you can only await it once. You can `.then` a JavaScript promise as often as you want which makes a promise an excellent primitive for caching things that take time (eg network requests) without triggering the same request twice.


You can leave a process around as a cache, sending it messages when its values need to change. And if it dies for whatever reason, its supervisor can restart it to a known good state. You shouldn't concern yourself as the caller of how the cache updates itself, you should send it an explicit message like "the cache for X is out of date" and let that go and do its thing accordingly.


Yes and that's a lot of work for what would be a two-liner in JS.

Every time someone says "just make a process and a supervisor, put it elegantly in the supervisor tree, go think through the failure modes, ponder a bit about these 3 or 4 edge cases and potential race conditions, and you're done! hooray!" I get a bit grumpy.

Elixir is awesome, but there's a simplicity to a single-threaded execution model that takes away a lot of the things that make concurrency hard.


At the small prize of parallelism. And now that you are dealing with workers, task queues, etc. a native process + supervisor solution doesn't seem bad at all.


It's not a contest!

All I'm saying is that there are things that Elixir isn't the best at. Is that so weird? I feel like everybody responding to me has this strong emotional need to defend Elixir. Why? it's just a tool. Like any tool it has pros and cons.


Elixir is amazing at creating parrallelized network heavy iobound applications. obviously its going to be shit if you want to do machine learning or anything computationally bound. Thats why it has amazing first class integrations!

https://github.com/fazibear/export and https://github.com/hansihe/rustler


I recently started working through the tutorials here [1] and they're excellent, but kinda like you I get grumpy when I have to do Django at work now.

[1] https://www.phoenix-tutorial.com/


Nice! You wrote flask-security etc. right? I actually reported a bug again flask-login years ago because of the way Elixir handled a certain security issue. Small world...


Flask Security isn't my work, that would be Matt Wright [1], though coincidentally I did link to an article by Matt on here a few days ago.

http://mattupstate.com


Just wanted to know what's wrong with Django?


Slow, GIL, Python, ORM, no otp, no supervisions, no runtime code inspection, bad code formatter, error handling, no macros, no pipe, not immutable, not functional, very complex writing multithreaded code, 100 different ways to write doc strings, ugly docs, hidden state everywhere... etc.


I would disagree with many of these, but thanks anyway.


I agree with some of andy_ppp's list below, but my main gripe is how much Django hides from the user through magic and endless inheritance chains. When I write code I like to know what it's going to do, and when it goes wrong I like to be able to find where it went wrong without jumping through 20 classes in 10 files.


Appreciate your response. I would happy to see your reaction when you're working with Laravel, lol.

I fully agree that too much of magic makes Django difficult and easy to newcomers at the same time. So I switched to Flask which is much pleasure to work with and now I'm very happy.


This is covered in hidden state everywhere...


Nothing major. Elixir is just such a good experience it has the effect of making everything else feel really clunky.


Erlang is great - we ended up settling on it as the basis of our platform. It was between Erlang and C++ for our real time highly available system and in hindsight Erlang was a no brainier. So far it hasn’t disappointed.


You won’t be disappointed. I have an Erlang/OTP system in operations since nearly 10 years. Exactly zero downtime since first day of operation. And we did upgrades and change of hardware. The customer is in love.


Zero downtime in 10 years - sounds impressive.

What is your system doing if you don't mind the question?


It is an automatic train supervision system for a metro line. It tracks and displays the position of trains and automatically routes them depending on schedule, delays and incidents. It also offer control and commands of the traction power and the devices in station such as elevators, gates, doors, passenger information display, etc. It also logs all commands and events on the system. It’s the « command center « of a railway if you prefer. So it must never stops.


Ahh, logging, a subject dear to my heart.

At the last project I worked on, we did a lot of work to optimize our logging, as it was being done on low spec TV set-top-boxes, and we were burning a ton of CPU cycles just doing formatting (lots of JSON serialization). What many devs fail to realise is that, unless extra effort is made (as in the Erlang system in the OP), every log operation you call is a synchronous, blocking call that can actually be relatively expensive due to all the serialization and I/O that goes on.

Async logging can help of course (necessary if you want to send logs over the network). But you can still spend a lot of CPU just formatting the log lines themselves.

The solution we decided on was similar to what you'd do in a C/C++ project: for release/production, add a build step that removes all logging commands below the logging level for the target. For example in C, you'd usually use macros for this, and if your macro wasn't defined in release builds, none of the logging code would be included in your executable.

This is great for performance but does have an operational trade off. Ops people might want to be able to change the logging level of a system to diagnose live issues. But if I think about all the CPU cycles and HDD storage spent on logging (and often those wasted cycles have a very real impact on UX or server latency), then I think in many cases it is worth stripping out anything lower than Error level as standard.


There is a lot of care in Elixir Logger (https://hexdocs.pm/logger/Logger.html) with regard to this.

For instance, "arguments given to info/2 will only be evaluated if a message is logged".

You can also trim out (at compile time, like you did with the build step) all the logging commands below the selected level, with the ":compile_time_purge_matching" option.


That’s really awesome. All modern platforms should do this


yeah, Erlang-wise, most loggers have done these checks for years. There's always a big concern in terms of avoiding the most work possible as early as possible so all that filtering takes place early on.

The handler and filter structure with both global and per-handler filters is fairly clever as a way to turn these things on and off easily, and the module and level checks are done the earliest since they can allow to eliminate the most amounts of logs the cheapest way possible.


Notably with good support for structured logging, which is usually badly supported by log4j descendants (like e.g. Python's logging module).


Agree with the sentiment behind a lot of the comments above. Elixir/Erlang are awesome and there is a lot of vitality in those communities at the moment (more in Elixir perhaps).

Elixir is definitely a darling of HN but there are some areas where there are some shortcomings in my point of view:

- A lot of people seem to love Elixir macros but I actually think that they result in a lot of compile time magic and make the code difficult to debug/read. Macros make code easy to write but supremely difficult to behind-the-scenes-understand and debug. The code that you write is transformed into something very different when you run it! Also every major Elixir library brings its own special macros which is a pain. Erlang OTOH does not use user defined code transformations (i.e. macros) so much so the code there is easier to comprehend. But the Erlang language is quite barebones and does not have the expressive power of some of its functional cousins like OCaml/ML/Haskell. Its seems that macros were an easy way to impart this expressive power to Elixir but it comes with the above mentioned costs.

- The Elixir (Erlang) runtime is stateful. This leads to many advantages but I kinda love the PHP model of every request is isolated/new from the rest of the system also. This may make PHP feel clunky/slowish but then its super easy to debug. PHP works out fine for _most_ systems even at scale. If your system does not have extreme requirements then PHP should work out fine.

- Elixir is not statically typed. In 2018 for a new project its hard to justify not using a typed system to make code more likely to "run" properly out of the gate or increase your confidence while refactoring it. Even a simple type system like golang's will do. No need for something complex like C++/Java/Rust etc.

In summary there is a lot of good in the Erlang/Elixir ecosystem. I think it makes sense to use this platform in many scenarios.

P.S. I've talked about PHP above. It continues to be fashionable to bash PHP and I feel this is a meme that must be combated. PHP 7.x is a much improved platform. Once you look past some of its historical warts you'll find that it can be very productive to work in. Of course it lacks static typing but if you specify the types of the your function parameters and (comment) annotate the type variables you can get some of the benefits of typing if you have the correct IDE/code analysis tools. (Elixir/Erlang also have "soft" static typing via dialyzer).


We're adressing the lack of static typing with the Gradualizer project. It's not ready for prime time yet but I find it very promising.

https://github.com/josefs/Gradualizer


1. Erlang has parse transforms, so it's not like Erlang has /nothing/ in this regard. They are painful to write, though, although that could probably be fixed by introducing a parse transform (apparently someone else thought the same, just found this: https://github.com/bucko909/uberpt). I just didn't really /need/ parse transforms. The only ones I ever used were lager and exprecs, and regarding the latter it turned out to be much easier to get the specs right by just writing modules manually.

2. Depending on the framework you use (we just use cowboy directly), Erlang HTTP pages/endpoints are pretty light on state as well, you need to pass them information or call global functions to make them stateful. Maybe I'm not getting you right here.

3. This is an issue to me as well, but I think most Elixir and Erlang developers would agree that this would really be nice to have, at least optionally. We're right now in the process of making our codebase "dialyzer-safe" but there are so many parts of it that could be checked statically. From what I got from different talks and articles on the subject, the main issues are the dynamic "dispatch" mechanisms, like throw/error/exit and general message passing that allow for arbitrarily typed data to be thrown at arbitrary other code.


> Elixir is not statically typed. In 2018 for a new project its hard to justify not using a typed system to make code more likely to "run" properly out of the gate or increase your confidence while refactoring it. Even a simple type system like golang's will do. No need for something complex like C++/Java/Rust etc.

Elixir was inspired by Ruby iirc and it was a project started in 2012 according to the elixir documentary by Honey.

Scripting languages without static typing have its place.

I don't believe all languages even new one starting in 2018 should be type. It depends on the language goals.

If you want a actor concurrency model with type then try Pony.


> every major Elixir library brings its own special macros which is a pain.

Is this actually a problem? The macros are not really overused, most library developers are sane about limiting the use of macros, and the lexical scope of any given macros is limited by the require/include semantics. "use" is pretty bizarre, but I've never had a problem with "just roll with it"


Its a problem while debugging. The code has often been so transformed that it can be unrecognizable.


Is this from experience with elixir specifically, or is it theoretical? I've never had any problems. Can you tell me where you had a problem?


regarding point 3: In 2018, for an actor model on the BEAM, it is really easy to justify not being statically type. For a single simple reason.

No one knows how to do it. It is an open research problem and we are far from a solution still.


https://github.com/josefs/Gradualizer already introduced above by Zalastax is one project which shows how to do it. It implements the gradual typing model, which merges static typing a'la ML/Haskell, with dynamic typing of Erlang. Other real-world examples of gradually typed languages are Facebook's Hack and TypeScript.

Erlang already has a dual nature:

- the sequential part, which strives to be a pure and mostly side-effect free functional language

- the concurrent part, i.e. send, receive, process preemption, message delivery semantics, etc, which is inherently stateful due to interactions between stateful processes

Gradual typing fits this model well, because it provides soundness to the sequential part of the language, while leaves the message passing parts which _no one knows how to_ type dynamically typed.

BTW, I've spent a while to integrate Gradualizer with Vim - https://github.com/neomake/neomake/pull/2115


So basically what i said. We do not know how to statically type it, just try to provide some false safety.


That is true. But some other models of concurrency (e.g. golang channels) _do admit_ static typing. This is a weakness of actor model adopted in BEAM.

Its a tradeoff -- do you go with untyped BEAM concurrency or typed concurrency like in golang.


In Erlang you only care about cases that you care about. The let it crash mentality. Pattern matching is also good here too.

In Go you have to write code to catch unwanted error and figure out what can go wrong.

So there are many things in Erlang where it soften this no type weakness.


Idle curiosity: I wonder if they fixed the 'bug' where if you had a big binary in an error, it got translated to a string representation of a binary, which absolutely blows up memory usage as it works its way through the logger.

In Erlang for instance, you might have a binary like <<"foo">>, which is the 3 bytes, 'f', 'o', 'o'. The display representation of it - <<"foo">> - though, as an Erlang string (which is a linked list of numbers) takes up 18 bytes according to erts_debug:size, which is a huge multiplier.

As always, though, excellent writing by Fred.


This is mostly fixed. First, the loggers tend to use unicode-aware pretty-printing, which lets you have rarer instances of a binary getting blown to its integer bytes representation.

Then there's also ways to prevent just printing huge terms out. That's what parameters like `term_depth` in many handlers do (I believe it's just `depth` in the default handlers). They limit how deep you go into outputting some data. So even if you have some very large strings or deeply nested structure, they can get elided when they reach a certain threshold.


I really like the help in the console of Elixir, but I prefer Erlang's syntax. Is there similar tools for Erlang?





Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: