[1] https://hal.archives-ouvertes.fr/hal-01890044 [2] http://www.cs.pomona.edu/~michael/

But my personal takeaway from this sort of research is that the posix shell is a truly insane language beyond even its readily apparent shortcomings. It's worthwhile to figure out the exact semantics of a language with so much existing code and have a precise interpreter for it, but frankly I think the more important task is to move away from such terrible languages. Bourne-derived shells have some great ideas, but these could be formalized into more sensible languages. In particular, I think the best direction is to embed shell languages inside good general-purpose languages. I'm biased to that end since I've written a shell embedded in Racket. But there is no shortage of better shell languages that benefit from better language design, such as Powershell and Elvish.

The way I think of it is that I can't tell my (former) coworkers or my sister to learn to program shell in 2018 "with a straight face". More than 50% of them had Ph.D.'s in some other field like statistics, math, biology, or physics -- i.e. the kind of person who has no problem with Python. But they would struggle with shell or make.

They're the kind of people Mike Bostock (author of d3) wrote this post about Make for [1]. He's absolutely right that they should use Make, because like shell it's a great tool to combine different programs that weren't designed to be combined.

But I feel like this is basically a losing battle -- I don't see that shell or make is an exciting thing for most people to learn, even though it would make them more productive and increase the quality of their work. I think that a language free of legacy would go a long way.

I guess I can see POSIX sh being used when confined to the lower level of an OS, by specialists. But I think it has a lot more potential than that. I'd say it's more important now than 10 years ago, because there's so much more software out there and systems are increasingly built of large heterogeneous parts.

----

FWIW I mentioned Morbig here [2], along with several other shell parsers, like ShellCheck in Haskell and shfmt in Go.

[1] https://bost.ocks.org/mike/make/

[2] https://github.com/oilshell/oil/wiki/ExternalResources

I could see that perhaps low level scripts need to be portable to different Unixes. But again I would say that's maybe 5% of what shell is good for.

Also, someone once said "it's easier to port a shell than a shell script". So that's why I'm paying careful attention to Oil build dependencies. For example, Rust and Go are significantly less portable than C, and Oil is self-contained C if you look at it the right way :)

----

EDIT: Also I think there's no problem adding local variables to POSIX, since all shells implement the same way as far as I can tell. That would be the much better solution, rather than trying to convince people to use only globals.

There are many useful / real shell scripts that are thousands of lines long, so you really need locals.

A global in a subshell, so the caller's globals aren't affected.

>Also, someone once said "it's easier to port a shell than a shell script".

Yeah, but it's pretty damn presumptuous to make someone install some random shell on my machine to build your code, imo. The same goes double for e.g. GNU coreutils extensions, since I have to set up some kind of separate root for you where true --version works (ditto for assuming /bin/sh is bash, or even dash so you can use 'local'). It's like encouraging people to use a fucked up version of libc where the args to strcat are reversed. Fuck that. The standard exists to define the environment. Sometimes it's within reason to have dependencies outside of that - but the shell is a good example of where the case for it is very weak.

For the record, my motivations for working on Simon's mrsh project is basically 50:50 between "I want to support POSIX-compatible shell scripts" and "I want a POSIX shell whose interactive mode isn't dogshit". Both are equally important to me.

>EDIT: Also I think there's no problem adding local variables to POSIX, since all shells implement the same way as far as I can tell. That would be the much better solution, rather than trying to convince people to use only globals.

That's all well and good. The Austin Group has public meetings, a public bug tracker, and a public mailing list. Go there and make a case for it! But be prepared for "all shells" to include more than dash, bash, ksh, and zsh.

>There are many useful / real shell scripts that are thousands of lines long, so you really need locals.

Length of shell script is not directly correleated for need for locals.

Is it more presumptuous to make people install a language you used than to make them install a library you used? Either way people need to get your dependencies if they want to run your code.

> The standard exists to define the environment.

Standards are great, but not all environments need to be standard. Most shell scripts don't need to be ported to dozens of different platforms, and most shell script authors would be much better served by a language with fewer weird gotchas, saner semantics, and the benefit of a lot of language research that's happened in the past few decades. For instance, as I've been porting my personal shell scripts to Rash (my shell in Racket), I've been able to use all the patterns I use in posix shells but with benefits of better readability, easier reasoning, better means of abstraction, etc. It's been a solid win that I would recommend to anybody. Sure, if someone else wants to run my scripts they need to install Rash. But few people (in practice: nobody) are really interested in running my personal shell scripts besides myself. If somebody does want to run them, installing the dependencies is not very hard.

There is a small set of scripts that needs to be portable and be written in a shell-like language. Until standards catch up, posix is perhaps the best thing we have. But most of the time when people write shell scripts they could be writing in any language but choose shell because they want to automate something they do manually or they really want a processes-and-files DSL. Choosing a better shell that fits the processes-and-files domain and allows easy interactive use with better features and less insanity in these situations is a slam dunk.

A C compiler is also required by POSIX.

>There is a small set of scripts that needs to be portable and be written in a shell-like language

I reject the assumption that the smaller of the two sets is the set which should be portable.

Your Racket-based shell requires a supported Racket platform, which is honestly a pathetic group: Linux, macOS, and Windows, on x86_64, i686, and ARM(?). Maybe if your software is written in Racket in the first place this makes sense, but otherwise it doesn't. It's not about you: it's about everyone. I have a RISC-V machine on my desk and if I wanted to get your stuff working on it I'd have to start by writing a new JIT for Racket. On the other hand POSIX & C runs just about everywhere. You mentioned using these for your personal scripts - fine, whatever floats your boat. You'll regret it if you ever think about playing with BSD, POWER9, or the platform of tomorrow (spoiler: I guarantee you that platform will support POSIX and C).

It's funny that everyone in this thread who's come out against POSIX shell in one breath is pushing their own shell in the next ;) maybe when those shells start to number their users in the double digits, we can talk.

>easy interactive use

A comfortable interactive shell experience is a good thing, and has nothing to do with POSIX.

But the platform of tomorrow is clearly a resurrected Lisp Machine! In all seriousness, though, I'm pretty sure Racket runs on BSD, and it would probably not be that hard to get it working on any Unix. (In fact, portability is one of the main thrusts right now in core Racket development.)

>Your Racket-based shell requires a supported Racket platform, which is honestly a pathetic group: Linux, macOS, and Windows, on x86_64, i686, and ARM

>It's not about you: it's about everyone.

The same could be said about pioneering new architectures. I think you're doing great work on eg. getting RISC-V available on your build platform. But by the time I will realistically buy a RISC-V machine to replace the server I run most of my shell scripts on a lot more software will be ported to it, probably including Racket. I'm under no delusions that the masses will be flocking to my shell, but there are so many shells that are better than Posix for (at least what I perceive to be) most shell scripting. If any of these better shells becomes very widely used I'm sure it will be ported to $platform_of_tomorrow (oops! I mean "$platform_of_tomorrow"!) before it is very widely deployed.

>A comfortable interactive shell experience is a good thing, and has nothing to do with POSIX.

At least we agree there!

As an aside, I think your projects (sway, wlroots, sr.ht) are really cool, and I like your blog, despite our disagreement about things like sticking to Posix shell and C. Cheers!

https://yash.osdn.jp/index.html.en

Yash, yet another shell, is a POSIX-compliant command line shell written in C99 (ISO/IEC 9899:1999). Yash is intended to be the most POSIX-compliant shell in the world while supporting features for daily interactive and scripting use.

I've installed the Ubuntu package and poked around at it a bit. Its source code seems well-written, and it even contains its own line editing library (i.e. the functionality of GNU readline).

It looks like it's still being developed, and magicant is the original author: https://github.com/magicant/yash

I mentioned it here (and just added a proper link): https://github.com/oilshell/oil/wiki/ExternalResources

I wrote some notes about POSIX here: http://www.oilshell.org/blog/2018/01/28.html#limit-to-posix

- Support the proliferation of portable shell scripts and discourage the use of non-standard extensions

- Provide a "POSIX shell as a library" to have a useful standalone parser and pluggable event loop

- Provide a moderately comfortable interactive shell experience OOTB

Personally, I'm going to eventually use the second goal to make a new shell based on libmrsh which has a fish-like interactive experience but with a strict POSIX syntax.

My personal grudges:

- no local variables (even dash implements them)

- no process substitution [0] so one has to use temp files, temp named pipes or possibly joggle fd numbers. I once did an external implementation of process substitution, so it's doable with POSIX shell, but then pointless.

- set -o pipefail

I understand that you know about and get over those limitations. But isn't it too cumbersome when Linux world settled on bash and BSD on ksh? Especially since ksh is a bit like a subset of bash. At least the things I mentioned are common for both.

It's sad that shell makes it so hard to make more complicated pipelines than | the | old | classic. Fds, processes and pipelines should all be first class. Maybe that's the point to make it harder to make those complicated scripts in something other than shell. In other languages it may be easier to make correct pipelines, but with heavy boilerplate. Then it's probably easier to count on libraries instead of pipelines, even if those libraries spawn processes in background.

I'm hoping for oil shell more as it means to address all those issues and more with an escape hatch to convert old scripts. Although I'm not so sure about the implementation details. For those in the know it made more sense to me after I read that OVM itself would become a VM for oil. So that there would be a single VM to run and not two as it is now. At least that's how I understand it.

POSIX shell is kinda magical, because it's nominally portable. However the rabbit seems to have died in the hat, because it's really that old. Shell was conceived in a different era with different limitations. I think that we should go further.

[0] take-in-and-out <(process-out) >(filter-this | grep that) | grep msg

They're not standard, so don't use them. Instead I often find a subshell to be sufficient. You can declare a function like so:

    my_func() (
        local_variable=10
        echo $local_variable
    )

>no process substitution, so one has to use temp files, temp named pipes or possibly joggle fd numbers

I don't find myself doing this often enough to care. It is possible, and if you're writing something which heavily relies on this perhaps you're better off with Go or Python.

>set -o pipefail

I agree, we've been talking to the Austin Group about getting this standardized.

I think we should go further, too, not by attempting to replace the shell with something that tries to do shell scripts better, but by making full-blown programming languages in which shell things are nicer to use.

But I think the efficiency problem is easier than the problem of running big hairy unmodified shell scripts, which I view as essential to replacing bash.

That is, it's easier to make something that works efficient, as opposed to thinking you have 60% of a shell and really having 20% of one :-/ That was the case for awhile, but since OSH can run many real programs, I'm pretty confident in its feature set.

There are a few places in POSIX where the parser has to be invoked recursively:

- command sub: $() and ``

- eval

- alias expansion. (I found some divergence in how shells implement this, but it does involve the parser.)

In Oil I also used the parser as a library in several other places:

- For interactive completion. Bash does not do this, and I don't believe any other POSIX-ish shell does. I wrote a bit about that in the latest blog post [1]. This turned out really well.

- For history expansion, because unevaluated words have to be picked out of previous command lines. Bash does not do this either.

Consider:

    $ echo ${x:-a b c}
    a b c
    $ echo !$
    echo c}
    c}

    $ if ec<TAB>
    $ for i in 1 2 3; do ec<TAB>

It also chooses to treat = and : as completion word delimiters, even though they don't delimit normal words, and this causes a lot of problems that the bash-completion project patches over in a very ugly fashion.

----

As for the runtime, one problem is that the shell inherently modifies global process state. So there is a limit to the abstraction you can provide over it. For example consider this program:

    { echo hi
      ls / 
      echo bye
    } > out.txt

----

The general idea of a shell that conforms to POSIX but provides a better interactive experience is a good one. (That seems to be the feedback on the Fish Shell 3.0 thread on the front page).

Although it is a huge amount of work! I hope that I will be able to metaprogram / compile Oil into something more compact, but that's an open problem now :) It is shaping up to be a better interactive shell than I originally thought though. Treating the parser as a library was a big win.

(Among other reasons, it's not a library in bash because it uses many global variables.)

[1] http://www.oilshell.org/blog/2018/12/16.html

>There are a few places in POSIX where the parser has to be invoked recursively

In your examples, we have the runtime invoke the parser as necessary. The parser doesn't know about the runtime. For alias resolution, we have a callback function, which hooks into the runtime but is pretty thin and abstract.

>For history expansion

Thankfully, this is non-POSIX so mrsh doesn't have to worry about it.

>one problem is that the shell inherently modifies global process state [...] i.e. my definition of library is that you can make instantiate multiple instances of it with different parameters

My definition doesn't line up with yours. My definition is a shared object or static archive and a bunch of headers with an API you can link to instead of implementing something yourself.

IIRC a case that really seals the deal is:

    $ echo $(case x in x) echo foo;; esac)
    foo

In other words, bash has had parsing bugs with PAREN MATCHING for 20 years (I have a case in my suite that was fixed between bash 4.3 and 4.4). If you just statically parse then you can get it right all on the first try.

It can get arbitrarily complicated, you can add a subshell and nested command subs in there too, etc.:

    $ echo $( ( case x in $(echo x)) echo foo;; esac) )
    foo

[1] http://www.oilshell.org/blog/2016/10/13.html

    ~/s/m/build > cat test.sh
    #!/bin/sh
    echo $(case x in x) echo foo;; esac)
    ~/s/m/build > mrsh -n test.sh
    program
    program
    └─command_list ─ pipeline
      └─simple_command
        ├─name ─ word_string [2:1 → 2:5] echo
        └─argument 1 ─ word_command ─ program
          └─command_list ─ pipeline
            └─case_clause
              ├─word ─ word_string [2:13 → 2:14] x
              └─items
                └─case_item
                  ├─patterns
                  │ └─word_string [2:18 → 2:19] x
                  └─body
                    └─command_list ─ pipeline
                      └─simple_command
                        ├─name ─ word_string [2:21 → 2:25] echo
                        └─argument 1 ─ word_string [2:26 → 2:29] foo

And some writeups, if you're into shells:

- https://nixpulvis.com/ramblings/2018-07-11-building-a-shell-...

- https://nixpulvis.com/ramblings/2018-10-15-building-a-shell-...