One of the fish devs here. I definitely didn’t come up with this hack but I’ve worked on improving some edge cases in the past. Our implementation differs from zsh some although I think we’d originally adopted their approach.
This feature is still surprisingly complicated with many caveats. For example, some terminals will wrap to the next line if you are at $COLUMNS, some at $COLUMNS+1, and some not at all. Some advertise they do one thing and do another. If you miscalculate, you’ll end up skipping an extra line or none at all, maybe overwriting existing output or emitting the missing line break glyph when it should be hidden. We’ve ironed out most of the cases but every once in a while someone will be running an esoteric Pseudo tty (usually not under X) and run into issues.
There are also issues with selection of content. Some terminal emulators will bug out when you try to copy-and-paste content because they get lost with the output past the current location in cases where a line break was already emitted. It’s always the terminal emulator that’s broken but you’d be surprised at how broken some of the most popular emulators are. The entire codebase is littered with workarounds for tty emulators; if someone comes to you and says “fish doesn’t work in $terminal but bash does” (either because it’s minimalistic and is missing a feature that would cause the breakage or because it has a workaround), we almost always end up treating it as a fish bug unless we’re able to get it fixed in the upstream terminal package/OS/software/whatever, whom we file reports with all the time.
It’s insane the configurations and platforms people use shells under and I’m personally deeply appreciative of everyone that takes the time to file these bugs - in addition to making the software better it really does shed light on some extremely archaic bits of posix standards and Unix behavior and I’ve learned so much from the many deep-dives that start off with “this is kind of quirky behavior observed under certain circumstances” and ends up requiring architectural changes deep within because some underlying assumption doesn’t hold universally true as exemplified by a *nix that hasn’t seen a new user in fifteen years or something. It’s why I love hacking on fish; I must have one of the craziest collection of virtual machines to test edge cases, as do many of the other devs (especially @zanchey and @faho, although no one does a deep dive like @ridiculousfish).
Out of curiosity, I installed fish and set my prompt to be blank[1]. Based on this article, I would expect to see the missing line feed character as the first character on the line after every command, but I didn't. Do you know why?
[1] My `fish_prompt` function just does `echo -n ''`. I also tried `true` and an empty function body, I don't know much about fish scripting.
Nice catch and great show of initiative there! It's actually not overwritten by the prompt but rather by the "hack"/post-execution handler itself, depending on where the glyph was emitted (which in turn depends on if the command output included a trailing \n or not) when the line is cleared after guaranteeing we're at col 0 of the next line. If the glyph wrapped to this line, then it would be cleared. If it didn't, it'll show on the line above.
I think the missing clue is that the hack is followed by `^[[K` (clear line from the cursor to the right), before control is passed to `fish_prompt` to paint the prompt or whatever else.
Here's some sample output (stripped of the hack spaces and any ansii escapes):
`fish_prompt` isn't called until the cursor is already at column 0 and the line is clear; the "missing line ending glyph" (here ¶ rather than ⏎ because fish correctly detected that the latter is likely not supported by the font used by my terminal emulator in yet another hack) is emitted at the end of the previous line.
You can see all this and more by piping the output of a `fish --interactive` session to `xxd -C` or `hexyl` or whatever (that'll disable tty feature detection and force ansii escapes to be emitted that you can then inspect).`
(You can also verify by simply using `sleep 1` as your fish_prompt - you'd expect to see the glyph appear at least while the `sleep` was running if fish were waiting on/expecting the prompt to overwrite the glyph.)
I did indeed skip some details in the article. In the case of Zsh, the output actually ends with `\r \r` which is probably meant to handle this exact case by having a space overwrite any marker in case the prompt doesn't.
Ah, I couldn't remember what it was that we did differently from zsh, but I knew there was something.
I would imagine - depending on the terminal emulator - the zsh behavior would cause copy-paste-cruft in the form of trailing whitespace while appearing correct, and would guess this was why we changed our implementation.
I assume that there is control character after carriage return that deletes everything from there to the EOL, so this do not break copy functionality or "go to EOL" shortcuts.
Oh, I just saw your reply after posting my own, though you beat me to it by quite some margin of time. That's correct, fish sends ^[[K to clear to the EOL after correctly setting the cursor position via this hack to erase any evidence of the hack and for posterity of your ptty contents.
Fantastic rabbit hole to dive into, thanks for linking!
The "good news" is that you can't even use tput rev under fish because it saves and restores attributes each time it regains control of the tty with very few exceptions , rev not being one of them. You can't break what you don't even support!
(Seriously though, we should add rev support.)
EDIT: I just saw that you're the same person that answered that post, which very much deserved the upvote. Your knowledge of tty behavior is impressive. It is also rare to find another person that cares about the kvt - a lot of apps that manipulate terminal attributes or use any vte codes are completely broken under even the well-written kernel virtual terminals.
If you want another rabbit hole with respect to supporting attributes, consider that the Windows Terminal people are considering adding ECMA-48 standards conformance that no other terminal emulator has, the various graphic renditions from ECMA-48:1991 that no-one else has implemented in 30 years. (I saw the issue and added overline, framed, and encircled to my terminal emulator's framebuffer front-end. It will be in version 1.41 of the toolset, but that is not out yet.)
It brought me up short to be described as caring about KVTs. (-: I've mooted doing VTs in application mode code for a couple of decades, now. Here's a white paper from 2006, which was several years in.
I believe you are misunderstanding. The feature ensures that your (empty) prompt is printed at the start of a line, even if the preceding command's output doesn't end with a newline.
Start up fish and run `echo -n foo` to see `foo⏎` in your terminal.
My understanding is that the ⏎ character (and some whitespace) is printed unconditionally, but is overwritten by my prompt if the previous command ended with a newline.
If there is no prompt to overwrite the ⏎, why can I not see it?
That would actually not be OK because we can't count on the prompt to overwrite n characters, and the resulting cruft would give you an incorrect representation of the output if you captured a snapshot of the terminal state (or tried to copy and select). Fish clears the line itself after finding itself at col 0 of the correct row. See my other response for more.
Isn't this sort of quirk what the terminfo/termcap databases are for? I guess adding new entries may no longer be feasible, but an entry there like "wraps at column" would allow every interactive command line program to benefit from fixes for new terminals.
am: Flag whose presence means that writing a character in the last column causes the cursor to wrap to the beginning of the next line. If `am' is not present, writing in the last column leaves the cursor at the place where the character was written. Writing in the last column of the last line should be avoided on terminals with `am', as it may or may not cause scrolling to occur (see section Scrolling). Scrolling is surely not what you would intend. If your program needs to check the `am' flag, then it also needs to check the `xn' flag which indicates that wrapping happens in a strange way. Many common terminals have the `xn' flag.
Yup. Fish checks both and acts accordingly, but they're kind of useless because `xn` could mean anything.
The real problem is I'm not sure when the last boolean capability was added to the termcap standard. While new terminals continue to add and update their termcaps, the spec itself is missing a lot of features that would help enormously.
I think the whole terminfo/termcap situation could be improved a lot. For one thing, most terminal emulators just report "xterm" or "xterm-256color" as the TERM environment variable, even though different terminals that report the same thing have different capabilities. A couple terminals, like alacritty and kitty, do ship with their own terminfo files, but that causes problems if you ssh to a box that doesn't have the terminfo file for that terminal.
It can definitely be improved. The model has some significant flaws. I started again from the fundamentals and came up with a rather different model for differentiating terminal capabilities.
Note, in fairness, that not having the terminfo file is not a huge problem, as terminfo is designed so that one can transport information from one system to another, and place local database entries in one's home directory if necessary.
It was supposed to refer to a particular behaviour of Concept terminals. In practice, it is used in many terminfo records to denote DEC pending wrap, which the terminfo commentary even itself records as being different to the Concept glitch. Yes, terminfo has highly problematic paradigms that simply do not match how terminals and their emulators actually work.
Regarding your second link, I cannot comment on the linked item as it has been archived by HN but as an additional reference for others and possibly a fun read for you: https://bugzilla.gnome.org/show_bug.cgi?id=754596
I wondered if anyone remembered those things. So many people seem to be hard-coding escape sequences these days, making untold amounts of work for themselves.
I've always wondered, would it be reasonable / possible for a shell to run everythhing in a virtual tty? I would love a shell when programs couldn't effect each other's output, it was easy to dynamically put programs in the background while hiding their output, etc.
I'm working on a shell that does this, for exactly the reasons you gave and more. It doesn't work well enough to announce here, but you can find an index of shells (including others that have had this idea) here:
https://github.com/evmar/smash/wiki/Related-projects
That came out of a discussion with someone doing a really ambitious overhaul of terminals and display servers (linked in the issue).
I believe you could do some interesting things UI-wise with a shell that doesn't run in a terminal itself. The subprocesses would require a terminal, but the shell wouldn't. For example I've heard the suggestion to always keep the prompt at the top of the screen from a few people. That seems like it would be straightforward in an HTML UI, but it probably requires some cooperation from the shell itself.
This is a more Emacs-like architecture for a shell, with the interface being written in a high-level language rather than C/C++. That came out of running parts of ble.sh, which is an interactive shell written in bash !! (I believe it's the biggest bash program in the world, in both number of lines and sophistication)
If you have any ideas or suggestions, I'm interested. In the interest of time, I'm cutting out the interactive parts of the project and making it more like a library that other people can build shell UIs on top of.
I collected these links awhile ago but haven't done anything with them, there is some overlap:
reptyr can reparent a program if you only realise after the fact.
I try and remember to have that installed on the heavy-interactive-use servers I admin for when users have an "oshi-" moment because they forgot to start something under screen.
Thanks for the post! These things absolutely should be documented and dissected in posts like your; there’s way too much voodoo in dealing with terminals.
Woo! I have the same trick in my .bashrc, love to see it spread.
If you can, please add a note that if you want to add it to bash you _have_ to use an echo/printf from PROMPT_CMD, if you add it to PS1 bash's (or readline's?) own logic will get confused as to how long the prompt is and what column the cursor is at.
(If anyone wants to try it, personally the only caveat i noticed is that when you copy a command from the terminal window any spaces not overwritten by something else will appear in the output.)
This is the kind of rabbit hole that gives me nightmares any time I even consider the idea of writing my own shell. Fish nails the UX so well that even doing half as good a job seems like a truly massive amount of work.
As someone who has written their own shell I can still recommend it as a project. Actually writing the readline implementation was much easier than I was expecting. Not suggesting there aren't edge cases I've missed but at least that is a longer term issue you can resolve as and when people report issues (and if nobody else's uses your shell then the other use case you need to worry about it your own).
I do have some recommendations however:
- first of all use a 3rd party readline implementation. This will allow you to write your language parser, any builtins you'd need (eg `cd`) and forking routines. These are the bigger / harder problems.
- then once you have a POC you're happy with, rewriting your readline implementation becomes a natural progression. By that point you know how you want your shell to look and feel.
I will second that. We provide VM solutions on cloud marketplace and all our VMs are using fish as the default shell. Both us and our customers love it.
Thanks for building it.
We wouldn’t run an external command because of the overhead, compatibility, and portability, but I’ll treat your question as “why not do what tput el would do?”
It all depends on where the input cursor is when the command terminated. You need something that works with both cases (line break and no line break) because you can’t introspect the output.
tput el clears the output to the end of line so you still have to figure out if you’re on the same line as the output or if you’re on a new line. The goal of spaces is not to clear content but to invoke the wrapping behavior of the terminal which will either cause you to be on the same line only in the case that you started off at col 0 or else somewhere in the middle of the next line, but in both cases, you’re on the row you need to be on (after taking into account whether wrapping happens at max column or max column plus one).
Edit: It’s actually a lot like rounding by adding 0.5 and truncating. And the complexities are akin to trying to do it with non-standardized or different standards of doubles and dealing with epsilon and NaN :D
No worries! (Now try figuring out what it does and why it does it from the code that you found spread out over a few hundred lines of C interspersed with curses calls and no comments.)
Not directly in fish, because we manually set the tty flags and restore them after each execution (it's why your fish prompt doesn't break when you ^C in the middle of a redirect or when you accidentally output binary to the terminal), but e.g. in bash or if fish were to emit those escape codes directly. But yeah, it definitely seems like a good hack and it would cut down on the bandwidth requirements (you can notice the lag when using fish/zsh over ssh on a lossy connection).
Note that depending on the xn attribute, \r may or may not end up taking you to col 0 of the same row, depending on if the previous output caused a soft wrap but didn't include the trailing new line - you could wind up at col 0 of the previous row instead, if I'm reading it correctly!
This is an XTerm FAQ, and you are not in the analogous situation that you purport to be in. You are abusing "VT100" to describe the current state of affairs in the world; which is most definitely not a DEC VT100, not like a DEC VT100, and most definitely different in many important respects (such as colour, for starters) to a DEC VT100. It's even different to "ANSI" (again, which should be ECMA-48) in that no terminal or terminal emulator (that I know of) actually provides standard function keys (function keys being another thing that a DEC VT100 does not have). A few of them, sort of, provide the non-standard function keys from a VT220, which isn't a VT100 either.
Far from "winning", "ANSI/VT100" was superseded (by things that were better) within a few years, long ago.
waves Great article about shell hackery. It's good to clear up the misconception that shells know what's on the screen. They don't even know where the cursor is!
One important reason for this hack is right prompts. If rprompt has width 7, the shell moves right by $COLUMNS-7, outputs the rprompt, and then moves left to return the cursor. What happens if during the move-right phase, the cursor wraps to the next line? move-left doesn't "wrap back", it just pins against the left side! So your prompts get split across lines, your right prompt floats somewhere in space, and your input may even overlap it. It's quite confusing to the user.
So if you have a right prompt, the shell has to be very sure it's on a new line when it starts!
You probably know some of them along the lines of : think of a number, then do a series of operations, and then I can guess the result. It can be done with cards too. In reality no matter what your initial choice is, the end result is always the same. The trick is to combine relatively complex functions (ex: "take the sum all digits of the number", "add 1 if odd") in a way that produce something simple (ex: "the result is 5").
Here the trick is to express the function "pos(x,y)=(0, y+1)" in term of the functions "pos(x,y)=(0,y)" and "pos(x,y)=if x<$column then (x+1,y) else (0,y+1).
It is also the basis of the "abstraction inversion" anti-pattern.
You probably would feel the same if you would implement your own pty including re-implementing entire terminal emulation. That is until someone would show this solution.
Can't disagree with that, as that may be my hobby project in the next few weeks. I'm currently evaluating options for implementing a decent terminal emulator for jailbroken iPadOS. 95% chance I'm just gonna go with a webview and hterm just for simplicity on the development end, but there's a sick and twisted masochistic part of me that wants to write a full terminal emulator myself.
You'd think working tech support would've beaten the masochism out of me.
Well, sure, but it doesn't necessarily incite to the most elegant solution. I mean, the article gives a sample of other possible approaches that one might fall into which are all far most eager it seems.
Even taking into account ComputerGuru insights on all the bumps of the actual implementation, it's still seem far more cleaver and elegant, at least to my mind.
I just discovered this yesterday when a user filed a bug against iTerm2. It caused a problem for reflow because it looks like one long line. Took a minute to makes sense of what was going on, and then I thought it was a neat trick but it wouldn’t have occurred to me to post on hn about it. Glad others appreciate it!
Since it can’t be uttered often enough: thank you for iTerm2. I’m not a power user, but it and Emacs are generally my first two software installs on any new Mac.
This made me think how trailing newline is actually bit weird. Wouldn't it make more sense to just have shell prompts always start with newline and commands not outputting trailing newlines as gratuitously as they do now?
In unix every text file is basically a (possibly degenerate, Nx1) matrix and every row is terminated (and demarcated) by a newline. This makes it much easier to filter, concatenate and rearrange rows. Try rewriting this:
I use a custom prompt that indeed starts with a newline. I like that it gives better visual distinction between the output of commands and the prompt; it's also indeed a workaround for when commands don't put a newline at the end of their output.
Same here (except the #). The advantage is not only that it prevents problems with very long paths and/or commands, but it also makes every command I type start at the same offset from the left, irrespective of the length of the path. Much cleaner that way, IMO.
I use the same pattern, if only because I've not mastered the art of naming directories with clear and short names, so my $PWD tends to get rather long. Add in any git nonsense, and I end up line wrapping anyways.
Much cleaner to always put a newline after the $PWD
If's impossible for the program to know which print is the last one, as the user may kill the program during the sleep. I guess in a parallel universe you could have print issue a newline first. Not sure I want to live in that universe.
The shell would know when the program has been finished for the same reason it knows when to print the shell prompt. The issue isn't that the shell is unaware of the state of the process, it's that the shell simply doesn't know where the cursor is nor what output a process has written to a TTY. And in fact a TTY can even be written to by processes outside of the same shell process group (since they're all just files in /dev).
> The shell could use pipes to intercept all output, and relay it onto the terminal. While it works in trivial cases like whoami, some programs check whether stdout is a terminal and change their behavior, others go over your head and talk to the TTY directly (e.g. ssh‘s password prompt), and some use TTY specific ioctls that fail if the output is not a TTY, such as querying window size or disabling local echo for password input.
Where do I read more about this? I'd like to understand terminals, pipes and the differences. And how does the mouse (in case of terminal apps which support mouse, e.g. Midnight Commander) fit in this model.
A pipe is like a special file that takes text from one program and puts it into another.
A terminal is either a physical object with a screen and a keyboard, or software that pretends to be such an object.
The terminal can display text, but also certain sequences of text do special things. For example, if the terminal receives the character for the escape key followed by [?1000h then it doesn't display it. Instead it turns on mouse click support. So a program connected to your terminal that wants mouse support can send that sequence. Then if you click, your terminal types an escape key, followed by [M, followed by three characters to say what buttons you're holding and where the mouse is located. Now the program will know you clicked.
Both the terminal or pipe show up as a preopened file descriptor to your process. You can pass that file descriptor to the isatty posix function call which will inform you whether the file is a tty (in most cases a terminal) or not.
OMG, as a developer of a terminal application I've encountered this little symbol but just couldn't figure out where it was coming from. I assumed it was something I was doing wrong on the emulation side.
Interrogating the terminal emulator to get the current column is in fact a way smarter, more robust solution that will work regardless of the terminal's behavior when printing at the rightmost column. Also, fewer characters are exchanged with the TTY in the happy case.
Proof of concept, using Bash on Ubuntu 18.04:
Define this function:
getcol()
{
local savetty=$(stty -g < /dev/tty)
local ttyesp
stty raw min 16 time 100 < /dev/tty
printf '\e[6n' > /dev/tty
read -s -r -n 16 -d R ttyresp < /dev/tty
stty $savetty < /dev/tty
printf "%s\n" ${ttyresp#*;}
}
Because you are claiming to be providing something that is "smarter" and "more robust". Not actually testing it enough to find the common case where it doesn't work gives the lie to that claim. It indicates if anything a lack of smartness, and is not robust. (Even Thomas Dickey's Stack Exchange answer on the subject, which is better than what you've given here, isn't robust, as M. Dickey didn't describe how to robustly parse a CPR sequence.)
When people talk about "ANSI sequences" and "ANSI conformant", they also indicate that likely their knowledge comes from old wives' tales and samizdat. The actual existing standards are ECMA-35 and ECMA-48, and the "E" in "ECMA" does not stand for "ANSI". The world of terminals is alas full of people who've worked off informal "escape code lists", or the Appendix for ANSI.SYS in the back of the Microsoft manual for MS-DOS, and don't actually understand in the first place what it is to be standards conformant.
For starters, it is ECMA-48 in both directions, and your code has to be standards conformant, which that absurdly fragile string matching is not. Would you have your own "it's not conformant, it can be disregarded" applied to you? Because it very much does.
In truth, (partly as a result of people working from samizdat and old wives' tales) most terminals and terminal emulators are like your code, not conformant, and by your metric everything should be "casually disregarded", leaving nothing at all; which is hardly a sensible position.
> When people talk about "ANSI sequences" and "ANSI conformant", they also indicate that likely their knowledge comes from old wives' tales and samizdat.
Or maybe some of them are actually referring to ANSI X3.64.
> Not actually testing it enough to find the common case where it doesn't work gives the lie to that claim.
I happen to maintain code that outputs VT100 sequences to control the screen; it's tested on GNU/Linuxes (various emulators), Cygwin, Solaris, MacOS, FreeBSD. Plus remote terminals like PuTTY, various Android SSH clients (ConnectBot, JuiceSSH, ...). So from that and other past work I'm confident that the escape sequence (which can be found in the DEC VT100 manual) works fine in more places than I care to support.
(If the only piece of code in this area I had ever written were the above script, then of course I wouldn't be.)
The script itself isn't portable; it uses nonportable features of stty, and Bash extensions of read.
The parsing of the terminal's response is probably adequate for the intended use, and was labeled clearly as "proof of concept". It won't handle serial line noise. (Actually, nothing will do that under all conditions anyway. Even if parity is enforced, the response can be damaged in such a way that it has valid syntax, but wrong digits giving incorrect coordinates.) Virtual terminals transmit the response reliably.
If you can think of a way that a malicious terminal emulator could target and exploit the flimsy parsing, do share.
I just tested in Terminal.app (macOS) with zsh and several things happen:
If you do nothing but run `echo -n 'hello world'` and then resize the window, the next line prompt will be jittery (it'll jump left and right as you resize but ultimately return to the correct position). My prompt is "14:41:44 ~ ️ <cursor>" and if I resize fast enough, the previous line (the one with the output `hello world%` starts displaying characters on the right side. Characters from the next line. If I do it fast, it becomes obvious what it's doing: Fragments of the next line is copied to the previous line, so the right part of 'hello world%' is filled with: `111141414:314:3914:3914:39:` after a few resizes.
If you press return again (executing no command) then the buggy output is now fixed and no magical behavior occurs.
It depends if you make it bigger or smaller and if the line break was missing or not. Under some combinations of those two parameters, previous lines overflow with white space.
Mostly, terminals don't reinterpret old escape sequences whe resized. xterm and alacritty just cut off text from the right of the window. rxvt-unicode can reflow long lines but seems to preserve it nicely. Presumably they accounted for carriage-return following wrapped text and have that move to a new line in their buffer. I can get it to glitch from inside a tmux.
Zsh will only do anything about the current active prompt on SIGWINCH. Old prompts back in the scroll buffer can't be redrawn even if zsh wanted to. And while a different command is running, even something like `sleep` will do if you're testing, that command has control of the terminal and it, rather than zsh, receives SIGWINCH.
I love this article, as a long time ZSH user it's one of these things I've always wondered how they were implemented but never cared to investigate myself. It's really a clever hack too, if I had been asked to implement this feature my default approach would definitely have been to find a way to intercept the command's output somehow.
This is such a wonderful little hack, I've placed it into my .bashrc. Thank you!
I did actually modify it a little bit, got rid of the two beginning %s in favor of ⏎. It's not a character I expect to really happen to end legitimate output of a program, so it seems more than good enough to keep it from being accidentally intended.
Apply some attributes to it if you want it to be clearer that it isn't part of the legitimate output of a program. This is why the zsh default is a % in standout.
And for zsh user's if you have a setup where unicode fonts can be replied upon, `PROMPT_EOL_MARK` can be set to change the mark. It allows all the usual zsh prompt expansions for stuff like coloring.
I want to swap the layering shell and the terminal emulator, so each shell command (that needs it) gets it's own pty. I think this will enable a much nicer UI and also simplify things.
No this is not what tmux is fore. IMO tmux and screens are giant hacks:
- multiplexing: better to use ssh multiplexing, which is just nice in general
- persistence: Yes, you want to open a pty on a the host, but you should actually emulate the terminal on the guest: one should just forward all the pty messages between the guest and host (graphical emlator). In other words, a lot more like regular ssh
- sharing: graphical emulators should just understand that multiple can be hooked up, have some support for this, we can relay the input from one emulator to the others as needed.
For my terminal-inside-shell, I would use a customer server + protocol for managing all the ptys (remember because backgrounded commands there can be multiple).
I do not like this feature. How can you distinguish the output of a program that outputs a newline from one that doesn't? This is intentional obfuscation. If you are bothered from where your prompt starts, add a newline at the beginning of your prompt.
For better or for worse, unix simply wasn't designed that way. Changing this now would mean breaking so many assumptions hard-coded into a vast majority of unix applications. Considering how complex unix TTYs are, there is practically zero chance this can be done transparently to existing programs. It's going to need a redesign of the I/O system, at which point, we might as well call this new system Plan10.
I also think there are problems with the approach itself. Making the shell sit between programs would complicate the interaction between various programs and introduce new bottlenecks in the shell. Additionally, it likely won't be an improvement over the current situation. Rather, it would just introduce another layer of incompatibility in the shell that programs would have to deal with.
I did this originally with my shell and it's actually not as painful as you'd think. Most CLI tools don't care where fd 0, 1 and 2 are coming from / going to (after all, if you're piping those programs then they're no longer writing to the TTY anyway).
The only programs that cause an issue are tools that send ANSI escape sequences (eg for ncurses) and check if STDOUT is a TTY before sending them. In those cases you'll just get a message written saying something along the lines of "STDOUT is not a TTY".
In theory you could write another workaround to fix that workaround but it was a horrible kludge that I didn't like in the first place so ended up finding another solution (which isn't as elegant as the zsh / fish fix and I'll soon be adapting that into my own shell too).
> The only programs that cause an issue are tools that send ANSI escape sequences (eg for ncurses) and check if STDOUT is a TTY before sending them.
Sure, but aren't those programs the whole point of having sophisticated terminal emulators? If plain streams of text was all we cared about for CLI applications, it would indeed make things a lot easier. So much easier, in fact, that we can dump our current terminal emulators in favor of a more simple alternative without ever having to complicate the shell. But since people do care about terminal UIs, this isn't a realistic solution.
> Sure, but aren't those programs the whole point of having sophisticated terminal emulators?
What we currently have isn't sophisticated terminal emulators. It's a buggy superset of hundreds of kludges due to 60s years of legacy. I mean there's a lot I do love about the design but there's a lot to legitimately dislike as well.
> If plain streams of text was all we cared about for CLI applications, it would indeed make things a lot easier. So much easier, in fact, that we can dump our current terminal emulators in favour of a more simple alternative without ever having to complicate the shell.
The shell isn't the complication here. The shell is just a program launcher. The real issues with terminals is:
- Formatting is inlined with the data stream.
- There isn't any type information in pipes (it is just a raw byte stream) so applications can't easily do context sensitive processing.
- ASCII isn't just a text format but also a data format (there's ASCII characters for tables and records) and flow control (EOF) and job control (^c, ^z).
- It's that responsibility for all of this is divvied up between the TTY driver (in Linux's case, the kernel), the terminal emulator, and any user space applications reading and writing to their respective file descriptors.
- Terminals were never built to be API driven and while there are some interactive ANSI escape sequences they're not widely supported by terminal emulators. So from a shell or tty perspective, the terminal is a black box
- Plus since file descriptors are literally files and TTYs are literally files, it means actually any other process can write to the TTY even if they're not part of the shell's process group (hence why tools like `wall` can exist). So even if the shell could keep track of what was in STDOUT of the processes it spawned, it can't possibly know if anyone else has written to that same TTY.
There some ingenious design in Linux terminals but there are soooo many rough patches thrown in too.
> But since people do care about terminal UIs, this isn't a realistic solution.
Right, but I was never suggesting people don't care about terminal UIs nor that shells don't offer a valuable function. In fact the opposite is true: I've written my own shell because I thought I could create a better UI/UX than Bash.
It does kind of "sit between the program and the terminal," but it's not much useful in the context of this discussion. Going back to the starting point, the problem is that programs and terminal emulators have very limited means of communication. This is why a seemingly straightforward feature can't be implemented by the shell in a straightforward manner.
screen is just another terminal emulator and nothing more. While screen is undoubtedly a useful program, it offers no additional means to solve the problem.
That's because it's the baseline for comparison. Arguing that a baseline isn't slower than itself doesn't make any bit of sense at all. A more useful example would be a comparison against terminal multiplexers, and they can slow down terminal I/O quite noticeably.
> they can slow down terminal I/O quite noticeably
If you're using a GPU-accelerated terminal like alacritty to sink stdout many orders of magnitude faster than a human can possibly read it¹ it's noticeable, but otherwise it's not.
1: I personally don't understand the use case for this — when would you run a command that writes multiple screenfuls of output without anticipating it and not ^C and redirect it to a file or pipe it into a pager?
> If you're using a GPU-accelerated terminal like alacritty to sink stdout many orders of magnitude faster than a human can possibly read it it's noticeable
I wrote "noticeable" because I noticed. On a non-GPU accelerated terminal. When a single pane in tmux is generating a lot of output, it makes all other panes and tmux itself unresponsive, which is hard not to notice.
Yes, please do read that article. alacritty + tmux is not meaningfully faster than Terminal.app + tmux despite the fact their performance without tmux has a 2x difference. The author neatly sums it up with the following phrase:
> Tmux and latency: I tried tmux and various terminals and found that the the differences were within the range of measurement noise.
---
> I personally don't understand the use case for this — when would you run a command that writes multiple screenfuls of output without anticipating it and not ^C and redirect it to a file or pipe it into a pager?
I have three things to say to you. First, have you ever run "make" on a large project before? Second, terminal UIs are a thing, and they write "multiple screenfuls of output" because there is insufficient support for partial updates. Lastly, and more importantly, graphical performance is the least of our concerns if you're passing every bit of output through the shell. It's going to affect I/O throughput regardless of whether it's going to be displayed on the screen or not.
This feature is still surprisingly complicated with many caveats. For example, some terminals will wrap to the next line if you are at $COLUMNS, some at $COLUMNS+1, and some not at all. Some advertise they do one thing and do another. If you miscalculate, you’ll end up skipping an extra line or none at all, maybe overwriting existing output or emitting the missing line break glyph when it should be hidden. We’ve ironed out most of the cases but every once in a while someone will be running an esoteric Pseudo tty (usually not under X) and run into issues.
There are also issues with selection of content. Some terminal emulators will bug out when you try to copy-and-paste content because they get lost with the output past the current location in cases where a line break was already emitted. It’s always the terminal emulator that’s broken but you’d be surprised at how broken some of the most popular emulators are. The entire codebase is littered with workarounds for tty emulators; if someone comes to you and says “fish doesn’t work in $terminal but bash does” (either because it’s minimalistic and is missing a feature that would cause the breakage or because it has a workaround), we almost always end up treating it as a fish bug unless we’re able to get it fixed in the upstream terminal package/OS/software/whatever, whom we file reports with all the time.
It’s insane the configurations and platforms people use shells under and I’m personally deeply appreciative of everyone that takes the time to file these bugs - in addition to making the software better it really does shed light on some extremely archaic bits of posix standards and Unix behavior and I’ve learned so much from the many deep-dives that start off with “this is kind of quirky behavior observed under certain circumstances” and ends up requiring architectural changes deep within because some underlying assumption doesn’t hold universally true as exemplified by a *nix that hasn’t seen a new user in fifteen years or something. It’s why I love hacking on fish; I must have one of the craziest collection of virtual machines to test edge cases, as do many of the other devs (especially @zanchey and @faho, although no one does a deep dive like @ridiculousfish).