One of the fish devs here. I definitely didn’t come up with this hack but I’ve worked on improving some edge cases in the past. Our implementation differs from zsh some although I think we’d originally adopted their approach.
This feature is still surprisingly complicated with many caveats. For example, some terminals will wrap to the next line if you are at $COLUMNS, some at $COLUMNS+1, and some not at all. Some advertise they do one thing and do another. If you miscalculate, you’ll end up skipping an extra line or none at all, maybe overwriting existing output or emitting the missing line break glyph when it should be hidden. We’ve ironed out most of the cases but every once in a while someone will be running an esoteric Pseudo tty (usually not under X) and run into issues.
There are also issues with selection of content. Some terminal emulators will bug out when you try to copy-and-paste content because they get lost with the output past the current location in cases where a line break was already emitted. It’s always the terminal emulator that’s broken but you’d be surprised at how broken some of the most popular emulators are. The entire codebase is littered with workarounds for tty emulators; if someone comes to you and says “fish doesn’t work in $terminal but bash does” (either because it’s minimalistic and is missing a feature that would cause the breakage or because it has a workaround), we almost always end up treating it as a fish bug unless we’re able to get it fixed in the upstream terminal package/OS/software/whatever, whom we file reports with all the time.
It’s insane the configurations and platforms people use shells under and I’m personally deeply appreciative of everyone that takes the time to file these bugs - in addition to making the software better it really does shed light on some extremely archaic bits of posix standards and Unix behavior and I’ve learned so much from the many deep-dives that start off with “this is kind of quirky behavior observed under certain circumstances” and ends up requiring architectural changes deep within because some underlying assumption doesn’t hold universally true as exemplified by a *nix that hasn’t seen a new user in fifteen years or something. It’s why I love hacking on fish; I must have one of the craziest collection of virtual machines to test edge cases, as do many of the other devs (especially @zanchey and @faho, although no one does a deep dive like @ridiculousfish).
Out of curiosity, I installed fish and set my prompt to be blank[1]. Based on this article, I would expect to see the missing line feed character as the first character on the line after every command, but I didn't. Do you know why?
[1] My `fish_prompt` function just does `echo -n ''`. I also tried `true` and an empty function body, I don't know much about fish scripting.
Nice catch and great show of initiative there! It's actually not overwritten by the prompt but rather by the "hack"/post-execution handler itself, depending on where the glyph was emitted (which in turn depends on if the command output included a trailing \n or not) when the line is cleared after guaranteeing we're at col 0 of the next line. If the glyph wrapped to this line, then it would be cleared. If it didn't, it'll show on the line above.
I think the missing clue is that the hack is followed by `^[[K` (clear line from the cursor to the right), before control is passed to `fish_prompt` to paint the prompt or whatever else.
Here's some sample output (stripped of the hack spaces and any ansii escapes):
`fish_prompt` isn't called until the cursor is already at column 0 and the line is clear; the "missing line ending glyph" (here ¶ rather than ⏎ because fish correctly detected that the latter is likely not supported by the font used by my terminal emulator in yet another hack) is emitted at the end of the previous line.
You can see all this and more by piping the output of a `fish --interactive` session to `xxd -C` or `hexyl` or whatever (that'll disable tty feature detection and force ansii escapes to be emitted that you can then inspect).`
(You can also verify by simply using `sleep 1` as your fish_prompt - you'd expect to see the glyph appear at least while the `sleep` was running if fish were waiting on/expecting the prompt to overwrite the glyph.)
I did indeed skip some details in the article. In the case of Zsh, the output actually ends with `\r \r` which is probably meant to handle this exact case by having a space overwrite any marker in case the prompt doesn't.
Ah, I couldn't remember what it was that we did differently from zsh, but I knew there was something.
I would imagine - depending on the terminal emulator - the zsh behavior would cause copy-paste-cruft in the form of trailing whitespace while appearing correct, and would guess this was why we changed our implementation.
I assume that there is control character after carriage return that deletes everything from there to the EOL, so this do not break copy functionality or "go to EOL" shortcuts.
Oh, I just saw your reply after posting my own, though you beat me to it by quite some margin of time. That's correct, fish sends ^[[K to clear to the EOL after correctly setting the cursor position via this hack to erase any evidence of the hack and for posterity of your ptty contents.
Fantastic rabbit hole to dive into, thanks for linking!
The "good news" is that you can't even use tput rev under fish because it saves and restores attributes each time it regains control of the tty with very few exceptions , rev not being one of them. You can't break what you don't even support!
(Seriously though, we should add rev support.)
EDIT: I just saw that you're the same person that answered that post, which very much deserved the upvote. Your knowledge of tty behavior is impressive. It is also rare to find another person that cares about the kvt - a lot of apps that manipulate terminal attributes or use any vte codes are completely broken under even the well-written kernel virtual terminals.
If you want another rabbit hole with respect to supporting attributes, consider that the Windows Terminal people are considering adding ECMA-48 standards conformance that no other terminal emulator has, the various graphic renditions from ECMA-48:1991 that no-one else has implemented in 30 years. (I saw the issue and added overline, framed, and encircled to my terminal emulator's framebuffer front-end. It will be in version 1.41 of the toolset, but that is not out yet.)
It brought me up short to be described as caring about KVTs. (-: I've mooted doing VTs in application mode code for a couple of decades, now. Here's a white paper from 2006, which was several years in.
I believe you are misunderstanding. The feature ensures that your (empty) prompt is printed at the start of a line, even if the preceding command's output doesn't end with a newline.
Start up fish and run `echo -n foo` to see `foo⏎` in your terminal.
My understanding is that the ⏎ character (and some whitespace) is printed unconditionally, but is overwritten by my prompt if the previous command ended with a newline.
If there is no prompt to overwrite the ⏎, why can I not see it?
That would actually not be OK because we can't count on the prompt to overwrite n characters, and the resulting cruft would give you an incorrect representation of the output if you captured a snapshot of the terminal state (or tried to copy and select). Fish clears the line itself after finding itself at col 0 of the correct row. See my other response for more.
Isn't this sort of quirk what the terminfo/termcap databases are for? I guess adding new entries may no longer be feasible, but an entry there like "wraps at column" would allow every interactive command line program to benefit from fixes for new terminals.
am: Flag whose presence means that writing a character in the last column causes the cursor to wrap to the beginning of the next line. If `am' is not present, writing in the last column leaves the cursor at the place where the character was written. Writing in the last column of the last line should be avoided on terminals with `am', as it may or may not cause scrolling to occur (see section Scrolling). Scrolling is surely not what you would intend. If your program needs to check the `am' flag, then it also needs to check the `xn' flag which indicates that wrapping happens in a strange way. Many common terminals have the `xn' flag.
Yup. Fish checks both and acts accordingly, but they're kind of useless because `xn` could mean anything.
The real problem is I'm not sure when the last boolean capability was added to the termcap standard. While new terminals continue to add and update their termcaps, the spec itself is missing a lot of features that would help enormously.
I think the whole terminfo/termcap situation could be improved a lot. For one thing, most terminal emulators just report "xterm" or "xterm-256color" as the TERM environment variable, even though different terminals that report the same thing have different capabilities. A couple terminals, like alacritty and kitty, do ship with their own terminfo files, but that causes problems if you ssh to a box that doesn't have the terminfo file for that terminal.
It can definitely be improved. The model has some significant flaws. I started again from the fundamentals and came up with a rather different model for differentiating terminal capabilities.
Note, in fairness, that not having the terminfo file is not a huge problem, as terminfo is designed so that one can transport information from one system to another, and place local database entries in one's home directory if necessary.
It was supposed to refer to a particular behaviour of Concept terminals. In practice, it is used in many terminfo records to denote DEC pending wrap, which the terminfo commentary even itself records as being different to the Concept glitch. Yes, terminfo has highly problematic paradigms that simply do not match how terminals and their emulators actually work.
Regarding your second link, I cannot comment on the linked item as it has been archived by HN but as an additional reference for others and possibly a fun read for you: https://bugzilla.gnome.org/show_bug.cgi?id=754596
I wondered if anyone remembered those things. So many people seem to be hard-coding escape sequences these days, making untold amounts of work for themselves.
I've always wondered, would it be reasonable / possible for a shell to run everythhing in a virtual tty? I would love a shell when programs couldn't effect each other's output, it was easy to dynamically put programs in the background while hiding their output, etc.
I'm working on a shell that does this, for exactly the reasons you gave and more. It doesn't work well enough to announce here, but you can find an index of shells (including others that have had this idea) here:
https://github.com/evmar/smash/wiki/Related-projects
That came out of a discussion with someone doing a really ambitious overhaul of terminals and display servers (linked in the issue).
I believe you could do some interesting things UI-wise with a shell that doesn't run in a terminal itself. The subprocesses would require a terminal, but the shell wouldn't. For example I've heard the suggestion to always keep the prompt at the top of the screen from a few people. That seems like it would be straightforward in an HTML UI, but it probably requires some cooperation from the shell itself.
This is a more Emacs-like architecture for a shell, with the interface being written in a high-level language rather than C/C++. That came out of running parts of ble.sh, which is an interactive shell written in bash !! (I believe it's the biggest bash program in the world, in both number of lines and sophistication)
If you have any ideas or suggestions, I'm interested. In the interest of time, I'm cutting out the interactive parts of the project and making it more like a library that other people can build shell UIs on top of.
I collected these links awhile ago but haven't done anything with them, there is some overlap:
reptyr can reparent a program if you only realise after the fact.
I try and remember to have that installed on the heavy-interactive-use servers I admin for when users have an "oshi-" moment because they forgot to start something under screen.
Thanks for the post! These things absolutely should be documented and dissected in posts like your; there’s way too much voodoo in dealing with terminals.
Woo! I have the same trick in my .bashrc, love to see it spread.
If you can, please add a note that if you want to add it to bash you _have_ to use an echo/printf from PROMPT_CMD, if you add it to PS1 bash's (or readline's?) own logic will get confused as to how long the prompt is and what column the cursor is at.
(If anyone wants to try it, personally the only caveat i noticed is that when you copy a command from the terminal window any spaces not overwritten by something else will appear in the output.)
This is the kind of rabbit hole that gives me nightmares any time I even consider the idea of writing my own shell. Fish nails the UX so well that even doing half as good a job seems like a truly massive amount of work.
As someone who has written their own shell I can still recommend it as a project. Actually writing the readline implementation was much easier than I was expecting. Not suggesting there aren't edge cases I've missed but at least that is a longer term issue you can resolve as and when people report issues (and if nobody else's uses your shell then the other use case you need to worry about it your own).
I do have some recommendations however:
- first of all use a 3rd party readline implementation. This will allow you to write your language parser, any builtins you'd need (eg `cd`) and forking routines. These are the bigger / harder problems.
- then once you have a POC you're happy with, rewriting your readline implementation becomes a natural progression. By that point you know how you want your shell to look and feel.
I will second that. We provide VM solutions on cloud marketplace and all our VMs are using fish as the default shell. Both us and our customers love it.
Thanks for building it.
We wouldn’t run an external command because of the overhead, compatibility, and portability, but I’ll treat your question as “why not do what tput el would do?”
It all depends on where the input cursor is when the command terminated. You need something that works with both cases (line break and no line break) because you can’t introspect the output.
tput el clears the output to the end of line so you still have to figure out if you’re on the same line as the output or if you’re on a new line. The goal of spaces is not to clear content but to invoke the wrapping behavior of the terminal which will either cause you to be on the same line only in the case that you started off at col 0 or else somewhere in the middle of the next line, but in both cases, you’re on the row you need to be on (after taking into account whether wrapping happens at max column or max column plus one).
Edit: It’s actually a lot like rounding by adding 0.5 and truncating. And the complexities are akin to trying to do it with non-standardized or different standards of doubles and dealing with epsilon and NaN :D
No worries! (Now try figuring out what it does and why it does it from the code that you found spread out over a few hundred lines of C interspersed with curses calls and no comments.)
Not directly in fish, because we manually set the tty flags and restore them after each execution (it's why your fish prompt doesn't break when you ^C in the middle of a redirect or when you accidentally output binary to the terminal), but e.g. in bash or if fish were to emit those escape codes directly. But yeah, it definitely seems like a good hack and it would cut down on the bandwidth requirements (you can notice the lag when using fish/zsh over ssh on a lossy connection).
Note that depending on the xn attribute, \r may or may not end up taking you to col 0 of the same row, depending on if the previous output caused a soft wrap but didn't include the trailing new line - you could wind up at col 0 of the previous row instead, if I'm reading it correctly!
This is an XTerm FAQ, and you are not in the analogous situation that you purport to be in. You are abusing "VT100" to describe the current state of affairs in the world; which is most definitely not a DEC VT100, not like a DEC VT100, and most definitely different in many important respects (such as colour, for starters) to a DEC VT100. It's even different to "ANSI" (again, which should be ECMA-48) in that no terminal or terminal emulator (that I know of) actually provides standard function keys (function keys being another thing that a DEC VT100 does not have). A few of them, sort of, provide the non-standard function keys from a VT220, which isn't a VT100 either.
Far from "winning", "ANSI/VT100" was superseded (by things that were better) within a few years, long ago.
This feature is still surprisingly complicated with many caveats. For example, some terminals will wrap to the next line if you are at $COLUMNS, some at $COLUMNS+1, and some not at all. Some advertise they do one thing and do another. If you miscalculate, you’ll end up skipping an extra line or none at all, maybe overwriting existing output or emitting the missing line break glyph when it should be hidden. We’ve ironed out most of the cases but every once in a while someone will be running an esoteric Pseudo tty (usually not under X) and run into issues.
There are also issues with selection of content. Some terminal emulators will bug out when you try to copy-and-paste content because they get lost with the output past the current location in cases where a line break was already emitted. It’s always the terminal emulator that’s broken but you’d be surprised at how broken some of the most popular emulators are. The entire codebase is littered with workarounds for tty emulators; if someone comes to you and says “fish doesn’t work in $terminal but bash does” (either because it’s minimalistic and is missing a feature that would cause the breakage or because it has a workaround), we almost always end up treating it as a fish bug unless we’re able to get it fixed in the upstream terminal package/OS/software/whatever, whom we file reports with all the time.
It’s insane the configurations and platforms people use shells under and I’m personally deeply appreciative of everyone that takes the time to file these bugs - in addition to making the software better it really does shed light on some extremely archaic bits of posix standards and Unix behavior and I’ve learned so much from the many deep-dives that start off with “this is kind of quirky behavior observed under certain circumstances” and ends up requiring architectural changes deep within because some underlying assumption doesn’t hold universally true as exemplified by a *nix that hasn’t seen a new user in fifteen years or something. It’s why I love hacking on fish; I must have one of the craziest collection of virtual machines to test edge cases, as do many of the other devs (especially @zanchey and @faho, although no one does a deep dive like @ridiculousfish).