I've been thinking about something similar, that there should be a richer command line interface platform rethinked from the ground up. For example, it could support images or mouse input, which is one reason why web-based shells like Jupyter exist.
Basically, I wonder what would a shell look like if it was designed today without any concern for compatibility (with SSH etc.).
Of course SSH compatibility is still a useful feature. Unlike vt220 compatability -- which is the real thing that kept terminals back.
Also SSH is not an impediment: it's a byte-pipe so as long as your graphical shell is serialisable, SSH should be fine. (Although we might want to replace it with something like Mosh (https://mosh.mit.edu/) for other reasons).
IMO the biggest problem with terminals is poor input handling. Many useful key combinations are impossible to type. Paul Evans has a proposal for fixing this, but it seems to be mostly ignored:
Basically, I wonder what would a shell look like if it was designed today without any concern for compatibility
It would look pretty much like it does now. If you look at any professional software, that people spend their entire working day in, it tends to avoid any fancy stuff and rewards those who take the time to learn it (because they have to, because its their job) with superior productivity through keystroke shortcuts that take advantage of muscle memory.
People forget that technical development is bidirectional. Yes you make the best you can with the components available to you - but those components didn't come from nowhere, they were all made by people to solve their own problems, so what's possible and what's ideal eventually converge.
The major leap forward in terminals was to be able to have many of them open at once on your X11 display. It's done, problem solved. Even Apple who can't resist GUI-ing everything have kept the Terminal.app simple.
I remember seeing some amazing things in a video of a LISP-based shell & 'terminal emulator' many years ago. Unfortunately I can't remember what it was called... but as far as I remember, it was full of inline images & animations and listing a directory would include icons to indicate the file type, which could be clicked on to open a file manager, etc.
I don't want mouse input in the shell, half the reason I use it is to avoid the mouse. You could have it with current VTEs though, GPM works just fine and I use it with vim.
couple of notable features.
Tiled, not overlapping windows.
All three mouse button combinations have meaning (select, copy & paste can be done without a keyboard).
Any text can be higlighted and executed.
GUI approach is document based, with everything a text and texts being mostly, um, textual...
I cant find a nice writeup to link to, but there used to be a vm available.
3-button mouse operation seems underused in modern OSes. I miss the "adjust" mouse button in RiscOS. It made changes with the minimum possible state change, eg. adjust-dragging a window would move it without changing the z-order, and adjust-clicking menu items would click the menu item without closing the menu.
With acme, most of the time I only need to 2-click any shell code that I assembled in a scratch file/window, then a new window opens with the output. We perhaps don't need smarter terminals either. Look at it in an abstract way, all we need is an easy way to (a) pass a command to some program and (b) display its output. For (a), acme's input to rc through 2-click is really simple and effective, although it's only for text; but for (b), if we want to include anything more than text, nowadays browsers work in such a way, but we need orders of magnitude faster processors to make it usable.
Right on the money. curses needs to die and we need to make better use of our resources. It's eye-opening to explore these projects that "never caught on" even though they had great ideas that we still don't see today. (Granted, those projects truly never gained popularity for other reasons.) I'm glad to see that there are others thinking about these things. The current state of things bothers me enough that I will do something about it, but I'm afraid that I am something of an extremist and that my particular solution won't satisfy many other people. Because of this, I expect more popular alternatives to arise while I am hacking my own.
I end up using the mouse + terminal(copy/paste hotkeys) quite often for git add after a git status. It's usually faster to highlight+copy/paste than it is to auto-complete on a nested file.
I've often thought that there should be a better/faster interactive way to reference existing output in the next command.
With intelligent highlighting on double click, mouse is so much faster for small one-off operations like that. I wish CLI users/devs/ranters weren't so hostile to the thought of using more than a keyboard...
A shell with a great auto-complete and rich input would be killer. I've been using black screen and really like it. Although they use too much whitespace for my liking.
Basically, I wonder what would a shell look like if it was designed today without any concern for compatibility (with SSH etc.).