In the a similar spirit, here is another research paper that Simon Peyton Jones is a coauthor of: Build Systems à la Carte[1]. It describes a formalism for representing build systems and classifies the recalculation algorithm in Excel as a type of build system:
View maintenance, rendering and compilation are instances of the same underlying problem: efficient incremental calculation of an acyclic computation graph, a.k.a. differential dataflow. When the graph has cycles or feedback, you enter control theory country. It’s not surprising that neural nets look like feedback control systems.
I've been interested in the theoretical side of this since forever (my first love was node based CGI software which actually exposes the graph as your main interface) and never go to it. Any control theory book suggestion before I hit springer ?
SPJ has been coming at this from the angle of let’s add things to Excel because Excel itself is already established.
This, in my opinion, is putting lipstick on a pig. The inputs and outputs work represents a very stretched contortion of trying to do real programming in a limited environment.
I am coming at it (commented elsewhere in the submission my project, avoiding spam) from the other end: start with a fresh, language take all the tried and tested programming language research useful to this way of working and connect it to a UI which fully comprehends this language. Throw out the grid and coordinates.
Row types, polymorphic variants, FRP for time/streams/events, type classes for arithmetic, CAS for cells, etc. We really do know what works, and therefore we can actually start from scratch and have some thing that is actually nice to use.
To add to (usual/frequent) spreadsheet software as a database cons:
- No formula/value separation
- Only trees, no cycles
- No named references
- No (easy) functions
- No (easy) SQL-like
- Views
- Triggers
- Default values
- Constraints
- Advisory at least
- No linter
- No birdeye for a schema
- Shit typed
- Even if you insist
- No who when what
One could add to this more and more, but the biggest issue with SSAAD is that a seasoned developer can’t even help a user to maintain its sanity or find errors. Users will break every rule because a software development is full of unspoken practice but it didn’t bite them yet. And when it finally does, when a user feels an overwhelming complexity and that lack of control kills their operation, it already costs a small fortune to reveal and reimplement all nuances and edge cases into software requirements. I’ve seen a few businesses that became a function of what they can do with Excel and what Excel can do with them.
Theoretically these are all problems with Excel, not with spreadsheets in general.
Also Excel does support circular references; I once worked for a guy who used them to iteratively fit a model!
I'm waiting for "the programmer's spreadsheet" where you have functions, clean separation of "data" and "display", proper types, and some kind of sensible version control.
I am working on a product which is pretty much exactly what you describe. I love spreadsheets but I wanted something that I wanted to actually program in right from the beginning to the end of my work process.
And it is a pure functional programming language, with cells, which are reactive, and with real data structures like lists and records. We have a work in progress beta. Sign up requires a paid subscription, but only because I wanted to make sure that the infrastructure was working. To try the work in progress product, you can just hit Try which lets you play in a sandbox. It doesn’t save or remember any of your work, and also doesn’t even hit our databases. (Safe from the hug of death.)
It’s pretty bare-bones right now but it’s based on ideas from Haskell, and Unison which is a content addressable language. (Every cell has a SHA512 hash of its content.) That makes versioning very easy, but this is not seen in the UI yet.
The table editing and what not is a little cumbersome right now, as there are a lot of things to come together and I wanted to demonstrate at least some of them.
Speaking of clean separation between data and display, The plan is to build for example reports using a simple HTML like DSL. Also, things like time and external data sources will be treated in the type system as explicit streams, which is an architecture that has been fully explored in Haskell under the guise of FRP.
But I’m working on this in my spare time and therefore progress is turtle pace rather than silicon valley pace. :)
Maybe something like Observable (the product not the pattern) + TypeScript?
I’ve been thinking recently that TypeScript, when considered by itself (ie, just the type system), is quickly becoming a kind of flexible type database around a very loose language.
You can force TypeScript to figure things like:
type One = 1;
type Two = 2;
type addOneAndTwo = One + Two;
But its very brittle and difficult to accomplish, especially if you wish to couple this back to the runtime values.
If these ever become language features, I can see this powering the thing you’re describing: a capability that gives you flexible loose JavaScript for the non-programmers that can also be iteratively tightened, possibly with a reactive view fronting all of it.
I couldn't agree more with the notion of "the programmer's spreadsheet"; I've often wondered what it would take to get something like R/Python notebooks up and running with some more spreadsheet features.
Excel is a nice job-creation measure for freelancers.
Re-implementing such sheet in C# was a full time job for several years for me.
Modern development tools (in contrast to C or Assembler) help the user to maintain its sanity by preventing him from hurting himself. You need much less of unspoken practices to develop in Java then in C. And you will hurt yourself much later.
The same goes for databases. You need much less foresight when you write SQL, but the optimizer will be overwhelmed sooner or later, if you are doing something complex.
I would say that CubeWeaver prevents the user from shooting himself in the foot for much longer than Excel.
Sure, it wasn’t a criticism for $subj, just mere observations from the first part of my “career”, cause it feels like I spent a good half of it trying to unfuck xls and its consequences for orgs.
By shit typed I meant entering or pasting 1.200 into a numeric field and getting -600,000 something. This is exclusive to excel, which is “smart” enough to detect patterns in input, but not smart enough to introduce/look at the cell affinity, at locale settings or at common sense.
This is near and dear to my heart since Excel is a great platform, but you can't just slap Excel in a server (for most apps).
Being reactive was a core design decision with my programming language ( http://www.adama-lang.org/ ), and it simplifies my life greatly. I treat the back-end like an Excel sheet that I can send messages to, and the server will ingest those messages into some data changes (typically table changes), and then all the formulas that depend on those sources of data get recomputed.
Since, I'm using websocket, I can leverage knowledge of the prior state and just send deltas down to the client. It's super efficient, and I'm having great velocity with just manually data binding DOM elements to a giant reactive document.
Beyond the current market of board games, I have an ambition that I'll scale it out of the current niche to compete with spreadsheets. I'll probably turn the infrastructure into an open source firebase like product. Not sure yet.
There was this company I worked for, that realised the customers (insurance biz) had a lot of proficiency in Excel, and basically half of the use case of the biz was converting these spreadsheets into web apps. Of course this a) takes a lot of time and b) isn't very agile in face of spreadsheet updates.
So my idea was to process the spreadsheet as follows:
- list the output cells
- list the input cells
- parse the formulas
- walk from the output cells to the input cells
- use the resulting graph to transpile the spreadsheet
Can you imagine the consequences of the resulting transpiled artifacts. Source control. Diffs. Conformance tests. And more.
So I got it to an Excel+OpenDocument formula parser plus essential standard library, an AST and prototype transpiler into Ruby and Go, ready to bundle in your monolith or run in a lambda.
The thing was shot down because "If we build this, how could we bill our customers big $$$ for non-work? This would drive our gross income down, so we'd put ourselves out of business!".
Right.
And now the IP is owned by that shitty company, which sits on it to extract cash for customers.
I wrote something like that many years ago using OpenOffice format which is basically XML, and then I translated to PHP. (http://jeffrey.io/wall-of-shame.html KaPowie)
If I didn't have so many different ideas, then I'd love to return to it.
Ultimately, I'm realizing if I want to storm this market, I need to create a successful revenue engine to explore the space. How I would do it now is provide two hacks:
(1) special format for tagging regions within the spreadsheet as either tables, objects, or values with static types.
(2) a wizard to walk through the transpilation to provide guidance and look into oddities in the sheet.
The purpose of (1) is to automate the tagging process in (2), and the key is to turn the process into a predictable game so you can move back and forth easily. Now, if you transpile from Excel/OpenSheet to Adama, you not only have source control but a reactive database with full history and auditability.
MS slapped Excel in a server : I remember developing a webapp with a backend which calculated ... things ... through an embedded, user-supplied Excel sheet.
If it was Sharepoint or pure .NET, I don't recall.
As I age, I realize the importance of it based on people I know.
I am pondering two things at the moment.
(1) how to introduce high contrast for important details. There is a lot of cosmetic noise in board games, and I'm looking at how to rendering JSON into an image. What annotations can guide this rendering such that important details are more obvious.
(2) of what I'm looking at, what is the (a) important stuff right now and (2) what happened. If I can make it easy to describe the picture with words, then not only can I solve the problem for blind people but I can bring home assistants into the picture.
For (2), I can solve (a) easy enough. For (b), I'm still sorting that out since I am playing with my hands behind my back. Like, I can describe things as "cards entered your hand" or "cards left your hand", but I'm struggling to figure out how to distinguish between "you got two cards from the deck" versus "you took a card from Zem into your hand". Similarly, I'm having the same problem as "you trashed a card" versus "you discarded a card", and I'm struggling with the in-between for animation. The core reason for this problem is that the client just sees an array changing in elements, and I don't know the origin or destination for those elements. My hope is that if I can solve this, then I get both the change-log and animation within a unified idea.
is it too expensive to send an event log to the client and have it be replayed? i did that for a stateless scrabble bot and it worked out pretty well, the bot got an event log from the server, replayed it to construct a board position, and sent back a move which was added to the event log sent to the other player.
that's what I am doing, but my events are data transformations with zero game-context. For instance, I have an array that goes from [1,2,5,7] (card ids in your hand). I will know that array will change into [1,2,5], and that's great. However, why did it change? Did I play a card? was a card taken from my hand? Did I trash a card?
The immediate problem is how do I animate the card going away? The next question is how do I convert that to text like "You drew a card" versus "You stole a card from player X"
Now, ideally, I want to make this very simple and also privacy respecting as well. I can hack the card animation problem directly by having the UI detect connect the dots between cards appearing and being orphaned. This may be good enough for many scenarios.
Another thing I'm thinking about is contextualizing the events the client sends (which do have more domain context). This is probably the right way to go, but this adds a new layer to think about.
it would work better if the event was something like "PLAY, played=7, drawn=K, trashed=K, hand=[1,2,5]" with the client perhaps validating that applying the play does lead to the hand [1,2,5] as an error correction mechanism.
you need to make sure not to leak information about one player's hidden moves to another player, of course, but in general i would say have your events be as rich as possible so that clients can interpret them for animations, screen reading etc.
Spreadsheets might be the best possible tool and represents an inflection point where any improvements detract from other dimensions. So it’s like the Pareto frontier for software/data projects.
I’ve been waiting years for someone to solve this as there are serious flaws that I’d like fixed.
But all the “improvements” end up breaking other things. For example, airtable seems cool but its cost makes it non-comparable (eg, email a spreadsheet to a million people, a million can “fork” try pricing that with airtable; save a spreadsheet to disk, archive forever, etc).
I think that everyone can program, but few will be professional programmers. And many of the fixes for Excel are assuming that all people who program should act like professional programmers. And this isn’t possible or likely.
What about strongly-typed spreadsheets? I’d kill for a strongly-typed spreadsheet. So many literal million-dollar mistakes are caused by typing (not “typing”) errors in spreadsheets, and Excel makes it easy to do the wrong thing.
Ad-hoc SQL workday be nice too. PowerQuery is nice (great, even) but it doesn’t feel like it’s a part of Excel - and while it’s language is better than SQL, I really don’t want to have to learn a whole new syntax to express the same simple ideas.
During a massive migration to a different device management service I opened my CSV and saved it as xlsx as the site didn't support csv. Instantly 12% of the serial numbers had their leading zeroes removed and needed a corrective action. ~1800 devices.
Excel "helpfully" auto formatting zip codes, part numbers, etc with leading 0s into integer columns deserves to replace null references as the billion dollar mistake.
I try to make my analyses in something reproducible and then have it “still work” in Excel. So the strongly typed stuff is done elsewhere.
It’s funny though because so many people who crank on excel don’t know what types are. And don’t know what variables are. And don’t have the layers of mental abstraction that programmers who want types have and need.
So the expected protections don’t really happen. So while it’s nice to not convert “01” to “1” automatically, lots of errors will probably be caused by people trying to add “01” and “02” and getting “0102” and being super confused.
You can’t in excel, but I was referring to if strong types the language could understand what to do with “+” for strings so could concat instead of fail.
I'd prefer it to fail rather than overload the `+` operator for unrelated types. VB gets few things right, but of the things it does get right: using the `&` operator for string concatenation makes far more sense than overloading `+`.
I commented elsewhere in this submission that I am working on a strongly typed spreadsheet. (Don’t want to spam my product link.)
I do think that bringing new syntax to the table is justifiable if it brings new concepts with it.
I agree, power query is essentially a separate product and doesn’t feel first class. I think that probably explains its relative obscurity, despite being highly powerful.
Airtable is strongly typed in the fields of the tables, but the expression language is not. It is my conclusion based on researching Airtable, that the decision to have well typed tables is so that relational operations work nicely, and so that they can implement interfaces on top of these tables easily, such as calendars and work planning things etc. I don’t think that there is an underlying motivation to fix the ill-type code of the world, so to speak. Any non-trivial code work is done in JavaScript, anyway.
> Any non-trivial code work is done in JavaScript, anyway.
JavaScript is strongly-typed though, it just lacks a way to declare types. One can easily restrict types in script code (using `typeof` for primitives and `instanceof` for object prototypes).
People refer to it as "weakly-typed" because most (but not all) of the built-in types have implicit conversions to other types; and many of which are non-obvious: https://github.com/denysdovhan/wtfjs
> So it’s like the Pareto frontier for software/data projects.
Yes, I like this point of view a lot. Like probably the majority of HN, I dread the very idea of a .xls file, but most problems are related to people not understanding the tradeoffs as opposed to shortcomings of Excel itself. "Usability by non-technical people" is a respectable if underappreciated dimension.
I have a friend who tells a story of how “Excel saved the world” because during the 2014/15 Ebola outbreak Excel was used so much.
So it’s not so much to exclude, but to figure out how to use it and include it.
I think Excel is like the saturated fat of the data world. Widely used, found bad in many situations, tried to be removed completely, replacements caused more harm (sugar), now figuring out the right kind and what combinations are good.
Excel is just reactive programming. One thing that I always thought was missing was the ability to do looping until a condition is met in Excel. It’s actual very possible if you enable “iterative calculation” in the settings. This will allow you to walk data structures and perform complex calculations until you reach a goal.
I've been working on/with spreadsheet users (and some hardcore spreadsheet financial models) for the past 1.5 years - and I like to think about spreadsheets as a reactive, visual (and 2d!), _functional_ programming language. Except if you use circular references, in which case it's no longer functional (b/c you can use use circular references to store internal state!).
If you're interested, I've spent the past 6 months working on Mito -- https://trymito.io/launch1 -- - it's a spreadsheet that sits inside a Python data analytics workflow, and generates Python for you while you edit. Our approach to spreadsheet programming is a little bit more on the nose :)
I like your work on Mito, and I feel like it’s the python answer to a project that I’m working on which is the equivalent but with Haskell, or rather a riff on Haskell to be more amenable to spreadsheets-like programming.
One thing that you can do when you have a pure functional programming language is rather than generating separate cells for modifications, you can go into the code and modify it directly fairly reliably.
I’ve linked my project elsewhere, don’t want to spam the thread.
I think two stated downsides have been somewhat addressed in excel in recent years:
The “high code duplication” downside has definitely been addressed in recent versions of excel, first with dynamic array formulas (which let you apply one formula to a number of cells, which can resize according to the data with use of unique/filter) and then with new expressions such as Let / Lambda.
Secondly Vlookup has been replaced with the faster, simpler and easier to read Xlookup. Using this with named ranges and dynamic arrays, if you wanted to get a students exam result from a table for example, you could have a formula that looks something like =XLookup(A2, Results_Name, Results_Score)
I don't think you can address the downsides without completely redesigning Excel.
There were some attempts to add loops, IFs and so on to the assembly language. Somehow this approach could not compete with Java: https://en.wikipedia.org/wiki/Assembly_language#Support_for_...
I think there's a lot of potential for tools like this that live on either side of the spreadsheet, i.e. that build on top of spreadsheets to provide more functionality, or that create more powerful ways of creating spreadsheets like the tool described in the article. For example GRID(https://grid.is) also works with existing spreadsheet software instead of asking people to learn a new tool in order to become more productive. It is essentially a presentation layer on top of spreadsheets, allowing the user to create custom web apps by using a proprietary, browser-based spreadsheet engine to run all of the spreadsheet computation on the fly.
Full disclosure, I am closely affiliated with people that work there.
I like your post and it makes sense to me. I went to check your product homepage and feel it is too busy.
I want to give the product a try. I am seriously interested in this space as I am building my own product in it.
Is CubeWeaver a SaaS product? The reason I ask is that the user might think that the added benefits CubeWeaver brings are going to be locked in a new (proprietary) file format and can not be shared/owned. The Excel export is great but I guess that reduces the benefits you mention.
One of the huge benefits of Excel, in my opinion, is that people have an app and share the file around. Yes, to the software engineer, this is horrible. But to the accountant/business manager it is a feature.
CubeWeaver is a .NET web application. It is available as a SaaS product and as a downloadable Desktop Edition (based on electron.js). Check out the download page: https://cubeweaver.com/download
I think you shouldn’t try to replace a spreadsheet with yet another spreadsheet that just has one extra feature. Many, many start-ups have done that. They make some money, but it’s not really anything new.
The first 3D spreadsheet I encountered was the very pricey Boeing Calc in the mid 80's. I also had a beta version of the never released VP Planner 3D. Seems they were litigated out of existence by Lotus with a look-feel lawsuit. I was later happy to dance on 1-2-3's grave...
Apparently nobody has mentioned Airtable yet. It appears to be a spreadsheet-database hybrid (also in the spirit of Improv), but is advertised as a low-code app development platform, aimed towards organization, and has minimal numerical processing tools. Instead, I want a number crunching environment, something with SQL's strong schemas and multiple tables, Excel's data entry/inspection interface and reactive formulas, R or numpy (idk about pandas)'s flexibility, and Excel/matplotlib/ggplot-like graphing.
Airtable is pretty, but it took me about 5 minutes to hit hard limits on its capabilities, for doing basic things I've done in spreadsheets for 20 years.
This is why MS created PowerApps or whatever they’re called. Basically Excel but UI controls instead of a spreadsheet. They’ve realized millions of people are programming without realizing it. So they created a step up from excel and are selling it as a “low-code” (new buzz word) platform.
Salesforce has been producing low-code solutions for several years, so it's not a new buzzword at all. Power Apps is competing with Salesforce on that front, as it's built on top of Dynamics 365, MS' original CRM solution.
For those interested in trying out the spreadsheet model in actual programming i recommend Mobx. It tracks what values are used in which calculation, and auto-runs them again, just like a spreadsheet.
Have you thought about not having the external components, and moving the logic/code into the spreadsheet itself? In other words, just have a single spreadsheet and the whole UX in that, and the things you have as external components integrate right into the user's sheet. You can have both secondary notations and conditional projectional editors based on the parsing of the sheet.
I cobbled in a spreadsheet, where one page convers numbers to words (10 to Ten) any amount till $1,000,000. No vba. One sheet is database for cheques. One sheet has three pages, each printing a version of cheque on bank cheque leaves. One other prints an electronic cheque. One last one prints complete cheque on blank stock paper.
All of these sheets gets all data from cheque database & bank database.
Same thing. I started working on a financial statement excel sheet like 4 years ago, refining it, then standardizing it, then decided to make data entry in a sheet. Then added more stuff.
4 years later I have import data from old file, previous year data import, PDF output, customised displays, multi page PDFs and I am just getting started. Come financial year, this is my most cherished work and I do it all the time. (Staff has to keep up with version updates every next week)
Every time I come across Blockpad I wish I could get my company to provide it. Unfortunately it’s huge group so that won’t happen, but it does look great.
I put my hopes on Python or Julia notebooks now, but it’s not accesible enough for most colleagues.
I see that it supports units in terms of scale - does it also prevent units that measure different things from being combined? E.g. 2 kg + 3cm (units mismatch) or 3$ * 7$ (two monies can’t be multiplied).
I'd be more interested in which country/ies where Excel is NOT used by MOST businesses. I'm sure such country exists, but I just can't imagine there is, considering it's also in widespread use in my country.
the reason excel sheets work well is most data is simple. the underlying tool may have complex features but most don’t use them. the value from simple features is so high it always is worth it
As someone that often finds myself using spreadsheets to merge, refine, organize, or otherwise work with different datasets, I would say that the simplest 10% of features cover about 90% of my use-cases. It's rare that I need to go digging for anything truly complex (usually VLOOKUP is as hard as it gets). Most data is indeed simple.
I agree, I think most people just need something one step up from a calculator that they can persist on a hard drive and create basic pivot tables/charts from.
Having this ability is a huge advantage over a calculator.
100% agree. There's a good reason spreadsheets have lasted since lotus 123 (1983) to now, you can get so much millage from the simple idea of of a grid of values.
Spreadsheet certainly are visual programming languages: by any measure, by far one of the most common most widely used types of visual programming languages in the world.
Taxonomies of Visual Programming (1990) [pdf] (cmu.edu)
>DonHopkins on July 13, 2019 | on: I was wrong about spreadsheets (2017)
>Google sheets (and other google docs) can be programmed in "serverless" JavaScript that runs in the cloud somewhere. It's hellishly slow making sheets API calls, though. Feels like some kind of remote procedure call. (Slower than driving Excel via OLE Automation even, and that's saying something!) Then it times out on a wall clock (not cpu time) limit, and breaks if you take too long.
>A CS grad student friend of mine was in a programming language class, and the instructor was lecturing about visual programming languages, and claimed that there weren't any widely used visual programming languages. (This was in the late 80's, but some people are still under the same impression.)
>He raised his hand and pointed out that spreadsheets qualified as visual programming languages, and were pretty darn common.
>They're quite visual and popular because of their 2D spatial nature, relative and absolute 2D addressing modes, declarative functions and constraints, visual presentation of live directly manipulatable data, fonts, text attributes, background and foreground colors, lines, patterns, etc. Some even support procedural scripting languages whose statements are written in columns of cells.
>Maybe "real programmers" would have accepted spreadsheets more readily had Lotus named their product "Lotus 012"? (But then normal people would have hated it!)
Excerpt from "Taxonomies of Visual Programming and Program Visualization", by Brad A Myers, 1990/3/1, Journal of Visual Languages & Computing, Volume 1, Issue 1, pages 97-123:
Spreadsheets, such as those in VisiCalc or Lotus 1-2-3, were designed to help nonprogrammers manage finances. Spreadsheets incorporate programming features and can be made to do general purpose calculations [71] and therefore qualify as a very-high level Visual Programming Language. Some of the reasons that spreadsheets are so popular are (from [43] and [1]):
1. the graphics on the screen use familiar, concrete, and visible representation which directly maps to the user's natural model of the data,
2. they are nonmodal and interpretive and therefore provide immediate feedback,
3. they supply aggregate and high-level operations,
4. they avoid the notion of variables (all data is visible),
5. the inner world of computation is suppressed,
6. each cell typically has a single value throughout the computation,
7. they are nondeclarative and typeless,
8. consistency is automatically maintained, and
9. the order of evaluation (flow of control) is entirely derived from the declared cell dependencies.
The first point differentiates spreadsheets from many other Visual Programming Languages including flowcharts which are graphical representations derived from textual (linear) languages. With spreadsheets, the original representation in graphical and there is no natural textual language.
Action Graphics [41] uses ideas from spreadsheets to try to make it easier to program graphical animations. The 'Forms' system [43] uses a more conventional spreadsheet format, but adds sub-sheets (to provide procedural abstraction) which can have an unbounded size (to handle arbitrary parameters).
A different style of system is SIL-ICON [49], which allows the user to construct 'iconic sentences' consisting of graphics arranged in a meaningful two-dimensional fashion, as shown in Figure 5. The SIL-ICON interpreter then parses the picture to determine what it means. The interpreter itself is generated from a description of the legal pictures, in the same way that conventional compilers can be generated from BNF descriptions of the grammar.
10. Conclusions
Visual Programming and Program Visualization are interesting areas that show promise for improving the programming process, especially for non-programmers, but more work needs to be done. The success of spreadsheets demonstrates that if we find the appropriate paradigms, graphical techniques can revolutionize the way people interact with computers.
>lmm 9 months ago | on: Maybe visual programming is the answer, maybe not
>If there was a visual programming language with anywhere near the popularity of Ruby, I'd be willing to consider that maybe the idea has some merit.
>DonHopkins 9 months ago [–]
>Excel.
>I could turn your argument around: If Ruby were anywhere near as popular, widely used, and successful as Excel, I'd be willing to consider that maybe the idea that Ruby is a viable programming language has some merit.
>But I won't, because whether or not something is a visual programming language isn't up to a popularity contest.
>Can you come up with a plausible definition of visual programming languages that excludes Excel, without being so hopelessly contrived and gerrymandered that it also arbitrarily excludes other visual programming languages?
[...] (TL;DR: he couldn't, since he was under the mistaken impression that Excel not programmable, and was less popular than Ruby...)
That thread was on an earlier discussion about a blog posting from 2020 about the same 1989 paper by Brad Myers that we're currently discussing.
>Maybe Visual Programming is The Answer. Maybe Not
>Whenever discussing problems with programming today and potential solutions, invariably someone will pop up and declare that the problem is obviously the fact that programs are linear text and if only programming were visual, all problems would immediately disappear in some unspecified way.
>I understand the attraction of visual programming, particularly for visual thinkers. However, it's not as if this hasn't been tried, with so far very limited success. Brad Myers, in his 1989 paper Taxonomies of Visual Programming gave, along with the titular taxonomy, a non-exhaustive summary of the problems, starting with visual languages in general:
I feel like I've been reading variants of this post since 1997. I'm nodding along while simultaneously face-palming. I feel like you have to address the chestnut, at least when writing about it. It's possible that a naive charge is better than a valiant one... but I can't help thinking that excel is the walls of constantinople, and the walls have stood since the first days of mass computing by non computer specialists.
The "spreadsheet for non-programmer programmer" take is probably the most common. For some reason, Sridhar Vembu's take is the one which comes to my mind, mid-200s. I'd be curious to hear his current take, considering how relevant "programmable by non-programmers" is to a lot of Zoho's products.
Spolsky, an OG excel PM, decided to finally notice the over non-intentional reasons that excel is persistent: It makes tables, which is useful for lists. His job was to make excel programmable at one point. It's interesting that he came to the opposite conclusion. Less, not more sophistication.
"We found a fellow whose entire job consisted of maintaining the “number of injuries this week” spreadsheet ...
..He typed the current date in the top of the spreadsheet, printed a copy, put it in a three-ring binder, and that was pretty much his whole, entire job...
..Suddenly we understood why Lotus Improv, which was this fancy futuristic spreadsheet that was going to make Excel obsolete, had failed completely: because it was great at calculations, but terrible at creating tables, and everyone was using Excel for tables, not calculations.
Bing! A light went off in my head.
The great horizontal killer applications are actually just fancy data structures.
Spreadsheets are not just tools for doing “what-if” analysis. They provide a specific data structure: a table. Most Excel users never enter a formula. They use Excel when they need a table. The gridlines are the most important feature of Excel, not recalc."
Word processors are not just tools for writing books, reports, and letters. They provide a specific data structure: lines of text which automatically wrap and split into pages.
PowerPoint is not just a tool for making boring meetings. It provides a specific data structure: an array of full-screen images..."
Anyway.... I feel like "futuristic spreadsheet that will make Excel obsolete" is that dangerous mix of understated ambition and foolhardiness. It probably would have been impossible to invent excel intentionally.
A spreadsheet compiler is an interesting concept. Relational data is often where excel users are often the most sophisticated already and have hard problems to seeking solutions.It's also treacherous, potentially. There's a very sparse no man's land between "real" programming environment and the space excel operates in. Adding power to excel without stepping into that no man's land is a Hard problem. Maybe it's worth biting the bullet, and approaching it as a "my first programming environment." That has learning curve implications, but it would make porting solutions from the "real" programming world a lot easier.
Spolsky's point is relevant. Users are simpler than you think. "Tables & lists" are where most users are. Functions are an intermediate feature. Pivot tables vlookups are advanced features. A small minority use them, or even believe they could learn to. Trello went after sophisticated novices. A spreadsheet compiler is kind of beyond "advanced," and <1%^ of users live here. But... in an country of 1bn people, .1% is still a lot of people. >0.1% of excel users are already programmers.
Excel is the worst of the Office tools, but it is hard to beat precisely because it is part of office.
On top of all the other challenges of replacing it, you are probably going to use Word and Powerpoint and maybe even some others and since it is bundled your employer won't save money ditching it.
A huge portion of the world runs on excel, it is one of the most stand out pieces in office. There is literally no competition that comes close to the level of functionality that excel has.
Over the past decade, I've worked with teams that dumped Word for Google Docs or MS Project for Basecamp/Jira. But Excel is always there. Nobody ever asks for a Google Sheet with product performance or financial projections data.
[1] Applying Programming Language Research Ideas to Transform Spreadsheets -- https://www.microsoft.com/en-us/research/blog/influencing-ma...
[2] McCutchen et al. 2020, "Elastic Sheet-Defined Functions: Generalising Spreadsheet Functions to Variable-Size Input Arrays" -- https://www.microsoft.com/en-us/research/publication/elastic...
[3] Gordon & Peyton-Jones 2021, "LAMBDA: The Ultimate Spreadsheet Function" -- https://www.microsoft.com/en-us/research/blog/lambda-the-ult...