SQL was invented in the 70s for business analysts to write reports without having to go bug the programmers.
So what? Unix was invented in the 70s because Ken Thompson wanted to play Space Travel. [1]
Queries are not composable.
Per your example, you want to throw out SQL because using AND and OR doesn't feel right. I'd love to see you compose a 10-line SQL query. Also, views ARE query composition.
PL/SQL is primitive by today's standards, yet it's the only game in town for certain classes of features, like stored procedures and triggers.
Untrue. Oracle, the creators of PL/SQL, now let you write stored procedures in any JVM or .NET language (so Jython, Clojure, etc.). MS SQL supports .NET stored procedures. PostgreSQL supports Tcl, Perl and Python. Also, your alternative suggestion of a simple and stupid data store DOES involve fetching all the data from the database into RAM and operating on it in your application. You can already do that with SQL.
It's not obvious which queries are slow.
Untrue. MySQL can log slow queries and help you analyze them in real time using tools like mytop. I assume similar tools are available for all databases. When you write code it's not always obvious if it will be slow, either.
No back door for map, filter, reduce
Isn't the WHERE clause filter? Isn't GROUP BY very similar to reduce? SELECT UPPER(firstname) is map? With the ability to define custom functions in different languages that's a fairly powerful tool. Also, comparisons between MapReduce and SQL are misguided, because MapReduce does not operate on tabular data.
Finally, your solution is based entirely on the idea that people will prefer Lisp to SQL.
SQL is not perfect for everything, but if you want to suggest a better alternative, start with a solid argument.
How about an RDBMS that exposes a relational algebra api and then people can experiment with developing alternative languages/interfaces instead of having a single designed by committee string-eval based atrocity.
What percent of the top N (your pick of N and your choice of big-ness metric) websites do you suppose are using predominantly string-eval'd SQL? I don't think it's very many; I'd wager that most big shops are quite happily using SQL, using stored procedures, views, and parameterized queries far more often than using "jam SQL into an nvarchar and execute the resulting string".
SQL may have its warts, but in the world of "I gotta get things done and not sweat data persistence", I think it's pretty wildly, successfully, and fairly universally used, almost like the cockroach of the data center.
When I look out and think about what's going to kill our ability to take orders, scale to new levels of growth, or to have development be efficient at developing new features, SQL the language doesn't make my top 20, and probably doesn't even rise to the level of "all other receiving votes". MS-SQL the database engine probably does make my top 10, but that's only because we're continually chopping away at the items on that list. It does take us a while to train brand-new devs on how to write effective SQL, but it takes us time to train them on how to write C#, how to write good test cases, how to structure code for testability and write unit tests, etc. IOW, it's just one of a laundry list of needs, and a proposed alternative interface would also be something on that list to teach them.
I should have been more clear in the article. There are two different ways SQL is not composable.
1) predicates can be composed, but only by resorting to string manipulation (i.e. str-join([a,b,c], " AND "), which sucks.
2) Views and subselects CAN compose, but only a few levels. Every DB I've ever seen has huge performance problems with just a few levels of nesting. Imagine your programming language only allowing 3-4 stack frames before it just dies. That's not acceptable in your programming language, so why is it acceptable in your database?
It's not obvious which queries are slow
Any decently experienced programmer can determine the correct big-O complexity just by reading the code. With databases, you have to read the query, and understand what optimizations the current version of your DB is capable of making.
No back door for map, filter, reduce
The reason I want that feature is for the case where the DB pukes on your query. What happens when your database is not capable of generating an efficient query for the data you want? Pulling everything into RAM isn't an acceptable solution either. When the DB pukes, you should be able to say "fine, let me give you the exact plan". Then I expect the DB to go off and execute that plan, and give me a stream of data using limit, offset, etc.
The DB allows you to define custom predicates. It doesn't solve the problems of needing to do string manipulation to piece together a query, and it doesn't solve the performance problems of nesting views and subselects.
I never compared SQL to MapReduce. MapReduce is a project created by Google. map, reduce are functions for working on sequences of data created decades before Google was founded.
Off topic, I wonder if it's a subjective bias among hardware producers against large amounts of RAM. Cost-wise, I don't think there is a problem to have multi-tera servers or desktops with tens or hundreds of gigs. And yet, you can't buy them. Even though as a software model it makes a lot more sense to keep all the data in ram and only ship the changes to an external memory.
I'm a bit annoyed they don't offer prices up-front, but a very nice product. I wonder if you can use a violin device as primary memory or you can only mount it as a fs.
I think they are mostly used as external storage, sometimes backed by spindles as a very fast SAN.
If you need huge amounts of local RAM then I think you're in the realm of the IBM zSeries and other mainframe devices. Nice stuff indeed, when you have the spare change...
Yeah, I didn't mean to really enforce '>' so much as to discourage the indenting (which is causing annoying horizontal scrolling on my side of things).
Finally, your solution is based entirely on the idea that people will prefer Lisp to SQL.
SQL is not perfect for everything, but if you want to suggest a better alternative, start with a solid argument.
[1] http://en.wikipedia.org/wiki/Unix#History