There are a lot of different solutions knocking around for running queries on one or more CSV files. This is a web-based notebook that prioritizes loading large files quickly so you can get running queries on them as soon as possible. It supports most text files, Excel files, JSON. It can also display your results as graphs.
Visidata[0] is another great tool in this vein I often forget the name of when I really need it. Python, in the terminal, kind of like vim on CSV with graphing capabilities.
Second this recommendation for the terminal. For my CLI toolbox, VisiData is my favorite.
I find VisiData is great for quickly exploring and querying data from the CLI. It can handle many types of files (SQLite, CSV, TSV, Excel, JSON, YAML, etc). Visidata loads all the data into memory, and so is very responsive when exploring the data. It allows you to quickly do all sorts of of adhoc queries interactively, without having to write a valid SQL query.
I haven't used Q. When I first heard of it, I liked the idea that Q allowed you to run random queries on CSV and TSV files. However, it seemed like it would be slow if you wanted to do follow up queries, since it had to repopulate the in memory SQLite file for each query. Though it looks like the latest version has a way to cache the generated sqlite file. So that seems like it could help.
Also, if I have some CSV, TSV, JSONL data sqlite-utils is useful for converting them to SQLite, and then exploring with Visidata or SQL queries.
Not sure what's your definition of "vanilla JavaScript", but https://dirtylittlesql.com/separators.js is clearly produced by a bundler, bundling a bunch of npm packages like events, buffer, etc. I would be pretty alarmed if someone wrote a single 7659-line vanilla JavaScript file by hand in 2021.
Vanilla JS nowadays means not using a framework, in yesteryears it meant not using jQuery, in either case, it doesn't preclude from having dependencies.
to be fair, in case the dependencies are pegged to specific known-good versions and are bundled into one or a few files then two concerns with dependencies go away, namely, insanely deep and wide file system trees and the lingering danger of any one of hundreds of software titles getting malware-ized. One could even add that it's probably a good idea to prefer tried-and-tested existing software over writing everything from scratch.
Why does it matter that it's vanilla Javascript? I don't think this would actually fit the general definition of vanilla Javascript, but I'm more curious why it matters at all?
This is really wonderful! The discussion about lay people's knowledge of sql reminded me that the Pandas API is often useful for non-sql folk. Likewise there are some projects similar to dirtylittlesql to bring Python data manipulation to the browser.
I’m skeptical of the utility of building for non techie people. It takes literally tens of minutes to Learn SQL necessary to a single csv file; and the overlap of people needing to query csv files and capable of learning basic select syntax is probably nearly perfectly overlapping.
I don’t knock the research at all, it’s very cool. But it has to be basically perfect for someone even slightly experienced to consider using, otherwise it will cause more frustration then time saving.
> It takes literally tens of minutes to Learn SQL necessary to a single csv file
It might take you, as a programmer or technically literate person, tens of minutes to learn the SQL necessary to parse a csv file.
But I have met a whole raft of people that would firstly take much longer to learn enough SQL to get by, then would still need help when basic syntax error messages came up, and then would get frustrated before getting someone else to do basic analysis for them.
I have met those people to, but none of those people have ever had a csv file to run queries on.
Analysis has a lot of meanings. Getting basic facts (there are N rows that X) and selecting subsets of data are realistic analysis non technical people might do on csv files.
And I want to be clear that parsing a csv file with sqlite or MySQL is much more complex than learning “SELECT .. FROM .. WHERE” (with no joins!) when there is a tool like OPs that magically does the “parsing”.
In reality, a basic query might be something like "Show me how many units of product code 010201 were sold each month this year", which actually isn't that easy to write in SQL (but very easy to express in plain English).
> And I want to be clear that parsing a csv file with sqlite or MySQL is much more complex than learning “SELECT .. FROM .. WHERE” (with no joins!) when there is a tool like OPs that magically does the “parsing”.
Here's another one I'm working on [0]. Run it as a desktop Electron app or as a server/web app. There's also a serverless/in-memory web app demo running here [1].
