Thanks, but I haven't really done much in jq before. Doing the above script involved looking a lot through the manual and experimenting. This exercise was a learning experience for me. I was only able to do this because the manpage is well written.
I was expecting some mention of jq for this post, and you did not disappoint. Thank you for the script -- it works great and I'm adding it to my collection.
That's insane. Here I was thinking about how much more challenging it would be to parse and reconstruct the object from jq, and you got the idea to take advantage of the syntax similarity to parse it as jq code itself. Nice. And so, that means the inverse of the jq code I posted would simply be:
Parsing is awkward in jq, but setpath(PATHS; VALUE) will create necessary structure. PATHS uses the array form, like ["movie", "cast", 5] not .movie.cast[5]. Since 1.5, jq has PCRE regex, so could remove ], and separate by . and [.
This would be more suitable to large json files if used with the `--stream` flag. Here's my take on it:
jq -c --stream '
. as $in
| select(length == 2)
| (
$in[0] | map(
if type == "number"
then "[" + tostring + "]"
else "." + .
end
) | add
) + " = " + ($in[1] | tostring)'
Using `--stream` allows jq to start before parsing the entire json file. In my experience, a 700mb json file can take up 5gb of ram in either jq or python -m json.
You're right about `--stream`, but you didn't need the variable assignment. Also, `-c`, besides the fact that it's not available in jq-1.5 which some people are using, is very pointless in this situation, since we're looking to output text, and `-c` is for outputting objects/arrays in a compact format. The fact that `-r` wasn't used causes jq to output the text encoded as json strings. So, instead of outputting:
".movie.name = Interstellar"
".movie.year = 2014"
".movie.is_released = true"
".movie.else = Christopher Nolan"
".movie.cast[0] = Matthew McConaughey"
".movie.cast[1] = Anne Hathaway"
".movie.cast[2] = Jessica Chastain"
".movie.cast[3] = Bill Irwin"
".movie.cast[4] = Ellen \\\\ Burstyn"
".movie.cast[5] = Michael Caine"
Another point is how the strings at the right of the `=` are displayed. They should be quoted. The reason why they're not is because you piped the second element to `tostring` instead of `@json`.
A better version of your suggestion would've been:
jq -r --stream '
select(length > 1)
| (
.[0] | map(
if type == "number"
then "[" + tostring + "]"
else "." + .
end
) | add
) + " = " + (.[1] | @json)
'
The use of `length > 1` instead of `length == 2` is a minor point, but if a future version jq decides to sometimes put 3 elements in these arrays, your filter would ignore those when we're likely to also want those. `length > 1` ensures what we need, that there are at least the elements that we're going to be using, while `length == 2` might filter some of those out, even if it's not right now.
Your use of `add` is neat, though. I wouldn't have thought of that.
You whitelist against what the syntax allows for identifiers and then you blacklist reserved keywords. Writing it this way makes it easier to verify for correctness when comparing with the ECMAScript Specs. This is still a non-exhaustive blacklist and the whitelist regex lacks allowed unicode characters.
Hmm... I just downgraded to 1.5 and it seems to work. jq just has 2 numbers in its versioning[1]. The other number must be from your distribution's package building. Maybe the issue is with something your distribution did while building the package (like adding a patch)? It might also be that the syntax error is not with the script, but with the JSON you input. Sorry, without being able to reproduce the error, I can't help more.
Like another thread mentioned, shebang (#!) parsing is non-standard. In macOS, I think what you tried would work like you'd expect, but it'd work differently on linux. The reason is that in linux, after parsing the path to the executable and a space, everything else is taken as a single argument. So if you were in bash, what you did would be the equivalent of doing:
jq "--stream -rf" path/to/script
and jq doesn't know of any one option called "--stream -rf".
I haven't seen the discussions around these design decisions in the different OSes, but I imagine the crux of the matter is that you have to pick somewhere to stop, and where you chose to stop is largely arbitrary.
I mean, you can have the OS interpret shebangs with multiple arguments, but then you'll want to be able to put spaces in these arguments, so you'll want quoting, and then you'll want to put special characters like newlines inside, so you'll want escaping, etc.
The OS can implement all these things in execve()'s logic, but it might also be preferable to keep the logic simple in the interest of avoiding security-harming bugs. You know, less code, less bugs, less vulnerabilities.
If --stream had a single letter option equivalent, you could stick it together with the other ones. However, since it doesn't, your only option to make a portable script is to use a shell shebang like #!/bin/bash, and then do:
exec jq --stream -rf ...
You might feel that this single argument restriction sucks and is definitely inferior to any implementation of multiple argument shebangs. I don't know if macOS shebangs support quoting, but if they don't and simply split on spaces, then I can tell you they can't do hacky stuff like writing code in a shebang like this:
macOS appears to support multiple args just fine. Which is why it annoys me that Shellcheck bitches about using more than one arg even though I'm writing a script for macOS specifically.
ShellCheck is right insofar as compatibility is concerned. You can only rely on the shebang supporting one argument. I'd personally just ignore that warning if I were writing for MacOS specifically, but you can configure ShellCheck to ignore certain errors that you don't care about[1].
Yeah, but it's simpler just to move the additional args to `set` calls than to remember the syntax for disabling the directive. It's just irritating, and especially so because, for some reason, in VSCode it ends up highlighting literally the entire file as an error instead of just the shebang.
Have you seen gron[0]? It's similar: flattens JSON documents to make them easily greppable. But it also can revert (ie, ungron) so you can pipe json to gron, grep -v to remove some data, then ungron to get it back to json.
That's actually a nice lifehack. Much simplier than jq. Unfortunately, would be harder to make all kinds of logical conditions for which jq allows (even if not that intuitively).
It still feels like there must be something in between, some way to make queries with json more naturally, than with jq, yet with enough power.
jq is certainly a unique language, which makes it unfamiliar to work with. Intuitiveness and natural feeling comes when one has gotten familiar with it after a bit of practice and reading the manual, though. It's a very well thought-out language. A very nice design.
It might help to recognize how it's influenced by shell languages and XPath, if you're familiar with those.
Well, no arguing there, it is indeed. And I use it from time to time. However, it's not like I need a tool like that every day, and if I'm no using it for a week I usually need to "learn" it all over again.
jq seems very powerful. I don't deal with json all that often and my most common use case (by far) is `jq '.' -C` and it took a few tries for me to remember that syntax.
The idea of flattening, grepping, then reverting sounds very appealing and sounds like a better fit for me.
It does look like neither are needed if you pipe a file in jq, but `jq . file.json` requires the `.` and if you're pipeing into a pager, like less, you need both `.` and `-C` to get colored output (that was the case with the alias I had pulled up). I am using 1.5 and haven't looked to see if 1.6 changes this.
`-C` would be required when piping because most of the time (with the exception of piping into less) when stdout is not a terminal, it doesn't make sense to include terminal color escape sequences. You'd end up with those codes in your files, and grep would be looking at them for matches, for example.
`.` would be required when passing the file as an argument instead of stdin, because jq interprets the first argument as jq-code. If you don't include `.` it would interpret the filename as jq-code.
`.` is still needed if I'm pipeing in json--but only when I'm piping out. Otherwise help goes to stderr and nothing goes to stdout.
I do honestly think jq is a cool and powerful tool. I also appreciate little things like auto-color when appropriate--git also does this. Git also uses your pager, which might trivialize my personal use case.
There are cases when you have some complicated json and just want to search for stuff. Then you use grep + gron.
There are cases when you want a complete json processing tool. Then you use jq.
You can probably simulate each approach with the other approach, but the code needed to this is just too tedious to write. So you use whatever tool fits your use case.
I find it useful when I don't know what the json schema is.
Then you can just do a quick gron + grep and find where the interesting parts of a large json document are.
Maybe GP meant that jq can do selection as well, i.e. that grepping is redundant after jq. But jq is much more complicated to learn and grep works on all inputs (not just json), so it makes a lot more sense to learn and use grep properly.
There is actually a standard around writing paths into JSON objects: JSON Pointer, https://tools.ietf.org/html/rfc6901. It’s straightforward, and avoids ambiguity between separator and key name by simple replacement, e.g. `/foo/bar~0baz~1quux` looks up a key named "foo", then a key named "bar~baz/quux" inside it. It’s not particularly widely used, but I’ve come across it in a few places over the years (it’s not a common thing to need to do), and probably most recently JMAP uses it for backreferences.
(I haven’t run it, but a skim of the code suggests that this tool will turn `{"foo.bar": "baz", "foo": {"bar": "baz"}}` into `["foo.bar"] = "baz"` and `.foo.bar = "baz"`, resolving the separator ambiguity in a pretty JavaScripty way.)
When XML came out into popularity I think the first thing I wrote was a small Python program to flatten/unflatten XML into per-line entries quite similar to the example output in the article.
The text streams that are processed line-by-line by dozens or hundreds of line-based tools are immensely powerful and universal. It's all Unix heritage and often overlooked by fancy modern designs that more often follow a fashion rather than root themselves in substance.
Surely text streams have their share of limitations like everything else but in practise you can retrofit nearly anything into line-based text streams and get an immediate productivity multiplier by being able to apply a whole array of established tools to process that data. Proof of that power is that it has been worthwhile to write converters to and from text and other formats. Not only you can find translators to turn various hierarchical or object-oriented formats into text but you can even convert a PNG into text and back (with SNG).
Text streams are like roads with lanes. They're ages old, they're pretty good at separating and guiding traffic, and they're somehow suboptimal in several senses yet rarely can anyone point out a single, clear practical improvement on laned roads, not to mention a system for containing traffic flows that is superior to them.
This looks cool! I'm curious: did you consider using FUSE for this, instead of taking a shell? Using FUSE might make installation harder for less technical users, but the upside is that they could use their choice of file manager (shell commands, curses-based or GUI).
As you've already implemented most relevant commands (cd, ls, cat), it would probably be easy to make a FUSE version using fs-fuse / fuse-bindings
What an interesting idea! On my last contract, I had to deal with 100mb+ JSON responses from a barely-documented API. This would have come in handy when I was figuring it out.
I've used JSONExplorer for this purpose, but it is web based and doesn't handle files this large.
Extending the filesystem metaphor to JSON data and re-using the same commands strikes me as a great idea.
Did another project inspire you, or did you come up with the concept yourself?
this is something between a joke and a thought experiment.
function cason(x){
switch(x[0]){
case "movie": switch(x[1]) {
case "name" : return "Interstellar";
case "year" : return 2014;
case "is_released": return true;
case "director" : return "Christopher Nolan";
case "cast": switch(x[2]){
case 0: return "Matthew McConaughey";
case 1: return "Anne Hathaway";
case 2: return "Jessica Chastain";
case 3: return "Bill Irwin";
case 4: return "Ellen Burstyn";
case 5: return "Michael Caine";
}
}
}
}
I don't see what jsonlines has over yaml; in fact the jsonlines examples presented there are almost trivially converted to yaml.
e.g. the first example is valid yaml if you add a '- ' to each line.
And JSON is (almost) a perfect subset of yaml.
I've been using csv lately. It's reputation is overstated.
What I like is that it's far more compact than yaml or json and trivially pulled into sqlite for ad-hoc queries.
One thing I like about JSON Lines is robustness to bad data - if an individual line is corrupted, you can discard it / print a warning and move on, and the parser can start again at the next newline. This makes it useful for log messages / metrics, because if something crashes while emitting a log line, you can recover. If something crashes while emitting an item in a YAML list, you might corrupt the entire rest of the document.
Another is that it makes streaming processing a little easier. Once you have a line, you know you can attempt to process it, and you can shard processing on newlines without a full YAML processor. Tools that work on newlines or tools that can just split on lines can handle the first level of JSON Lines output.
Oh no. This looks like the horrible config format a colleague of mine invented at Yahoo when making an absolutely horrific config system. This brings back bad memories.
This may look cute but it is horrific when dealing with large configs and you have to reconstruct all the structure in your head.
Also, when you have a format that nests using brackets, braces and parenthesis, you can get help from the editor. This format does not give you that.
I'm not a huge fan of JSON (and the above mentioned format was invented because none of us were fans of XML at the time), but it turns out that both XML and JSON are actually easier to work with in practice than this format. Not least because there is ample tooling for JSON (and XML).
The lesson I learnt: I may hate XML (or in this case JSON), but finding an alternative that is better is not easy.
I think the idea is not that this is for storage or editing, but just for querying - you should keep your data in JSON, but if you want to find where something is, do `catj foo.json | grep something` and it'll tell you all the paths in the document where you can find the string "something". The intent is not to open catj format in a text editor, or to use the output of catj for anything other than ad-hoc purposes.
This is really similar to the format that AWS uses to represent recursive structures (arrays, maps) as a single array of key-value pairs for their APIs. Your catj could potentially be used to create that if ever working with the raw API directly instead of a SDK.
Indent each line with four spaces. Please please keep line length very short (under forty?) as HN's pre tags are absolutely not mobile friendly and can trash the entire page.
Edit: actually it seems to at least scroll within the comment div on overflow now, that's a huge improvement!
Hm, essentially JSON->Properties file. Cool. I wrote a little script the other day and decided Properties format was a pleasant way to define the config, maybe this could dovetail with similar future endeavors.
That's really smart, in fact this was - maybe until now - the only reason for me to resort to csv. (With pandas it's by the way really easy to flatten jsons into csv) JSON is such a nice format but the tools are really not there yet. I guess should should make it then possible to combine with line-based tools like head, tail, sort, uniq etc.
The output format like a slightly better INI (in that it would support deeper hierarchies), although arrays would be a pain to write an index at a time like that.
And for purely aesthetic, nitpicky reasons I think the leading period in each line is redundant.
Sweet! I find this tool extremely useful for parsing dmn files in javascript (via xml2js). But right now I am more interested in a similar feature for xml/html documents. Hoping if someone can point me to it. Thanks.
Author here. I use it all the time when working with jq [1] to find the path of the nodes that I want to select or filter. It also makes it much easier to understand the structure of deeply nested JSON files.
I could see it being useful when using grep or writing awk scripts. For the latter you usually have to carry around some state if you want to deduce the path to a desired leaf node.
I could also see it useful for just finding the index of a particular array entity.
In the very rare cases that you really need to serialize complicated data structures, something like s-expressions or (gasp) json, or even a memory dump is perfectly appropriate. However, most of the times your data structures should be lists of numbers or lists of strings. For those, you do not really need a "format", you can simply print and scan them from a text file.
If you come from xml hell, then yes, json may even seem a reasonable option. But if you are used to the terse beauty of printf and scanf, then a json file looks like bad sarcasm.
While I don’t disagree with you, if you look at the comment above this you’ll see @enriquto mentions just dumping lists to a text file. Pretty sure that’s what was implied by ‘unix’: use a simple text file that is dumped to the fs that can be easily parsed by standard tools on any Unix-like.
JSON is usually mocked by people who don't get to use Javascript objects natively. The benefits aren't obvious or tangible to them.
XML is native to nothing. It's annoying to use everywhere.
UNIX tools just don't compare. Sure, they stick around, because sometimes they're the easiest tool for some job, until they aren't and you regret starting out with them.
In a few years, Javascript and Javascript-compatible languages are likely to be bigger than ever before. A whole generation of developers has been trained mainly on them. Billions of dollars have been invested into the ecosystem. Whether we like it or not, that's the reality. For that reason alone, JSON will stick around.
EDIT 2: For those who want to save this script, you can put just the jq code in an executable file with the shebang: