> Unless you can carefully code, it's not worth pretending a script to be idempotent.

Just because you can't reach perfection doesn't mean it's useless to strive towards it.

Idempotency is a very useful property that more shell script writers should be made aware of, even if they only learn basic things like "mkdir -p" or "rf -f SOMEFILE"

This whole thing is also loaded with TOCTOU problems. It's almost always best just to try to do what you want to do and check for failure.

I had to look up TOCTOU https://en.wikipedia.org/wiki/Time-of-check_to_time-of-use

Very interesting!

That link also describes the Catholic schoolboy algorithm: ask forgiveness not permission.

Orchestration/install/setup scripts are usually run while nothing else is running. Bash is not ideal for this. But it's not ideal for almost everything. Still making things idempotent-ish is better than bailing on every recoverable error and making the admin do the cleanup manually so the script can finally do what it supposed to do.

(Of course a few years ago immutable infrastructure was the rage, because it means if the script runs once, pack it up as a VM and you're done. And that's how the docker sausage images are made.)

Is there a tool that treats txt files as a Set?

Been looking for one a while a go and it feels like it’s something sound be built in.

You can use a combination of `comm` and `uniq` to implement set intersection and union. https://ss64.com/bash/comm.html

> Return the unique lines in the file words.txt that don't exist in countries.txt

    comm -23 <(sort words.txt | uniq) <(sort countries.txt | uniq)

>Return the lines that are in both words.txt and countries.txt:

    comm -12 <(sort words.txt | uniq) <(sort countries.txt | uniq)

I wrote these years ago. They're damn handy. It's true that they're not implemented in Bash (that would be nuts), but having them on hand lets me do much more on the command line than would otherwise be possible.

  ~/bin/union
  ===========
  #! /usr/bin/awk -f

  !acc[$0]++


  ~/bin/intersection
  ==================
  #! /usr/bin/awk -f

  !buf[$0]++ {acc[$0] += 1}

  ENDFILE {
    delete buf;
    files++
  }

  END {
    for (k in acc) if (acc[k] == files) print k
  }

  ~/bin/set-diff
  ==============
  #! /usr/bin/awk -f

  ! filenum { acc[$0] = 1    }
  filenum   { delete acc[$0] }

  ENDFILE { filenum++ }

  END {
    for (k in acc) print k
  }

In bash? How would you even implement a set in bash without just doing linear greps? Or did someone add sets to bash 20 years ago and I never got the memo?

You could use the 'look' command, which does a binary search.

It's basically meant to look up spellings in /usr/share/dict/words, but it can work on any file. It will match any line that your pattern is a prefix of, so you'd have to add logic to eliminate longer matches.

But if you had some huge file to search and you wanted to do it from a shell script, that would be one way. Caveat: although it's fairly standard, 'look' might not be installed on every system.

Also, you have to be sure to maintain your file in sorted order. So no adding things by appending to the end, and checking if something is in the set is much quicker at the expense of adding things being much slower.

When you’re paying six figures careful coding isn’t too much to ask.

I agree with you. Bash is a wonderful tool, but it's not very well designed for sophisticated control flow. Gotta know when to say when, and switch to something else.

Do-one-thing-and-only-one-thing-well bash scripts coupled with a Makefile gives one “sophisticated control flow”.