I'm one of the hard-headed privacy freaks usually sharpening my pitchfork when there is an outrage against civil liberties. I'm that guy.
I once had a job that involved investigations of criminal activity (not law enforcement or government related, just a company protecting its own users and employees).
In this case, I had identified, with certainty, one individual that was engaging in significant fraud. He appeared to have several accounts, and it was appearing highly likely that he had a few accomplices.
During the investigation, I was fully willing to violate everyone's privacy to find everyone in the fraud network. This included data that was already submitted voluntarily, private communications, as well as embedding tracking objects and invisible flash objects to retrieve IP addresses of users surfing behind proxies (this used to be an effective way to unmask users). I didn't have a second thought about it. Why would I? I didn't care what the legitimate users were doing, wasn't going to stalk them, wasn't going to pay any attention to their personal affairs. But, to weed out this problem effectively, I needed to sweep everything. I'm trustworthy, just doing my job, and I certainly trust myself enough to disregard or ignore information that wasn't pertinent.
After being entrenched in the investigation, I had a fairly exhaustive list of the bad actors. Initially this was just basic hard data, (such as correlating IP addresses), but then there was kind of a "sixth sense" that I also started relying on, where I couldn't articulate the signal, but some behavioral cues just felt like they were related. You know, "gut instinct". So I ended up digging into those accounts, and confirmation bias took over. I did find many more bad actors, but I was thoroughly convinced that a few cases were also related, which ended up being suspended, and it turned out that they were actually unrelated and legitimate. That's when I started to reflect a bit.
I didn't go through with the most blatant of the proposed violations, although at the time I was willing to initially. I now realized how egregious that was, and noticed how easily I fell into that mindset. If asked, I think the words "If you've got nothing to hide, you've got nothing to fear" could have naturally rolled off my tongue (though, this certainly would have alerted me to the errors of my thought process).
So I concluded a few things:
- Most of the time, these blatant, sweeping violations, are most likely not malicious and probably do have good intentions. I very much understand what frame of mind most of those people are in. It's not an opaque three letter agency, it's made up of regular individuals with tunnel vision on their legitimate objectives (stopping crime).
- When you look at criminals day in and day out, and are on a mission, everybody starts to look like a criminal.
- The "working backwards" approach - finding signatures of bad activity, and applying it to other data, then "confirming" the new matches, is a well-understood statistical fallacy, aptly named, the prosecutor's fallacy[1]. If you spot it in court, your defense attorney can try and point it out to the jury - and good luck explaining it to your "peers" who probably play the same lotto numbers because theirs is "due eventually". But let's face it - your life is already ruined by then. You're on all the watch lists, your vehicles are bugged, you've got huge legal bills and no job, and maybe if you're extremely unlucky, you're even in Guantanamo. Everything prior had little or no judicial oversight, no way to defend yourself, and is from a system that is invariably full of investigators who are not self-aware enough to always catch themselves doing this, especially when the cost of missing an actual threat is extremely high.
And for bonus points:
The interface that a coworker created to do some of the data mining (let's call it the "lawful intercept interface") had an SQL injection bug in the logic that parsed login history. It wouldn't have been difficult to discover and exploit without even knowing this interface existed, due to the error a user would see on login if they had certain bad characters in the affected field. I found it roughly a year later and reported it to the CTO in a message from his own account, after using the bug to take his auth cookie out of the DB (we were friends, so I knew he would be a good sport).
tl;dr It's mostly good intentioned individuals with tunnel vision, who are very misguided, and who don't understand the side effects and costs of what they propose.
Please tell which US citizens are in Guantanamo because they were "extremely unlucky". You make it sound as if the FBI is picking up Soccer Moms for no reason and mysteriously spiriting them away to Guantanamo.
No, I don't make it sound like that at all. Hence, "extremely unlucky." And I didn't say US citizens went to Guantanamo.
If you aren't aware that there were many documented false positives who were sent to Guantanamo or other CIA detention facilities, you aren't paying attention, because there were some very high profile cases. Here is one example:
Khalid El-Masri is a German citizen who was mistakenly abducted by the Macedonian Police, and handed-over to the U.S. CIA, whose officers interrogated, sodomized and tortured him. While in CIA hands, he was flown to Afghanistan, where he was held in a black site, interrogated, beaten, strip-searched and subjected to inhuman and degrading treatment, tantamount to torture. After El-Masri held hunger strikes, and was detained for four months in the "Salt Pit", the CIA finally admitted his arrest and torture were a mistake and released him.
In April 2004, CIA Director George Tenet was told by his staff that El-Masri was being wrongfully detained. National Security Adviser Condoleezza Rice learned of the German citizen's detention in early May and ordered his release. Shortly before el-Masri was released, in May 2004 the US ambassador to Germany informed the government for the first time of his detention.
* According to a December 4, 2005, article in the Washington Post, CIA agents discussed whether they should remove El-Masri from Macedonia in an extraordinary rendition. The decision to do so was made by the head of the al Qaeda division of the CIA's Counter-Terrorism Center, on the basis of "a hunch" that El-Masri was involved in terrorism; his name was similar to Khalid al-Masri, strongly suspected as a terrorist.*
I can't think of a worse way to completely ruin an innocent man's life. He was basically a "Soccer Dad".
I once had a job that involved investigations of criminal activity (not law enforcement or government related, just a company protecting its own users and employees).
In this case, I had identified, with certainty, one individual that was engaging in significant fraud. He appeared to have several accounts, and it was appearing highly likely that he had a few accomplices.
During the investigation, I was fully willing to violate everyone's privacy to find everyone in the fraud network. This included data that was already submitted voluntarily, private communications, as well as embedding tracking objects and invisible flash objects to retrieve IP addresses of users surfing behind proxies (this used to be an effective way to unmask users). I didn't have a second thought about it. Why would I? I didn't care what the legitimate users were doing, wasn't going to stalk them, wasn't going to pay any attention to their personal affairs. But, to weed out this problem effectively, I needed to sweep everything. I'm trustworthy, just doing my job, and I certainly trust myself enough to disregard or ignore information that wasn't pertinent.
After being entrenched in the investigation, I had a fairly exhaustive list of the bad actors. Initially this was just basic hard data, (such as correlating IP addresses), but then there was kind of a "sixth sense" that I also started relying on, where I couldn't articulate the signal, but some behavioral cues just felt like they were related. You know, "gut instinct". So I ended up digging into those accounts, and confirmation bias took over. I did find many more bad actors, but I was thoroughly convinced that a few cases were also related, which ended up being suspended, and it turned out that they were actually unrelated and legitimate. That's when I started to reflect a bit.
I didn't go through with the most blatant of the proposed violations, although at the time I was willing to initially. I now realized how egregious that was, and noticed how easily I fell into that mindset. If asked, I think the words "If you've got nothing to hide, you've got nothing to fear" could have naturally rolled off my tongue (though, this certainly would have alerted me to the errors of my thought process).
So I concluded a few things:
- Most of the time, these blatant, sweeping violations, are most likely not malicious and probably do have good intentions. I very much understand what frame of mind most of those people are in. It's not an opaque three letter agency, it's made up of regular individuals with tunnel vision on their legitimate objectives (stopping crime).
- When you look at criminals day in and day out, and are on a mission, everybody starts to look like a criminal.
- The "working backwards" approach - finding signatures of bad activity, and applying it to other data, then "confirming" the new matches, is a well-understood statistical fallacy, aptly named, the prosecutor's fallacy[1]. If you spot it in court, your defense attorney can try and point it out to the jury - and good luck explaining it to your "peers" who probably play the same lotto numbers because theirs is "due eventually". But let's face it - your life is already ruined by then. You're on all the watch lists, your vehicles are bugged, you've got huge legal bills and no job, and maybe if you're extremely unlucky, you're even in Guantanamo. Everything prior had little or no judicial oversight, no way to defend yourself, and is from a system that is invariably full of investigators who are not self-aware enough to always catch themselves doing this, especially when the cost of missing an actual threat is extremely high.
And for bonus points:
The interface that a coworker created to do some of the data mining (let's call it the "lawful intercept interface") had an SQL injection bug in the logic that parsed login history. It wouldn't have been difficult to discover and exploit without even knowing this interface existed, due to the error a user would see on login if they had certain bad characters in the affected field. I found it roughly a year later and reported it to the CTO in a message from his own account, after using the bug to take his auth cookie out of the DB (we were friends, so I knew he would be a good sport).
tl;dr It's mostly good intentioned individuals with tunnel vision, who are very misguided, and who don't understand the side effects and costs of what they propose.
[1] http://en.wikipedia.org/wiki/Prosecutor%27s_fallacy