The DoJ filed a motion on Wednesday to delay the release of the Clinton Foundation emails for 27 months. The reason cited was that there were errors in how the search was conducted.
Given that, it sounds like there is also a fundamental problem in how information is stored and classified. The FOIA requests were for emails, so I would think that the retrieval would be simple. They must be vetting each email for sensitive (secret) information, otherwise I can't imagine how it would take 2.25 years to fulfill that request.
I can't tell if this is a stonewalling effort or a technology shortfall in the FOIA process.
I'm vetting FIOA-like request emails right now and yes each email is vetted for sensitive information. And not just secret info, there are a bunch of exemptions you have to watch out for. If anything the list of exemptions is too short. A lot of stuff that seems like it's private isn't exempted.
It's incredibly time consuming and very very expensive. I'm working one person's emails from a 4 month span and we have ~dozen people working on it for a week.
The more specific request the easier it is. The ones asking for "everything" are just hard as hell.
Cry me a river. It's hard only because there is too much utterly worthless, indeed counterproductive, secrecy. A more open government could not do some things. Good. It shouldn't.
The constitution of the United States + the Bill of Rights were all designed and argued about in secrecy. This was so the politicians in the room could actually be honest and negotiate rather than pander to their states short sighted interests.
Working for the Government isn't "just a job" though. The Government holds a huge amount of power over people - and not just the people who pass the laws, but every department that has to come up with its own policies and processes.
Everything the Government does affects somebody in a way that few businesses can. A bad process can literally result in death in some cases (although the third sector attempts to help when Government fails).
People already get into Kafkaesque situations on a regular basis due to either the department's employees not understanding its own policies, or not knowing the magic words to use. The only way to protect against that is to allow people to force the Government to give them information. Personally, I've used FOIA responses in the UK to great effect in forcing the NHS to follow their own procedures (not medical ones, admin ones).
Few business, but gradually more and more. The decisions that Google, Monsanto, MSNBC, Fox News, Dow, or even Walmart make are starting to have more impact than US government decisions.
And I absolutely agree with your comment, but per my comment below, I think 100% transparency carries its own risks in making government less effective.
If the UN and large bureaucratic organizations have taught us anything it's that position jockeying gets ruthless when promotions are a zero sum game because of no organizational growth.
I'm not talking about malfeasance and corruption. I'm talking about embarrassment and enablement of dubious practices. FOIA should rightly make that painful.
On one hand you have dubious practices. On the other hand you have political football / back-stabbing.
My point is that I believe the public interest is best served by a balance between "the public knows how this business was conducted" vs "any rival who wants my job can dig through everything I've done to find some dirt of me."
Politicians, functionaries, and executive staff are people too. And I think HN can be a bit schizophrenic decrying "people in government are so ineffectual and unwilling to take risks" while simultaneously designing a purity litmus test where only the most boring, unadventurous apparatchiks are able to keep their jobs.
> My point is that I believe the public interest is best served by a balance between "the public knows how this business was conducted" vs "any rival who wants my job can dig through everything I've done to find some dirt of me."
Was that ever a standard for exemption from FOIA? Much less the intent of anyone writing any part of FOIA? Stupidity should hurt.
Your original point was that "99.999% [of FOIA exemptions are] ass-coverage through secrecy" and that's a bad thing. I disagree for the reasons I specified.
If stupidity were cause for dismissal, we'd have a hard time staffing most of the jobs in the world.
So if someone asks for emails, no one should care if some of them are employees asking for medical leave? HIPAA just shouldn't apply to government employees? Emails containing complaints about harassment or assaults, the victims who complained should be just named to anyone who asks the government? Employees have to provide SSNs and banking info to HR, should that be publicized for scammers?
"The Government" is also a workplace, and deals with all sorts of personal information on a daily basis. Their privacy is hardly "worthless," their emails deserve at least a quick sanity check before being released potentially to the entire world.
> They must be vetting each email for sensitive (secret) information, otherwise I can't imagine how it would take 2.25 years to fulfill that request.
Yes, that's the gist of it. From the filing in question:
"Furthermore, in this case in particular, the substantive review process is challenging given the wide-ranging subject matters covered by the documents at issue in this case. Id. ¶ 25. The documents implicate subject areas governed by multiple different bureaus within the State Department, all of which must be consulted before materials may be released. Id. State must also consult with other agencies within the U.S. Government. Id. Moreover, because Plaintiffs’ requests cover communications with outside organizations, State must consult with such outside organizations if the organizations’ confidential business information is at issue."
Though there is certainly tech incompetence involved as well:
"Specifically, it found that one office had not conducted the search by keyword, but rather had only searched the ‘To’ and ‘From’ line of the latest e-mail in an e-mail communication, a method that would not capture responsive e-mails found only in older communications that were part of an e-mail chain."
"In order to do so, as explained in Mr. Stein’s Declaration, IPS must manually open and review every e-mail with an attachment amongst the 34,116 potentially responsive emails in order to determine whether any responsive attachments had inadvertently been omitted."
("Responsive" in this case means "sent or received by four current or former State employees – Michael Fuchs, Melanne Verveer, Cheryl Mills, and Huma Abedin – to or from individuals associated with the Clinton Foundation and Teneo Holdings".)
This is just anecdotal, but it speaks to the technology problem. I live in DC; a few weeks ago, I found myself in a queue with some rather loud young women who all work/worked in a FOIA department of some federal agency (they didn't say which). They spent maybe 15 minutes complaining about various coworkers and their general incompetency. I especially remember their remarks about one man who absolutely could not learn about the redaction feature in Adobe Acrobat. He always printed documents, manually redacted them with a marker, and scanned them again. Not out of any kind of malice--he just couldn't/wouldn't learn a better way.
This is an excellent example of fud created by security pundits wanting to say "actually, that's not secure". If you read that link carefully you will find that the problems discussed are not unique to digital redaction (size/length of the redaction, height of the redaction, etc). The accepted answer also clearly stated that if you use pixel-replacement (this is what DoD calls it) you're fine. Further, there have been plenty of times I've seen a redaction-by-marker break down when the underlying ink bleeds through in photocopy.
NSA published guidelines in 2005 for how to safely redact PDF (as well as word) documents. The corresponding physical redaction guidance begins by suggesting that the inspector physically remove the information to be redacted with an xacto knife. They redact and release thousands of documents, if nobody has reversed their redaction process yet then it's good enough.
Also note the answer that points out underhanded ways to leak data.
You then go ahead to recommend just the thing that gp was making fun of: printing and physically altering the document instead of using a software tool.
You should re-read my post. I said that there were standards for digital redaction which are considered sufficient and equivalent to physical redaction. Break. If you still want to manually redact... iow, manual redaction is a waste of time, but if you have time to waste here's an effective way to waste it.
May is a pretty clear clue that the security "advice" which follows is fud. Your house may flood even if it's on top of a high hill, buy my insurance product.
Seriously. Anyone who has ever done FOIA requests knows how laughably incorrect that is. If it's not an exemption and it's not classified, you can't just deny it.
Furthermore, if it is denied, you can appeal that decision. It's not going to go back to the same person either, it'll go a lot higher on the foodchain. Do you think a commander/director is going to put up with a FOIA officer who is constantly creating work for them by improperly denying requests? Here's what a response to a FOIA appeal looks like[1]. It's comprehensive and needs to give its reasoning.
The article says there is now a 25 year limit on redacting, if I read it right. Obviously the DoD isn't going to play, so how are they getting around it? Is there a generic "security can trump this law"?
"This category was added after witnesses testified that geological maps based on explorations by private oil companies were not covered by “trade secrets” provisions of present laws. Details of oil and gas findings must be filed with Federal agencies by companies which want to lease Government-owned land. Current regulations of the Bureau of Land Management prohibit disclosure of these details only if the disclosure “would be prejudicial to the Government” (43 CFR, pt. 2). Witnesses contended that disclosure of seismic reports and other exploratory findings of oil companies would give speculators an unfair advantage over the companies which spent millions of dollars in exploration."
So it's another bill that has no funding to implement it (likely because of "budget hawks" who take any chance they can get to tighten government spending... well, at least government spending not benefitting their district/state directly).
When will we get sane legislation back? Or are unfunded mandates the future of all such improvements?
That's not the way the appropriations process works. Congress does not set aside money for FOIA or for this bill. Generally appropriations are made to each agency, and each agency must carry out its operations in a manner consistent with all laws, including FOIA.
Yeah, those darn "budget hawks." They only let the feds borrow and spend $439,000,000,000 last year. That isn't even half a trillion! Just awful. If only we could chase off these budget hawks and borrow as many trillions as we need.
This is an improvement, but ultimately we will not have an open government until the following conditions are met:
1. Openness must be the default, assumed state. When our founding fathers addressed the problem of unfair imprisonment, they didn't do so by giving people tools to prove their innocence. Instead, they started from the assumption of innocence, putting the burden of proving guilt on the prosecution. An open government will not be achieved by giving people tools to open the government. Instead we should start from the assumption of openness, publishing everything by default, and place the burden of proving a need for secrecy on members of our government. One example of why this is necessary is that we don't even know what to request; FOIA powers are only useful for opening up known unknowns.
2. Openness must be checked and balanced. Who watches the watchmen? Right now, no one. The honor system doesn't work here: members of government often have strong incentives to hide information from the public, and these are often the instances where openness is most important. Yet members of government can deny FOIA requests for all sorts of reasons, and requesters have little recourse. Without an impartial third party to assess whether secrecy is needed, members of government can simply claim anything needs secrecy, and there is no one to investigate whether this is actually true. One example of why this is necessary is the content of the Snowden leaks: the NSA, operating in secrecy with no oversight, has denied requests for information which we now know only because Snowden blew the whistle.
3. Openness must be a technical requirement. In order for openness to be the default, we need to change the way the government stores and distributes information. Software projects recognize this need for their projects, which is why systems like Git store every action. The technical requirements as I see them are: a) Accessibility Often members of government obstruct FOIA requests by distributing large amounts of data in unusable formats such as PDF. Distribution in a format which can be searched and analyzed is a necessity. b) Reliability There have been a few high-profile cases recently where data has been "lost" in hard drive crashes, accidental deletions, etc. We have no way of knowing whether these data losses were the result of actual technical problems or foul play. Data should be stored and backed up sufficiently that technical failure can't be used as plausible deniability for foul play. c) Distribution A recent FOIA request for Clinton Foundation emails has recently been delayed by 27 months. Whether this is stonewalling or technological failure is unknown but the effect is the same. If we started from the assumption of openness, the emails should easy to retrieve, and in fact should be published to an RSS feed or similar on a tight deadline. These technologies already exist; we should be applying them to government.
4. Openness must be incentivized. Currently, all the incentives are for members of government to obstruct openness. Wrongdoers obviously have incentives to hide their wrongdoing, but the problem goes deeper. Whistleblowers are ostracized when they aren't outright jailed. And there are no incentives that counteract these. Members of government delay requests, deny requests without justification, provide data in unusable formats, cover up wrongdoing, or even in some cases outright counterattack the person asking for information, and these members of government are never punished with more than slap on the wrist. In fact, they are often rewarded. We need to provide strong incentives for openness. Whistleblowers should have strong protections. Data should be released on a tight deadline and in a usable format with fines and firings for failure--even if there are technical reasons, because people unable to meet the openness requirements of their jobs should not hold those jobs. People who cover up wrongdoing in government should be prosecuted along with the wrongdoers (by an impartial third party, since checks and balances are needed). Any set of laws intended to promote openness will be ineffective if it doesn't have teeth.
The largest issue with FOIA requests is that we don't know what to request, and literally anyone can just make up a reason to deny them.
Hell, even after stuff is released publicly, sometimes the government decides to label it secret.
[1] http://austingwalters.com/foia-requesting-100-universities/