The JEDI RFP history can be found here[1], including Oracle's protest[2] circa Aug 2018--essentially 40 pages of legal whining about prejudice because they can't possibly deliver on a single award IDIQ but still want a "fair opportunity" to parasitically leech off of it...and the lawyer firm filing the protest wants a fat paycheck.
To get an idea of what companies comprise a fair chunk of the DoD general enterprise IT landscape, see the agreements here[3].
For once, the DoD looks to be on track towards infrastructure that has a good chance of not absolutely sucking hind tit right out the gate.
Seems like the good ol' revolving door trick. Guy works for Amazon in charge of getting companies to use Amazon products more. Then the love for the "American people" suddenly strikes him
> He wanted to use his skills not “to make a search engine more performant, or help a box of stuff get to a customer faster; but rather towards service of the American people,
Then, after the contract looked like would be going through, his love the the American people waned and he joined Amazon again
> At the end of October 2017, Mr. Ubhi recused himself from JEDI, saying Amazon and his restaurant start-up, Tablehero, “may soon engage in further partnership discussions.” Two weeks later, he resigned and then rejoined Amazon
It feels dirty saying it but maybe Oracle has a point here.
Though in general this is how the game works in Washington.
> The Pentagon released the JEDI request for proposals nine months after Mr. Ubhi recused himself.
> Amazon has countered that the Pentagon identified 72 people substantially involved in developing the contract and its requirements, and that Mr. Ubhi worked on JEDI for only seven weeks, in the early stages.
Which one would you rather believe:
Some random noob, one of 72, works for 7 weeks, and magically has such a major impact that it's still felt 9 months later?
Or
Amazon, which has been the leader in Cloud for about a decade, is just the better (safer) option?
> Some random noob, one of 72, works for 7 weeks, and magically has such a major impact that it's still felt 9 months later?
9 months is not a lot of time in government land. Before the request for proposal is sent out, often, a pick is already made informally. In fact the best time to lobby is exactly that time to ensure the bid is written to one company's specifications (even though it is illegal). And surprise, the company is the company this person used to work for, and also surprise, surprise advocating its use to other companies. After he quits, then promptly goes back to the same company.
This is not unusual and pretty sure Oracle would be doing the same thing, it just they missed the boat, so now are happy to "expose the corruption".
> After he quits, then promptly goes back to the same company.
In isolation, sure, this seems weird. But remember that around that time (Obama admin), a lot of techies were being invited to join the government in the "Digital Service". Many of them took leaves of absence (I know a couple) to work 6 months or a year in the government, fixing their systems, and when done, rejoined their previous companies. So, in context, it's not that unusual.
It's probably less true now with Microsoft's improvement and growth, but early in this process there was only one company reliably capable of supporting a cloud contract this big and security critical. The idea of trusting Oracle to run the DoD cloud is terrifying.
Employees that are okay with indirectly working for the US government? They aren't even competing for JEDI due to employee backlash.
But they are also lacking gov certifications and have a weaker history with government contracts than Amazon or Microsoft, which is important because government procurement processes can be pretty arcane.
Oracle has burned a lot of bridges in DoD- they have their enterprise DoD agreements for DB and milk those for far more than they're worth. Lots of individual programs would love to get off Oracle, but find themselves tied because in theory it's already paid for.
Definitely. There was an article here not too long ago about how even at an Oracle conference someone casually polled the attendants and found most would be happy to use something else and were exploring PostgreSQL or other options.
I've heard requests for proposals (RFPs) in Washington will often be specified so that only a single vendor could meet the requirements. Is that what happened here, where the Pentagon's Joint Enterprise Defense Infrastructure (JEDI) RFP depended on implementation details of AWS?
And to a larger point, can anyone elaborate on whether what I've heard is true? Has litigation or sunlight through advocacy against the government worked in those cases?
The trick is to find out about the RFP and then write your proposal so that if you win the proposal when they put it out for bid, you're the only person who can win.
This happens any time any government entity goes through the RFP>RFQ> process; local schools, city government, municipal water departments, state agencies, the FBI, etc
I onced snipped an almost million dollar deal at the department of justice. HP tape guys thought they had the deal zipped up as the spec called for only equipment they could provide.
Well they forgot[1] that they oem'ed all their tape hardware and software to the company I worked for and we had no non compete clause in our contract with them. I found the deal on FedBid, under bid them by like 70K with their own hardware that matched the spec 100%, won the deal and took the rest of the month off.
Dont even get me started on the dirty tricks that can be done around deal registration. Enterprise IT sales more shady than a used car lot.
[1] So they didnt forget as much as the fact that HP Gov sales is an entirely different division than the OEM team with essentially no communication between the two.
I learned about this working on election integrity.
After spending years learning the laws, rules, regulations, principles, policy, tech stack... it really just came down to who's getting paid.
I don't know how to mitigate this. More eyeballs?
Regulation isn't sufficient and susceptible to capture.
Sunshine laws aren't sufficient. FOIA is a step forward, but you're always two steps behind (don't even know what questions to ask).
Whistleblowers are routinely stomped, so less likely to speak up over time.
Few can afford to do investigative journalism any more.
So I was big fan of Obama's data.gov initiative. Publish all (non-sensitive) data by default. Calendars (hearings), procurement process, contracts, appropriations.
> I've heard requests for proposals (RFPs) in Washington will often be specified so that only a single vendor could meet the requirements.
The usual term for this is "single-sourcing", and it applies to much more than vendor selection in RFPs. You can single-source materials, vendors, locations, or anything else, and it's routinely done in legislation.
Single-sourcing is a procurement strategy, not an outcome of RFP manipulation. When a purchase is put forth, the end-user[1] writes the specifications, and procurement staff[2] determines the procurement strategy. If the end-user can convince procurement staff that only a single provider can meet their needs, then they move forward with a single-source procurement.
When a specification is written specifically for a single vendor, it's not called single-source; although the end-user probably advocated for single-source during negotiations with the procurement department. Not surprisingly, this is a constant source of friction for procurement departments. End-users almost always want to buy their preferred solution, and especially in IT. There are often good reasons, but I digress.
In some contexts, the single-source procurement strategy is simply not available; usually due to procurement law. This is when you most commonly see manipulation of specification. However, competing vendors can challenge the procurement in court through a procedure usually called a "bid protest". That's not exactly what's happening here, but it's similar.
1: The person who will ultimately use/implement the purchased goods/services.
2: Usually a separate department responsible for ensuring that goods/services are "responsibly" procured.
A single-source approach requires a lengthy justification as to why they are the single option. Manipulation of the specifications so that you believe there is only one option does not require this justification because it's not explicit. In fact, it's possible that you are mistaken and another vendor could win the open (not single-source) contract.
Well, yes and no. Yes, when a single-source strategy is adopted, the spec is written for that supplier, but that does not mean that all cases where the spec is written with a specific supplier in mind are single-source bids. Sometimes it's an end-user trying to manipulate a bid so that only one supplier can answer. That's the distinction I'm trying to make.
In fact, many products have language that’s tailor-made to copy/paste into requirements documents to either/both: easily describe in a generic way and confine to the extent possible the selection to this product.
The photo of Bezos at the Pentagon confirms that nobody knows what to wear to work anymore. I see cargo pants, a cocktail dress, a hoodie, and a couple power suits.
I know AWS and GCP provide isolated hosting if required by companies or governments.
However it is truly amazing and frightening that a crappy node.js website could be neighbored with the Pentagon's logistics control software or something even more important.
Well, can you honestly say to a certain degree of certainty where your EC2 instance is at any given time?
It might be sharing a blade server hosting Netflix services.
Equally, governments are notoriously incompetent and it wouldn't surprise me that some ill configured IAM profile or security group causes mayhem.
While we've taken pains to isolate workloads from one another on EC2 and can show there's no known vulnerabilities that would allow information leakage between instances, we have customers who want that extra level of assurance that their EC2 instance is not sharing hardware with another customer. Dedicated instances fulfil this role.
https://aws.amazon.com/ec2/purchasing-options/dedicated-inst...
As for IAM profiles, security groups, and the other, they're all security tools that can be used, but they're only as good as the admin who wields them. But the same can be said for on-premise (non-cloud) workloads that are internet facing. The times I'd found SSH (or egads, telnet!) ports exposed to the internet on a boundary router were too many to mention. It's not a cloud-specific problem, but a IT training problem.
The good thing about cloud in this aspect is that when a customer is all in on say, AWS, you can enforce a high level of consistency in what those security controls should look like, which makes it easier to identify deviations. You can then pair that with features like AWS Config to automatically detect when you say, have an IAM profile that's been changed from the secure baseline.
So cloud can actually make your environment more secure, because you have security tools built into all of your services, they're consistent, and you benefit from all of the engineering and research that AWS puts into them, instead of having to design solutions for everything yourself or manage lots of differing products that all require their own training, runbooks, configuration, etc.
But regardless of the solution you choose, it will always come down to training your personnel to use those tools properly, because bad configurations can occur anywhere. Cloud can help minimize the occurrences of that, but only training will eliminate it.
The public's conception that there is a competitive bid process in the federal government is laughable. Sole source justification is the magic phrase and trillions of dollars of government contracts have been awarded under this auspice.
Back when I worked in the Beltway it was called FAR-12 evasion and you paid for quality counsel to advise you on all of the legal ways and means to have lunch with GSA employees for purposes of being awarded with sole source justification contracts under FAR-12.
Having bid and won contracts, it is oft-gamed. but plenty of room in most reqs for a small player to slip in and win projexts if you are otherwise qualified and more flexible than the average goverment contractor. Not a dauntingly high bar honestly.
To get an idea of what companies comprise a fair chunk of the DoD general enterprise IT landscape, see the agreements here[3].
For once, the DoD looks to be on track towards infrastructure that has a good chance of not absolutely sucking hind tit right out the gate.
[1] https://www.fbo.gov/index?s=opportunity&mode=form&id=7a17a56...
[2] https://www.fbo.gov/utils/view?id=6cd8017d52d2b832855c41fb74...
[3] http://www.esi.mil/