I'm glad you're contributing to genomics technology, but I feel compelled to point out that zlib is a terrible algorithm for compressing BAM files. The reason is that over 50% of the compressed BAM file space is spent on encoding the quality scores at full resolution, which is just wasted effort. Quality scores are very hard to compress- they are very close to completely random data- and the best compressed use complex probabilistic models to switch the encoding technology depending on what category quality scores fall into.
I'm all for a replacement for BAM file format and the quality scores. Ideally something that supports delta based encoding to a reference genome similar to CRAM and something that more compactly represents quality of bases sequenced.
In the meantime as a user of BAM I'm very very very grateful for faster zlib as it makes life a lot better with the current large installed base of programs that use BAM and data sets in BAM. These sort of improvements really do make a meaningful difference in the same way that switching to SSDs from magnetic disks didn't improve the implementation of my software but makes my computer a lot faster to boot up and compile things.
You know what, I was trying to find an example for Matt3o12_ where it was #included, but I can't even find the file at the tip of any of the branches. Can a better git archaeologist explain what happened to it? Did it get merged into another file?
A lot of serious security advisories affecting closed source programs often don't become exploits in the wild due to the nature of closed source and the impied secrecy involved.
For example, IE had a major, major issue regarding SChannel that more or less trivially gave attackers an ability to run arbitary code at either admin or SYSTEM level. It was scary. It was reported privately and then patched. Never in this process did anyone have the source code to analyze and publish an early PoC like they did shellshock and heartbleed. When the patch was released, it was a binary, so no one could just compare the old code to the new and figure out exactly what the problem was and launch an attack. Sure, they could analyze the binary, but that gives limited and often unusable results. Or at the very least puts up enough barriers to buy time for patch installs.
Its funny, years ago we used to worry about our Windows servers, now only worry about our linux servers. FOSS's transparency is ugly when it comes to exploits because they go from discovered to in the wild very, very quickly. Even when they don't, once the patch is released, the hackers have the exploit instantly, and that means if your organization can't patch for a couple hours, you're screwed. The recent Drupal exploit is a good example of this. It went from published to bots hacking Drupal installs within seven hours. Millions of sites were affected.
You're right, but there's no question that being somewhat vague with the patch details and producing only a changed binary means attackers will be slowed down a bit from producing a functional exploit. Even if you're only buying yourself maybe ~6-24 hours, that might mean N extra hours for millions of machines to patch.
This is the debate about Full/Responsible/Non-Disclosure. With Full Disclosure, users, admins and pirates alike get the same information at the same time, meaning that you, user might be able to protect yourself (add a line to your WAF filter, add a block on your FW, etc.).
On the other hand, I note that proprietary software is flawed with tons of 0 day (I'm thinking about Flash lately), whereas the self-proclaimed most security-oriented open-source projects only have a tiny number of unsafe code (I'm thinking about OpenBSD "Only 2 remote holes in the default install, in a heck of a long time!")
Except no one uses the default install and these types of claims just incentivize making the default as sparse as possible. Things change when you deploy your stack, use ssl, etc, etc.
You're actually making the argument that Windows is now secure and Linux isn't, because Linux gets attention around security events? I'm afraid that seems outlandish to me.
I think he is talking about smaller and less mature projects than Linux or BSD variants. Say you're a small developer who's spent years on a new platform that will power your social network with millions of people. You might want to keep it closed source for a few years and only release the source code to a select number of people until you've let security researchers take cracks a it. Once it's had a few years of battle testing, you can release it. As opposed to releasing the source code to your social network and making it 100x easier for anyone to come up with an exploit against your network.
In fact I'd say it might be better that a relatively new app (especially ANY APP that powers servers) to remain closed source until given the green light by security researchers. And even then...
Imagine if Facebook open sourced the code running their social network? I guess the question could equally be ... is there ever a social good from centralizing your social network and not letting it be distributed across all machines in the world?
I don't think Facebook's business would be at risk at all if they open sourced most of the backend code. As long they kept things like spam/malware detections closed. Facebook's main value proposition is the existing network. Another network could possibly overtake them some day, but they're definitely not going to be doing it with Facebook's source; not even a heavy fork of that source.
Right. But imagine if Zuck opensourced the exact facebook source from day 1 or day 100. It would have been hacked a long time ago to the point of collapse, and who would that have benefited?
The answer depends somewhat on the license. However, for the purpose of my answer I shall assume a product or service for which releasing the source code would destroy an ability to generate a return on investment. (Of course this isn't always true, however I suspect it's not always false, either).
If the only way to develop your product or service is to generate a return on investment, and society will benefit as a result of the provision of said product/service. I think in many cases it's safe to say that society has benefitted if you generate a return on your investment.
Having said this, in principle I would advocate for an enthusiastic open-sourcing strategy once the societal benefit has been realised. Practically, it's a bit more complicated than that. For example: are you able to continue to generate a greater societal benefit by maintaining a monopoly on your source code? (Tough to answer, I'm sure).
If your industry is run by unethical people who would use your code, if developed in the open, in a way that would put your firm out of business and create orders-of-magnitude-fewer jobs than your firm would IF it kept that code proprietary and used it as a way to leapfrog the status quo... then I think you have a very good argument for your company keeping its core code under wraps.
And even once you're at the scale where you can compete head-to-head with those people, you still might want to keep things proprietary so that you can encourage ethical and aligned behavior across the industry. Because the stakes are much higher than falling asleep happy because you maintained the "purity" of the open-source-software movement.
You'd still share your changes to zlib and other non-core components, though. Because of articles exactly like this. But none of the secret sauce.
As a side note, it is not at all my intent to imply that any of this refers to the finance industry. Not at all. Nope. (But if this piques your interest, send me a message at the email in my profile.)
Yes, if your product provides a social benefit but would not be able to exist if its source code were free. Google's ranking algorithms would seem like a good example of this.
If the algorithm's nooks and crannies were fully known, website owners could manipulate it to display their results at the top, which may not be in the web's best interest.
I'm not OP but my guess is that without the revenue generated by the proprietary algorithm, there wouldn't be enough money or incentive to keep improving it.
With an open source google algorithm there would be possibly 1000's of lesser google's (lesser as in inferior search) the lesser googles might even get eclipsed by Bing or some other proprietary search which has the resources to improve it's search.
I think it's a myth that you can flip a switch and open source your code. Technically, you can, but in many cases it would be no better than closed anyway. Proprietary code often has a lot of dependencies and it can be hard to understand without being devoted full-time. Just building it often requires a fairly complex procedure with its own dependencies (often on specific versions of libraries and compilers).
So throwing the code up on github isn't really "open sourcing" it.
The question then becomes: is it a net social benefit to spend the large effort to truly open source a particular code base? And the answer is that it depends.
Absolutely. I contributed heavily to the code that ran an anti-spam DNS blocklist (that is used by most of the world). Open sourcing that code would have given away how we populated that blacklist, which would have allowed spammers to bypass it completely. After 6 years the technology and concept is still working strong.
I've never seen an open source AAA game, perhaps it's impossible go get enough money to pay the developers and designers of a AAA game with an open source project.
The social benefit is that some members of the society can get fun from a AAA game if they want.
There's a few.
ID software[0] open source their games, but require the assets to actually build and run the game (which you obtain by buying the game).
Epic [1] have open sourced Unreal Tournament (note I'm not sure on the licensing, just that the code is available).
Natural Selection [2] source code was released
Star wars Jedi Knight [3] had the source released.
Note that all the AAA games that I'm aware of that have been open sourced are older games, I don't know of any in development AAA games that are open source.
ID, at least while Carmack was there, was very very good about releasing their last-gen engine tech once all licensees had published their games. A whole lot of people learned about how to do game development from those things.
Unknown Worlds only released the source code for the NS1 mod for Half-Life--and as Valve has yet to release the Gold Src source code, that's of questionable utility.
And second, I think (but cannot prove) that people need time to adapt to technological progress, which is also driven by software. For example, we cannot figure out how to integrate Google Glass into our social norms over night. And we needed time to respond to the Snowden revelations. Who knows how we will handle drones and the IoT? Accelerating progress has upsides and downsides.
(I feel this is more about hardware than software, though. More efficient Big Data might be scary?)
If there is a private benefit that can be realized then it will incentivize people to write more code, making more tools and solutions available to everyone, even if for a fee.
I've heard server licensing suggested as a reason the web displaced Gopher.
I remember being amazed by AOLserver which I was surprised to discover is now open-source; possibly by that point it was too late though? My impression was that everyone used Apache because it was FOSS, but that could be wrong.
You could argue it's beneficial to open source and make it public since it would raise awareness and probably get the root problem fixed rather quickly, as opposed to keeping it to yourself.
Manage the risk. Lots of people knowing = many new threats. keep that stuff a secret or hard to access then only a certain class of threat will present itself. still not great but better.
Things like backoffice code which are very specific to one business. Or something like the Javascript that ties together the widgets of a GUI. These types of things are basically only useful in exactly the context they're currently being used. When your potential user base are people who'd like to run an exact clone of your business, open sourcing is not a good idea (EDIT: and it's potentially a social benefit to maintain your business's existence and unique identity). I guess you could step back and question the utility of your business altogether, but that's a separate consideration.
The previous discussion of another zlib alternative last month mentioned unexpected issues incorporating them into other software (that reaches into internals), as well as the potential for licensing issues.
I've experienced something similar, although on a much smaller scale.
A few years back, I got an e-mail that one of my open source tools was being used as a monitoring solution on a hospital ship. I felt pretty damn proud too, not having imagined the impact of open sourcing a project could have.
I've /never/ seen -O3 used in the wild (nor -O0 actually). I've essentially only seen -O2 and -Os actually being used in-production. Perhaps that has just been luck of the draw…
> The files used for this kind of research reach hundreds of gigabytes and every time they are compressed and decompressed with our library many important seconds are saved, bringing the cure for cancer that much closer. At least that's what I am going to tell myself when I go to bed.
I realise it's not meant entirely seriously and I applaud any effort that helps speed up the exchange and storage of information and this type of research. But "bringing the cure for cancer that much closer" I find a bit of a stretch.
