There's a lot of truth in this post but this line of thinking also plays into our own prejudices. I do often look at third party code and think it's pretty crappy, but then I need to remind myself that my own code is far from perfect!
More importantly, though, we tend to underestimate both how long it will take to implement something ourselves and especially how much it costs to maintain over time. If you can use third party code from a popular and active project you get a huge amount of testing, bug fixing and other maintenance from the community over time. If you implement in house then you've just given yourself an extra burden for the life of your project.
I think it's also worth mentioning that choosing to use third party code is an opportunity for you to give back to the community.
Most of the PRs on the projects I follow come from people that are using the code in their own applications. They'll find a small problem and push a fix for everyone's benefit, instead of leaving it up to the core maintainers.
Absolutely, this is why popularity of the project matters, as it means it is more likely to be tested and actively maintained. And as a good citizen you should participate!
Also: if you find you need to write this non-core functionality in-house you could consider open sourcing it yourself. If you're right that this is a need not filled by existing libraries you might just find a community grows around your version to help you maintain it.
> If you implement in house then you've just given yourself an extra burden for the life of your project.
That is true, but maintaining third-party dependencies is also a burden for the life of your project, especially if your project outlives its dependencies. It may be well worth the burden in most cases, but it is not free.
That depends on how much time it takes to develop and maintain the interface to the external dependency. Also, there's little risk if it's a well supported standard open source library with many users, but if it's a proprietary library or a tiny project with a small user base, what can be done to mitigate the risk of it going in a direction that is not compatible with your own use case?
In the end, you may end up having to work with and maintain your own fork of the code anyway, and I doubt that it would be easier to do so than something you wrote that was tailored to your own needs.
> If you implement in house then you've just given yourself an extra burden for the life of your project.
If you are doing this work for an organisation, you can also argue that it is professionally irresponsible. You are certainly unlikely to be there forever to pick up the tab for that. Coding in house is a toxic line of thought endemic in mid-level developers who haven't yet had exposure to full lifecycle costs of custom code.
(And ironically the problem of so many crappy libraries is precisely due to people succumbing to NIH rather than uniting around a standard solution.)
From my experience so far, I'd rather inherit an in-house solution to something than a hacked-in didn't-exactly-meet-the-requirements-in-the-first-place third party library. I find the latter cases particularly painful to keep running well when the need to update the original project comes along.
If you're going to pick a third party library, try to make sure you can depend on it to be supported for as long as your project will be. Otherwise it's not much different than inheriting code that's not supported because the original author left the company.
The built-in-house solution tends to have less useless-to-me stuff bolted on in an attempt to be "versatile" but that ends up getting in the way down the line. (This is hardly universally true, though: the built-in-house swiss-army-knife attempt is probably the worst of all worlds.)
While I agree it'd be nice for their to be unification, rather than a fragmented set of varying libraries, the one size fits all solution tends to be a maintenance nightmare. I guess we just have wildly different experiences, but mine is such that adding a third party dependency is typically a major liability and often originates from less experienced developers.
Adding a dependency is a burden. You don't have direct control over the evolution of that code. You have to keep track of the changes. If you're lucky, the library will have a changelog, but that's sorely lacking in most projects. Blindly upgrading will bite you, so if you have a product that can't suffer downtime, you really need audit that code for every upgrade. Good luck if you've determined that the version you have is sufficient, so you've locked it down, but then a security issue is found . . . winding back patches like that can be an exercise in futility. Oh, and then there's the wonderful world of transitive dependencies.
NIH for everything isn't ideal, naturally. But pushing everything out to 3rd party libs has its own set of problems that in many cases are worse. In my experience, a more experienced developer knows when to make the trade-off and solve the problem in-house with a much smaller body of code that's far easier to maintain.
> a more experienced developer knows when to make the trade-off and solve the problem in-house with a much smaller body of code that's far easier to maintain
Indeed, but this is hopefully obvious to all of us. My point is that the cost of maintenance is vastly higher than is ever really understood. Most code is written in the context of delivering a project, and here the highest cost is considered to be getting functional code out of the door (even the first set of bugs is often overlooked).
I'd propose the counterintuitive and unpopular view that the cost of development is the tiniest fraction of the lifecycle cost of lines of code when you factor in things like bugs, keeping up with security, adapting to environment changes, modifications due to new requirements, and probably most importantly the multiplying effect of adding new moving parts which makes each piece of code in a solution more expensive to write and maintain than the last.
I'm perfectly aware that this will seem ridiculous and in some ways it is a little aggressive to make the point, but as evidence I would consider how many work years are spent on libraries which contain a tiny part of the functionality that is contained in useful software.
Therefore the trade-off, while being about the right things, is very rarely correctly sized.
I don't know. Based on my experience I firmly believe that between not-invented-here vs invented-here syndromes, many more projects fail because they try to do too much themselves. Especially in startups. For anything that is not the laser-focused core offering of your product, pick a good enough library and be done with it.
Invariably, most problems are such that you can get 70% of the implementation done (the fun part) with a reasonable amount of work, but the last 30% (all the little details that you didn't thought of) take +70% of the time. And you also need battle-test it. For example, React is interesting, because it is battle-tested by Facebook, not just because of the approach it took to the rendering.
Good candidates for underestimating the amount of work:
Anything with i18n and l10n (dates, money).
Anything with threads (futures and promises library).
Anything with small errors causing cumulatively bigger errors (scientific calculation and money).
Anything integrating with the real-world outside of your app (sending email)
I think the real problem is actually the sunk cost fallacy, where often by the time you discover the integration wasn't worth the effort put into it you feel that you should just move on and leave it be. But in practice, something that's a pain to integrate is usually also a pain to maintain that integration.
In the end you can't really know if something is going to save time unless you actually try it most of the time. But if you try it and it becomes clear it's going to be a drain it might be time to throw it away and do it better.
Of course, there are a few vultures out there that hang on the entire industry because the task they perform is so complex almost any attempt at reinventing their wheel will fail in some non-obvious way. See for example ICU for unicode text processing.
I've done this a few times in recent memory and on at least two of them it's bitten me already. In one instance I wanted to use an external service to handle email drip campaigns. It turns out that introducing an external service added a lot of synchronization problems I was not prepared for, and I'm still dealing with those today. In another instance, I wanted to use a workflow management system. The DSL for job specification ended up causing more headaches than it resolved and the tradeoff for the additional monitoring capability it was adding wasn't worth it. I try to think harder about the specific parts that I do want to write and which parts I will really gain from by outsourcing.
Email, deliverability in particular, is a rabbit hole.
Its easy enough to get it set up and "basically" working, but not. Basically, you're using in your favorite language's mail sending library to send an email out.
The next level is figuring out all the standards surrounding email that those third party systems know and implement for you. This will help deliverability, to a point.
Beyond that, once you hit enough volume, there's best practices should have always been doing but you learn from experience and need to abide by or else you're not getting anything through even though you do all the previously mentioned stuff.
If I were to develop an app with low amount of sent email, I would be okay with doing the above myself. But I'm also familiar with the process by now.
Beyond that, theres continuous monitoring of deliverability and reputation, tweaking your sending algorithms (you don't just send email as fast as you can generate them, right?), keeping up with how email providers are changing their rules over time, maintaining your relationship with people who work at major email providers, and more I probably never had to get into.
You can easily have a team of people dedicated to ensuring your email doesn't just end up in the spam folder, or being rejected outright.
Agreed. That's why it might make sense to fight NIH on the general problem of sending emails, but to give into NIH when deciding to use an "external service to handle email drip campaigns" vs. building it in-house. That type of problem is much more specific to your needs and attempting to outsource it could cost you more time and complexity than it saves.
Using an ESP for delivering email is a no brainer in most cases.
Email is a perfect example of a time where you should just leave the whole thing to someone else. It is such a headache to get right that the amount of hours spent trying to do so is just a waste. Better to focus on what your application actually does.
I've seen more junior developers fall prey to this because they were not comfortable writing the functionality. A good engineer balances the two needs and gets it done right. Sometimes that involves contributing to the third party libraries themselves.
I'm not sure the authors understands what the NIH syndrome (http://en.wikipedia.org/wiki/Not_invented_here) is about. It is about writing everything yourself. IMHO the following paragraphe is highly symptomatic for that:
A great mass of libraries, modules, code, and other
software products, available on the net are either
crappy in their own right, or simply not suitable to our
project. Forcing things to work together can be costlier
than simply writing the required code on our own.
In the context of software development, NIH syndrome doesn't mean that you use libraries not invented here but that you re-implement everything on your own.
I think the author understands the NIH syndrome just fine, and starts the whole post describing the IH syndrome as the extreme opposite of NIH pretty well, where one is ready bend over backwards to use a subset of a 3rd party library, even if it might be easier and quicker to implement the desired functionality yourself.
Are you afraid to write code? Does the thought
linger in your brain that somewhere out there
somebody has already done this? Do you find yourself
trapped in an analysis cycle where nothing is getting
done? Is your product mutating to accommodate third
party components? If yes, then perhaps you are
suffering from invented-here syndrome.
I can relate to this. I researched a bunch of Ruby gems for "feature flags" a while back and didn't find any that worked for our needs, so I wrote the code myself, and it was super easy. On the other hand I'll choose Devise over DIY authentication every time. . . . :-)
My general heuristic goes on the following questions:
-> Is this project going to outlast my time on it?
-> Is the problem I'm solving something that can be done in less than a couple pages of code with comments?
-> Is there a minimal open-source library out there that solves this problem for my language?
-> How many variants of this problem am I going to face?
-> Are other engineers than me going to be working on it concurrently?
-> If there is a library already out there, does it have good documentation?
Those usually guide me pretty well, and in the event where I end up writing my own code, I look to either contribute to an existing library (yay pull requests) or to publish my library for other folks to use.
One advantage of rolling your own is that you only need to consider your (your own, your employer's, ...) specific needs, and sometimes that added flexibility is well worth the effort.
Especially once those needs change, you can simply change your solution rather than beg some company or open source project to implement that change for you or try to get some patch accepted or maintaining it yourself. Sometimes that is a big advantage.
On the other hand, if your hand-rolled library/application/service/whatever is built by a single person and that person leaves the company, you might be royally screwed.
About a year ago I had the, um, honor of taking over maintenance of a bespoke application developed in-house (by a single guy) to make up for some shortcomings in our ERP system (which can, in theory, be customized, but 1. it has to be customized in Visual Basic Script, and 2. the vendor makes no guarantees whatsoever that your customizations will still work after the next upgrade).
It has mostly worked out OK, because the guy that left coded in a very readable style, but when I deployed an updated version of the application, the accounting department complained that things stopped working which used to work. But when I looked at the source code, they should not have worked at all, because the code was not there. So probably the version the accountants had been using before my update was not built from the version of the source code I was given (not to mention that the code was not maintained using any kind of versioning system).
TL;DR - in the case of small-ish applications (depending on how unusual/special your needs are and what amount of resources you can afford to invest) I believe there can be big advantages in rolling your own, but you need to be aware of the risks. I've seen a few cases where companies had kept using bespoke applications for ten years and longer, even though they were clearly broken / out of sync with the company's needs, because nobody was left to maintain them.
EDIT: Just to be clear, I am thinking more at the level of (small) applications than libraries. With libraries, I prefer to use existing solutions, for a lot of reasons, many of which other participants have already pointed out.
I've seen this happen so many times. "We can't use X because it doesn't come with library Y!" And then they point to their favorite alternative to X, and note that it does have library Y, and ignore just how utterly terrible that library is.
Sometimes it's better to rewrite a library yourself than use an existing one that gets in the way as often as it helps.
> Sometimes it's better to rewrite a library yourself than use an existing one that gets in the way as often as it helps.
Sometimes, but not most of the time. Most of the time your own library will be just as terrible as the one you were trying to replace. This is especially true when it is done by someone who is not good at designing APIs. In then end you then get something that is barely documented, slow and full of bugs. I would know, I have removed more than my fair share of crappy inhouse libraries from projects.
Or maybe different people have different strengths? Look, I like to tinker at all levels of the stack, but I know what it takes to, say, write a correct web server. I am not going to write my own unless I must (did this at least once).
Some people prefer to re-use existing code and are good at integrating existing pieces. Others are good at writing purpose built stuff from scratch quickly. And even that is spread across technologies. I can write an async task manage, but can't easily create a CSS grid system. Just don't get stuck in the "built here" or "not built here" camp and you'll be fine.
Additional bonus for rolling your own: not having to deal with restrictive, ambiguous or outright bizarre licensing terms and, in the case of SAAS, that the service or its best API will become unavailable with minimal warning.
I haven't yet worked in a shop where I thought they should have spent less time buying products, and more time inventing things. Obviously there have been some library choices that have been regrettable but there is no shop I've been to that has a general cultural problem of not building enough things. (Background enterprise)
But when I have seen someones make an error in judgement, I have noticed that they realize it much more quickly when integrating software then when building it themselves. I suspect because of the attachment that quickly grows for one's own code.
My company so prefers outside solutions that sometimes we engineers joke that the best way to get your code used is to open source it and let management think it's from outside the company!
Some well structured projects actually do allow you to use only part of the library - in JS-land you can:
require('library/just/this.js');
Of course, it takes effort to make that work, but I'd still try to do it, because if something is confusing me about almost any library then the answer is a quick Google search away. If I wrote it myself I'm stuck.
I briefly worked with a startup that has built their whole system using Python (though they use things like ElasticSearch, so they also run the JVM). The programmers at this startup are very intelligent, and very thoughtful. There are 4 main programmers. Their background is a mix: some of them were initially scientists, some were computer science engineers. They are all in their late 20s and early 30s. The startup began in 2011.
Because they are so thoughtful and talented, whenever they see a flaw in a 3rd party library, they give up on that library, and they try to write their own.
So for instance:
They tried using Chef for configuration, but they didn't like Ruby. Then they tried Ansible, but they found it immature. The tried Puppet, but didn't like that it had a central daemon. So they started writing their build/config system, which they released as open source. Their system has some unusual ideas, a heavy reliance on bash, and an emphasis on supporting a polyglot system. However, this took a lot of time that could have been directed at the specific needs of the startup.
They decided they would need a system of dependency injection. In Python it is common to configure some variables at the level of the module, but they felt this lead to tight coupling. They did not like any existing system for dependency injection for Python, so they wrote their own. They released this as open source. Their DI system borrows a lot of ideas from Angular. It is clever and they have long debates about how to make it better. But this also took time that could have gone to the specific needs of the startup.
They needed a framework that would allow them to serve a website out over Nginx. They reviewed all of the existing Python web/application servers. They found flaws in all of them: Zope, Pylons, Flask, etc. They decided to build their own. They based their work on Flask but they added a lot of their own unique ideas. They released it as open source. It has been a big effort and it is still missing certain features. For instance, they started off assuming that all of the pages of their site would require a login. When they eventually had to add in some pages that did not require a login, they had to re-work the way they were handling security and authorization. All of this has taken time away from the specific needs of the startup. And, of course, their software may have security flaws which they have not yet discovered. Although it is open source, I don't believe anyone else is using their software.
They decided they needed a way to enforce contracts on the structure of the documents moving through their system, and they wanted a system that could work almost exactly the same in Python and in Angular. So they built their own library for validating the structure of documents. They built out a whole vocabulary for the "types" in their system, using types which were generic enough that they could work in both Python and Javascript. It is an interesting system, but it takes at least 25% of the time of the most senior engineer. If they did not write their own system, he would have at least an extra 10 hours a week to put toward tasks that are specific to the needs of the startup.
They wanted to store JSON but they did not like any of the JSON document stores that were available in 2011/2012. They love PostGres. This was before PostGres had a JSON column type. So they decided they would store JSON in PostGres as a string in a TEXT column. They wrote a library that serializes and deserializes the string. They built this as a layer on top of SqlAlchemy. Their code is very clever: you can query the database with their ORM and you always get back objects, you never have to think about JSON as a string. In some ways, they have reinvented the idea of an "object database" that was popular in the late 1980s and early to mid 1990s. These are clever ideas, though it took time away from the actual needs of the startup.
I could go on. They wrote their own cache system, their own system for transforming data structures, etc.
They are thoughtful. When adding a new feature to one of their libraries, they have respectful debates that sometimes last 2 or 3 or even 4 hours. They release everything as open source. They are trying to help the Python community. But it adds up to a huge tax on their time, and it takes them away from the needs of the startup. The founder/CEO is not technical, and I think they often wonder why things take so much time.
With startups, it is especially true that "the perfect is the enemy of the good." Their perfectionist streak takes them away from the needs of the company. They pursue an ideal of absolute brilliance, though they often don't have the time they need to make their libraries as good as what they dream of.
Having worked with them, I've gained a new respect for the idea that startups should, as much as possible, just use what libraries they can find. All libraries have their quirks, but it is an illusion to think it is cost effective to spend your time re-inventing what's out there.
Here's an analogy: McDonald's Big Macs. Is the code in question part of the Special Sauce(TM), or is it just something like the buns? McDonald's makes the Special Sauce, but I bet they just buy the buns.
If it's not part of your special sauce, if you can buy it, do so. Spend your time on the special sauce.
More importantly, though, we tend to underestimate both how long it will take to implement something ourselves and especially how much it costs to maintain over time. If you can use third party code from a popular and active project you get a huge amount of testing, bug fixing and other maintenance from the community over time. If you implement in house then you've just given yourself an extra burden for the life of your project.