I'm not a CTO but I do lead the dev team at our agency (was previously 16 devs, but we've slimmed down to 7 currently). I want to preface this by saying that at an agency, your biggest enemy is always time; sales teams have to sell projects for the absolute minimum in order to get a contract, so you can't waste time on non-essentials for most projects.

That said, the biggest resistance I have found is "this feature is due in three days, I need two and a half to finish, and then we have another half to review and find bugs." In the end, the biggest issue is that we have time to test on the spot or write tests, but not both. You can scrape by with just manual testing, but I don't think anyone would ever rely on automated tests 100%.

Our larger projects are test-backed, and our largest even reaches 90% coverage, but the only reasons we wrote tests for those was because we knew we would be working on them for 2-3 years and it was worth the time in that case. I wish this wasn't the case, but I've found it's always the argument against automated tests in my corner of the market

In my previous agency life, this was something that I experienced as well. A short lived product that was due in less time than any sane dev would estimate. We all knew that we "should" write tests, but there just wasn't time. And in 6 weeks the project would be relegated to living in source control because the campaign was over.

It made hiring devs fun. Trying to explain to people why it was that way, and their insistence that software development doesn't work that way.

> A short lived product that was due in less time than any sane dev would estimate. > And in 6 weeks the project would be relegated to living in source control because the campaign was over.

That is exactly it for 90% of agency projects. Underquoted to get the deal, a rapid development cycle that leaves the devs feeling dead, and then once that first release is out, you have maybe 1 or 2 small updates and the project is never touched again, or at least not for a year or two.

There is no world where it makes sense to write tests for these projects.

What does 'agency' refer to in this subthread?

Agency for what?

Advertising agency, marketing agencies where technology takes a back seat to marketing/promotions etc. Places where projects number in the thousands on websites, apps, games, many systems clients, new technologies etc.

Every developer/engineer should work in an agency for a while because of the amount of sheer work and lifeline of said work is short, projects are primarily promotions and one and dones in many cases.

What we did at the agency I worked at was try to harvest systems from common work. Landing page systems that then had base code that was testable and common across all, create a content management system that supports agency specifics. Promotions/instant win systems that had common code across all and could live longer than the 3 week promotion, create a prize/promotions system that ran all future promotions and improved AFTER most promotions due to time constraints. Game systems for promotional games / advergaming, after new games and types became common or re-usable etc.

Many times, you have to take an after the ship approach and harvest systems that make sense from the sheer amount of work you are going across hundreds of projects. Where good engineering really comes along on subsequent systems where promotions, projects or games/apps were initially made and proved a need or prototype for how to do future projects quicker and with more robust systems.

Testing and doing code specifically for that campaign may be usable or not, but later you can harvest the good ideas and try to formulate a time saving system for the next, including better testing and backbone/baseline libs/tools/tests etc.

I have worked in agencies 5+ years and game studios 5+ years and both are extremely fast paced, usually the harvesting approach is the one that is workable in very pressurized project environments like that. Initial projects/games/apps etc are harvested for good ideas and the first one might even be more prototype like where testing/continuous integration might not fit in the schedule the first time around, or might not even be clear what to standardize and test until multiple types of those projects it out. Starting out with verbose development on new systems/prototypes/promotions/campaigns/games might not be budget capable or time allowed to do so on the first versions as they might be throwaway after just a few weeks or months. There is a delicate balance in agencies/game studios like that where the product and shipping on time is more important on the first go around as the project timeline and lifeline may be short. Subsequent projects that fit that form are where improvements can be made.

I remember my agency days.

Now days I work on a single long-running legacy project where tests make sense. Back then, I read a lot about how testing was the "right thing to do." But I also realized that most of the time (a) the client wasn't going to be willing to pay for the tests and (b) odds are that once we launch the product, that will be last time I ever look at it.

Maintenance will occur in five years when sales talks the client into scrapping the entire thing and rewriting it -- the client won't be willing to pay for maintenance or automated tests, but somehow sales could always sell them on a total rewrite.

It's a very interesting setup: all prod code is "throwaway".

I wonder if each such project is built completely from scratch. If not, the reusable parts can be improved over time, and covered by tests.

> I wonder if each such project is built completely from scratch.

For us, it's a mixed bag. We have a CMS we use for most projects that we did not develop, but we have developed our own packages/blocks for it that are included in every project that bootstrap and whitelabel the hell out of the CMS to provide the functionality we need in every project. From a data standpoint, one of our packages replaces several dozen classes and hundreds, if not thousands, of lines of custom code in every project.

When it comes to more custom projects, specifically ones that never see public use (like a custom CRM, admin dashboard, CRUD-based system, API backend, etc.), we build using the Laravel framework which bootstraps away all of the authentication, permissions, routing, middleware, etc. and gives us a very good blank slate to work with. For these, everything is mostly from scratch, minus what we can use third-party packages for (such as the awesome Bouncer ACL). We have a front-end library that I wrote to abstract away common tasks into single line instantiations, but it's our experience that these projects are being built on a blank slate for a reason. These are the projects that may actually see tests written for as well, although not all will.

The typical stuff an agency can reuse is all covered by frameworks and libraries anyway.

You take an existing CMS or shop software and customize it, or take a web framework and build a very customer-specific service on top of it. Most everything you can share between CMS projects is already part of that CMS.

I find this view (and the replies) interesting. One thing that I've experienced after writing tests a lot is that once you know the patterns, implementing TDD becomes effortless. Eg (python)

Need to test interfacing with an SDK correctly?

Sure, patch the SDK methods and ensure they are called with the proper parameters

Also, for extra coverage, add a long running test that makes actual calls using the SDK. Run these only when lines that directly call the SDK change (and ideally there should only be a few of those).

Need to mock a system class?

Sure - Here's the saved snippet on how to do that

---

This of course applies only if you repeatedly access projects that use the same stack. If you don't then I understand that it can be pretty hard. But basically over time, writing tests must become easier else that's a sign that something in the process is not working correctly. Knowledge isn't being transferred. Or things aren't being done uniformly.

Ideally once you get past a certain point, testing should be just a selection of patterns from which you can choose your desired solution to implement against a given scenario.

I accept that I could be missing something here so please take what I say within the context that my thinking applies to work that can be described as technologically similar.

I always find code coverage such a useless metric: if you have two independent ifs next to each other and one test goes in one if and another test in the other you have 100% coverage. Congratulations. But you've never tested what happens when you go in both

I agree that it is a useless statistic, especially when comparing unit vs integration vs functional vs smoke testing. There are different types of tests and just because you are reaching 90% of your code does not mean you are thoroughly testing it.

The only reason I brought it up was to show that we don't skip test writing entirely and the projects where we do write them, it isn't like we just wrote a test to check that "Project Name" is returned on the homepage and called it a day.

A few years back a person I worked with was tasked with implementing code coverage. Part of that task was they also had to get our code base up to 80% coverage or so.

They wrote stupid test after stupid test after stupid test. Hundreds of them. Oh Em Gee. It was like that story of Mr T. Where the army sergeant punished him by telling him to go chop trees down, only to come back and find Mr T had cut down the whole forest.

That's just the basic coverage metrics, there's more than that: https://en.wikipedia.org/wiki/Code_coverage

There are different kinds of coverage metrics. If line coverage is not enough for your liking you can always go for full path coverage. You'd have to write an exponential number of tests though.

I find that tests pay off pretty quickly in terms of productivity -- somewhere around a week. There are a couple of caveats, though. First, you have to have a team that's already good at TDD (and not just test-after-the-fact). What I mean by TDD is hard to describe succinctly and especially since I said it's not test-after-the-fact, it's easy to think that I mean test first. I don't. To me TDD is a way of structuring the code so that it is easy to insert "probes" into places where you want to see what the values will be. You can think of it a bit like having a whole set of predefined debugger watch points.

With good TDD (or at least my definition of it :-) ), the programmer is constantly thinking about branch complexity and defining "units" that have very low branch complexity. In that way you minimise the number of tests that you have to write (every branch multiplies the number of tests you need by 2). The common idea that a "unit" is a class and "unit tests" are tests that test a class in isolation is pretty poor in practice, IMHO. Rather it's the other way around (hence test driven design, not design driven tests). Classes fall out of the units that you discover. I wish I could explain it better, but after a few years of thinking about it I'm still falling short. Maybe in a few more years :-)

In any case, my experience is that good TDD practitioners can write code faster than non-TDD practitioners. That's because they can use their tests to reason about the system. It's very similar to the way that Haskell programmers can use the type system to reason about their code. There is an upfront cost, but the ability to reduce overall complexity by logically deducing how it goes together more than pays off the up front cost.

But that leads us to our second caveat. If you already have code in place that wasn't TDDed, the return can be much lower. Good test systems will run in seconds because you are relying on the tests to remove a large part of the reasoning that you would otherwise have to do. You need to have it check your assumptions regularly -- normally I like to run the tests every 1-2 minutes. Clearly if it takes even 1 minute to run the tests, then I'm in big trouble. IMHO good TDD practitioners are absolutely anal about their build systems and how efficient they are. If you don't have that all set up, it's going to be a problem. On a new project, it's not a big deal for an experienced team. On legacy projects -- it will almost certainly be a big deal. Whether or not you can jury rig something to get you most of the way there will depend a lot on the situation.

So, if I were doing agency work on a legacy system... Probably I wouldn't be doing TDD either. I might still write some tests in areas where I think there is a payoff, but I would be pretty careful about picking and choosing my targets. On a greenfield project of non-toy size, though, I would definitely be doing TDD (if my teammates were also on board).

tdd can be faster for some but you are forced into a funnel that involves another step.

If you know exactly what you are writing it is quicker to add your changes jump to the next file add your changes. If you are constantly checking the browser to see if what you wrote works Tdd can help.

> I find that tests pay off pretty quickly in terms of productivity -- somewhere around a week.

I think you overestimate the agency project life cycle. Most of our projects are built and ready for client review in 2-3 weeks total. Once the client makes a few days worth of changes, the project is shipped and we likely do not look at it again for another year or three.

That said, there are always long-running projects and those are the ones you try to include tests in.

Interesting. I worked very briefly in an agency a long time ago. Our projects were on the 2-3 month time frame. I suppose it depends on what you are doing.

We have plenty of those as well, but the overwhelming majority of them are about 2-3 weeks of work once we get started

It's much easier to convince people that tests are necessary if you're starting a new project. Problems arise when you're working on a legacy codebase that never had tests to begin with. Often, the code isn't testable at all.

IME, there are far too many "senior" devs (who absolutely should know better) who never worked on any testing-heavy teams that just don't see the point. After all, there's QA, and it's not like THIS code should break THAT code in a seemingly-unrelated part of the codebase...

I'm CTO at a bootstrapped agency that has now 30+ ppl, and I couldn't agree more with the author.

Sure thing that you can use your authority to force ppl, but should you?

Smart people are hard to come by but once you have them you should let them work, and when you tell them how to do their job you implicitly assuming that you know better. Besides if you force them you achieve nothing but some brain-dead tests that are going to hunt you later and getting a budget to "rewrite tests" is a fairytale.

The art here is to build a culture that embrace the test as a powerful tool. So newcomers are quickly seeing benefits and start to write the tests in right places, not for the sake of an artificial metric.

Besides, there are plenty of places where having high coverage is going to be a waste of time: - throwaway prototypes, - heavy UI code full of animations - they need to look right that is hard to test, - infrastructure code if you have just a few servers of a particular type, - customer projects with unreasonable deadlines that are not going to be touched again,

So having your team that writes tests is a hard job and using PR policy won't help much.

The things that worked for me were: - write tests that make sense your self in early stages of the project - pair with your employees and write tests with them, - do peer reviews and suggest what could be tested and why it make sense.

Getting a team to write tests is change management 101:

People are resistant to change when they don't know how it benefits them directly and immediately.

My suggestions have been:

- By giving developers slight nudges every time they get frustrated with developing when tests aren't present is a good way to help them see the benefit. "Imagine how much easier it would be to write this piece of code if you had tests in places where this function calls other things".

- Enforcing it during commits (as you suggest, using PRs)

- Reminding your whole organization that while you migrate to implementing more testing that velocity of development will be impacted. This is really important, because it means people outside of the dev team also need to see the benefit.

- Eliminating "deadline cultures" first and then implementing unit testing

At my old telecom job, we had weekly MOP (method of procedure) meetings, basically the gate process from lab to production.

One of the MOP checkoff boxes, test results.

So many times you could tank someone by asking "Where are the test results" and they would have to reschedule their maintenance window. If you pissed some ops engineer off, expect the question "Where are the test results" every MOP meeting.

Good times.

Team Lead/Manager here. I frequently have the opposite problem: people who think code coverage (quantity) is as important as or worse more important than test quality. I'd much rather have low percentage high quality coverage than high-percentage low quality coverage.

I completely agree. I've met a few developers who think that 100% coverage is required, and that a complete set of tests will save them from all bugs. Perhaps unsurprisingly, they wrote pretty crap code. Passed the tests, but it was unpleasant to work with.

I like to see good coverage (say, 85%) because the act of trying to cover that much has led us to discovering some bugs that would otherwise have gone unnoticed until someone ran into them in production. But 100% line coverage is still a tiny, tiny fraction of covering all permutations of how that code is used, so I feel like trying to hit some kind of holy grail perfect coverage target over-emphasizes the value of tests. While tests can absolutely be very useful, it's the actual running code that needs to be high quality, the tests are just helpers.

How many tests does a PR need? One? Five? When would you not write tests for something(that something not necessarily being a PR, but maybe a unit or feature)?

Tests, like any process, should be serving you and your goals.. You shouldn't be serving your processes or testing practices. This sort of un-nuanced thinking isn't indicative of a high performing startup or CTO IMHO. Perhaps your policies are not directly indicative of your real thoughts on the matter?

I guess it might sometimes be fine to be a relativist and write off the need for tests as a result of "nuanced thinking," but I think you have to accept that you are running a risk by shipping untested code into your product.

As others have said, line coverage is a misleading metric. Ideally, your tests would fully cover all _program states_, and even 100% line coverage doesn't guarantee full state coverage. If you have untested states, then the following facts are true:

- You don't have a formalized way of modeling or understanding what will happen to your program when it enters an untested state.

- You have no way to detect when a different change causes that state to behave in an undesired way.

So the answer to how many tests does a PR needs- as many as needed to reduce your software's risk of failure to a minimal level... And this is failure right now, and in the future, because you will likely be stuck with this code for a while. Since it's difficult to know how much a future failure will cost your company, IMO I always try to err on testing as much as possible. Plus, good comprehensive tests have other benefits, such as making other changes / cleanups safer by reducing the risk that they unintentionally side-effect other code.

Those facts are untrue. If i am using a sound static type system, I have a formal way of modeling and understanding what will happen to my program, even without tests.

If a function has been statically proven to return an int, I know it will either return an int or not return at all. It can't suddenly return a hashmap at runtime, no matter what untested state it enters.

Code without test code doesn't mean untested code. And vice-versa.

Unless you're actually writing complex tools - no, you're probably not getting a "formalized way of modeling" what happens to your program.

If somebody tells me "hey, I have to keep manually testing this and that, I'm losing a lot of time, how about I spend 2 days writing my test thing?" - I'll say Sure!

But if someone tries to convince me in the abstract - I'll be skeptical. Developer busy-work is real.

If you have any concurrency in your system then you aren't going to cover all the states using unit tests. You'll need some sort of formal model for that.

> How many tests does a PR need?

Enough test for each of your spec. Adding a new functionality to your product? Your test should cover and the cases you put in your specs. Correcting a bug? You test should trigger it with the old code.

You can have 100% code coverage with unit testing, it will do jack-shit for your users when they enter a negative number and there was no code meant to manage this case so it never was tested.

Be careful that coverage is a proxy metric to good tests. Striving for high coverage can mislead you on the quality of your tests.

On the flip side, encouraging good coverage usually ends up uncovering some bugs that might otherwise have gone unnoticed until they bit someone.

On the other flip side it encourages writing test that have zero business value.

High coverage is necessary but not sufficient, sure. I don't think you can have a good test suite with low coverage (< 80%).

As an engineer, I would now be very wary of joining any company unless "Do they write tests for their code?" as part of the hiring criteria. If you want to have something be part of your culture, it needs to be part of the judgement exercised by humans in the hiring/performance evaluation process. I say "by humans" because you do need someone exercising actual judgement rather than checking a box.

What you outline seems reasonable, at least in an environment where you sometimes have hard deadlines (eg. Ticket sales for this festival go live next week). Outside of that, I'm curious what cases there are where you can have a PR which is both critical to merge and doesn't need tests. When I review a PR, I look at the tests as one way of thinking through "what edge cases have already been accounted for here?"

I totally agree that requiring PRs to have tests is a good way to solve this - it's what we've adopted after trying a few approaches

I've never found developers resisted reasonable unit tests, though teams may squabble about what _real_ unit tests are, or may have been burned by poor approaches to unit testing in the past. If you can find the cause of resistance, all teams I've worked with have been happy (even excited) to get better testing in place. It makes them more productive and more successful when done properly.

What I find more common is for the business to be unprepared to make lateral changes to a product. Even rational unit tests are a medium term investment. You need to spend time developing features customers don't see, and apply those tools for some time, to see quality differences. That can be difficult to justify in a number fairly normal business scenarios (low cashflow/reserves, high tech debt/regret, etc.).

To help offset the cost (and delayed benefits), I've always suggested phasing in unit test strategically. Pick a module or cross-section of the product that is suffering from bugs that customers see (i.e., affecting revenue) and add the minimum viable tests to that. Repeat as needed, and within months/years, you'll have coverage that fits the business needs well.

I haven't had this problem either. We also tend to hire more experienced folks though. A lot of startups are hiring junior folks who might not know the best way to structure things. There aren't a ton of incentives to write tests at startups and it's hard to get right. Very different when you write an infra product though :). It likely depends on the kind of product you're shipping.

Consumer might not need that as much as enterprise.

Most test writing feels unnatural (especially lower level unit tests). I think this is a tooling issue and an inherent problem with unit tests.

Can you recommend good resources (article/pdf, lynda, etc.) on writing tests? Less about the how (though important too) - but more about when, why, and for what parts/functionality. I find when I do write tests some are so trivial and then I stop doing them altogether.

I've actually had a bit of luck in the past with the idea that every PR required a test in some unrelated bit of code. This accomplished two things: coverage went up, and a junior Dev had to explore the code base and learn

I have not had this problem as a CTO. I try to lead by example with comprehensive tests for everything I commit. We also track test coverage with CodeClimate and make checking tests a part of our human code reviews.

Very hard to write tests in ES5 and angular 1 and karma. I am testing way more with React, jest with snapshots and enzyme, redux, my team wouldn't have 100% coverage without easy to test code

Now say I have a CTO who doesn’t like (automated) tests be cause he thinks they’re a waste of time and mostly something a type system should do for you (never mind we’re in dynamic loosetyped land). How do I introduce tests? Tried many times, always end up not getting adopted.

Some options:

- Educate your CTO.

- Just start writing tests. Consider whether you can pull this off.

- Wait for something to go wrong where tests would have caught the problem earlier. That would be a good time to bring testing up again.

- Find another job.

A and C if not done tactfully could lead to D

Do it for yourself - tell the CTO you're doing TDD to make your own process sane. It would be weird for others to object to you adding tests to your own code. Then others will see your awesome code with great coverage and no bugs in PRs and it may even catch on. Convince one or two colleagues to do it with you.

Respectfully I disagree. I’ve found that ‘talking about doing it’ will cause problems if you aren’t able to do it and hold schedule with everything else. From personal experience, for parent post, I’ve found it’s been best to implement what you can for unit tests and grow it yourself slowly or at whatever pace you are comfortable with, but set the bar high for yourself, beat it, and then maybe try to get others on board. This only holds if there is no mentor you could talk into helping you. Just personal experience. :-/ If the culture doesn’t encourage them now, announcing you will be doing it might only hinder the effort.

forgiveness > permission

Oh yeah, go ahead and do TDD to prove to your CTO that he is right.

TDD is a tool for specific needs, just doing TDD because you heard it's great will just kill the team productivity and whatever product they are working on.

Guessing because it's usually presented as something they should do without changing timelines or resource levels.

Sure, it will pay off, but not right away. Something needs to cover the interim.