* The sample size was tiny. (20 students)

* The participants were selected by convenience. (They were students in the researcher's class.)

* The majority of participants had no professional experience. (Six students had prior professional experience. Only three had more than two years' experience.)

* The programming problems were trivial. (The Bowling Kata and an 'equivalent complexity' Mars Rover API problem.)

Maybe, maybe you could use this to draw conclusions about how TDD affects novices working on simple algorithmic problems. Given the tiny sample size and sampling by convenience, I'm not sure you can even draw that much of a conclusion.

But it won't tell you anything about whether or not TDD impacts development time or code quality in real-world development.

The top-most comment on that page emphasizes this point:

Here's my hypothesis, based on personal experience: the benefits of TDD begin to manifest when they are applied at scale. During design and development, if a single developer can plausibly understand an entire system in their head, the benefits of TDD (and, in fact, unit testing) are negligible. However, there's a non-linear benefit as systems become larger, particularly in the diagnosis of large and complex system failures.

However, the moment it becomes more than just a one-off project this debt becomes a problem.

In terms of technical debt, does it matter when the tests are written?

Conversely, extensible code can have the tests written before or after without much change to the amount of technical debt. The code is written to adapt to change, so tacking on robust tests wouldn't require too much technical debt.

Note that there is a balance. If you know your goals and have no reasonable expectation for them to change, then making everything extensible is unnecessary overhead. You can also have a mix of abstract functions that are extensible alongside immutable functions that handle something specific. It's all about what works best for your project.

Forcing myself to think "how would this be tested?" helped, but doing some of the tests first got me in that frame of mind.

And by practicing that, you're writing code that - by definition - is reusable - you're using it in the tests, and in your production code. The 'reuse' thing doesn't have to mean 'reuse on 8 other projects' (a common rebuttal I've heard). Reusing the functionality in another part of your project 3 months from now is reuse too.

Looks like you misunderstood what TDD is. TDD is not a method of testing, it's the method of design of system architecture, in which you formalize the contracts and design APIs by writing examples of their use in tests. Because of that TDD is applicable on every scale (e.g. even when you design complicated algorithm that can fit in just couple of screens) and because of that TDD must be compared not to other test methods, but to other methods of design.

Of course, that comparison will not show the supremacy of TDD for many reasons, like, for example, insufficient expressiveness of testing framework or domain specifics. TDD is useful as one of the many tools we have today, but it shall never be the only tool.

Also note that TDD is not the only way of getting this benefit. IMHO Paul Graham's essay Programming Bottom Up[1] and Casey Muratori's concept of 'compression-oriented programming'[2] both espouse this same idea of improving API design by "writing examples of their use."

[1] http://www.paulgraham.com/progbot.html

[2] https://mollyrocket.com/casey/stream_0019.html

Mostly because TDD process of red-green-refactor is an evolutionary process. Like taking mother nature to design.

It's un-intelligent design....hahahha

Anyway, tests first or tests later both are good. No tests is a nightmare as a couple of projects I've just inherited are reminding me today.

Edit: typoes

But maybe that's what's needed - a placebo to feel you better as a programmer, and thus making you more productive through your feelings.

This is my experience. I've watched teams do TDD, only to come up against their first refactor and have to change 80% or more of the tests. Just the other day, they were griping about a code review that they had to do which involved changes to 2,000 lines of test code over a code change with minimal functionality change.

TDD has the effect of making you think differently about how you code. Thinking about how you write code is a good thing, but you don't need TDD to make you think differently about your code. You just need to think.

It's like observing two fans debate which Star Wars movie is the best and the merits of Jar Jar Binks, and concluding that because they disagree, in fact all Star Wars movies must be pretty bad.

No, I am not concluding that the TDD is bad, if anything, I am concluding that it doesn't actually matter. Anyway, I am not even clearly doing that, I just wanted to entertain the idea that some practices (such as TDD) can be placebo, so they subjectively feel as a good thing despite the fact that we cannot measure any effect.

Neither was my intent to touch the moral issue about placebos.. I think if it works for you, do it.

In fact, thinking about it some more, there can even be practices in SW development that have nocebo effect, that is, have no measurable impact, but make you feel worse. Daily scrum meeting comes to mind for me.

* The type of code I'm working on

* How well I remember/understand the full requirements.

* How much of the testing framework is built already and how much I have to build.

* The type of test I'm writing.

* How new the project is.

Writing the test also gives you a second chance to think deeply about the requirement and fix any problems with it before wasting time implementing the wrong thing.

If this is the crutch that makes you a better programmer, who am I to discredit the practice? But dogma has no place in software engineering.

Sometimes using something to help you write better code is just a plain old good idea. You can write it off as unnecessary dogma, but if it actually makes your code better you're shooting yourself in the foot a bit.

It could be argued that an experienced programmer doesn't need TDD to write the same code that TDD could have produced.

However the experienced programmer will be very happy when they don't have to re-test all of the old functionality manually after a large re-factor of some core code.

Tests aren't just for initial correctness---their benefit is largely in maintenance. They allow you to make changes or refactor the system with guarantees that the covered portions still operate as they were originally designed. There is really no excuse for those types of breaks.

Then there's the team setting: I'm not the only one writing my code. Someone else has to get in there and make changes. I might have to come back months or years later and make changes. At work, we have five developers touching over 100 distinct projects at any point. We have observed that the lack of tests essentially guarantees breaks---we have QA, and there is a direct correlation between bugs and whether code is comprehensively tested.

With regards to TDD: we also observe time and time again that writing code after tests yields untested behavior. There are always odd cases that might not be immediately obvious. There might be a bug that was inadvertently introduced, and now that bug is tested as part of the implementation. Certain branches may not be fully tested. I've seen comments here about this being an experience thing---that more senior developers won't have this problem. That's essentially saying that one is infallible; it doesn't make sense.

TDD also prohibits rushing. Not writing tests is also an excuse to rush the implementation and produce a shitton of unnecessary code. Tests slow you down and force you to think.

Small projects grow. If you don't write tests upfront, when do you write them? We develop incrementally---all of our projects start small, and some of them will remain small for perhaps a year or more until we revisit it. Writing tests at that point loses a lot: we may have forgotten the details of the implementation by then, and the person who originally wrote it might not be involved at all in the changes.

Tests are important, you should not have "the new guy" write them, because they should already have been written, by someone who understood the code being tested. Having some junior dealing with testing is a sure way of producing useless, slow and incorrect tests.

We've changed the title along with the url. Please see https://news.ycombinator.com/item?id=12742120. The submitted title was “TDD has little or no impact on development time or code quality” (including the quotes).

Also I don't think convenience sampling is bad in this case (and they clearly indicate they did it etc.) as it allows you to control for TDD experience and the like. I'd even go further and say the typical "lol students as participants" isn't an issue either.

"How effective is test-Driven Development. Making Software: What Really Works, and Why We Believe It" by Turhan et al. is a good meta analysis if you want more studies.

As an aside I like the fact that this is a replication study and that there have been a couple of these.

I mean worst case you can uses theses studies to guestimate better priors for your own Bayesian analysis :)

Being mindful of the conditions of the study is important. Shrugging it off isn't helpful in the least.

http://infohost.nmt.edu/~al/cseet-paper.html

EDIT: This is now it's own submission below if anyone wants to discuss it there.

https://news.ycombinator.com/item?id=12741237

This is not a claim that TDD is useless.

* There is no way to scientifically measure absolute code quality.

There are numerous subjective indicators (tabs vs. spaces) and the objective indicators only really become apparent once you are dealing with more than one component/service.

However, there are other considerations with TDD, probably on a more theoretical level that:

* There is no other major engineering branch that uses testing this way in production (would you bang a car against the guard rail to see if it works?)

* There is one thing common for sure with all of the bugs you catch in production, they all passed the unit tests

I think testing has it benefits but driving(!) your software development efforts is probably too much, at least for me.

This hints at why the philosophy of TDD is not as helpful as it seems: it confuses modelling/analysis with implementation.

Your testing methodology can be perfectly implemented, but if your problem analysis is wrong your software will still be useless and broken.

TDD would be more interesting if it understood the distinction between specific behavioural expectations (easy to test for, but incomplete), generic applicability and model robustness (much harder to test for, but more complete), and complete formal correctness (often unreachable, but always an interesting goal.)

TDD is better than nothing. If all you can manage is a set of behavioural tests, that should still improve reliability.

But it's a big leap from there to the suggestion that if you define your spec as a series of behavioural tests and your code passes them all, you have a full and correct solution for the initial requirement.

That is just plain wrong, and in untrained hands it can be dangerous.

TDD probably pays its best rewards with relatively simple well-defined projects, and needs to be deployed with a modularization strategy. Possibly the wishful-thinking strategy of top-down design would be a natural complement, since when you wishfully think of a function you can hopefully quickly write a test or two for it.

This is why this newer study is interesting; it compares to TLD, which allows you to say something about TDD rather than just about testing.

I'm ambivalent on TDD, but the arguments defending this study here have been ridiculous and anti-intellectual.

The truth is that it depends on the team writing the code not how you write it. While consistency and a method may help to organize the group it doesn't really say anything about any quality what so ever.

They're dressed up versions of "correlation != causation". Low thought comments appealing to people with not-even stats 101 knowledge.

There is no such thing as a perfect study, and a generic version of the OPs comment could be copy and pasted on any study ever. Unless you study the entire population of the planet, you're going to miss subgroups. Unless you study the entire population of the planet, you're going to need some sort of selection criteria. Unless you have infinite funding and time, you're going to need to make trade offs and sacrifices in your experiment design.

A study will disclose these shortcomings for readers to balance the significance of results against.

Take this complaint from OP:

>* The sample size was tiny. (20 students)

What sample size would satisfy him? Why is 20 too small? 40? 80? 1037? Is he basing his opinion of a proper sample size on his gut? 20 just doesn't feel right?

Experienced devs do what works best for them, which may be TDD but probably in most cases not.

In a graduate-level engineering class, the students were making similar statements about all the studies we read. One day the professor said: It's easy to find flaws in someone else's work; humans are flawed. The real challenge and benefit is to find the value in their work - find what has lasting value, learn from it, and carry it forward.

It might be a "real challenge" to find value in every one of those blog posts too, but it's by no means useful or valuable. It's a waste of time if not outright counterproductive.

If all the studies are flawed, maybe the field is flawed. Maybe the subject is just not susceptible to (cheap) experimental studies. It's hard to trust peer review when there might be problems with the whole field. We may very well be better off relying on experience and opinions because realistic experiments are so far off the mark.

It can be even worse: generalizing the results of a bad experiment might even be dangerously wrong. Psychology results from experiments on young, Western college students are a great example—we don't want to make laws or base diagnoses purely on experiments like that because that could actively harm groups that are fundamentally unlike young, Western college students.

And all that is pretty much exactly where I see experimental software engineering: the results just don't generalize. And sometimes, I suspect, results generalize in ways that are counterproductive to experienced programmers working on large projects—exactly the people I actually care about. And yet empirical studies (even bad ones) still inherently carry a lot of unearned cachet. The real challenge at the end is overcoming this cachet, not finding value where there just might not be all that much.

The whole point of careful statistics and well-designed experiments is so that we can learn whether a premise is true or not. Without rigor, we prove nothing; this study for example, neither proves NOR disproves anything about TDD in a professional setting. For us professionals, it's noise. Yet it's sitting at the #1 spot on HN, fulfilling people's preconceptions.

Also, the replication crisis [1] would disagree with you.

[1] https://en.wikipedia.org/wiki/Replication_crisis

That's where you're wrong... even with "careful statistics", the point of sample-based studies is to disprove the null hypothesis with some confidence level. There is no requirement that it be 100% confident. In fact, with p<0.05, you might expect some small fraction of repeat experiments to show no significant effect.

(See, it's easy to get this wrong!)

This study is meaningful in that it provides some limited evidence. It's fine to question biases and confounding factors... but that doesn't change the relevance of their results, merely the scope. In this case, what the researchers actually found: "At <X> confidence interval, TDD doesn't work for white, male graduate students at <Y> University working on <Z> problem. Generalize at your own peril." But that's a shitty headline.

Neither jdlshore nor I were talking about "proof" or "truth" in the mathematical sense.

> Results are based on statistical relevance subject to the sampling (and their biases).

The point was that this study doesn't have sufficient statistical relevance to give any evidence whether TDD is effective in general. It doesn't matter if this study gives any evidence whether TDD is effective when used by graduate students working on toy problems, because that was not the intention of the study (besides, nobody cares about this highly specific case).

If you read the actual article (even just the abstract), you would realize that evaluating TDD in a general professional settings was neither the goal nor the conclusion of the authors. You are arguing a straw man and hoping to make inferences that are unsupported by the paper's claims. The authors do a good job of scientific communication about their methods, results, and limitations. This is good practice for scientific communications. Don't think it's sufficiently generalizable? Fine, feel free to expand upon their work. That's how science functions.

("truth", "prove", and "rigor" were verbatim, primary components of jdlshore's comment. The two former words have very specific scientific meaning.)

Science does work pretty well; it predicts things with accuracy and reliability otherwise unknown by humanity (AFAIK). It's an interesting question: If not by hewing close to this ideal, how does it actually achieve results?

You should be a lot more sceptical of supposed facts in soft sciences than in hard sciences. Not because of prejudice or arrogance, but because it's much harder to be reasonably certain about anything in a soft science than in a hard science.

Equating a somewhat flawed study to 'a persuasive essay' is wholly disingenuous. As is claiming n=20 to be 'tiny'. It's small, sure, but not ridiculously small.

This kind of study isn't a final nail in the coffin, but a data point to add to the discussion.

After months of calculations and material research, he determines that only if he builds his machine out of birch will it be light enough to fly. According to his calculations, nothing else will work.

Days later, his best friend is visiting him in the hospital. He carries over a piece of the broken machine. With a puzzled look on his face, the friend asks, "Why did you build it out of pine when you knew you needed birch?"

"Because I didn't have any birch."

Le mieux est l'ennemi du bien. It was right here not a couple of days ago.

Sheesh why do I bother?

Regarding the replication crisis...this is actually a replication and there's actually more replication for TDD than for most topics I read about.

In contrast, validating studies _should_ have large enough sample sizes to be statistically significant, precommit to sharing results in order to avoid publication bias, have well-validated experimental designs, etc.

The problem is that these two are often conflated, so most people either blindly trust studies or put no faith in them whatsoever.

I do feel that small studies like this one serve a very important purpose: to let scientists hone their experiments. Most any large study should first be attempted as a small study to work out any kinks.

Based on my admittedly poor understanding, I would expect to see a sample size of around 70 to have the power to detect a small effect in a single-tailed T test. I would also expect to have the samples selected and balanced carefully, preferably from the target group (i.e. experienced professional programmers in the language at hand)

From what I can tell, for an effect to be detected at least 80% of the time in a sample of 20 people, you'd need the effect to be >60%. I would definitely not expect that a short term project would show 60% improvement when employing similar levels of testing, changing only either before or after writing the code under test.

This study doesn't bring anything new to the table except: "in this manufactured environment we found a single point of data that equates to noise."

I'm guessing the only reason the story was upvoted at all in the first place is because some people who agree with the title clicked the up arrow without looking at the article.

Er, no. The studies I've read all end up showing that principled testing helps, but test-first and TDD (strict red/green cycle, code only enough to pass the new test, etc) provide no additional benefit over anything else that gets the tests written.

The "it depends" always comes from the echo chamber trying to justify their desire to believe that TDD isn't completely useless. It actually feels quite similar to the claims I've seen from practitioners that reikei, faith healing, etc aren't complete bunk.

Isn't this enough though? The tests gets written, which probably was the main point in the first place?

Which is what most of these mechanism are about: Finding what causes sufficient friction to get the tests written.

(I'm making this comment because I know of a team with a coverage tool tied into CI where the coverage has been ignored for years)

By writing tests early, you make sure the code you are testing is testable and your knowledge about the code is fresh.

IME you also end up writing tests which are far too tied to the implementation. (With the resulting churn that that implies when the implementation changes.)

You get far more mileage from QuickCheck-type tests IME. Granted, not everything is very amenable to testing using QC-type tests, but a lot of stuff is.

I'd rather just trust programmers to consider what approach works best for their problem and mindset and go from there.

I personally don't TDD most things, but it's a tool I have available and I bring it out when a situation arises.

I find it surprising that's not slower than Test-Last (maybe if you really leave it for last then you'll need some time to fit your functions to your test)

> I'd rather just trust programmers to consider what approach works best for their problem and mindset and go from there.

People are often surprisingly good at fooling themselves. I'd rather have actual empirical validation. And also a pony.

In fact, isn't this a good thing? If it has little impact on development time, we should totally be doing it. That way, we have tests. We have them from the start. We met the conditions we wanted instead of guessing conditions at the end.

That's easily the best part of HN and reddit. A great help with the Murray Gell-Mann Amnesia effect.

I don't think it's that HN is negative (though HN can certainly be negative), I think most articles really are just kinda crummy. Could be oversimplified, probably overhyped, maybe some wildly skewed sense of "providing both sides", or one of a million other possible problems. The comments will tell you about these problems, and that makes them a better barometer of "is this worth it?" than the title or even the content.

This study has the potential to be misused by people to get their senseless argument across and could cause a lot of headache for people in the industry. It's hard enough as it is for a consultant to get people to pay for testable software.

That line about your professor is cute, but the issue is the media gobbles up studies and reports on them no matter what the quality leaving large chunks of the population misinformed. So publishing a study that is inaccurate is frankly irresponsible.

The sooner you can remove worthless information the better.

Yeah, that's because they are all terrible.

The fact that it matches my experience (that TDD has benefits but only for projects of a certain size with programmers of a certain minimum experience level) helps with that, but it is always good to be reminded of the importance of sample size and other priors related to a study that makes a very bold claim.

So... where is the money coming from? Who is going to pay for this multitude of professional programmers to converge to the same environment, be vetted, and spend a non-trivial amount of time coding the same thing as the others in the group?

Of course the researchers in the article would have loved to have those kind of resources and do the perfect, wide-ranging, deeply detailed study, but the OP's criticisms just show how divorced the OP is from experimenting with real-world humans in real-world situations, and with real-world resources.

The research presented here is weak. Honestly pointing that out without pulling punches is better than simply giving them a pass because 'doing good research is hard'.

It'd still not be easy, and of course there'd still be issues (e.g. is there anything about the corporate culture or training in that company that would affect the result?), but it'd still be far better than a bunch of students and toy problems.

Though I think we could derive some more realistic scenarios, like evolving requirements and switching developers mid project that would be more enlightening.

As others have said, good research costs money...

Hackatons seem to manage. Why not set one up?

But maybe I missed something.

The top comment is a book writer on TDD who is shocked and downright appalled

TDD provides absolutely no value over TLD.

TDD is a process where you write empty shims for your code and tests for it first, failing because there is no implementation, and then you write your code that passes the tests.

Frankly I find this style to be completely opposite of how I code - getting something working ASAP, plugging it in to the big picture and then figuring out the problems with my approach and designing with the insight gained. Then I spec out the behavior with tests. TDD assumes you have the design/spec right from the start and all you need to do is write the implementation - very little of my work falls in to that category - perhaps it's different for others.

One segment where I found TDD useful is writing story level E2E tests that spec out the requirements before the code is written - this is the least ambiguous way to spec the problem I've found - the downside being that the person doing the spec needs to know how to write E2E tests.

This is not how TDD works, and I can definitely see how writing code in that manner wouldn't be very productive at all. I also used to think that was how TDD was done until I read how it's really supposed to work by experts.

You iteratively build up code in a small tight cyclic manner. You do not write 100% of your test in 1 go, and you do not write 100% of the code in one go. You are also missing a step. TDD is a 3 step process, red-green-refactor, and the last missing step is refactoring. You start off with a small atomic behaviour you need in your API/product. You then make a test for that behaviour. The behaviour is not related to a particular class or method. Perhaps it is 1 class by coincidence , but maybe its 3 or 4 classes together. A good example of this would be if I had a method that dealt with multiple items. I would start off by constructing my test to use 1 item, and and the code to deal with 1 item. Then the next iteration maybe a list of 1 item in the test and code. After that, maybe I'd move to multiple items in a list.

How do you know you made API mistakes from writing unit tests ? API mistakes become apparent when you integrate stuff and use the API in conjunction with other things (using API in isolated scenarios like unit tests is not really insightful you can figure out those flaws just by looking at the API). This is my primary criticism of TDD - you write tests around the initial bad API and you end up keeping that bad design because you already spent so much time testing that. Implementation errors caught by unit tests are cheaper to fix afterwards then design errors.

My approach is hack together a POC -> refactor that and write tests. If you discard the POC then TDD makes sense to me.

I don't find unit tests all that useful for integration code - either as tests that find bugs or for doing TDD.

Do you mean "with TDD", by any chance?

The rest of your post says that you observed better results when tests were written before the code (rather than later). But as far as I understand, writing the tests before the code is indeed one of TDD's commandments.

But I know some people are wired differently; they'll think a lot more about scenarios first, then code after they have everything accounted for. For them, TDD as a philosophy seems more fitting.

I think the chasm exists between _untested_ code and code that has tests. I've never understood the seemingly-religious zealotry behind TDD as an XP practice. Just like pair programming... if it works for you and your coding style, awesome. But don't force it down my throat or act like it's the One True Path to clean code.

80% of software development is designing correct abstractions/interfaces/APIs. If you have the correct abstractions, everything else is easy by comparison. And both tests and code are fundamentally founded on these early design decisions.

So whether I do TDD or tests-after-code, I'm confronted with the 80% first: designing the interfaces (either in writing or mentally).

Naturally, I never get this part right at first, and so I wind up refactoring a lot as I go along. ("Build one to throw away", I believe this has been called.)

Now in my experience, TDD requires me to refactor more during this process than tests-after-code. But suit yourself. Changing the order within the last 20% won't be earth-shattering one way or the other.

In my experience, using a type system to do this requires much less effort and refactoring.

Yes. TDD came from developers who worked in weak/dynamic typed languages.

It is mandatory to discover errors, that are otherwise discovered instantly for free by compilers in the stronger typed languages.

Strangely, the TDD folks never outline this point.

Unit tests can demonstrate that my code is correct; it can say nothing about how my code is used by others.

In contrast, a type system can extend those protections downstream.

(Yes, yes, unless you go way off the deep end of dependent types, you still need tests. But a powerful type system can systemically prevent a huge number of very common bugs.)

As for dependent types, I'm not sure that even then you would be able to ensure that, say, your cache is correctly invalidated.

I never catch all the design errors with non-coding design work - this gives me the high level view which is critical - but there are always some logical errors and bugs I've failed to model correctly which early tests catch.

But yeah, the optimal design-implememt-test probably depends probably very much on the implememter and on the problem to be solved.

The third meta-advantage is that if I'm writing hobby code for myself and the profject is large compared to the weekly time allocations available in my sparetime I get something to help me remember where I was (say a month back) and gives me something to compile and run immediately (which is a huge motivational booster for me).

Not sure if that claim is wrong or not, but I will state it anyway. If you write in TDD manner and you decide to change the API/abstraction a little bit because of something you didn't think of, you will have to correct both tests and the code. When you do test-after-code it's much easier (IMO) to hit correct abstractions and interfaces because you are working on a living thing, not some artificial test cases you have to figure out. Then you write tests to ensure those abstractions are respected and in working order.

TDD people, please correct me if I'm wrong.

EDIT: disclaimer: I'm a front-end developer working mostly in React and Redux

This applies to so many things in Software Development - text-editors, variable names, plugins, architectures, operating systems...

Sometimes writing tests first helps me think about the high-level design of my code. Other times I've got to try a few things before I have any idea what the code should look like, and writing tests first would just create a lot of extra code churn. I feel that, as time goes on, I'm getting better at anticipating which situation I'm in. I also switch back and forth between the two testing methods even more rapidly.

I'm personally a fan of "readme driven development" (https://news.ycombinator.com/item?id=1627246). It's a nice compromise between tons of upfront planning and nothing at all.

I had thought of involving users in the design of new features via the support forum but I'm afraid of how that might turn out. A readme type document would probably be about as good a way of doing it as any though.

When doing UI code (with something like React) it's next to impossible to do TDD - you can't know the structure until you actually implement it.

If I'm fixing a bug in some sort of non-ui code, I often find TDD to be very helpful. Create a test case that demonstrates that failure, then fix the implementation to make the test pass.

* write a failing test (RED) * implement minimal functionality so all tests pass (GREEN) * REFACTOR

and overcome the initial revulsion you will get a ton of benefits that you didn't know existed. I'll first give a few that are more on the code quality side

* You have a complete test suite, without redundant tests * Tests are usually not that mock-focused [] as the implementation-after-test didn't introduce random roadblockers for testing. tests do assert the relevant properties (tests-later code bases often have code bases that don't assert much or properties you don't care about). * When you write your test first, you basically write a small example how it feels like to use the API you have in mind for solving your problem. This drastically improves quality.

Some non-technical benefits of TDD

* I find TDD helps me tackle bigger code-pieces where I just don't know how to start implementing. * TDD is programming gamification. You are rewarded with regular, small successes of having added another test and made it green. So whenever you ask yourself: OMG, have I made any progress in the last 3 days, you can actually tell: Yes, I added 15 tests and made them pass - I objectively got closer to implementing that feature. * Not an issue where I work, but have often heard about this from other places: If tests are there first, no one can tell you to not spend so much time on writing tests. TDD-style is protecting your boss from shipping that car without having the brakes checked. * The same line: TDD gives explicit room for refactoring.

[*] obviously one cannot avoid mocking especially when dealing with resources external to your code base, yet I have hardly seen TDD-originated tests that mocked

This made me think. There are many axes of software development, and aside from TDD/non-TDD axis, there's also the bottom-up vs. top-down axis. By top-down, I mean you start with a skeleton that provides an interface to the outside world, and the fill in the implementation details. Bottom-up, on the other hand, means you first write some routine that you know will be vital in some form or another, but without immediately seeing how it will be hooked into the interface to the outside world.

Top-down has the advantage that you can have something runnable quickly, and it seems to me that TDD requires top-down development.

But somehow, there have been times when I was working on a piece of code and it just flowed more naturally in a bottom-up manner. It has happened that I went for a day (and very rarely longer) working on tricky algorithmic code before I even tried to compile it for the first time.

This is definitely something that I try to avoid when I can, it's just that sometimes it works out that way. And it is totally incompatible with the TDD way of doing things.

How often this happens surely depends on the type of project you're working on. As usual with these things, it seems like TDD can be a useful inspiration, but it shouldn't be taken as dogma.

I dont think writing Tests first makes you work top down or bottom up.

If you want to work bottom-up (I like to work bottom up) you just start by writing tests for your bottom up functionality.

And of course I don't want to impose TDD on others, but I would like others to try it out sincerely (recommending kent Becks book for this. Don't do it without guidance, many people who reject TDD after trying it out were doing some kind of Test-first method, but didn't get the benefit because they were not aware of stuff like get one test at a time to complete, etc.)

---------

> shouldn't be taken as dogma.

Life in software engineering becomes much more relaxed when you realize that every piece of advice and technology is in fact based on dogmas and is not based on sound empirical, statistically relevant evidence :-) (apart from the pure math part).

It is just really really hard to study software quality in an objective way that satisfies scientific criteria. So we are kind of stuck with hearsay in IT

So TDD is kind of dogmatic. So is OOP, or the rejection of gotos.

Writing a list of tests first is quicker, though, and working through the list, updating it as I learn more, tells me that my initial ideas were wrong often enough that I'm almost convinced that the true secret of TDD isn't the tests at all. It's that checklist, and the purely mechanical process of working through it, checking them off one-by-one and fixing it up as I go along.

To relate this to your situation, I often find that I'll only get started with a fairly fuzzy idea, and the checklist will be short. I can still expand it as I go, knowing that I'm working on as solid foundations as I can given the constraints.

I think there's a danger in writing off practices like TDD with "if it works for you..." because it avoids discussion of why it works for some people and not others. That in turn gives people license to ignore it simply because it's unfamiliar, and the brain doesn't like unfamiliar things by default. That's a shame, because for some of those people they'll be stuck in local optima, writing off the one thing which could break them out of it.

When I was a programming beginner and started learning about TDD, I was always thinking how soon to start using it because everyone says it's such a good idea. If you would search anything about it, you'd rarely find a resource not marvelling at its surprising benefits (at least, that is how it was few years back). Granted, it may have its merits but is it really a good idea for everything? No way!

One of the anti-patterns I have seen in programming world is how everyone jumps on adopting / advocating a 'good' practice without understanding how it fits the project. I wrote about this a while back[1]. Cue in — git flow, TDD, not using goto, DOM Parser instead of RegEx.

[1]: https://shubhamjain.co/2015/10/26/imperfect-best-practices/

    Modularity, DRY, SRP, etc. is never a goal, *it’s a trait*.
    [...] understand that they’re approaches and not achievements.

IMHO untested code isn't always a bad thing. Anything related to privacy, security, or data integrity should be heavily tested.

But beyond that, code should need to earn its tests. (At least in a web startup context.) After all, the core of agility is being able and willing to go in a different direction when something isn't working.

In addition, if you are truly in the earliest stages of your startup (pre-revenue/traction) the tests are not needed -- but you have to be _very_ aware that you are making a tradeoff for sheer velocity and you will have to pay the price later on. I've seen too many startups fail to pay the price and it comes back to haunt them and reduces their overall velocity.

"If your code doesn't have tests then you can't refactor" is pure dogma and trivial to falsify (go on, refactor some untested code now!). More likely to introduce bugs? Perhaps. But relying on that absolutist statement about refactoring will alienate me every time.

I generally find it better to engage in "top-down" development - specification->tests->high level code->low level code as it's cheaper to fix mistakes at higher levels first.

If I'm doing purely experimental code though - where I'm not sure of what I even want or what the output should be, tests are just a waste of time.

It also loses its effectiveness the more declarative the code is. If I'm essentially tweaking configuration or HTML there's no point.

I'm one of those people. But I find it hard to believe that a test suite can reflect everything I understand about a problem domain. Focusing too much on unit tests makes it difficult to find the general rules that govern the behavior of a system. Have too few unit tests, and they won't capture all the subtleties and corner cases. Have too many unit tests, and they won't fit in your head.

If you have no idea what the results should look like, then TDD is pointless.

Sometimes it is hard to write the tests, but it is easy to see if your code works - sometimes it is easy to write the tests, but not the code.

I would call it 'do first what is easier'.

But it is important to write the tests quickly after you wrote the piece of code - so that you don't forget something.

In contrast, TDD has been a waste of time for me for UI-based work, as the effort needed to properly expose the functionality under test is too great and the requirements and design change too quickly to be worth it.

In the latter case, writing some deterministic UI tests against mock data after the requirements and implementation have settled has proven much more effective in preventing regressions.

On the other hand, TDD for operations that involve I/O (including user interactions) were not helpful at all.

It also produces superior software in my experience.

So it's win win then ;)

Another approach is to partition your code into what Gary Bernhardt calls an "imperative shell" of code that touches the outside world and a "functional core" that does not. Then unit test the functional core, which shouldn't require special testing libraries, and validate that the imperative shell works by the occasional manual or "outside" test.

1) http://www.webtigerteam.com/johan/en/blog/error_driven_devel...

  * fire your QA team

  * dev team is the level 2 production support, and

  * get to continuous integration nirvana

Teams that aren't testing constantly, well, they've got tons of escape defects on every release. And those devs are constantly in fire-fighting mode, it's miserable for them. And I see that leading to compressed schedules for them and more reckless behavior like asking to push their releases during the holidays where there could be severe financial consequences to bugs.

As far as I'm concerned, in an environment like mine, where developers can no longer hide their incompetence behind bureaucracy like a QA team, it is official insanity to not spend inordinate amounts of development time writing automated tests. You should be spending 70% of your dev time writing tests and doing devops and 30% writing features.

I read in these comments a lot of bellyaching about how much time it takes to write tests. First, TDD is a skill that you can get good at, and it won't take as much time as you think once you get good. Second, I just don't think you have a choice to not test comprehensively when escape defects become a mark of shame in the organization.

Highly agreed, anytime there has been QA available, the code base starts to look more hacky and fewer tests are written and only the tests that cover the happy path are written. When things are looking bleak, that's when I start to think we should have certifications for developers, optional of course, like the PMP or CAPM, but they should be there as a rough indicator that a dev won't skimp on QAing their own code.

>I read in these comments a lot of bellyaching about how much time it takes to write tests. First, TDD is a skill that you can get good at, and it won't take as much time as you think once you get good. Second, I just don't think you have a choice to not test comprehensively when escape defects become a mark of shame in the organization.

After years of looking at tests written poorly, I've come to the conclusion that you have to treat test code like your regular code and do code reviews on it and refactor the heck out of it.

If you don't, you'll end up in the situation I'm in, where our company has almost 4000 spec tests in ruby/rails that can take an hour to run on a local dev box and still take 20 min when parallelized across 5 boxes. Shaving off a couple of seconds here and there by removing unneeded tests or refactoring to simplify them or to share mock objects is a worthwhile endeavour but because they're "just tests", the time to do that isn't allocated and everyone on the team merrily continues to write and modify tests in a sloppy way.

Personally, I don't care how you test. Just get decent coverage over all the control paths and have a test suite that catches regression bugs. Ideally also have integration tests to prove your connections to your partners are good and maybe a smoke test to prove your code is wired up right once it hits an environment.

I'm still on the journey myself, so I can't claim to be an expert. I know there are people in my org that are suffering because of code quality issues. I call them incompetent because they're producing shoddy work products with lots of defects. A smart way to combat bugs and kill them for good is to test, so that's why I connect not testing to incompetence. A dumb way to combat bugs is heroics and working 18 hour work days trying to debug with "GOT HERE" logging.

Developing software and developing tests and test infrastructure are two different skills and mindsets that are often inversely coupled.

It's not that "incompetence" is revealed by firing the test team (although it sometimes is) - it's that "being bad at writing tests" is revealed. A team with 3 dedicated testers and 7 devs will probably outperform (in code output and reliability) 10 devs spending 70% of their time on testing.

The "We" and "Them" distinctions made people in different teams and different roles ignore the needs of the others. The change brought a culture change and also gave a view on the needs of the other side. This could make it possible that for the first time the development of tests and features could really be done in parallel (officially that was the methodology before, but never really worked out well, because the testability considerations were usually ignored at design time, and the docs were lagging, because of the bad bandwagoning culture. With the mixed team where everyone was treated as an equal these problems dissolved surprisingly quickly (in less than half a year)).

So my point is having dedicated testers can give base for bad culture which hurts the product and the company. Having everyone do the same job with regards to development and testing is better.

Small teams almost always worked out well for me if a single leader person was present to sort out initial problems.

I like to believe that I sometimes will forget to test for edge cases, but have mastered how to write tests. I don't think that how to write tests is easily acquired by any descend programmer. However, you need to learn a few patterns you otherwise won't need.

If so, they have missed the point of TDD.

In the short term, TDD probably doesn't make a difference, one way or another.

But software as a business is not a short-term game.

I would love to see a study where the participants are, over a period of six months, given the same series of features (including both incremental improvements, as well as major changes in direction).

In my experience, teams that don't test at all quickly get buried in technical debt.

Untested code is nigh impossible to refactor, so nobody ever does, and the end result is usually piles of hacks upon piles of hacks.

As far as testing after development goes, there are three problems that I see regularly:

One, tests just don't get written. I have never seen a TLD (Test Later Development) team that had comprehensive code coverage. If a push to production on Friday at 6pm sounds scary, then your tests (and/or infrastructure) aren't good enough.

Two, tests written after code tend reflect what was implemented, not necessarily what was requested. This might work for open-source projects, where the developers are also the users, but not so much when building, say, software to automate small-scale farm management.

Three, you lose the benefit of tests as a design tool. Code that is hard to test is probably not well-factored, and it is much easer to fix that when writing tests, then it is to change the code.

As for your points why TLD is theoretically worse:

1. "in TLD tests just don't get written" - this is a dual argument to "in TDD the code just doesn't get refactored". After tests are green, there is no motivation to do so and it feels like a waste of time (if it works, why change it, right?)". I think the latter is much, much worse for longevity of the project than low test coverage. Not enough coverage doesn't mean automatically bad structure of the code and you can always fix that by adding more tests later (and maybe fixing a few bugs detected by them). But writing code quickly to just make the tests green and then not doing refactoring quickly leads to bad structure of the code very quickly. Whether you chose TDD or TLD, you need to apply some discipline: in TDD to refactor often, in TLD to keep high test coverage.

2."tests written after code tend reflect what was implemented, not necessarily what was requested" - a dual argument for this exists as well: "code written after tests tends to reflect the tests on case-by-case basis, not necessarily the minimal general solution covering all possible inputs that should be the true goal of the implementation". Whenever I hear the argument that tests help write better code I always remind myself a famous Sudoku Solver written by Ron Jeffreys, TDD proponent: http://ravimohan.blogspot.com/2007/04/learning-from-sudoku-s... I also saw that happen in a few real world projects - the code written after tests was just a giant if-else (or switch) ladder handling each test case separately. Bad, bad code, additionally missing a few important cases. And the most funny thing was that after seeing this during the code review, I rewrote the implementation in a more general way, got two tests failing and after investigation it turned out the tests were wrong and the code was good. Lol, verifying tests by implementation :D

3. Tests are not as a design tool. Tests are for... testing. They are often a reason for over-engineering and over-abstraction that makes the code more complex and harder to read. See FizzBuzz Enterprise Edition. It is definitely testable to death.

The mistake is in creating unrefactorable code. TDD may be a possible solution if done correctly but there are other ways to skin that cat.

Whether people code tests before thinking interfaces, before writing a prototype, before implementing, during the inevitable interface rewriting, after coding, after manual verification that it seems to work, or right before submitting the branch to review, doesn't matter as long as someone on the team looks over and sees that "yup, here are tests and they seem to cover the important parts, the interface makes sense, and the documentation is useful".

Then people can do TDD, TLD, TWD or whatever they personally feel most productive with. Developers being happy and feeling in control of their own work does more for quality than enforcing a shared philosophy.

But moreover, I think it's important to understand why Beck pushed for TDD.

TDD is like saying "I'm going to floss before I brush every time, no matter what."

But, when people don't do TDD they typically aren't all saying "I'm going to brush and floss afterwards every time, no matter what."

Instead, most say "I'll floss regularly at some point, but I don't have time now, and it takes too much effort. I'll floss here and there periodically, maybe before my monthly meeting or big date night."

Another reason Beck pushed for TDD was method and solution complexity reduction which results in lower time and cost required for maintenance because code is simpler to read and understand. Again, with ITL, you're still writing tests for everything, so you'll see those benefits. However, if you fail to write some or most tests, some developers will write overengineered solutions to things and have overly long difficult to follow methods that will make maintenance suck more resources.

If you want to go beyond this study, though, Beck, Fowler, and DHH had a critical discussion about TDD in 2014 that's worth checking out:

http://martinfowler.com/articles/is-tdd-dead/

I think it would be very hard to argue that Waterfall ~= $CURRENT_BEST_KNOWN_METHODOLOGY.

[1] Of course, this isn't usually what happens in practice when pharmaceutical companies are doing their own testing, but it's what should happen if you actually care about efficacy and not just PR/sales.

The research shows no significant difference between TDD and iterative test-last (ITL) development.

Could the title be updated? To show that it is a comparison of TDD vs ITL/TLD.

I'm not making a stand about TDD here - I just think we need to have much better computer engineering science studies if we want to have significant results.

I also think of TDD as a sustainability practice. If I'm writing a small thing that I do not intend to maintain, I won't bother with TDD (or with tests at all). But I'll definitely TDD something where I expect to come back to it frequently, especially when I initially don't know the requirements, and I expect requirements to change over time.

In practice, I suspect a lot of the interesting questions about software are effectively unanswerable with the budgets available to CS profs. I can't imagine really answering this question without doing something of the scope of a substantial medical study.

FTA: In this paper we reported a replication of an experiment in which TDD was compared to a test-last approach.

Very different title.

I view it more as important for breaking the growth of testing effort in an iterative project. With each release the scope of what should be tested to fully test a project climbs and unless a team wishes to linearly increase the size of its test team its all but certain tests will be skipped.

TDD gives us the ability to always full regression test as its just machine time. Its a safety factor in knowing nothing is broken which in turn gives us confidence we can refactor.

This makes it a fact and a great counter argument for helping a lot of programmers who are being forced to practice TDD because of the generally accepted claims in productivity and code quality associated with doing it.

[1] http://david.heinemeierhansson.com/2014/tdd-is-dead-long-liv...

- Sample size: 21 students

- Study duration: 2 days

- Team size: Individual

Tests are most useful when refactoring someone else's long-forgotten code; the sort of thing that happens frequently in long-running projects consisting of large teams. In other words, the "real world".

Show me that study.

https://www.microsoft.com/en-us/research/exploding-software-...

This finding contradicts the headline. TDD impacted both development time and code quality in that study.

TDD solved a major problem for me which I have seen a lot of people suffer with. _Where do I start ?_ . The thing is TDD and refactoring go hand in hand. I cannot imagine doing TDD if I was not using an IDE like Intellij or something. When you normally start writing code first(typical TLD) then you need to have a plan before hand. This plan cannot change much because you really do not get feedback till you complete major segments of the code. TDD ensures you keep getting nibble sized feedbacks which assure you that what you are writing works. This according to me is the single most beneficial point of the system. TDD or TLD would allow maintainable code too.And often while doing TDD too,you can strictly follow TDD. It might not have an impact on code quality for seasoned developers(coding for years on the same codebase) but it does help for the others .It also reduces my inertia considerably too. So while it might not have impacts on development time or code quality. I tend to sleep well without large UML Diagrams floating in my head and knowing that each unit of my code works independently.

A good plan is modular and therefore flexible. I get lots of feedback as I'm writing the code. When I find myself writing very similar code over and over or an API feels unwieldy I take that feedback and refactor without delay.

I think what really happens in big groups of developers is that TDD forces everyone to delay agreement on the interfaces between code modules until the process of writing tests has uncovered most of the problems. With out TDD the devs who just want to get their part done and go home plow ahead with the first draft of the API locking it in stone before it's been vetted and then dig in their heels creating technical debt. TDD appears to be the saviour.

This is a major factor. the Pareto principle applies in workplaces too. TDD tries to even the balance a bit.

Regarding feedback.I should have clarified that I am a mobile dev. I need to build everything from the backend services to the UI to be able to get viable feedback without tests. And if any other mobile dev has any other approach to this problem . Please let me know,I've been trying different approaches for a while now. None seem viable to me apart from TDD.

I would expect a determined attempt at TDD to solve the "no tests" problem, because it is so utterly insistent on tests. It should also solve the "don't know how to start" problem, because it de-emphasizes planning and design in favor of just jumping in; you write the tests, and then you do the bare minimum to make them pass.

That said, I would expect a TDD-based project to have the "bad architecture" problem: messy interfaces and sort of ad-hoc separation of concerns, because it makes no time for up-front analysis and design. It's always focused on the current feature and doing whatever it takes to make it work now.

In fairness, it does include a refactoring step, which is supposed to clean up the mess after the fact. Color me skeptical. Refactoring is hard, and people tend to do it on a large scale only when they have to.

It doesn't SOLVE this problem. It only pushes it in time, making it actually worse, because you're wasting time on stupid tests instead of actively researching a solution. No amount of tests is going to help you find the right solution if you don't know what you are doing. See http://ravimohan.blogspot.com/2007/04/learning-from-sudoku-s...

But I fear this incremental approach is going to produce a very baroque solution that is going to have to be rewritten completely once the bell goes "bing" and the programmer actually understand the underlying problem well enough to produce a clean solution.

I think the larger problem with TDD is that there are at least four parts to software design, and TDD bets it all on two of them. There's requirements analysis, architectural design, construction, and finally testing. TDD is really all about the construction and testing bits. It doesn't address requirements analysis at all, and it doesn't seem to want to do architectural design, it just constructs and tests with great passion. It's imbalanced.

Developers vary as well. Some temperamentally find something like TDD useful. Others do not.

There is no one software development methodology to rule them all.

https://news.ycombinator.com/item?id=3033446

Practitioners have a problem recruiting subjects. There is often a tradeoff between applying more rigorous experiment design and using convenience sampling (students) versus sacrificing controlled environments (so that professionals would actually join the study).

It's easy to condemn work like this but there's no other option. In this case the researchers chose to replicate a study (which often risks similar ire for telling us nothing new) with a commendable level of rigour and have provided more evidence that, for the scope of experiments we can construct that TDD is probably no different to TLD when using a population of relatively unqualified developers (students).

As to the problem being trivial, what else can be done? There's a finite time you can ethically expect participants to give to you, even if you pay them. If anything the criticism of this work is better directed at the limitations academics are forced to bear.

"A policeman sees a drunk man searching for something under a streetlight and asks what the drunk has lost. He says he lost his keys and they both look under the streetlight together. After a few minutes the policeman asks if he is sure he lost them here, and the drunk replies, no, and that he lost them in the park. The policeman asks why he is searching here, and the drunk replies, 'this is where the light is.'" [https://en.wikipedia.org/wiki/Streetlight_effect]

I agree that this is a well-designed study given its constraints. And it's admirable that it's a replication study.

That doesn't change the fact that it's largely irrelevant to professionals. It doesn't test the claims made by TDD proponents (TDD leads to better design, reduces long-term maintenance, allows for team coordination, etc.), nor does it address any of the interesting questions about TDD:

* Is TDD more effective in a professional setting than commonly-used alternatives?

* Is a mock-heavy approach to TDD more effective than a mock-light approach?

* Do people using TDD refactor their code more or less than people using a different but equally rigorous approach?

* Is the code done with TDD more maintainable than code done rigorously in another way?

* Is TDD easier or harder to sustain than equivalently-effective alternatives?

As a study, it's fine, if only of interest to academics. The problem isn't the study. It's the credulous response on the part of industry developers who then turn the false authority of the study into statements like "TDD doesn't lead to higher quality or productivity."

"Students should generally not be used in studies that depend on specific expertise which requires significant experience and a long learning curve to achieve, or in studies of professional practices. Such studies are best performed by observing and interviewing professionals, not by controlled experiments."

This seems directly relevant to this TDD study. Every one of these contraindications are true in this case.

That's all well and fine, and possibly completely accurate. However, many people's takeaway is going to be the out-of-context & incorrect title of this post. (It does not say TDD is worthless - it says it is essentially the same as TLD)

I've always looked at TDD as a tool to help push less experienced, less "educated" developers into 1) even using tests at any point of the development cycle, 2) creating tighter, cleaner and MORE TESTABLE code by the time they've reached the end of the cycle.

So, if your team is and always will be well-educated, experienced programmers who already understand how to always do everything correctly from the beginning, feel free to use either method.

Otherwise, I'd urge you to consider TDD.

I've met a few that couldn't code let alone write proper tests, even though their foci were not policy or say the lighter side of user experience.

That's only personal experience, however, and mostly out-of-date

That said, I would hope any of them would be a step up on the new programmers from the "churn 'em out" short courses many are forced to take as the only affordable way they have to get into the industry, but then are left with little knowledge beyond how to get the requirements fulfilled.

Perhaps because it's less stressful. You think about system design as you code, instead of only when you hit a wall and have to rewrite everything, or when you have to clean up for code review.

Either way, if it has little to no compact on dev team or code quality, I bet the positive impact TDD has on team morale would make it worthwhile.

After and during the experimental phase, it depends. Both before and after I may write tests before or "test with" for gnarly logic or algorithm-y stuff. Otherwise and in addition I do copious amounts of manual testing. Manual testing is a must for much of what I do, so I augment or substitute automated testing as appropriate. Automated testing is great, but sometimes the overhead is too expensive.

And generally, I would say that the more interesting sorts of tests (fuzz testing, large-scale system testing) are extremely unpopular with software engineers because "they suddenly fail without reason". Not quite as unpopular as actual proof that an algorithm works, like implementing it in coq for instance, but very unpopular.