Hacker News new | past | comments | ask | show | jobs | submit login

I split tests into two categories. One is "I'm testing a bunch of different variations of the same thing." Table tests are great for this, they make the code pretty readable and, perhaps more importantly, the process of writing the inner part of the table test seems to nudge you to think about all the permutations and combinations of what you're testing can be. It leads to good tests and helps you reason about your inputs.

The other kind of test is what I call a "storybook test." Table tests are less useful here, this is usually for testing CRUD in a coherent manner but can apply to any kind of object that has a lifecycle that you need to transition between various states. You may argue: aha! This is a state machine and you ought to simply independently test each state transition! Fair enough, I say, and yes in some cases I might actually do so if the API that's exposed lends itself to having its internal state set so explicitly. But more often than not I argue that the story itself is more valuable than exactly what permutations of the inputs and state get tested at each call. What the test represents is that some flow is important, gives an example of it, and helps show that this general use-case is what we want to keep working, not necessarily these specific state transitions.

Mind you reality is messy so tests often blur and blend and so on. But when I sit down and write a new test, I'm usually imagining it as one of these two archetypes of tests.




You might like Hypothesis' approach to stateful testing. See https://hypothesis.readthedocs.io/en/latest/stateful.html I think what they call stateful testing is close to what you describe as a storybook test.

Hypothesis is a property based testing library.


> [T]he process of writing the inner part of the table test seems to nudge you to think about all the permutations and combinations of what you're testing can be. It leads to good tests and helps you reason about your inputs.

I think this is not inherent. If you need to test all permutations and combinations, do the exhaustive property testing instead. Table tests (ideally) represent a pruned tree of possibilities that make intuitive sense to humans, but that pruning itself doesn't always neatly fit into tables. But otherwise I generally agree.

When I write a specific set of tests I first start with representative positive cases, and I think you would call them as a storybook test (because it serves both as a test and an example). Then I map out edge and negative cases, but with a knowledge of some internal working in order to prune them. For example if some function `f(x, y)` requires both `x` and `y` to be a square number, I can reasonably expect that such check happens before anything else so we can only feed square numbers when we are confident enough that checks are correct. And then I optionally write more expensive tests to fill any remaining gaps---exhaustive testing, randomized testing, property testing and so on.

Note that these tests are essentially sequential because each test relies on assumptions that are thought to be verified by earlier tests. Some of them can make use of tables, but the entire test should look like a gradient of increasing complexity and assurance in my opinion. Table tests are only a good fit for some portion of that gradient. And I don't like per-case subtests suggested in the OP---it just feels like a workaround over Go's inability to provide more contexts on panic.


Btw, those tables making intuitive sense to humans sounds like a strength, but it's also a big weakness. Humans aren't always good with thinking about corner cases.

Property based testing is one way to get past this limitation. (As you also imply in the later part of your comment.)


One of difficulties with property testing is that humans are notably bad at specifying good enough properties. A canonical example is sorting: if your property is that every consecutive element should be ordered, an implementation that duplicates some element in place of others won't be caught. We can't always come up with a complete property, but intuitive cases can improve otherwise incomplete properties.


Yes, coming up with good properties is a skill that requires practice.

I find that in practice, training that skills also helps people come up with better designs for their code. (And people are even more hopeless at coming up with good example-based test cases.)

Of course, property-based testing doesn't mean you have to swear off specifying examples by hand. You can mix and match.

When you are just starting out with property based testing, as a minimum you can come up with some examples, but then replace the parts that shouldn't matter (eg that your string is exactly 'foobar') with arbitrary values.

That's only slightly more complicated than a fixed example, and only slightly more comprehensive in testing; but it's much better in terms of how well your tests are documenting your code for fellow humans. (Eg you have to be explicit about whether the empty string would be ok.)


>The other kind of test is what I call a "storybook test." Table tests are less useful here

Storybook tests lend themselves well to forked tests. E.g.

  steps:
  ...
  - click: shopping basket

  variations:
    Empty basket:
      following steps:
      - click: empty basket
      - items in basket: 0

    Purchase basket:
      following steps:
      - click: purchase
      ...


Are you dogmatic about things such as only one assert per test case or similar?


Not at all! Quite the opposite. I encourage people to write tests that make sense, are readable, and are a good use of their time. I encourage people to NOT write tests if they don't actually believe that it improves their belief the code actually works in prod, and also if they think it's not worth the LOC.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: