I think their definition of automated testing is good. Myself I find unit testing to not be worth the trouble, most unit tests I've seen in mobile apps are just checking obvious inputs for obvious outputs. Unit tests take time away from writing good code, can never be complete (even 100% code coverage isn't 100%), and take time away from working on the most important part of any software, features.
I prefer to spend my extra time writing code that validates all inputs and all possible error conditions, and logging exceptional situations that can't be handled within the error system.
Tests I find valuable are ones that explore performance and data quality issues. For example in my current project the original developers decided to assume our server APIs always return nil data (hopefully with an error) or that they will return perfectly formed JSON, and if not, would crash. The truth is it's unlikely the server doesn't return poorly formed JSON for a "successful" request, but I wrote code to test 3,000 stores with various data inputs just to test a single API to find out if it did.
Of course I"m not writing a server API. If I was, I'd definitely be writing unit tests. But in a mobile app, there is plenty of work to do to improve the app directly that provide far more value than most unit tests can.
I have always thought that one of the most useful reasons for having unit tests is to help refactoring, but with the rapid change of js frameworks or even languages, it seems one is more likely to transition to something entirely different every 4 or 5 years. Which makes the roi much lower on some types of testing. Can anyone who has changed tech stacks/frameworks often for an app/site comment?
There are of course lots and lots of ways of writing useless tests, but I don't think it's right that the most important part of any software are its features. I for example prefer software with fewer features that works reliably over software that includes everything and the kitchen sink but crashes regularly.
Features have external and internal complexity. External complexity is documentation, usage, support while internal complexity is time spent on maintenance and development, increased complexity of the code, reduced stability etc.
External and internal complexity are unrelated, and when deciding whether to add some feature both must be considered against the assumed advantages of the feature.
I realize there are scenarios in theory where it's easier to verify output than create it, but in practical terms (e.g. A web application) what would the difference be between a testthat validates all possible inputs/outputs and the actual program you're writing?
Lots of mobile app code has inputs dependent upon a variety of events from network calls to user inputs, and it's "output" can be almost anything from UI state changes, pixel maps or database changes. Then your design group wants to change the UI, and you have to refactor everything, and rewrite every single unit test.
> if you want to check a single function's behavior with 10 different sets of primitive arguments, using integration tests to test it may mean you end up setting-up and tearing-down an application process 10 times: a slow and and wasteful use of your compute resources
This is only true if the test is run in process and there it is stateful. Performing out of process tests on stateless units are very effective.
I prefer to spend my extra time writing code that validates all inputs and all possible error conditions, and logging exceptional situations that can't be handled within the error system.
Tests I find valuable are ones that explore performance and data quality issues. For example in my current project the original developers decided to assume our server APIs always return nil data (hopefully with an error) or that they will return perfectly formed JSON, and if not, would crash. The truth is it's unlikely the server doesn't return poorly formed JSON for a "successful" request, but I wrote code to test 3,000 stores with various data inputs just to test a single API to find out if it did.
Of course I"m not writing a server API. If I was, I'd definitely be writing unit tests. But in a mobile app, there is plenty of work to do to improve the app directly that provide far more value than most unit tests can.