We all love automated tests, right? Unit tests, functional tests, all kinds of test are better when they are automated. Unless of course, they aren’t.
The general pattern of testing software is configure-execute-assert. That is, we attempt to set the software into some known state. That includes constructing particular objects (including parameters for methods), setting configuration and even building test databases and inserting known data. Then we poke the appropriate part of the system by calling a function or injecting a message or event. The we write some more code that accesses the state of the system again and checks it against some expected value. That expected value might be dynamic (like today’s date) or static (method should return 0). The simplest code to test can skip step 1 as there is no stored state (barring simple parameter inputs to the method) and the assert step is as simple as checking what the method returns.
The kind of test that won’t be squeezed to fit this pattern are the tests that involve users. You might very easily have a requirement that “the users of the website can find what they need easily”. This is an important requirement to be sure, but not a testable one. What do you do, ignore it? There are often less vague requirements like “the report should show the individual transactions on the front page when there are less than 10 of them, but they should be on another page when there are more than 10”. That is a very reasonable – and very common – requirement for reporting applications. Unless you have hand-crafted your own reporting framework then it is also impossible to make an automated test. The *stuff* that will make that report will be a mix of data, configuration in binary files, code and all of it will be hidden in the reporting system framework; the only input will be the data from the database, the only output will be the PDF report. Creating a test that relies on doing a diff of the actual PDF file with the expected PDF file is not really an option; the test will be brittle unless the report is very stable and the test will be constantly breaking as other parts of the report are updated.
So how do you automate? You don’t. At least, not fully.
You have to verify the requirement has been met and expecting the end-user to do all the work by clicking buttons is unreasonable. Particularly if you have to regression test. So what to do? My suggestion is that you only use people for the “assert” part and close up the rest with automation (See Scrum and XP from the trenches page 70).
So you make a test system that does the following for each test case:
- The configuration part. In this example, you have database entries that have 2 sets of report data; one with more than 10 transactions and one with less, you have the data stored in the test case so the user doesn’t need to remember it or input it.
- The execute part; the test system runs both reports
- The first part of assert is capturing the state; the system presents both reports to the tester in one screen
- The test case has a full description of what is expected; maybe including a diagram that shows what the tester is to check on the reports. Maybe some samples reports linked in.
- Some buttons to say “pass” and “fail” (and maybe “test broken”) that the user can click to store the result and move to the next test (N.B.: it would be good to store timing data too, be nice to know if some tests are ultra-time-consuming)
This means that the human testers are not wasting their time doing things the system can do and you don’t waste your time trying to program things that a computer can’t do. One example of this human computation in a talk from google is image recognition which simply can’t be done at present by computer, but humans can do very fast. This kind of Google approach to mass-collaboration also gives us a way around systems with hundreds of test cases. One person would go mad if they had to do them all, but if you provided a nice way for dozens of people to all take a few test cases each then no one would get too demotivated or lose concentration and the cycle-time could be kept short.
Of course, minimizing human testing is important as even the smoothest human computation is hundreds of times slower than true automation. Also, these tests would generally be system integration tests and whilst they are more important than unit tests, they won’t be helpful for catching all the bugs. So it is best to save the human-driven regression tests for the big releases and the parts that other testing approaches can’t reach.