Principles of Software Testing

“First of all, testers must want software to fail.” – Udacity CS258 John Regehr
See the original video here.

1. First of all, testers must want software to fail.
2. Second, testers are like detectives who are hunting down bugs.
   • And as detectives, testers will have to be observant to all sorts of suspicious behaviors and anomalies in the software of your test.
3. All available test oracles should be used as a basis for testing.
   • All of the oracles should be used because they can all detect different kinds of faults, and even if they detect the same faults, weak oracles might be much cheaper to use.
4. Test cases should contain values selected from the entire input domain
   • and if there’s doubt about what exactly that domain is, this is something that will be good to hash out with the developers.
5. Interfaces that cross a trust boundary need to be tested with all representable values, not just those from the ostensible input domain.
6. A little brute force goes a long ways when we’re testing.
   • And what I mean in particular is in certain restricted circumstances, we can do exhaustive testing
   • and almost anything else can be randomly tested.
7. Quality cannot be tested into bad software.
   • So in contrast with examples like a fair at , testable software has a few of the following qualities:
   1. No hidden coupling between modules and side channels where modules can share information without being visible to the system developers.
   2. A few variables but are shared between threads.
   3. Few global variables shared between modules.
   4. And no pointer soup, and that is to say, no huge data structures with pointers going everywhere where it can’t possibly keep track of who’s changing what and what’s valid and what’s not.
8. Codes should be self checking whenever possible using plenty of assertions. However
   • these assertions are never used for error checking, rather they’re used to check for logically impossible conditions. It implies some sort of an internal consistency violation.
   • Assertions must never be side effecting, because if they are, and you turn them off, the system behavior will change, and this leads to madness among developers.
   • Finally, these assertions should never be silly or trivial ones
9. When appropriate, all sources of input to a piece of software under test should be used a a basis for testing.
   • And those include the obvious API’s, provided by the software on your test which can be tested directly.
   • API’s used by the software under test can be tested using fault injections techniques. So recall that these are things like substituting the library that provides these API’s with a different library the inject faults, well perhaps just hacking the layer underneath to inject faults.
   • Finally, non-functional inputs such as thread schedules need to be tested using whatever method you can get that will work to actually test these things.
10. And finally, the last principle for testing is that failed code coverage do not provide a mandate to cover the failed items
    • no matter how tempting that might be, but rather, they give clues to ways in which the test we did is inadequate.