Archive for May, 2012

Does Karl Popper have anything to say on testing?

Posted in Uncategorized on May 24, 2012 by Crazy Eddie

I have noticed a tendency among many to hold what I consider a misunderstanding about testing, especially unit testing. For example, when I brought up the topic of testing critical sections, the first response and many after the fact seemed to focus on an inability to prove, through testing, that a multi-threaded component contains no bugs. Being the empiricist that I am, I tend to find this kind of response and this assumption about testing to be extremely questionable at best.

In some respects, and at a very theoretical level, computer programming is an a priori application of logic. Functions are mathematical constructs of pure logic that at least technically can be proven to be correct or incorrect through the rigorous application of predicate logic. In the older days I gather that this was the primary view of software development and programmers proved their code from start to finish.

Nobody does this today; nobody sane anyway. There are still a few adherents to the idea of proven code but instead of proving an entire program they prove individual functions. This seems a lot more realistic and practical, but it still runs into a fundamental problem with a priori understanding: the outcome of your proof is dictated more upon your understanding of the problem than on the reality of it.

Almost all developers in today’s landscape have learned that testing a product before releasing it is an absolute requirement. Many have also learned that you need to go even further than this and test each individual module or unit within the product as well. This is, however, a completely different approach from proofing and would be utterly unnecessary if proofing were possible or practical. Modern software engineering though has become way too complex to support the idea of proofing as a practical exercise and the difficulty in predicting behavior in parallel processing has perhaps made the idea ridiculous as well.

We don’t need to assert the impracticality or impossibility of proving a program at this point though. What only needs to be considered is how different the approaches between proof and testing are and to realize that while one method is entirely philosophical in its application, the other is empirical and scientific. Although math is considered a science by many, it is actually more in line with philosophical thinking and mathematical formulas, no matter how correct they seem…must always be checked against the real world to make sure the universe they model is the one we live in.

If proving your code is the philosophical approach then testing is the empirical one, the scientific one. I believe it worth reviewing then what this means in other sciences and what the philosophy of science itself says on the matter–unlike what Lawrence Krause might say I think the philosophy of science has done a great deal to improve the practice of science itself, especially the views of Karl Popper who did much to hone science into the workhorse it is today.

The definitive work on modern science and how it differs from pseudo-sciences like astrology was laid out in Popper’s article, Science as Falsification. Through it and other works he explains how science increases human knowledge not by showing what is true, but by showing what is false. We know that bricks of gold don’t fly not because we’ve proven that they don’t…but that we’ve shown that they do not. Relativity is viewed as true not because it’s been proven so, but because it’s the only theory that has not been proven false yet under most conditions. When a scientist comes up with a new theory it’s because they’re showing some previous theory wrong. When they test the theory they don’t go looking for ways to validate it, they look for ways to falsify it and if that cannot be done then the theory stands until it can be. This is the main reason why unfalsifiable statements are not scientific and I would go further to assert that they’re nonsense anyway.

This is the view that people should have toward software testing. When you’re writing your tests, be they integration, unit, or some other level of testing, you should focus not on proving that your code works but on showing that it doesn’t NOT work. Your code is your theory about how to solve a particular problem; your tests should attempt to falsify that theory. You should not view your tests as validating or justifying your code, but as a continuing effort to fix code that fails to work in some way.

With this in mind then it would become absurd to claim that tests prove code works or that tested code contains no bugs just like it is absurd to claim that any one of our scientific theories represents the whole and entire truth. Science works better than anything else for the very reason that it does NOT work in proofs nor make grandiose claims of absolute knowledge. Science works because the exercise of filtering away bad ideas, mistaken assumptions, and incomplete understanding sharpens our ability to manipulate the universe toward our own ends. Software development, as both a science and discipline of engineering, can and should be approached with the same understanding.

Advertisements