Testing, Testing

By John Parkinson  |  Posted 10-28-2009

Ever since I started out writing code in the 1960s, I've been interested in the processes used to demonstrate that the code always does what it is supposed to do, and nothing else.

I even worked with methods that aimed to create correctness proofs for either designs or code based on designs. Even 40 years ago, this seemed easier to aspire to than to do, but at least back then we wrote almost all of the code.

Today, most of the code we use is written by someone else--and it can't always be tested very easily. My company's code base has become so large and complex that exhaustive testing isn't economically feasible, even if we figure out what an exhaustive test actually requires.

Modern code has millions (perhaps billions) of possible execution paths and program states, and we cannot test every unique combination. So we have evolved as an industry, consciously or unconsciously, toward a testing strategy based on a combination of materiality and risk.

Nothing wrong with that--and done right the strategy works pretty well. But to do it right, you have to do it consciously. And I suspect that in some shops this isn't always the case.

One consequence of the risk-based approach is that some defects can get into production. Most IT shops get pretty good at watching new code to make sure that there aren't any major defects that testing missed, but the older the code gets, the more confident we generally get that there are no big issues lurking.

This can be a mistake.

Over time, production code undergoes a steady process of changes. As hard as it is to test code when it's being developed, it's a lot harder to test it when it's undergoing maintenance or enhancement. Now you not only have to look for defects in the new code, you also have to ensure that it didn't break anything.

Large-scale regression testing is slow and expensive, and generally not feasible for every minor change that code is exposed to. And new code probably adds new and initially undetected defects to the code base. It might be a better idea to assume that as code gets older it gets worse, not better, in terms of defect density.

So if we want to deliver defect-free (or at least defect-light) code, what should we do?

First, as 20 years of experience should have taught us, it's better to prevent defects from occurring than to detect them once they've occurred. So design and coding processes should be as defect-resistant as possible.

There is really no excuse for poor design and coding practices, whatever approach we take. But requirements are a different animal, and no matter how well we define them, humans make mistakes, and misunderstandings occur.

So testing won't go away--it will just have to get more focused and more thorough.

We will also need to add more kinds of tests. The industry has always focused on functional testing (Do the functions work as specified and do they do what we expect?) and we have added testing for performance (Does it go fast enough?), integration (Does it play nice with everyone else?) and regression (Did I break anything?) over time. Tools for most of these areas exist and are steadily improving.

Now we have to add testing for security (Can it be hacked?) and scalability (What if we throw a party and everyone shows up?), and these are requiring new skills and tools to be effective.

Bottom line: Testing is becoming as necessary a "profession" as design and coding. Skills and experience matter. Process matters. Tools matter. Let the tests begin.

Back to CIO Insight