Quality Is NOT Equal To Testing
What’s the biggest cultural change that companies who use CAST undergo?
That is the question that Lev Lesokhin and I were asked last week. We were talking to Margo Visitacion and Mike Gualtieri of Forrester.
The answer: The realization that software quality is not equal to testing.
There’s a light switch that flips when organizations realize there’s much more to quality than functional testing. There’s non-functional testing, and even beyond that, “dependability testing” (to borrow a phrase from our Chief Scientist, Bill Curtis).
Let’s have a look.
Everyone realizes that functional testing is nowhere close to enough. If all that mattered is the *what*, then every car that lines up at the start of a race will win — after all, they all satisfy functional specifications!
But winning the race is not only about what you come to the starting line with. It also depends on how well that thing works during the race! In fact, you can have the car that satisfies the functional specs the best but fail to even finish the race — just ask poor Sebastian Vettel and the Red Bull Formula 1 team!
There are two ways to tackle the “how well”.
The first way is to make sure the car performs in race conditions. That’s the equivalent of non-functional testing — you simulate real-world conditions as best you can and fix the problems that appear.
To have any confidence in such testing, you must be confident that your simulation replicates race conditions (or the critical elements of it), you know what to test, and you know how to interpret test results and use them to improve (rather than just have terabytes of test data sitting in a data warehouse somewhere).
The dirty secret of non-functional testing is it’s too little, too late. The result: low confidence in how this thing will perform when the rubber hits the road. Production problems. Business disruption. A ruined business case.
No amount of non-functional testing can give you confidence in the car’s dependability. Ensuring dependability is the second way to go beyond functional testing.
Dependability is about how the car will perform in those conditions you haven’t yet tested. Can we overtake on turn 3 if the tank is quarter full, the tires are worn, it’s beginning to drizzle, and the wind blowing from the south east at 34mph?
Dependability is about how the car will perform in conditions you couldn’t possibly test for. How quickly can you make a gearbox adjustment to help your driver on a straightaway made slick by an oil spill? What if you had to use a non-factory replacement part to do it? Will the car still perform well enough to win?
So, you’ve spent tens of millions of dollars on the stuff. The business case depends on it performing up to snuff. You’ve tested it as best you can. Yet, the day you roll this out feels like a roll at the craps table.
To make it feel less like that, you need to have dependability – the confidence that you have an effective plan for the unknown unknowns.
The realization that quality is not equal to testing fundamentally changes the way IT organizations develop, enhance and maintain business applications. It fundamentally changes the way they manage their software assets.
Yesterday CAST hosted a breakfast for IT leaders in government agencies – one of the themes was moving applications to the Cloud. I was surprised (and relieved) to see so many agencies represented – FEMA, DHS, SEC, NASA, and the Defense Intelligence Agency just to name a few. 