Scripted Automation as a Magic Eight Ball

First, a little background. In American billiards, the game of "Eight Ball" is a game where the main goal is to sink the number "8" ball after you have sunk all of your others (either striped or solid). However, if you "scratch" (sink the cue ball) while trying to sink the eight ball, you lose the game. As a result, the game outcome is always in doubt.

A brilliant person once thought of the idea of creating a "Magic Eight Ball". This is a toy that looks like an eight ball, but has a flat side with a window in it. Inside is a fluid with a multi-sided object that has a phrase written on each side. You ask the Magic Eight Ball a question, shake it, then turn it over and see the displayed answer, which is always a vague answer like "It is possible" or "Who knows?".

So, how does this relate to scripted automation? A major problem with the larger test tool suites is that they are set up to relate script outcomes directly to requirements on a pass/fail basis. This leads to automated reports that provide coverage in terms of requirements passed, etc. This can be not only misleading, but dangerously so. It inevitably leads to a false sense of security or panic that degrades the credibility of the test team.

Scripted test tools do not test or verify functional requirements. Instead, they check specific parameters at specific interface points. When I report scripted results, I only report a "negative" result and always state what was expected and what was measured. That allows me to quickly validate the test failure before entering a bug.

If you have to associate tests directly to functional requirements, you can't say it "passed" and you definitely can't say it "failed" without a verification. Instead, it would be better to phrase it in less absolute terms. For passing tests, you could report that "The outcome is unclear" and for failing tests you could report "The signs are troubling". That way, no one jumps to conclusions and you end up performing exploratory test sessions to provide more specific outcomes. Add a little randomization and ... voila!  The Magic Eight Ball test automation suite that combines the simplicity of direct association to requirements combined with just enough uncertainty to insure thorough testing.

I wonder if I could add a tarot card interface for projected test completion dates? Hmmm ...