What is an Appropriate Software Testing Standard?

Recently, I wrote a blog at the Software Testing Club on why I didn't sign the petition opposed to the ISO 29119 standard. In the blog, I laid out my opposition to the manner in which the ISO organization developed and published the standard. The process should have been much more visible with free drafts of the standard available for public review. As a result, I took the stance that I would simply not review the standard - either by paying for draft copies of the standard or by reviewing second-hand descriptions. That led me to the difficult position of not being willing (or ethically in a position) to oppose it.

The ensuing discussions in the comments were lively and interesting. I invited - and actually promoted - criticism of my decision in order to gain some insights into the discussion. Many of the answers focused on the mechanics of getting me (and thus others) to sign the petition, mainly based on the limited information freely available. I agreed with many of the arguments and actually said many times that indications are that I would oppose using the standard when it actually was published. All of the comments were valuable and I thanked everyone for contributing.

One extremely interesting topic that I think hasn't gotten a lot of press is whether testing can never be standardized (or regulated) in any way. In fact, a couple of times I actually asked that question explicitly. In general, no one was willing to tackle this question with a common answer being that testing is a purely intellectual process. That answer is helpful in separating the content of software testing from the goals or intent. One person actually said that they felt software testing could never be standardized. I laid some bait by actually stating the opposite stance: I had no problem with industries such as automobile manufacturers, militaries, and hospitals standardizing their software testing. (Note the lack of any qualifiers on my part!) Regretably, nobody took the bait. :(  That could have led to an interesting discussion to shed some light on the boundaries of the topic.

I think the question of standardization is an important question for two reasons. First, there are many heavily regulated industries existing now that governments and industries feel compelled to regulate in general. Secondly, the specific topics of automation (such as drones and self-driving cars) and machine intelligence will lead inevitably to the regulation of  software content. Will that regulation spill over to mandating testing for machine intelligence or control? How does that differ from mandating testing practices? Where do they overlap?

One point I would like to make here is that I don't differentiate between industry standards, government regulations, or internal corporate guidelines. Each address a given context (an important point that I will expand on later) and all mandate behavior of teams (developers and testers) in some way. The only difference between them is the authority behind the mandates. As a result, for the purpose of this blog, I will refer to them all as "standards" in a more generic sense.

There are many blogs written opposing ISO 29119 that give some interesting insights into the boundaries of standards. In general, the authors don't oppose standards in general. One interesting point was that many appropriate standards deal with interfaces and not content - very similar to the definition of software testing as an intellectual process mentioned earlier. That is a very important point - almost every standard I could think of did not deal with mandates of content or behavior and only with the interface. A good example of this is the US government Section 508 standards for accessibility. Although they contain a shopping list of development coding practices (such as using the alt-text fields in web development), these are treated as development suggestions and the overall implementation concentrates on functional testing of the products with actual users.

Based on what I have determined so far, here is a list that (to me) measures whether a proposed software testing standard would be appropriate (any criticisms or suggestions for expanding the list would be appreciated):

• Measurable - The standard should clearly outline the problem or issue it is attempting to resolve. It should be stated in a way that allows the creation of one or more metrics that can be used to determine the effectiveness.
• Context - To be effective, a standard has to have a clear context that shows the limits of the intended domain of the problem written above. If the standard is then extrapolated to other contexts, then some statement of the new context and problem description should be created. An example would be the US military adopting a standard to address weapon safety that was originally written to address patient safety in a hospital outpatient setting. This implies that a context of "universal" is inappropriate for a standard.
• Interface vs Content - The standard should address target goals and end effects and not mandate specific processes or methods. An example would be mandating sufficient testing to insure PII is protected in a banking industry environment. As with the Section 508 standards, it may address some suggested methods or practices for helping to achieve the goals, but implementation of the practices should not be used to measure compliance with the standard.
• Temporal vs Indefinite - Another indication of an acceptable standard is a mandate to evaluate the standard at pre-defined periods to evaluate the effectiveness and potential changes (e.g. every four years or so). That gives the targeted industry time to prepare potential changes and collect metrics to support those changes.

Based on what I know of the proposed ISO 29119 standard, it violates every one of these criteria I have outlined. Is that enough to finally sign the petition opposing the standard? I'll have to think about that.