Q&A: Why testing guru James Bach is against the ISO 29119 testing standard

James Bach is probably best known for his role in co-developing and advocating for what’s known as the Context-Driven School of Testing.

Among other things, that line of thought essentially holds that effective testing can be done only through judgment and skill. “There are good practices in context, but there are no best practices,” one of the School’s principles says.

So it may not be surprising that Bach, who runs an Eastsound, Wash.-based testing education and consulting firm called Satisfice, is a firm opponent of a would-be standard for software testing called ISO 29119.

What follows is the first of a two-part interview I did with Bach about ISO 29119. I also chatted with another opponent of the standard, James Christie, and we’re running his interview separately.

To begin, Bach gave me five reasons why he’s against the testing standard.

Question: Why do you oppose ISO 29119?

James Bach:

1. The first reason I oppose it is that the process by which it was created is illegitimate. ISO has no more right to declare its document a standard than I have to declare my Rapid Software Testing methodology a standard. To make a real standard, there must be good evidence of consensus about practice in the field, and of course there must actually be some sort of consensus to collect evidence about. Neither of these things exist.

ISO could have used its resources to fund real sociologists to perform systematic research into the practices of our industry. This has been done before (see Harry Collins’ studies of gravity wave scientists,) but of course it is expensive and it takes years to do properly. And all they would find out in the end is that there are a great variety of practices and opinions about practices in our field. At that point they would know that there is no basis for a standard.

2. Software testing is not an activity that ought to be standardized, as such. There is no reason to standardize it, any more than there is a reason to force programmers to use one programming language or to force everyone on Earth to speak English. In our industry, testing and development practices are evolving, and standardization inhibits evolution. Software testing is a technical and social activity that takes different forms in different contexts. Contradictory ideas about how to test are competing with each other in the free market and that is how it should be.

3. The imposition of a “standard” by one faction of the testing community, apart from being illegitimate, has the potential to allow them to control who gets to call themselves a tester. The audience for their project is not other testers, but rather government organizations and large consulting companies that have little interest in testing and are not aware of the controversy in the field, nor the many options in the market for how to test.

4. It’s important to note that this “standardizer” faction has not been able to win on the merit of their work. They could make their ideas public and promote them at conferences and in training just like the rest of us. I have no problem with the sort of de facto standardization that comes from people freely adopting ideas they like. If these ideas gained acceptance naturally, that would be perfectly fine. But this faction has failed in that project since 1972. Now they want to change the rules of the game by declaring their ideas the official way to do testing.

My faction of the testing industry is content to compete in an open marketplace. We are not asking for any special status. Furthermore, we are happy to engage in debates with other the other factions. The position of the standardizers does not bear up under informed scrutiny, and indeed they have so far refused to respond seriously to our concerns. Dr. Stuart Reid recently scheduled and then backed out of a public debate with representatives of the International Society for Software Testing, which opposes the standard. (Ed. Note: Reid’s response to opponents of ISO 29119 is here.)

5. Bad software testing wastes an amazing amount of money (I have no statistics, except that for a couple of decades now I have been wading through this waste as I visited hundreds of companies and see what they do), and still companies are chronically surprised by security holes and other problems in their products. Apps are updated with bug fixes so often that a typical iPad user upgrades something almost every single day.

It is the responsibility of good testers everywhere to oppose this standard and preserve professional freedom, so that we can continue to strive to do our best work.

Look for Part II of our interview with James Bach next week.