Context Driven Testing and ISO 29119:
A Matter of Principle
The Association for Software Testing is explicitly Context-Driven. The Principles of Context-Driven Testing tell us that any practice’s value or effectiveness is determined by the context in which it is applied: the business drivers, risks, schedule, environment, and most importantly, the people involved in a project.
A well-understood corollary is that a given practice could work well in one context, but be inappropriate, ineffective, or even dangerous in another. Once this is understood, it’s easy to see that it is wrong to prescribe any practice as appropriate for every circumstance, yet ISO 29119 explicitly claims it “can be used by any organization when performing any form of software testing.”
On that basis alone, ISO 29119 is a mistake, without consideration of its content. We have four categories of additional objection to the standard, and briefly discuss each of these here.
Regulatory Theory and Practice
As with all practices, context should be examined to understand the standard’s purpose, effectiveness, and risks. Standards have credibility and huge influence simply from their status as standards. If we are to have standards it is essential that they are relevant, credible and framed in a way that does not mislead investigators. Every standard is potentially an auditing and regulatory instrument, so these consequences must be considered in any discussion of standards.
Consider the difference between principles and rules: principles provide guidance, rules dictate what must be done. We believe that the complex, context-dependent, and cognitively demanding activity of software testing would be more effectively aided by a set of general principles, leaving the setting of specific rules to particular organizations and settings (contexts). A standard appropriate for software testing would therefore be brief, clear, and non-specific. ISO 29119 is not this kind of standard.
One common argument for standards is the requirements of industry regulators. Members of our community have experience working in these contexts and meeting these requirements. We suggest testing standards should be framed as principles that are consistent with the requirements, not a series of new rules that must be reconciled with them.
Another justification of standards is that testers should be accountable to stakeholders: testers must demonstrate that they are providing valuable information, effectively and efficiently. We agree accountability is important, but do not believe that standards meet this need. Consider that the auditing profession does not demand generic, prescriptive testing standards. The 1st edition of the Institute of Internal Auditors’ Global Technology Audit Guide described the Snowflake Theory of IT Audit: “Every IT environment is unique and represents a unique set of risks. The differences make it increasingly difficult to take a generic or checklist approach to auditing.”
The Nature of Software Development and Testing
Testers must offer more valuable information about quality of products than how many test cases were run and passed. Testers need to tell a valuable story, not fill in templates.
Software development and testing are inherently complex, and therefore “best practice” and prescriptive standards are inappropriate. A flexible, iterative approach is the state of the art, and has been for over a decade. This makes a heavy investment in advance testing documentation a risky investment, and without detailed documentation of requirements, perhaps impossible. Yet, standards still demand heavy investment in planning and documentation before the software has been written.
Good testing is a process of exploration and learning about how the product works. Testers must offer more valuable information than how many test cases were run and passed. Collecting the status of pass/fail checks is an appropriate task for automation. To add value, testers must tell a valuable story.
The Social Sciences
Prescriptive standards do not take account of how people acquire skills and apply their knowledge and experience in cognitively demanding jobs such as testing. Consider “tacit knowledge” – the knowledge we have and use but cannot articulate. Creative professionals such as software designers or architects have an iterative approach to developing ideas; much of their knowledge is tacit. They can’t write it all down as a neatly defined process, because they make many situational judgements based on their tacit knowledge. To gain access to what they know, they must perform the creative act so they can learn, reflect on what they’ve learned, and then apply the new knowledge.
Another risk of prescriptive standards is goal displacement: losing sight of the true goals, and instead focusing on conforming to methods, processes, and documentation. Stakeholders care about results. Foisting prescriptive standards onto skilled people is counterproductive, and encourages them to concentrate on the means rather than the ends.
In order to cope with the stress and anxiety of difficult jobs we build “social defenses” to help us cope. We use methodology and prescriptive standards to avoid the risks and uncertainties of real engagement with people and problems. This might make the work easier to manage, but it distances us from our true role and reduces the quality of our work.
Effective two way communication requires effort. The anthropologist David Graeber argues that the greater the degree of force or compulsion, and the greater the bureaucratic regime of rules and forms, the less the communication. Those who issue the orders don’t think they need to understand the complexities of the situation they’re managing, and don’t take the trouble to.
The Need for Competition
ISO is the global leader in standardization. Any standard it issues has a huge advantage over alternative methods, because of the ISO brand. Any ISO standard is likely to be referenced in contracts by lawyers and managers, introducing compulsion. It is vital that any testing standard is both credible and widely accepted on its own merits. Our view is that a detailed, prescriptive standard cannot meet those tests.
Buyers of testing services are often ill-informed about the quality of the service that they buy. Any prescriptive testing standard will help sellers of poor testing services to sell plausible services. If the standard is used as a shield for credibility, purporting to guarantee good results, it will be harder for sellers of high quality testing services to gain a worthwhile price, and the quality of what is sold will drift downwards when testing is commoditized, or sold on price rather than quality.
In Conclusion
As practitioners and advocates for the profession of testing, we oppose the creation, publication, purchase, and implementation of ISO 29119 or any other prescriptive standard, on principle.
We disagree with the notion advanced by some defenders that the standard should be accommodated as a “take it or leave it” collection of suggestions. It was deliberately published as a Standard because of the connotations, implications, perspective, motives, authority, and power that any ISO Standard would have. Denying that ISO 29119 has those meanings baked in is disingenuous at best.
James Christie and Eric Proegler
Other Resources
James Christie has written and spoken extensively about ISO 29119. Huib Schoots has compiled a comprehensive collection of links on ISO 29119 responses
Recent Comments