Why Not a Testing Standard?

by | Syndicated

The Cambridge Tester Meetup last night was a discussion on testing standards. Not on the specific question of ISO 29119 (for which see Huib Schoots’ excellent resource) but more generally on the possibility of there being a standard at all. It was structured along the lines of Lean Coffee with thoughts and questions being thrown down on post-its and then grouped together for brief discussion.

I’ve recorded the content of the stickies here with just a little post-hoc editing to remove some duplication or occasionally disambiguate. The titles were just handles to attach stickies to once we had a few in an area and I haven’t tried to rationalise them or rearrange the content.

Enhance/Inhibit Testing

  • Testing is a creative process so can’t be standardised.
  • Testing doesn’t fit into a standard format, so how can there be a standard for it? (Do we mean “good testing” whatever that is?)
  • New tools, technology might not fit into a standard.
  • Standardisation destroys great design ideas by encouraging/forcing overly broad application.
  • Can a general standard really fit specific project constraints?
  • Each tester is different.
  • A standard limits new thinking.
  • Could a standard be simply “Do the best you can in the time you have”?

Who Benefits?

  • Who do certifications serve anyway? What do they want from them?
  • As litigation becomes more prevalent who is protected by a standard? Customers, producers, users?
  • With a standard, companies can be “trusted” (QA-approved sticker).
  • People outside of test are usually very opinionated. Do standards help or hinder?
  • End users care because of the possible added costs.
  • A testing standard would provide false reassurance for companies.

Communication

  • How does an agile team fit in the standard?
  • Too much documentation? Standards may cause the need for more documentation to show compliance.
  • Standard language for communicating test ideas.
  • Divide the testing community – good or bad?
  • Respond to feedback and criticism.

How Much? or Alternatives

  • Do we need an alternative at all?
  • Where are the standards for science, consultancy, product management, development?
  • Use as much or as little of a standards as needed?
  • Could a standard be subjective?
  • Standards for products, or the process of creating products?
  • What else do we need or want instead?
  • Could a standard cover the minimum at least?
  • A standard should be flexible to adapt to project constraints.

Useful Subsets

  • Can a single standard fit different products? (Angry Birds vs nuclear reactor).
  • Uniformisation of some testing (bring up the baseline).
  • There are already some government standards.
  • Infinite space of testing. Can a standard capture that?
  • Can some aspects of testing be covered by standards? If so, which?

Can’t we Just Explore?

  • Scientists do. Why can’t we? (But what about mandated science?)
  • Approaches in methodologies used set out in a common understood format could help consistency.

Fear of Being Assessed?

  • Are testers just scared of being evaluated or taking responsibility?
  • I’m too shy.
  • Could it open up law suites, blame and other consequences?
  • Should you insure yourself or your company against any not conforming to the standard?
  • Anything unstructured used as an addition to, rather than part of, the primary approach. Stops people hiding?

Show Me the Money?

  • What is the motivation of those seeking to create certification? (Rent-seekers?) 
  • It’s just to make money for ISO companies.
  • Adds organisation to a “messy” activity.

Certify Testers Not Testing

  • Can you differentiate certifications for testers from certifications for pieces of work? (c.f. Kaner) 
  • Can you say “product tester by a tester certified XYZ”?
  • How would recruitment distinguish between testers and checkers?
  • An independent body to audit the testing/tester on real project work? (Who audits the auditor?)
  • Qualification vs certification vs standardisation.

Standards in Other Industries

  • Learn more about standards in other industries and how they dealt with their first standard.
  • Standards in e.g. car safety are on the result of the work not the methodology? 
  • Universities and schools start teaching testing. Should they teach about the standards?
  • Standards to help produce evidence of testing not just test plans, which are usually fiction.
  • “Informed” standards (courses, talks etc), “in-house” standards?

Misc

  • Are objections to certification objections to theoretical risks but in practice it’s possible to have something good enough?
  • Would companies without testers need a testing standard?
  • Development standards to be closely linked to testing standards.
  • Easy to find jobs abroad (if there were standardisation).
  • A standard would be good as a product.
  • Would a standard really impact our day-to-day job?
  • Is the standard simply a reason to justify testing?
  • Is the idea of a standard predicated on an outdated idea of testing?

As you can see there was no shortage of ground to cover but, with only a couple of hours, plenty was necessarily shallow or not dug into at all.

To pull out a handful of  points that I found particularly interesting: we were not shy about asking questions and we were prepared to aim them at ourselves; we bumped into the distinction between certifying product, tester and testing multiple times; we didn’t really explore what we meant by standards, certification and qualification and what the differences between them might be; while the discussion was entered into with an open mind (which was the remit) there were sometimes implicit assumptions about what a standard must entail (inflexibility; lots of documentation etc) which were mostly negative and where positives were proposed they tended to be viewed more as possibilities.

P.S. There’s a few photos.
Image: https://flic.kr/p/7rddEr