Everyone is talking about “context driven testing” like it’s a revelation. I’m seeing lots of people talk about it at events. Even the other week, at Cambridge lean coffee, someone posed the topic “What is Context Driven testing?”, to which no-one around the table seem to have an answer, although not for want of trying! Several of us have gone through courses run by leaders in the context driven testing community, and others are active contributors in the testing community – but it became apparent that we did not have a clear answer to this fairly direct question. Maybe more fundamentally, we’re struggling with how (or “if”) OUR testing practices differ from this current ideal.

Surely ALL testing should be context driven? Without context nothing (metrics, bug reports, user research) has true meaning, and never has. As the title shows, even basic words in the English language have different meanings and pronunciations depending on the context they are being used in. Context matters, even in the basics of language. Am I wrong in thinking other people realise that context is important in most situations?

The Graduate Theory

On the one hand, this might be a reaction against (or training for) people new to testing, who are blindly following instructions. To people who are merely going through the motions and doing what they are told without questioning why, or maybe just implementing what they have learned, perhaps very badly or at least blindly. Maybe some testers don’t quite progress beyond this stage in their understanding of their profession.

When you’re learning a new skill (testing), you often need structure and boundaries, but maybe many new testers fail to progress beyond those boundaries and achieve true competency, developing to apply and adapt those skills based on the needs of the task at hand.  Is this what dictates the need to identify that context matters?

This article talks ShuHaRi, which is term used to describe the progression of learning.  It talks about the initial stage Shu where you are learning how to do something, without worrying too much about the theory, then Ha where you learn the principle and theory behind the techniques and finally the Ri stage where you learn from your own practice and adapt as you go.  So is context driven testing as a concept aimed at people in the Shu stage of testing or or is it a way practice for all to learn.  If it is a practice for all to learn the what are the basic principles of context driven testing that people need to grasp to enable them to successfully progress and how do they differ from other testing concepts?  Or is context driven testing really the name for people who have reached the ri stage in their testing abilities?

If it is the case that people are not graduating from this initial introduction of skills, than does having a term that seems almost like a job description aid these people in understanding what the crux of context driven testing is? Likewise do these ‘lay’ people make up the majority of testers attending conferences and getting involved in the community?  Even if the graduate theory holds true, and large swathes of our profession are just going through the motions, then how come so many of us, new and old to testing, do not feel equipped to answer what appears to be a simple question? What is context driven testing?

Is the term “context driven” being used to emphasise that the context in which a system will be used should force you to adapt and shape your approach techniques to match? Is it to remind people of the 5 W’s and a H that we should be asking so that we know that everyone involved in the product has the same understanding of the product and it’s aims?  

The Cynical Theory

It’s also possible that I simply enjoy understanding the systems I work with, and so my testing has always been framed and contextualised by what I know, and so what seems obvious to me is a revelation to many others. As a result, I should probably call this the “Maybe I am just lucky” Theory.

Even when I started (*cough cough* 15 years ago), I worked closely with developers to understand the products we were building and the changes that we were making. We sat in separate areas but it was a close, communicative relationship. I had the opportunity to go to customer sites and work with them, seeing how they were using the tools and the struggles they faced. We had clear build systems, with automated tests being written by both the developers and the testers, as well as a clear set of other test to run.

I’ve been in the enviable position where I’m invited to get involved from the start of a project right through to demo stands at conferences and customer visits. It’s possible that many testers aren’t in this position of inclusion, and just run through comprehensive checklists defined by their managers or the dev team around them. In that case, have we really only just realised that meaningful context yields significantly more effective tests?

This still doesn’t give an obvious answer to “What is context driven testing?”. Can I just call myself a context driven tester? Or is this, disappointingly, more about a new term for something good testers have been applying all along, and seems to be encouraging a segregation in the testing community? Which is my roundabout way of asking “Is this ultimately all about speaking and consultancy fees?”

The Underlying Point

When you first learn something, I absolutely agree that you need information boundaries to enable you to do directed, structured learning. However, if many of us within the software testing community are uncertain as to the actual definition of this term, then I begin to wonder whether those boundaries or explanations missing or maybe we’re looking for a deeper answer then actually exists.

I recently read Rethinking Expertise and encountered Harry Collins and Robert Evan’s idea of connoisseurship in respect of technical pursuits. This is taken from http://reagle.org/joseph/2009/01/CollinsEvans2007-expertise.html

Technical connoisseurship

“experience within the conventions of judgment rather than experience of the skill itself.” “turns on on interactional expertise alone”.

e.g., architect can recommend tiles, even if no tiling experience

Hearing about connoisseurship made me wonder how one gauges one’s own expertise in relation to others – do we just have to wait to be judged by connoisseurs who are somehow collectively nominated? Is there some test we need to pass to call ourselves a ‘context driven tester’? Even if we change the title, what is to say we are not hoaxers or posers or fakers?

Possibly the path I was lucky enough to follow in my career was not the norm, and this may be why I struggle to empathise with the need to distinguish this delineated approach from ‘good testing’.

I just don’t understand how Context Driven Testing is a new idea. Or at least, I firmly believe that it really really shouldn’t be. To test whether anything works, you have to know what it’s supposed to do, and that surely requires more than a modicum of context. There’s a critical distinction I want to make here (in case it’s not been obvious): My not understanding the term is not the same as not advocating that context is a pivotal part of determining what and how to test on a given project. I’m just not sure that changing the way I identify myself is the intention behind the phrase “context driven testing”. Should I change my job title to make it clear how I work, or should we assume that all testers are CDT unless specifically stated otherwise (Graduates & learners aside).

Overall I am just left feeling like I have missed something really important, or some critical nuance has passed me by :o(

Tagged: “context driven”, agile, Testing