We recently ran the first Rapid Testing Intensive on Orcas Island. One of the outcomes of this, as I have promised, is a public test report on what happened. This is my report report. I am reporting on my reporting…

What was the “Rapid Testing Intensive”?

Oh, it was just the most realistic and ambitious test training event ever. Sorry if you missed it. We (17 testers onsite and nearly ninety online) tested a part of eBay for four solid days, finding many important bugs. But it was more about training and coaching than bug finding.

Where am I with this report?

I am actually creating several reports. One is a report about our process. This is meant for people curious about Rapid Testing methodology, and in it I will contrast what we did with the “traditional” approach. I will also create a report for our primary client, eBay, whose software we tested.

There’s a lot to do and I am plugging away. I just finished the usability testing roll-up report. I’m going through 320+ bug reports now. Here’s my overall todo list:

Analyze HipChat traffic

  • who was talking most?
  • who was being responded to most?
  • when were people talking most?
  • were conversations more social or technical?
  • what proportion of the conversation were people seeking information, and what proportion were people providing information?
  • was HipChat helpful in coordinating a worldwide testing effort?

Analyze bug list

  • Assess bug report quality
  • Reproduce bugs
  • Close duplicates and non-bugs
  • Add labels and categorize according to risk
  • What did eBay know about before we started?
  • What does eBay consider important that we found?

Produce “official” test coverage outline

  • Analyze student TCOs

Produce “official” Risk List

Produce official sequence of events

Describe our test activities

  • Analyze student test session reports

Describe what we did not test

Analyze tools used

Identify outstanding student work

Write report narrative and summary

How “Rapid” is this?

Yeah, this seems like a lot of work and yes it will take a while (I’m working evenings on it while teaching during the day). But this is because it’s a seminar and demonstration project. In a real project, our reporting (other than the bug list itself) is either oral, or expressed in brief written statements. On a normal project, once we get organized (after a few days if it’s a new project) reporting gets pretty easy.