On CAST 2015 Selection
By Matthew Heusser – matt@xndev.com or @mheusser on twitter
I’ve heard it said that for any five years you are involved in AST, someone should apologize for two. Perhaps three. More recently, a respected member of the community said she “ran out of the patience” she once had for volunteer organizations.
Yet here I remain, pressing on.
I think it’s time I wrote something about this years CAST, the selection process, and my role in it as program chair.
Traditional Conference Methods
There are perhaps two classic ways to select conference speakers with a CFP. In the first, each program committee rates proposals on a 1-10 basis, averages and sorts. In the second, the team creates categories, like “reputation”, “speaking ability”, “mastery of subject matter.” 1-10, perhaps weighted, averaged and sorted. The first method is easy; the second allows the conference to give more objective feedback to those that don’t make it.
The second method also yields odd results, because there are factors that impact your judgement that are not listed on the form. It is more work and yields worse results; people end up fudging the numbers to get the speakers they want. The disincentives are too extreme.
One to ten overall isn’t much better. It is very hard to agree on a scale that high; your seven is my six. We end up arguing over why you gave this person an eight instead of a nine, and those thing points end up mattering. Yuck.
One thing we can usually agree on are three buckets – Yes, Maybe, and No. These are more defensible and clear. So we assigned three point scales – 1, 5, and 10. With four judges, the number of permutations, we thought, would give us a clear cut-line, without having to argue.
Oh boy were we wrong.
How it Played Out
Of 29 workshop or track slots, the first seventeen were 100% “yes.” That meant that if a single person said maybe, it drastically reduced your chances of being accepted.
I said maybe a lot.
Or at least, I was extremely careful about conflict of interest. Sometimes I had a business interest in the person who proposed – for example, one of our contractors applied and Carol Brands ran the first formal implementation of Lean Software Testing. In those cases, I abstained, we did not give a score, and the overall score was the average of the three other members. Erik did the same with the people who applied to speak from Hyland.
But then there were the people I knew, some I knew well, some I considered friends. There’s no business relationship there, but the very appearance of impropriety could damage the reputation of the conference. So, when those names came up, I gave a real hard look at the proposal. The number one question I asked was did the proposal match the format of CAST, which is historically first person experience reports and, more recently, workshops with peer learning.
I was probably too tough on my friends. Part of that was the format; I just did not realize how much damage a “maybe” could do. Perhaps I should have abstained. The net result was a few people who were newer got a chance to speak at CAST — and that was one of our goals.
The other thing we did was select a couple of talks below the cut-line – on purpose. We had a fully selected program, and then a few of the speakers were unable to make it to CAST. In these cases, we went to speakers with the next highest rating. Sometimes, these were talks where the speaker has a different perspective than classic context-driven, but seemed well thought out and explained. We told these speakers to expect tough questions during open season, including “That’s not testing.” The only way we can grow, as a community, is to change, and change means something different, so we brought in a couple ringers to spark debate.
Don’t worry; we told them to wear body armor.
The Bottom Line
Selecting the right speakers for CAST was incredibly challenging, more challenging, I believe, than judging the software testing world cup. Some of the people we rejected are fantastic and exceptional, and I hope we did not sour them on AST. I hope they do not lose patience.
If you think someone owes you an apology for your engagement with AST this year, well, I hope you will accept mine.
Now let’s press on together.
Recent Comments