Tools are not Judges: 30 Days of Accessibility Testing

by | Syndicated

The Ministry of Testing has declared that May should be “30 Days of Accessibility Testing“. As in the days of yore when I used to take on these challenges and blog regularly, I’m in the mood to get back to doing that. Therefore, I am looking to write a post every day around this topic and as a way to address each line of their checklist.


12. Read an article about accessibility and share it with someone.

There’s a lot of benefits that we can get from automation. It sequences a lot of repetitive tasks. We can run thousands of steps the same way or in random ways, over a period of time, that would drive a person insane. It lets us create assertions to see if an element is present or not. No argument from me, there’s a lot of good stuff that automation does. However, there are a few things that automation does not do well, and I have mentioned this in the past with talks I have given. One of the areas I’ve commonly stated as lacking by automation is that it cannot make a judgment call. 
Karl Groves article “Automated Web Accessibility Testing Tools Are Not Judges” takes a similar stance. Here’s an example that I feel provides a nice commentary on that:

“One thing that is missing across all automated testing tools, is contextual understanding. Automated accessibility testing tools provide the ability to test against highly specific heuristics that are very tightly scoped. They have no insight into the broader document being tested. They have no ability to determine the specific purpose of the page as a whole. I have been present at usability tests where test scenarios failed because of one highly important technical failing that – within its own context – was relatively minor. Nevertheless, it caused all participants to fail the test scenario.”

[…]

we can’t even definitively prove conformance with any given WCAG SC, so grading any specific SC as a “pass” is wholly impossible. This is important to consider, because it means that even if your tool of choice returns zero errors, you can still be non-conformant. A picture of a dog, with an alt attribute of “cat” will pass all automated accessibility tools, despite being completely inaccurate even in terms of what is displayed in the image.”

I encourage everyone to read Karl’s article, as it has some great food for thought.