Yesterday’s Lean Coffee was hosted by Jagex.  Here’s a brief note on the topics that made it to discussion in the group that I was in.

Automated testing.

  • A big topic but mostly restricted this time to the question of screenshot comparison for web testing.
  • Experience reports say it’s fragile.
  • Understanding what you want to achieve with it is crucial because maintenance costs will likely be high.
  • Looking to test at the lowest level possible, for the smallest testable element possible, can probably reduce the number of screenshots you will want to take.
  • For example, to check that a background image is visible for a page, you might check at a lower level that the image is served and assume that browsers are reliable enough to render it rather than taking a screenshot of the whole page which includes much more than the simple background image.

Why go to a testing conference?

  • It builds your confidence as a tester to find that other people think similar things, make similar decisions, have similar solutions.
  • It also reassures you when other people have similar problems.
  • You are exposed in a short space of time, in a sympathetic environment, to new ideas or new perspectives on old ideas.
  • You can meet people that you’ve only previously followed or tweeted at, and deepen the connection with them. “Testing royalty” is accessible!
  • When you come back, sharing what you found can clarify it for you and hopefully make positive changes to the way you work.

Strategies for compatibility testing.

  • Experience reports say that there’s reasonable success with online services – to avoid having a stack of devices in-house – although not when high data throughput is required.
  • Reduce the permutations with a risk analysis.
  • Reduce the permutations by taking guidance from the business. What is important to your context, your customers?

How do you know which automated tests to remove?

  • Some tests have been running for years and never failed. This is wasting time and resource. 
  • Perhaps you shouldn’t remove them if the impact of them failing is considered too serious.
  • Perhaps there’s other ways to save time and resource. Do you even need to save this time and resource?
  • Can you run them differently? e.g. prioritise each test and run higher priority tests with greater frequency?
  • Can you run them differently? e.g. run only those that could be affected by a code change?
  • Can you run them differently? e.g. use randomisation to run subsets and build coverage over time?
  • Can you run them differently? e.g. run every suite frequently, but some configurations less frequently?
  • Chris George has a good talk on legacy tests.

Why isn’t testing easier?

  • We’ve been testing software for decades now. Why hasn’t it got easy?
  • It’s bespoke to each solution.
  • Teams often want to reinvent the wheel (and all the mistakes that go into invention.)
  • You can’t test everything.
  • Complexity of the inputs and deployment contexts increases at least as fast as advances in testing.
  • Systems are so interconnected these days, and pull in dependencies from all over the place.
  • People don’t like to change and so get stuck with out of date ideas about testing that don’t fit the current context.