Paul Holland picked up where lunch left off, with some basic reminders.
Paul Holland
Sr. Director of the Test Engineering, Medidata Solutions, Inc.
Automation will not increase coverage, decrease costs, save time, or allow an org to reduce headcount, but it will let you do a lot of repetitive steps in sequence repetitively and very fast. In many organizations, that’s a win all by itself, but it can often be the lead into automating all of the things all of the time. A question that is worth asking is:
How many bugs were found by scripts vs. the eyes of a vigilant tester? In the example Paul gave from an experience report he showed that 70% of the bugs found were attributed to a vigilant tester, meaning 30% of the reported bugs were found by the scripts. In short, the 70% of the bugs in the list would not have been found in automation.
Paul made a comparison of a roadmap of the US to the various pathways that can take you from city to city in 1907. Then he showed a map of roadways in the US in 2017, which showed a much denser network of roads. In the 1907 comparison, there are a few paths people can take to get from one city to another. In 2017, it’s much more complex and interconnected. In short, it would be impossible to cover every path around a typical sized metropolitan city.
Another example is a company that Paul worked with that had made a decision where an automated test had to be written for every written requirement. Why was this asked? Maybe to save time, maybe to increase coverage, not sure. This company had an excellent automation framework with over 10,000 scripts (which is an irrelevant measure). They also had a fair number of implicit requirements that had nothing written (and hence no automation test for those). On the whole, they would get intermittent failures of the tests about 40% of the time. Many of these tests would run successfully on a second run. This means that a high maintenance cost is incurred for getting closer to 100% automation. It can also take a long time to determine if the errors are real or just intermittent line noise.
Paul suggests that automating the critical paths is still important. Automating the paths that are the most important based on your biggest client is a good suggestion. Writing automation for failures found in the field may or may not make sense. Putting in automation may prove to be expensive so consider the costs to the benefits. Also, have live active testers looking at the code to see if they can spot additional benefits.
Recent Comments