This blog post has been written by my colleague and software automation expert Scott Miles. As ever, both Scott and I will be glad to receive any feedback, comments or suggestion regarding major flaws in logic, about similar tools, questions about the proposed solution and about any grammatical or formatting errors that may have escaped my eyes.
-Raj
————————————–
Improving the cost-effectiveness of automation using adaptive tools
Automated checking (often incorrectly referred to as automated testing or test automation) is widely acknowledged as a useful tool within today’s software industry to augment an organisation’s testing processes. Many organisations implement some level of automated checking or are striving to do so with the same goals in mind; to improve test coverage and reduce the cost of testing. The expectation is that automation will accomplish these goals by allowing checks to run without manual effort, outside of regular business hours, be repeatable on demand, reusable across environments or configurations, run faster than comparable manual checking and more accurate than manual checking. Yet from my experience within the industry and through discussions with other industry experts, I have found that many organisations are dissatisfied with the cost-effectiveness of their automation frameworks. The return of investment an organisation achieves rarely seems to meet the claims made by the tools used and often falls short of expectations.
It is well documented that automation is not a cure-all remedy that is suited for every context. In fact there are a lot of circumstances in which the benefits of automation will not outweigh the cost. Instead of being considered as a silver bullet to fix all of your testing woes, automation needs to be considered as another weapon in your testing arsenal that may provide benefit when used in the correct situations, but may also wreak mayhem when used incorrectly.
Almost all of the negative sentiments about automation that I hear tend to stem from a single complaint; automation frameworks do not do enough for how much they cost. In this context cost may refer to financial expenses, effort or other resources. While this revelation is not surprising for anyone who has been involved in creating and maintaining an automation framework, it has led me to delve deeper into the question of why automation costs so much and more importantly what can be done about it.
Why does automation cost so much?
Whilst the main contributors to cost in automation vary between organisations, there are several factors that I have identified as being consistent across the majority of contexts. I believe these key costs to be:
-
Initial investment to get the framework off the ground
There are many automation tools available to choose from. Some that come with all of the bells and whistles, some that come with just the bells and others that will only provide a single feature. The price of these tools vary as widely as the features, ranging from astonishingly pricey to free (open source). Selecting cheaper/free tools may appear to decrease the cost of your initial investment, yet you will likely find that those savings will quickly dissipate due to the increased effort required when attempting to build a framework around multiple/partial tools. No matter your choices at this stage, you may be looking at a significant investment.
-
Maintenance to keep the framework running effectively
The magnitude of this aspect is what catches most organisations unaware. When a seemingly insignificant change is made to an application it can result in days worth of effort to update the affected automation scripts. Worse is that an automation script will often break due to a code change that has no effect on functionality. A common occurrence of this is when a developer changes the name property of an object. This property is only ever used by the code and will not affect the functionality of the product, yet the automated scripts may depend on this property when locating elements in a user interface.
-
Additional resource requirements
The cost of purchasing automation tools aside, an organisation will still find heavy expenses in acquiring additional software development tools, hardware for hosting tools and frameworks and additional environments for running automated checks. There is also likely to be additional costs in hiring people to assist in the implementation, as well as training existing team members in the use of new tools and technologies. And when onboarding new team members you may find that the additional requirement of automation skills increases the cost and selection of potential candidates.
-
The application needs to be modified or reverse engineered
One more factor of cost that needs to be considered is the complexity or the unreadable nature of the software. In some instances when creating a locator for a simple element such as a button, the only options available to the tester is to find the button based on its X and Y coordinates, and sometimes even that option doesn’t exist. In these scenarios the development team gets tasked with going back through years of legacy code in order to expose handles to these objects for the automation framework to use. The scope of this task by itself can be immense even without including the inherent risk involved when modifying any legacy code. However even in the most modern systems that were built with automation in mind, the tester will still run into the odd unexpected querk. A web page may be generating an object on-the-fly when the user scrolls to a certain point on the page or an application is using a custom object type that the tool cannot interpret. These situations often require reverse engineering of the code by the tester or developer in order to find or create a solution.
These costs can usually be foreseen and mitigated by an effective testing team with a little investigation, collaboration and good communication. However while I spend hours updating automation scripts with simple changes I can’t help but wonder why these expensive tools couldn’t have anticipated the update in the first place. After all, computers have been capable of solving advanced mathematical equations since the 1930’s and with current advancements are capable of driving cars, diagnosing medical conditions and even holding conversations?
With this in mind, I put forth three challenges that are and faced by many commercial automation tools but could be minimised by utilising simple artificial intelligence and machine learning approaches that are already prevalent in other industries.
What can be done about it?
Challenge 1: I never know if my scripts are broken
One of the great advantages of automation is that scripts can be scheduled to run in a specific order and usually at any hour. However when it comes to verifying the condition of your automated scripts this can also be a weakness. It is not unusual for an error to exist within a script and it is possible that these errors will not be detected until the script has been executed by an automation tool. An error within a script may exist due to the tester not being aware that a change had occurred, that a change would affect a particular script, that the tester missed a script when making the required updates or that the tester made a simple mistake during the update.
Due to the sequential nature of automation scripts, in order to verify a particular script, several other scripts may need to be executed first. This dependency can mean that it takes a significant time for the tool to reach the point you are interested in, possibly several hours. If the tool does encounter an error in a script you may not be confident with any of the results that come after the failure. When the script error is corrected the whole sequence will need to be executed again to be assured that all errors have been resolved. There may also be a number of factors such as shared resources and environment availability that require scripts to only run out of business hours or to be scheduled to run when an environment becomes available. The result is that a script may require several run throughs to find all of the errors in the script that have been recently introduced and those runs may need to occur over several days.
But should it be left to the user to find and make these adjustments? When checking changes to an API interface an automation tool could make several validations prior to runtime simply by inspecting the changed libraries or code source. Through this inspection the tool could determine which interfaces have been modified. Taking the inspection a step further could reveal useful information about the change such as if the number of required parameters or the type of one of the parameters has changed. The tool would then be able to notify a team member as soon as the change has been committed and identify every script that will be effected. Likewise when a GUI element has been modified the tool could use the same detection methods to discover the change and identify the affected scripts.
Presenting the tester with this information as soon as the change is made would save large amounts of effort on a frequent basis greatly reducing the maintenance cost of an automation framework.
Challenge 2: My scripts keep failing
An automation script failing is a common occurrence and only occasionally is the failure due to a defect in the application being checked. It is more likely that the failure is caused by an error in the script or even an issue in the environment. A failure in an automation script is mostly due to either not finding an expected element, not being able to set a value in an element or an element having an unexpected value.
When an automation tool fails to locate an item in the application being checked the majority of automation tools will handle the situation in one of the following two ways. They either skip the remaining steps and fail the whole script, or fail the single step and continue to next which in turn will most likely fail as a result of the skipped step.
There are many reasons an automation tool may fail to locate an item, but the most common reasons are:
-
The object has changed
Objects changing within an application is the natural outcome of software development. A good automation tool will select the parameters it uses to locate an element carefully with the intention of finding a set of parameters that are most likely to remain unique and are least likely to be modified. However even in the perfect automation tool these parameters are likely to require updating from time to time.
A solution to this issue has been partially addressed by the previous challenge, however we don’t always have access to libraries or code sources, particularly when checking off-the-shelf products. When a tool has not been able to detect a possible failure prior to execution there are several other approaches that can be used to minimise the impact of a missing element. One of these approaches is to have the automation tool attempt to identify the missing element based on a closest match technique. By comparing the known properties of our element to those currently present on the page it could identify the most likely object that the script was looking for. To further enhance results the tool could cache the set of objects on a page from the previous run and only include the delta set of objects in its comparison. -
An unexpected event interrupted the script
It is impossible to control every condition of your environment and application. Therefore, a tester cannot predict every event that may occur during an automated script. When a script encounters a warning screen, an unexpected dialog box, or even the lack of an expected dialog box, the script will not be able to locate the expected element and fail the step. These interruptions could be environment related or even caused by the application being checked. Common occurrences of these interruptions include: new updates are available, this is not your default browser, the terms and conditions have changed and another user completed this action recently message. There are countless variations of similar messages that can occur at any time interrupting your automated script.
When these interruptions occur an automation tool could use machine learning and artificial intelligence to identify the best action to take to continue the script. Sometimes these dialog boxes have only a single possible response such as OK, in which case the correct path is simple to identify. When more than one response is possible the tool could refer to a catalog of dialogs for common applications such as Internet Explorer or Microsoft Windows and respond with a predefined action. If the dialog is not contained in the list of known interruptions the tool could refer to existing scripts to identify if this interruption is expected elsewhere and what response is used. The last option for the tool would be a set of heuristics to determine what the best action would be based on the choices available and keywords detected in the text. -
An event timed out
Many organisations help minimise the cost of software development by providing limited and/or shared environments for testing. Unfortunately this means that response times can be sporadic and requests can occasionally timeout. This may occur due to another team performing a stress or performance check on the same environment or it could just be that there is just too much activity on a particular server. What ever the cause, a failure of a web page to load or of a database query to return a result within the expected timeframe is not always reason to fail a script.
A more advanced automation tool would be able to repeat a query, refresh a page or even repeat a login when the previous attempt timed out. Based on the configuration of the tool an action like this could be repeated multiple times in succession or with a fixed pause between to allow other tasks to complete.
Whenever the automation tool has recovered a possible failure the automated script would continue as scheduled. The final report would show all results including any recovery actions taken. The report would allow the tester to determine if the tool adapted to the failure correctly by accepting or rejecting a change. When the change is accepted the user would have a further choice of allowing the script to be updated with the change, further reducing maintenance time and cost.
Challenge 3: Creating scripts is slow and repetitive
When a page contains input fields there are a dozen checks that jump to mind in order to verify if the field is validating data correctly. By using simple heuristics I can instantly tell that a field labeled age should not accept negative values and a field labelled email should require the text to be in a specific format. Furthermore I know that pressing back should return me to the previous page and the total cost listed on every page should remain consistent until new items are added.
An automation tool could gather information from a page and make intelligent decisions on the purpose of most elements based on the type of objects, the available parameters such as name, and relative labels that are grouped by elements such as a div or pane. Similar techniques are already used by current web browsers in order to pre populate login or payment details on remembered pages. Having identified the purpose of a field, a set of checks can be generated on the fly to determine validation and consistency across pages. An automation tool could walk through entire web pages, applications or forms this way performing in-depth analysis with little to no generation or maintenance involved.
Making dreams a reality
Having read this article you may be thinking that these ideas are not possible with the technologies available today. You may be thinking the ideas are great but no company would invest into developing them. Or you may be thinking that the ideas mentioned in this article only scratch the surface of what is possible. So what are the next steps from here?
I am currently working with my esteemed mentor and friend Rajesh Mathur to produce an academic paper to promote the concept of smarter automation tools. The ideas presented in this article and the corresponding paper have been developed over long conversations and many coffees with Rajesh. The interest and support shown in the software community has also led me to begin work on a prototype tool to demonstrate some of the functionality discussed. Stay tuned to Rajesh’s blog http://www.dogmatictesting.com/ for further articles, the final academic paper and some early releases of a working prototype.
——————————–
References:
1 http://www.satisfice.com/blog/archives/category/testing-vs-checking
2 http://www.softwarequalitymethods.com/papers/star99%20model%20paper.pdf
3 http://worrydream.com/refs/Brooks-NoSilverBullet.pdf
4 http://mathworld.wolfram.com/TuringMachine.html
5 http://spectrum.ieee.org/automaton/robotics/artificial-intelligence/how-google-self-driving-car-works
6 http://www.wired.com/2014/06/ai-healthcare/
7 http://www.cleverbot.com/
Recent Comments