I’m catching up on past issues of the Coding QA Podcast and I’m up to episode 23. In this episode those ASP.NET test guys talk about the issues that they had with the heavyness of their divisional lab harness (familiar to me since I used to work in that division) and how they went about getting better. They talk about specific problems they had and how their lightweight approach solved those problems. Before I go further, some definitions may be good since the terminology we use on our team differs slightly from what they use. When I talk about the test automation my team uses, the following are definitions of those terms:

Lab Harness: The (usually distributed) application that is responsible for running tests in a test lab. This type of application manages the machines in the lab. Machines in a test automation lab will generally execute test suites based on requests. Those requests may contain a suggestion of how many machines to run the test suite on. The lab harness will allocate the machines to suites, initialize them, monitor them while the tests are running and potentially flagging them if something bad happens, and reclaim them back to the machine pool when the tests are done. They will also provide an interface where test suites can take the results of the test run and store them in a repository to help with investigation and later data analysis.

Automation Framework: This is the framework that provides support to the actual test code. In our case, the framework we use has many of the same attributes as the “portable test bed” referred to in the podcast. It was cool to hear that they faced a similar situation to what we did and came up with a similar solution.

If I had to pick one attribute that we strive for that makes life better for us it’s doing our best to minimize dependencies on particular aspect of any one environment that our tests run in. By keeping those dependencies low, we can run our tests in many different environments for many different purposes. This has proven to be a big win. I’ll write more about specific aspects of our framework in my next post.