OZTester has just published my article on test estimation in their year-end issue. The article can be read at:
Or you can read it below. I wrote this piece almost a year ago and since then few things have changed. I have understood #NoEstimates movement better and have developed a positive intellectual relationship with its proponent Vasco Duarte. There are few editing errors, for example the hyperlink for AST is incorrect in the magazine. The article also associates me with Telstra Corporation, but I have never worked with them. I have asked the magazine editor to correct it. I am open for feedback or discussion. Cheers!
————————————————–
The Pragmatism of Test Estimation
There are hundreds of approaches and methods available for test estimation. Some of them are very theoretical in nature and do not fit in a day-to-day project delivery environment; others are too casual and do not satisfy the need either. Software projects are ‘living creatures’ in nature and every change affects cost & schedule. Due to this dynamic nature of software projects, it is vital that estimation is done in a way that it becomes practical and pragmatic for the project.
My work on software projects during the last two decades has taught me that software estimation is not an easy task. In the actual fact, it is very difficult or even close to impossible. Close to impossible because no one can see the future, we can only guess. We can only imagine about how long it will take to test. Attaching a number on imagination is not good.
Test estimation is more difficult because testing is a cognitive process. Basing it on requirements is hard. Project sponsors ask for estimates for Initiation or Conceptualization phase. “Give us a number +/- 100%”. Hang on, how do you know that it will not be +/-1000% or +/- 10% of the given estimate? Requirements are often vague, or ambiguous or both. So if I estimate on wrong requirements, the estimate will be wrong, isn’t it?
But business owners still want an estimate and in certain circumstances it is possible to give a decent estimate. Often in the situation when you are doing exactly what you did before plus buffer for external factors that you cannot control.
I am following the #NoEstimates movement but I am not a follower of the movement. Think it this way. You want to build a house. You approach a builder/ contractor and describe your requirements to them. Your requirements are still a wish list and somewhat vague. It is possible that some of them are not doable, out of your budget or the builder does not have capability to do them. But that is okay because you have a budget and you want something done within that budget. You ask the builder how long will they take to build the house and they reply that they belong to #NoEstimates movement and will not tell you how much it will cost. You will agree with it only if you have won the Australian lotto or you are the heir of a massive fortune with a buffer more than your original budget.
Business owners want ‘some’ estimates. For software it is not entirely impossible to not estimate. Business needs it or they do not need you. Simple! So what to do to estimate for testing. One has to be aware of multiple aspects and dynamism of project delivery methodologies, people working on projects, organizational culture, nature of software solution, other supporting application like upstream, downstream, front end or backend applications, time constraints, dependencies on other external factors etc.
One thing that testers must keep in mind while estimating is that estimates are ‘just estimates’. I have seen project managers asking for ‘accurate estimates’, which is nothing more than a fallacy. If it is accurate, then it is not an estimate. My advice is to provide three different numbers to projects; optimistic, pessimistic and realistic.
Realistic are those number that you derive after studying and analyzing requirements, defining a clear cut scope of your tests, referring to architectural designs, viewing and skimming through any other supportive documents that you feel would be worth reading for estimation, and by considering all the external factors I mentioned earlier for including in your estimate.
Pessimistic estimates are those that are derived after a discussion with project manager, architects and business analysts. I would recommend a collaborative approach for these discussions. This will also save you from politics that often follows once projects start getting pear shaped. It is also always worthwhile educating your non-testing team members of testing approach if they are not aware of it. Most non-testers I have met believe that testing is easy and all we do is to ensure that software is working fine. My response to these people has been consistent that my job is not to make sure software is working fine; rather it is to discover that in what conditions or situations it does not work. What approaches or techniques I use for this discovery depends on the context. Another thing that I consistently mention is that I am not the one who ensures quality of software. I only show where are the issues. Quality Assurance is a broader term that is project team’s responsibility not just testers’.
Optimistic estimates provide an option to the project management to show that there is a possibility that testing might finish earlier than expected. See, I always make it very clear that complete testing is a myth. If this was not a myth, all those giant software firms out there would not have kept their testing teams after the first release of their software. They would not also need to release patches of bug fixes. There is a very good article by Doug Hoffman which talks about completeness of testing. If you have done #BBST course you will know what I am talking about. if not, you must visit www.associationforsoftwaretesting.org to find out.
The Optimistic estimates are based on a simple fact which is the Exit Criteria of your testing approach that states that if certain conditions meet, no further testing will be performed. Again, this totally depends on the context of the project.
In another context, your manager or client may give you a schedule or time to complete your testing. In this case you don’t really have to estimate.
As a tester you must understand that while test estimation is required to define a solution delivery timeline, there is no control over what drivers are pushing your business teams or the client. If the business is risk-averse, the chances are that you will get what you have asked for. If the risk-appetite is high (or business teams has no clue what they are doing), product will be shipped without much testing. In this case they are probably looking for a scapegoat or a box-ticker.
Whatever I mention here in terms of terminology, is purely context-driven. I do not follow so-called standards of testing and I do not want you to comment about a certain term referring to ISTQB or any other glossary of terms. I have my own views about testing and I believe I have not harmed any of my projects by using or not-using a specific set of standards.
If you are interested in learning or discussing more about test estimation, you may contact me. Or, if learning is the objective, you must look at Jerry Weinberg’s book ‘Perfect Software’ and blog posts by Michael Bolton on Developsense blog.
My work on software projects during the last two decades has taught me that software estimation is not an easy task. In the actual fact, it is very difficult or even close to impossible. Close to impossible because no one can see the future, we can only guess. We can only imagine about how long it will take to test. Attaching a number on imagination is not good.
Test estimation is more difficult because testing is a cognitive process. Basing it on requirements is hard. Project sponsors ask for estimates for Initiation or Conceptualization phase. “Give us a number +/- 100%”. Hang on, how do you know that it will not be +/-1000% or +/- 10% of the given estimate? Requirements are often vague, or ambiguous or both. So if I estimate on wrong requirements, the estimate will be wrong, isn’t it?
But business owners still want an estimate and in certain circumstances it is possible to give a decent estimate. Often in the situation when you are doing exactly what you did before plus buffer for external factors that you cannot control.
I am following the #NoEstimates movement but I am not a follower of the movement. Think it this way. You want to build a house. You approach a builder/ contractor and describe your requirements to them. Your requirements are still a wish list and somewhat vague. It is possible that some of them are not doable, out of your budget or the builder does not have capability to do them. But that is okay because you have a budget and you want something done within that budget. You ask the builder how long will they take to build the house and they reply that they belong to #NoEstimates movement and will not tell you how much it will cost. You will agree with it only if you have won the Australian lotto or you are the heir of a massive fortune with a buffer more than your original budget.
Business owners want ‘some’ estimates. For software it is not entirely impossible to not estimate. Business needs it or they do not need you. Simple! So what to do to estimate for testing. One has to be aware of multiple aspects and dynamism of project delivery methodologies, people working on projects, organizational culture, nature of software solution, other supporting application like upstream, downstream, front end or backend applications, time constraints, dependencies on other external factors etc.
One thing that testers must keep in mind while estimating is that estimates are ‘just estimates’. I have seen project managers asking for ‘accurate estimates’, which is nothing more than a fallacy. If it is accurate, then it is not an estimate. My advice is to provide three different numbers to projects; optimistic, pessimistic and realistic.
Realistic are those number that you derive after studying and analyzing requirements, defining a clear cut scope of your tests, referring to architectural designs, viewing and skimming through any other supportive documents that you feel would be worth reading for estimation, and by considering all the external factors I mentioned earlier for including in your estimate.
Pessimistic estimates are those that are derived after a discussion with project manager, architects and business analysts. I would recommend a collaborative approach for these discussions. This will also save you from politics that often follows once projects start getting pear shaped. It is also always worthwhile educating your non-testing team members of testing approach if they are not aware of it. Most non-testers I have met believe that testing is easy and all we do is to ensure that software is working fine. My response to these people has been consistent that my job is not to make sure software is working fine; rather it is to discover that in what conditions or situations it does not work. What approaches or techniques I use for this discovery depends on the context. Another thing that I consistently mention is that I am not the one who ensures quality of software. I only show where are the issues. Quality Assurance is a broader term that is project team’s responsibility not just testers’.
Optimistic estimates provide an option to the project management to show that there is a possibility that testing might finish earlier than expected. See, I always make it very clear that complete testing is a myth. If this was not a myth, all those giant software firms out there would not have kept their testing teams after the first release of their software. They would not also need to release patches of bug fixes. There is a very good article by Doug Hoffman which talks about completeness of testing. If you have done #BBST course you will know what I am talking about. if not, you must visit www.associationforsoftwaretesting.org to find out.
The Optimistic estimates are based on a simple fact which is the Exit Criteria of your testing approach that states that if certain conditions meet, no further testing will be performed. Again, this totally depends on the context of the project.
In another context, your manager or client may give you a schedule or time to complete your testing. In this case you don’t really have to estimate.
As a tester you must understand that while test estimation is required to define a solution delivery timeline, there is no control over what drivers are pushing your business teams or the client. If the business is risk-averse, the chances are that you will get what you have asked for. If the risk-appetite is high (or business teams has no clue what they are doing), product will be shipped without much testing. In this case they are probably looking for a scapegoat or a box-ticker.
Whatever I mention here in terms of terminology, is purely context-driven. I do not follow so-called standards of testing and I do not want you to comment about a certain term referring to ISTQB or any other glossary of terms. I have my own views about testing and I believe I have not harmed any of my projects by using or not-using a specific set of standards.
If you are interested in learning or discussing more about test estimation, you may contact me. Or, if learning is the objective, you must look at Jerry Weinberg’s book ‘Perfect Software’ and blog posts by Michael Bolton on Developsense blog.
Recent Comments