Ah, estimation!

For as long as I can remember, I’ve been asked to provide test estimates for project proposals. More often than not, the details are very high-level and there is never enough time given to you to digest the materials. The latest I received was this morning with 104 bulleted requirements (not yet including other background references) and they expect a reply by today otherwise they’d assume the test effort would be 25% of the development effort. Such short notice demands just make me wince. Didn’t they ever do math problems as a kid?

Juan renders 8.5 hours a day to Project X. Last week, Juan was on sick leave and is still feeling slightly under the weather (but this doesn’t really matter). How many minutes must Juan spend on each 104 requirements (not part of Project X) so that he does not have to do overtime?

Try as I might though, I can’t do away with test estimation. Estimation is part and parcel of the software engineering process, and is highly essential when you’re trying to bid for a project. Business needs to put a price. There’s no going around it. You, as a tester, will eventually be asked to estimate!

Lesson 1: Estimates are wrong most of the times

There are so much unknowns. You have to deal with the cone of uncertainty and all that jazz. Try googling why estimates are always wrong and you’d find a lot of results on the topic. So knowing this? What does it tell us? Nope, this doesn’t mean we just go off throwing numbers just for the sake of having a deliverable. Personally, it just tells me to be efficient with the effort that I put into estimation. I may only have half a day or half an hour even to come up with the numbers. I can’t develop an estimation model in that short a time, and even if I could the output can’t be validated and would most likely be wrong. Essentially, just estimate — don’t kill yourself over trying to come up with something perfect, and put aside your worries that it will be wrong (because it most likely is anyway). Chill! Don’t sweat it!

Lesson 2: Estimate like you’ll be the one to deliver

A reality is you often don’t have the authority over the final estimates. Countless of times, I’ve seen my initial estimates whittled down with each review. You have to remember that those who are working on those proposals are out there to win the bid. But as someone who got pulled in to do the estimates, do whatever’s in your power to not short change the team who will actually execute. Just keep adding buffers whenever and wherever you can!

Lesson 3: Best case scenarios hardly ever happen

In an ideal setup, there could be a team of 2-3 testers going over the requirements — each giving their estimate for the optimistic, most likely and pessimistic scenarios. In instances where there’s a huge discrepancy, you guys can talk it out until you come to a point where you all agree. From experience though, it’s almost always a lone effort. You don’t get the ideal situation for estimation, so it is safe to assume that you won’t get the ideal situation when it comes to delivery. Buffers and factoring in the pessimistic scenarios are always a good idea.

Lesson 4: Ask for models

For mature organizations, it’s possible that they’ve already collected some past metrics and have already crafted some magical software estimation models. Enter the size in function points and voila we have the estimates! Enter the development estimates and ta-dah you have got estimates for everything else! Just enter the parameters and it spews out the number! What number? THE number!!! Now I don’t want folks to be all reliant on these numbers since the parameters in which they may have worked in the past might be totally different from your current situation. I am wary of when they say that testing is just 25% of the dev effort when I know from first-hand experience that the effort in testing could go far more than that. What you can do is ask around and use these numbers as references. Only as references. You have to exercise judgement before deciding these numbers could be applicable.

Lesson 5: How I estimate all by my lonely self

So after making you painstakingly read through the previous items, here’s the actual content. When pushed into an estimation task for a new system to be built, I’m typically given a list of requirements or functionalities that the new system or app has to cater for. I’d go over each requirement to estimate for the following:

  • Test Execution – This covers the execution of the test cases, reporting the status of the tests, and logging bugs when found.
  • Test Planning – This includes studying the test basis, drafting test cases, identifying the needed test data.
  • Test Plan Rework – This is for reviewing and revising test case drafts.
  • Retest – For verifying bug-fixes.
  • Test Execution using a 2nd, 3rd, nth platform – It’s possible that you need to test the system/app using a different platform or device.

Now that’s for when you have the time to go over each and every requirement. Plan B would be to do the estimates for what would constitute as a Simple, Average, or Complex requirement. Then instead of estimating for each requirement, you just categorize them and then use the corresponding estimates for them.

Then you also add buffers to consider things like:

  • Things generally go bad, and there’s a general communication overhead
  • Technical difficulty — like if it is relatively unknown how to test a particular function, or if you’ll need to have some learning curve since you’ll need a tool to augment testing
  • Adjustments in case testing will be done by someone less experienced

Now, that was only for the functional test estimates. You’ll also need to estimate for the following:

  • Test Management – Especially for a large testing project, there’ll be management overhead to consider. There are also project deliverables like the Test Plan or Test Strategy document. There would be management meetings to attend to, reports to report, and project issues or escalations to deal with. There’d be onboarding of new folks into the team and essentially setting up the test framework, processes, standards, guidelines, etc. to keep all the testers in the same page, and the test team in the same page as the rest of the development team.
  • Integration / System Test – The focus of this test is for the end-to-end scenarios or business flows.
  • User Acceptance Test – The involvement of the test team during UAT will also need to be identified. Will they be facilitating (or would that be handled by the BAs)? Will we just provide support e.g., replicate issues reported, verify change requests, etc.?

And in the process of going over the requirements, you’ll also need to identify other types of testing which might be needed for the project and estimate for those that you are capable of or have experience on. It’s possible that the project could go with usability testing, accessibility testing, performance testing, etc.

Anyways, you’ve read this far. Thank you! I guess it’s too late to give out the disclaimer that I’m no expert. In case you have magical ratios or percentages that you’d like to share, I’d like to hear about them. Please share in the comments! And if there’s one take-away that you get from this super long post, please let it be: Do not short change the team who will actually deliver.

Two developers working at a rate of N hrs/day…

A friend told me an odd story. Two developers from another team were allocated at a total of 40% into a particular project. They were given a task which they’ve estimated they could complete by January.

The PM Prodded and pushed. “Finish it by December,” he demanded. Eventually, after tireless nagging, the two conceded and said they will deliver by December as requested.

“How will they do that,” I wondered. Apparently, nothing changed — only the deadline. The amount of work to be done is still the same. Their measly 0.25 and 0.15 allocations were retained as is. What was the January estimate then? Had they padded their initial estimates such that it would actually be feasible to complete the task by December? Or will they have to render overtime work to cope and meet the deadline?

Instances like these make me wonder whether the PM ever did math problems while he was back in school. Joe works at a rate of N hours per day. A given task takes X hours. How many days will it take for Joe to finish his work?  Judith needs to complete a task that requires X hours in N days? How many hours per day will Judith have to work? How many Judiths are needed if each Judith can only work Z hours per day?