Not every new thing is an innovation

My friend Alice shared that she conducted a talk and shared that they were using so-and-so tool in their project, whereas typically projects use this other so-and-so tool. That seemed to have wowed her audience and they said that could be considered as an innovation. Alice and I agreed that it felt like it wasn’t. Unless you’d call it an innovation if I suggested to use Google Docs as opposed to Microsoft Office.

But isn’t innovation simply a new idea, device or method? Something new that will make things easier for yourself and others? Yes and yes. But taking that definition kind of opens the floodgates where anything new could be taken as innovation — which shouldn’t be the case because not every new thing can be regarded as truly innovative.

In googling what is not innovation in the hopes of getting more insights on this, I found this whose points I do understand:

First, an improvement that only meets the market standard or reacts to innovation that your competitors have already introduced into the market is NOT innovation. It’s playing catch-up.

Second, introducing an improvement that does not significantly differentiate you from your competitors is NOT innovation. It’s simply just an improvement—evolutionary, not revolutionary.

And finally, introducing improvement that may give you a competitive advantage but also can be easily copied by your competitors is NOT innovation. It’s just a temporary advantage.

…Here’s the takeaway: It’s easy to confuse improvement with innovation. But only innovation creates a unique outcome that, despite the superior financial returns resulting from the action, competitors are either unwilling or unable to match.

Now, innovation or not, I still believe in looking for and sharing things that helps us make things easier for ourselves and others. Whatever ideas we put onto the table and help realize, just push for it if it holds the promise of making things better. And you can just let other people decide if it’s an innovation or not.

Trying out Postman

One of the tools that we’ve used in our project is Postman. It’s a Google Chrome App wherein we can send the web service request and then view the response in a prettier, much readable style.

In one of our user stories for this Sprint, we needed to come up with the list of profiles that needs to be deactivated because the employee had become inactive. To verify whether the employee had become inactive, we had to check against an employee database. To check, you can go visit a search site, search for that user, and view the search result. Using their provided API, you can pretty much do the same with Postman. Submit the request with the email passed as parameter and evaluate the result.

01-single-request

But then, sometimes you have to check several employees at a time and it turns out we can use Postman to test in batches. To accomplish that what I did was the following:

1. Whether you run it one at a time or in batches, I defined my tests. Below is the snippets I used in my tests which checks for a 200 status being returned, and for whether the employee status field has value not equal to Active.

tests["Status code is 200"] = responseCode.code === 200;

try {
    var responseJSON = JSON.parse(responseBody);
    tests["Employee is NOT Active"] = "Active" != responseJSON.result[0].empStatus;
}
catch (e) { }

2. I used a variable for the parameter e.g., {{email}}.

02-parameter

3. I saved the parameter values in .csv format where the first line (treated as the header) use the variable used as parameter, in this case “email”.

4. I saved the request (with the tests and params) into a collection.

5. Then I used the Collection Runner to select the collection I want to run, the csv file with the data to pass as parameter values, and to indicate the number of iterations. Results are then displayed at the rightmost panel.

3-collection-runner

So there. I probably haven’t even scratched the surface of how else Postman can be used. But it’s been pretty nifty for this particular use case.

Automated testing with CucumberJS with Selenium Webdriver using JavaScript

I’m currently picking up something new at work. And I guess the title of the post gives it away. It’s not a brand new thing altogether, just something that I haven’t tried out before. And in one of the blogs I stumbled upon, there was a line that captured what I wanted to do in this post (and I guess in a lot of other previous posts too): helping me remember what I figure[d] out. Anyways, this is just a gist of the basic setup. Much of the learning is in the actual scripting, and most of it is when things don’t work as expected.

First things first: Setting up

  • Download and install Node from https://nodejs.org/en/download. This will also install the Node Package Manager (NPM).
  • Download the drivers into a folder in your local.
  • Update your environment variables to include the drivers into your PATH variable.
    • Add SELENIUM_DRIVERS as a system variable and specify the folder where you placed the drivers
    • Update the PATH system variable by appending “;%SELENIUM_DRIVERS%;”
  • We used a starter found here. Download the files from there, and that’ll give us stuff that I have no idea how to write from scratch at this point:
    • package.json
    • Gruntfile.js
    • features/support folder contains world.js, env.js and hooks.js
  • Install the needed dependencies.
    • Dependencies like CucumberJS, Chai, Selenium Webdriver, etc. are indicated in the package.json file.
    • Open the Command Prompt, then go to the folder where the starter files are placed. Run “npm install”.

Writing our Feature File and Step Definitions

Since we’re using CucumberJS, we’ll write up a feature file in Gherkin format. This means that scenarios that we’ll be testing for are written in a Given-When-Then format. Given includes the needed setup for the test. When refer to actions taken from the initial state (as setup in the Givens). And Then includes the expected outcomes that will be checked for.

For each Given, When and Then, corresponding code has to be written to actually perform the automated steps. These steps are stored in step definition files stored under features/step_definitions.

Feature: Searching for cucumbers
  As an internet user
  In order to find out more about cucumbers
  I want to be able to search for information about cucumbers

  Scenario: Google cucumber search
    Given I go to the Google website
    When I search Google for "cucumbers"
    Then I should see some results
'use strict';

var expect = require('chai').expect;

module.exports = function() {
  this.World = require('../support/world.js').World;

  this.Given('I go to the Google website', function (next) {
    this.browser.get('http://www.google.com').then(function () {
      next();
    });
  });

  this.When('I search Google for "$searchQuery"', function (searchQuery, next) {

    this.browser.findElement(this.webdriver.By.css('input[name="q"]')).sendKeys(searchQuery);
    this.browser.findElement(this.webdriver.By.css('input[name="q"]')).sendKeys(this.webdriver.Key.ENTER)
    .then(function () {
      next();
    });
  });

  this.Then('I should see some results', function (next) {
    this.waitFor('div.g', 10000);
    this.browser.findElements(this.webdriver.By.css('div.g')).then(function (elements) {
      expect(elements.length).to.not.equal(0);
      next();
    });
  });
};

Okay, actually writing our step definitions

To interact with the web site under test, we use Selenium Webdriver. While working on a POC, we had to look for elements within the page under test, and then interact with those elements by either triggering an action like clicking or sending keys or we also interact by extracting information about the element I found. For the checks in the Then steps, we use Chai to do the comparisons. For the POC at least, I pretty much managed to get by with simple scenarios using the items below.


//Finding Elements with Selenium Webdriver
//Find Element by ID
this.browser.findElement(this.webdriver.By.id('sp-search-input'))

//Find Element by CSS
this.browser.findElement(this.webdriver.By.css('#sp-search-input'))
this.browser.findElement(this.webdriver.By.css('.ui-chip'))
this.browser.findElement(this.webdriver.By.css('.ui-chip[data-val="hello"]'))

//Interacting with Elements with Selenium Webdriver
this.browser.findElement(this.webdriver.By.id('sp-search-input')).sendKeys(searchText+"\n");
this.browser.findElement(this.webdriver.By.id('sp-add-widget')).click();
this.browser.findElement(this.webdriver.By.css('.ui-chip>span')).getText();
//looks for element with class "ui-chip" and then looks for a <span> and gets the text in that <span>

this.browser.findElement(this.webdriver.By.id('sp-search-input')).getAttribute('placeholder');
//e.g., <input id="sp-search-input" placeholder="Add Search Tags" …>
//above will return "Add Search Tags"

this.browser.isElementPresent(this.webdriver.By.css('.ui-toggle.checked'));
//returns boolean value

//Comparisons using Chai
expect(actualText).to.equal(expectedText);
expect(textFound).to.contain(expectedString);
expect(toggleState).to.be.ok;

Running our scripts

Using the Command Prompt, go to your test project folder, then run “cucumber-js”. There’s also an option to run it with other options. I find the tags to be most helpful. What I do is I tag the scenarios that I’m working on in the feature file. The format of the tag is “@tagname” in the line before the scenario you are tagging.

cucumber-js --tags @dev               //tagged with @dev
cucumber-js --tags ~@dev              //NOT tagged with @dev
cucumber-js --tags @foo,@bar          //tagged with @foo OR bar
cucumber-js --tags @foo --tags @bar   //tagged with @foo AND bar

Well, that’s about it. Again, the learning really comes when you’re actually scripting. Sometimes, you just need to dive in knowing you don’t know everything, and hopefully you’ll figure it out as you go along.

Summary of references

Example, Starter – https://github.com/Matt-B/cucumber-js-selenium-webdriver-example
CSS Selectors – http://www.w3schools.com/cssref/css_selectors.asp
Chai – http://chaijs.com/api/bdd/
Selenium Webdriver – http://www.seleniumhq.org/docs/03_webdriver.jsp
CucumberJS – https://github.com/cucumber/cucumber-js
Gherkin – https://github.com/cucumber/cucumber/wiki/Gherkin
Cucumber – http://cukes.info
Cucumber tags – https://github.com/cucumber/cucumber/wiki/Tags
Blog post – http://www.tcias.co.uk/blog/2014/09/03/writing-our-first-functional-test/ (2014 Sep 3)
Blog post – http://transitioning.to/2012/01/cucumber-and-js-getting-started-with-cucumber-js/ (2012 Jan 27)

Looking for that singular statement

There’s been some recent talk about vision and goals where I work, and that has somehow led me into thinking about what I personally stand for (at least professionally). Or maybe it could also be because I watched Batman v Superman last weekend and there was this scene wherein the lady senator wants to know what Superman stands for. Anyways, I tried to give it some thought. The truisms and platitudes that I often say crossed my mind. I thought of my default thoughts and my fave quotes. But it’s hard to capture what I stand for or what I’d like to stand for in a singular statement.

Be kind. Play fair.

Let your work speak for itself. Keep on learning. Keep the saw sharp.

No, not quite. Each statement doesn’t quite cut it.

Then something hit me. 16 years out of college, and it hit me! I don’t think I’ve ever been the type who’s so overly gung-ho about graduating from our state university. But I realized that the singular statement I was looking for was in our school motto all along:

Honor and Excellence.

Honor. This captures how it’s not about winning all the time, but more importantly it’s how you played the game. It covers honesty, integrity and respect. It covers selflessness, doing what is right, and giving credit where it’s due.

Excellence. This highlights competence. It covers knowing your craft and continuing to learn to be better than you were previously. It covers setting the bar high. It covers uncompromising standards. This also extends to paving the way for others to excel and being a spotlight to others when it’s their time to shine.

Honor and Excellence. You need both. It’s not one or the other. This means being good at what you do without the needless assholiness. This means rising above others while still remaining grounded. This means expanding your mind, capabilities, scope, or whatever; without getting so full of yourself.

This is a good reminder of what I’d like to see more of in myself, and what I should reward or encourage more in others.

Read: Leading the Transformation

Our product owner is one of the rare few individuals I know at work who actually still reads books. Last month, he recommended that we read Leading the Transformation: Applying Agile and DevOps Principles at Scale by Gary Gruver and Tommy Mouser. It’s a thin book with only 112 pages on paperback and around a 3-hour read. It’s intended for leaders/executives so it gives a high level overview of the changes teams and the organization need to make and the benefit of those changes, and it repeatedly emphasizes management’s role in pushing for those changes. In particular, the changes that they want to drive at center around Agile, DevOps and Continuous Delivery (CD).

At work, small teams have now been shifting to Agile, our own team has been in this Agile project since January of last year, and I’ve heard of proposals wherein the methodology they suggest is already Agile instead of Waterfall. But then, I pick up from the book that trying to scale up Agile adoption across the board with small teams as the starting point doesn’t quite work for large organizations. Whoops. The book suggests that if you want an enterprise-level change, you have to plan for it and drive it from the management level down to us lowly minions. A key difference though is that within our organization (at least locally that I know of), we don’t really have hundreds of developers working on the same product or code base. And in our case, we’re only under 20 in the team, but even so the book still offers a good introduction to a lot of mature development practices that we need to look into.

Key items highlighted in the book that I’d like to reiterate further:

Importance of having quick feedback loops

Unit tests and static analysis tools can already weed out a lot of problems so that defective code won’t even get committed to the repository to begin with. And having those fixes done even before passing it to the test team — instead of fixing them only after the code has been deployed and testers found issues that were caused by those defects — will definitely help reduce the turnaround time.

Quick feedback loops will also help the team work and resolve issues while the code or user story is still relatively fresh in their heads. It’s more difficult for both the devs and testers to fix and retest an issue on a behavior that they’ve pretty much forgotten about.

Having builds as release- or production-ready as possible

With regular and stable builds in place, it’ll be easier to identify when a commit breaks the build. Since you don’t have to backtrack through days or weeks of commits, it’ll be easier to narrow down and identify the problematic commit.

Having dev/test environments as close to production as possible

One problem that we’ve personally encountered in not having a test environment in sync with the production version was that whenever we encountered an odd behavior in the test environment we had to double check whether the issue was also in prod. We also had to be mindful of issues that were already resolved in prod but not in the test environment. But I guess this problem is a combination of why it’s good to have the test environment as close to prod as possible and the next item related to why it’s good to have good deployment procedures in place.

Having repeatable build, deploy and test processes

From experience and the example above, having a reliable and repeatable deployment process could’ve saved us all effort and heartache. It could be so frustrating to test the same build (supposedly) but then get different outputs even if you’ve done the same steps using the same test data. In the same vein, you’d hate for a feature not to work in prod even if it had already been thoroughly code reviewed, tested and signed-off in UAT/PO review.

And last, but not the least, having test automation

You simply will never achieve the full benefit of Agile development until you get your automated testing properly built out and integrated into the development pipeline.

Test automation is key to the first item I mentioned since it enables quick feedback loops. It also allows repeatable tests to be executed across the different environments, and it allows repeated execution of the regression tests which you might not be able to afford to do so manually.

Having test automation, by itself, will not suffice. Tests have to be designed such that it’ll be easy to localize the cause of failed tests should any be encountered. Maintainability of the automated tests also have to be considered. Otherwise, the benefits of test automation won’t be realized since the team ends up ignoring the test results on account of being not sure whether the issue encountered is a code issue or a test issue.

One last thing… it’s a cultural shift

You can’t just invest on tools for CD or test automation or announce “Let’s do the Agile thing”, and expect the benefits to magically follow right away. This kind of thing takes time because there’s the technical learning overhead, plus shifting to a new way of doing things requires discipline and resolve so that folks won’t revert to the old habits that they’re trying to change.

It is important for executives to understand early on if the organization is embracing this cultural change, because if it doesn’t, all the investments in technical changes will be a waste of time.

It’s not going to be enough for the project team alone to be invested in the changes. The management and executives need to be aligned with this. In fact, they should help drive it. Otherwise, they might give demands that would bypass the adoption of change and instead force people back to their old habits (just because it might appear faster but only in the short term).

The book, after all, isn’t entitled “Leading the Transformation” for nothing. Management’s presence and push isn’t merely a suggestion; it’s a necessity. Sure, the project teams are the ones making the technical changes; but management needs to understand and support the changes. Essentially, people need to be in the same page in order to move in the same direction.

Finished reading: Managing the Test People

It’s a quick and easy read as it promised to be. I’m not a manager and it’s not something I’m planning to be. But I am somewhat in a position of leadership so the book is still quite relevant to me. Judging by how much I’ve highlighted in the book, it’s undeniably quite relevant.

I’m also working with younger folks who I believe have great potential to be leaders. They can be even better leaders than who we have at the moment, but only if they’re positively influenced by the right mindset on both leadership and technical aspects.

I’d go recommend this book to them since the author really paints a great picture of a leader (or manager) to aspire to be. And with its focus on testing teams — or technical teams in general — it’s a perfect fit for us. Reading the book raises the bar for our expectations on managers but only as it should be because we can’t expect nothing less than for our managers to lead and empower their people. You also get insights on how managers should (better) deal with things. But more than that, and I guess what’s most important, you also get to pick up and be reminded on how you should be as a leader (even if not by title).

In closing, the author shares:

Stay on the right path by frequently asking yourself, “Am I being honest? Am I being consistent? Would I want to work with* me?”

*Originally “for”. But since we’re not bosses or managers, “with” seems more relatable.

Maybe that simple level of introspection — especially on that last question — is what we all need to remind us to be first and foremost good colleagues or team mates before even rising to becoming good leaders.

On valuing your time, Maker’s and manager’s schedules

Time and again, my hate for useless meetings seems to keep on drawing me to Paul Graham’s essay “Maker’s Schedule, Manager’s Schedule”. (And also, I did tell a friend I’ll go share this link with her). Every time I read it, I couldn’t help but agree to a lot of the things he said. So much so that I find it hard to cite just one particular line to quote here in this post. You really just have to read the whole thing yourself.

To me, this essay is a pretty good reminder of what we should all be doing (just in case I’ve lapsed, and have been setting meetings or following up like there’s no tomorrow), and that is to respect my own time and other people’s time. In doing that, you make a more conscious effort to (well, if i can help it):

  • avoid interrupting or disturbing people unnecessarily
  • express gratitude when someone obliges you with their time
  • be present in meetings where your inputs or feedback are actually needed
  • set up meetings with the implicit target of not wasting people’s time
  • decline meetings I’m pretty sure I won’t be engaged in
  • decline meetings when they’re in conflict of personal commitments — Those are just as important (and sometimes even more) as work commitments
  • honor commitments to yourself — Ages ago, I had to block of time just for my lunch or dinner, and I even missed that because of work. That just isn’t healthy. Also when you block off time to work on something, then use that time to be productive.

Discipline on how you manage your time or own your own calendar starts with one’s self. And how badly your time gets mistreated by others (and even by yourself) highly depends on how much you’d allow it. So for your sake, start respecting and managing your time.

What makes them leaders I want to work with

So leadership happens to be one of the things I think about, you know, just for fun. I’ve been working for quite some time now and I’ve had my fair share of leaders — some by title and some for real. Now, I don’t want to dive deep into my memory banks so I just thought of two leaders I’ve had the good fortune of working with recently. One is my former manager, Eric, and the other is my current project’s product owner, Maneesh. I posed the question: What about them makes it feel like it’s worth it to follow them?

Vision

They aspire for great things. They have an idea of what they want (or at least they seem to). You know they already have a picture of what they want to achieve or some sort of a plan in their heads (or at least it feels like it) so it doesn’t feel like they’re waiting on you to hand them the answer on a silver platter. More importantly, they are able to communicate their views clearly enough to get the team’s buy-in.

What you say matters

When they ask for your inputs, it doesn’t feel like they’re just asking out of courtesy (well, they could’ve been but you couldn’t tell). When you raise a concern, it doesn’t feel like it went in one ear and out the other. You go out of the discussion feeling like you’ve been heard.

Ego is thrown out of the door

They don’t insist on being right all the time. So if you happen to have conflicting ideas, they’re willing to hear you out and they’re also willing to explain where they’re coming from. The focus is on the problem that needs to be solved. So it doesn’t matter if you’ve disagreed on some points midway — what matters is you agree on a solution in the end.

Follow through

When something needs to get done and they’re the one who needs to do it, they act on it right away. Of course, you understand that they’re busy with a lot of other stuff but you can still count on them to follow through — without the need for multiple follow-ups from you.

Clarity, transparency and integrity

It’s hard to pinpoint if one is just the effect of the other. Or if they should be separate points instead of bundled as one. Either way, these three make it easier for us to understand the reason behind certain decisions or actions. Things don’t come in as a surprise (or a shock) because they’re communicated clearly, discussed openly or aligned with some bigger plan or purpose. What’s been done is congruent to what’s been said. There’s little to no room for second guessing or thinking “Oh, he probably means well…”

Your success in mind

Maybe it’s just me, but I think to have your success in mind is a basic expectation from your managers. Of course, I also expect that people help themselves but managers have the role of removing impediments on your way and setting you up for success. I don’t want to work for someone who just passes the buck (delegation doesn’t always equate to “empowerment”) or throws you out to the wolves.

As for our product owner, I expected that his focus would only be on the product that we’re working on. But for him to express interest in our career progression or on us maturing in our practice, that was something. I do understand that if we do improve ourselves, in turn we’ll provide better service to him. But what I like is that he considers wins for both sides — ours and not only his.

Coincidentally, there’s a buzz in the team on the “growth mindset” — the thinking that you just have to work at something hard enough in order to succeed. But while I understand that it is hugely up to the individual to drive his own success, you can’t discount how it’s also a matter of a bunch of other things including leaders who truly empower and enable you. I guess the paragraph below from this post states it much better:

“Ultimately, I’m convinced that there’s great power in starting from a place of believing that all people can improve themselves if the conditions are right, and I think that’s what Kohn is getting at when he worries about the growth mindset ideology being co-opted by personal responsibility advocates. Human beings are not vacuums. We rely on family, teachers, economics, societal expectations, and a range of other factors beyond ourselves to contribute to our success. The notion that personal responsibility is the only condition that matters for success, or the most important one, is just plain false.”

That’s about it

So those are just the stuff that’s on the top of my head. Ultimately (I just had to use “ultimately” myself), I think the best relationships are the ones that don’t feel forced, and that also goes for you and your leader. I guess one last thing I can add to the list is how they didn’t just ask for my respect — they earned it.

Web accessibility, testers’ start for building it in

So I haven’t had any first-hand working experience on web accessibility testing. I’ve read about it in passing, and once I attended this weekend testing session that tackled the subject. In my previous company where I did a lot of functional web testing, I recall some standard test cases that were related to usability or accessibility. This week though, the topics of accessibility testing and WCAG 2.0 came up, and I had to google a bit on the subjects.

One of the interesting things I found is this blog post on how expensive is accessibility. And what the writer, Karl Groves, said makes sense — that if it’s already part of how you do things, then it comes at no extra cost. And inversely, if it’s something you haven’t factored in at all, then pushing for accessibility would require changes and those changes would undeniably entail some cost.

Now I googled WCAG 2.0 and found really looong references. There’s a quick reference page, but seriously, that page felt anything but quick. In another post by the same person:

What if I told you that the WCAG 2.0 recommendation by the W3C is 36 pages, printed? In addition, “How to Meet WCAG 2.0” is 44 pages and “Understanding WCAG 2.0” 230 pages. Not only that, but the accompanying Techniques and Failures for WCAG 2.0 is 780 pages, printed.

That’s a bit daunting!

But thankfully, he also shared a post on the 6 simplest web accessibility tests anyone can do. And the Web Accessibility Initiative (WAI) has a reference on easy checks for web accessibility. These could be a start and could be things we start looking for in our web testing projects even though we weren’t really asked to do accessibility testing. It’s a chance to add value. Let’s check these out:

 

Ah, estimation!

For as long as I can remember, I’ve been asked to provide test estimates for project proposals. More often than not, the details are very high-level and there is never enough time given to you to digest the materials. The latest I received was this morning with 104 bulleted requirements (not yet including other background references) and they expect a reply by today otherwise they’d assume the test effort would be 25% of the development effort. Such short notice demands just make me wince. Didn’t they ever do math problems as a kid?

Juan renders 8.5 hours a day to Project X. Last week, Juan was on sick leave and is still feeling slightly under the weather (but this doesn’t really matter). How many minutes must Juan spend on each 104 requirements (not part of Project X) so that he does not have to do overtime?

Try as I might though, I can’t do away with test estimation. Estimation is part and parcel of the software engineering process, and is highly essential when you’re trying to bid for a project. Business needs to put a price. There’s no going around it. You, as a tester, will eventually be asked to estimate!

Lesson 1: Estimates are wrong most of the times

There are so much unknowns. You have to deal with the cone of uncertainty and all that jazz. Try googling why estimates are always wrong and you’d find a lot of results on the topic. So knowing this? What does it tell us? Nope, this doesn’t mean we just go off throwing numbers just for the sake of having a deliverable. Personally, it just tells me to be efficient with the effort that I put into estimation. I may only have half a day or half an hour even to come up with the numbers. I can’t develop an estimation model in that short a time, and even if I could the output can’t be validated and would most likely be wrong. Essentially, just estimate — don’t kill yourself over trying to come up with something perfect, and put aside your worries that it will be wrong (because it most likely is anyway). Chill! Don’t sweat it!

Lesson 2: Estimate like you’ll be the one to deliver

A reality is you often don’t have the authority over the final estimates. Countless of times, I’ve seen my initial estimates whittled down with each review. You have to remember that those who are working on those proposals are out there to win the bid. But as someone who got pulled in to do the estimates, do whatever’s in your power to not short change the team who will actually execute. Just keep adding buffers whenever and wherever you can!

Lesson 3: Best case scenarios hardly ever happen

In an ideal setup, there could be a team of 2-3 testers going over the requirements — each giving their estimate for the optimistic, most likely and pessimistic scenarios. In instances where there’s a huge discrepancy, you guys can talk it out until you come to a point where you all agree. From experience though, it’s almost always a lone effort. You don’t get the ideal situation for estimation, so it is safe to assume that you won’t get the ideal situation when it comes to delivery. Buffers and factoring in the pessimistic scenarios are always a good idea.

Lesson 4: Ask for models

For mature organizations, it’s possible that they’ve already collected some past metrics and have already crafted some magical software estimation models. Enter the size in function points and voila we have the estimates! Enter the development estimates and ta-dah you have got estimates for everything else! Just enter the parameters and it spews out the number! What number? THE number!!! Now I don’t want folks to be all reliant on these numbers since the parameters in which they may have worked in the past might be totally different from your current situation. I am wary of when they say that testing is just 25% of the dev effort when I know from first-hand experience that the effort in testing could go far more than that. What you can do is ask around and use these numbers as references. Only as references. You have to exercise judgement before deciding these numbers could be applicable.

Lesson 5: How I estimate all by my lonely self

So after making you painstakingly read through the previous items, here’s the actual content. When pushed into an estimation task for a new system to be built, I’m typically given a list of requirements or functionalities that the new system or app has to cater for. I’d go over each requirement to estimate for the following:

  • Test Execution – This covers the execution of the test cases, reporting the status of the tests, and logging bugs when found.
  • Test Planning – This includes studying the test basis, drafting test cases, identifying the needed test data.
  • Test Plan Rework – This is for reviewing and revising test case drafts.
  • Retest – For verifying bug-fixes.
  • Test Execution using a 2nd, 3rd, nth platform – It’s possible that you need to test the system/app using a different platform or device.

Now that’s for when you have the time to go over each and every requirement. Plan B would be to do the estimates for what would constitute as a Simple, Average, or Complex requirement. Then instead of estimating for each requirement, you just categorize them and then use the corresponding estimates for them.

Then you also add buffers to consider things like:

  • Things generally go bad, and there’s a general communication overhead
  • Technical difficulty — like if it is relatively unknown how to test a particular function, or if you’ll need to have some learning curve since you’ll need a tool to augment testing
  • Adjustments in case testing will be done by someone less experienced

Now, that was only for the functional test estimates. You’ll also need to estimate for the following:

  • Test Management – Especially for a large testing project, there’ll be management overhead to consider. There are also project deliverables like the Test Plan or Test Strategy document. There would be management meetings to attend to, reports to report, and project issues or escalations to deal with. There’d be onboarding of new folks into the team and essentially setting up the test framework, processes, standards, guidelines, etc. to keep all the testers in the same page, and the test team in the same page as the rest of the development team.
  • Integration / System Test – The focus of this test is for the end-to-end scenarios or business flows.
  • User Acceptance Test – The involvement of the test team during UAT will also need to be identified. Will they be facilitating (or would that be handled by the BAs)? Will we just provide support e.g., replicate issues reported, verify change requests, etc.?

And in the process of going over the requirements, you’ll also need to identify other types of testing which might be needed for the project and estimate for those that you are capable of or have experience on. It’s possible that the project could go with usability testing, accessibility testing, performance testing, etc.

Anyways, you’ve read this far. Thank you! I guess it’s too late to give out the disclaimer that I’m no expert. In case you have magical ratios or percentages that you’d like to share, I’d like to hear about them. Please share in the comments! And if there’s one take-away that you get from this super long post, please let it be: Do not short change the team who will actually deliver.