Is this a bug?

This is a fairly common question that pops up in projects. It could happen when requirements are ambiguous, or when the behavior could pass off as meeting the requirements and folks are on the fence about it, or when there’s a change in the requirements or implementation that didn’t get communicated across timely enough, etc. It just happens.

But as someone who has a testing background, there are admittedly times when I don’t like it when testers ask me this question. In general, I think it’s when very little to no analysis goes into the understanding the bug (or possible bug). Especially when there are provided references.

If I were the one testing, I would expect me to exercise due diligence. I have to try to crosscheck against the user story and other provided references. I expect me to understand what I’m supposed to be testing. I’ll need to try to do the RIMGEN / RIMGEA stuff. I won’t just say “I did this and this, this happened, is this a bug?” I will try to answer that question myself. And if it’s still not clear and I need to consult with a team mate, I would phrase my question slightly differently. Probably somewhere along the lines of: “I did this and this, this happened. It might be a bug because <so-and-so>. On the other hand, <reason why it might not be a bug>.”

The more brains you pull into consultation on this possible bug, the more costly it is. It also takes away other people’s time and focus from stuff that they’re actually supposed to be doing when they’re helping you do that analysis.

That analysis is part of the service we provide as testers. That analysis is part of the value we bring in to the team.

RIMGEA (2nd ed.) aka RIMGEN

Wow, it’s been 10 years since this blog post: RIMGEA – 6 approaches to bug reporting. There I wrote about a very useful mnemonic I picked up from a bug advocacy class (it’s more about bug reporting, than loving bugs). Lately, I’m not in a role where functional testing is part of the official R&R. But I think I’m a tester through and through — I still test or review, and I still log bugs or give feedback. And I still care about how bad bug reporting can give testing a bad rap.

So anyways, now’s a good time as any to retouch that post.

---

RIMGEN (and maybe plus S)

Originally, the mnemonic I picked up from the bug advocacy class was “RIMGEA“. It stood for the 6 factors or approaches to bug reporting which are: Replicate, Isolate, Maximize, Generalize, Externalize, And say it dispassionately. Then there was a suggestion in the comments before and I also found some recent BBST material that made the shift from “A” to “N” to suggest the neutrality in tone. The last letter S was from the comments, with a suggestion to spellcheck. I would generalize it as proofread but RIMGENP doesn’t roll off the tongue quite easily.

Replicate it – Try to replicate the bug.

If you can’t replicate it yourself, then it might be harder to persuade your developer to fix a problem that they can’t see. This also doesn’t only benefit the developer. Somewhere down the line, you or a fellow tester will also need the same information. The replication steps would be helpful in preparing test data or setting up the conditions to test the fix for the bug.

Related Link:  How helpful are my bug reports

Isolate it – Limit the steps or the conditions that trigger the bug.

Here you try to narrow down your repro steps or find what exactly are the critical conditions that led to the bug. Here you want to get to the bug in the easiest way possible. You want to rule out factors that have nothing to do with the bug.

Related Link: Bug isolation

Maximize it – Try to trigger a worse or bigger failure.

The bug you find might just be the tip of the iceberg, or a symptom of an even bigger bug. Follow-up tests could help uncover the bigger problem if there is indeed one. For instance, you find a bug regarding the positioning of a button in a modal when you tried it out on mobile. Turns out, if you try it on your laptop browser, the problem still exists. Turns out, other similar modals have a different positioning of the buttons. Turns out, the bigger problem is on the consistency of the modals across your app — and not just with particular modal screen you tried on mobile.

There are various tactics that you can try:

• vary your behavior – e.g., instead of doing A then B, try changing the sequence; or try a different way of doing the task like using a shortcut key instead of the button
• vary your program settings – e.g., there could be program settings that you can toggle on or off or adjust; for browser testing, you can try adjusting the zoom sizes or enable/disable caching
• vary your inputs – e.g., if the bug was encountered when file X was used, try a similar file Y or another file Z
• vary your configuration – e.g., try using a different OS/browser combination

Generalize it – Try to broaden the extent of the bug.

Here we try to uncorner corner cases. We try to find other ranges in which the bug can be reproduced. For instance, we find that a bug occurs when we have 1M records. In generalizing, we try to see if the bug occurs at lower more realistic values or if the 1M records is truly a critical condition.

I find this quite similar to Maximize. But essentially, with both, you are trying to push or find out how damaging the bug actually is in depth (how much does it cascade) and in breadth (in terms of the users and cases it can affect).

Externalize it – Try to see the value or impact of the bug from other stakeholders’ perspective.

Here we try to go beyond our roles as testers and try to get a sense of the bigger picture. In understanding the value that could be lost, we could paint a more compelling picture of why the bug needs to be fixed in our bug reports.

As testers, we might also find ourselves focusing too much on the functional specs. The devil’s in the details and zooming in on the details is important. But so is taking a step back and seeing the bigger picture on whether the functionality can actually deliver what the users need.

Neutral tone – Try to write our bug reports as clearly and as neutral as possible.

Having worked in building apps for over ten years, I know that there’s some egos that bruise far too easily. The bug report isn’t really about the people working on the project, so keep it that way. It only has to be about the bug. Keep it on point and add in only what would be relevant to help address the bug.

Related Link: Article: How to report bugs effectively

Spellcheck – Proofread your bug report.

It’s amazing how often people forget to proofread. So when and if you can, just do a quick pass to see there are no glaring errors and to also check if it’s clear enough. Be mindful that the audience of that bug report could possibly be future you, so you’d literally be helping yourself if you write it clearly.

---

So there. Let me awkwardly close with one of my favorite quotes. It’s one that I often associate with bug reporting.

A problem well stated is a problem half solved.

How our team “does Agile”

Super quick background: Our project started Jan 2015. To kick things off on the Agile methodology, our Scrum master conducted a brief training (less than half a day) to the team, and we’ve been playing it by ear ever since.

Over the course of many sprints, retros and releases, we’ve made adjustments on how we’re doing Agile. I’m not sure if there are Agile Purists who would frown down and shake their heads at us for the variations we’ve made. But the thing is, despite our possibly non-canon approaches, what still matters most is the team closely working together to deliver working software.

This post might be TL;DR. But in case you’re still interested in having a peek at how our little team does Agile Scrum, go on and have a read…

RIMGEA – 6 approaches to bug reporting

I’ve learned a very useful mnemonic from the bug advocacy class on some approaches to bug reporting. Here’s something I had written on the topic with some edits (I just linked to previous write-ups on the same or related topics).

---

The 6 factors or approaches to bug reporting are:

1. Replicate it – Try to see if you can replicate the bug.
If you can’t replicate it, it might be more difficult to provide information to the developer and persuade her to provide a fix for a problem she can’t see.

[Link:  How helpful are my bug reports]

2. Isolate it – Try to limit the steps or the conditions that trigger the bug.
Here you try to narrow down your repro steps or to find what exactly are the critical conditions. Here you want to get to the bug in the easiest way possible.

[Link: Bug isolation]

3. Maximize it – Try to do follow-up steps to see if you can trigger a worse failure.
The bug you find might be just the tip of the iceberg, or a symptom of an even bigger bug. Follow-up tests could help to uncover the bigger problem if there is indeed one. There are various tactics that can be tried:

• vary your behavior – e.g., instead of doing A then B, try changing the sequence; or try a different way of doing the task like using a shortcut key instead of the button
• vary your program settings – e.g., there could be program settings that you can toggle on or off or adjust; for browser testing, you can try adjusting the zoom sizes or enable/disable caching
• vary your inputs – e.g., if the bug was encountered when file X was used, try a similar file Y or another file Z
• vary your configuration – e.g., try using a different OS/browser combination

4. Generalize it – Try to broaden the extent of the bug
Here we try to uncorner corner cases. We try to find other ranges in which the bug can be reproduced. For instance, we find that a bug occurs when we have 1M records. In generalizing, we try to see if the bug occurs at lower more realistic values or if the 1M records is truly a critical condition.

5. Externalize it – Try to go see the value/impact of the bug in other stakeholders’ perspective.
Here we try to go beyond our roles as testers and try to get a sense of the bigger picture. In understanding the value that could be lost, we could paint a more compelling picture of why the bug needs to be fixed in our bug reports.

6. And, say it clearly and dispassionately – Try to have our bug reports as easy to understand and as neutral as possible.
Clarity will make it easier for the dev/triage team to find out what exactly needs fixing and hopefully why it needs fixing. Neutral bug reports will make our work easier to read as opposed to reports that are angry or disrespectful in tone. It is difficult enough to have to deal with a problem in the product, antagonistic reports don’t help as they can only increase the difficulty, cause conflict and discredit the reporter for lack of professionalism.

[Link: Article: How to report bugs effectively]

Email != defect tracker

I suppose one could expect that having a buglist or a defect tracking system should already be pretty standard in a software project. But it’s funny how it’s the basics that gets forgotten or foregone sometimes.

Well, we did have Quality Center set up, but one of the concerns was that the devs weren’t paying attention to it and testers weren’t logging into it. It was a chicken-and-egg thing. Email got inundated with issues, follow-ups and such which kinda sucked since I always got copied into issue email. Eventually, our PM (he got cc’d too) put his foot down and we’ve made the shift back into QC with me goading the testers to use our defect tracker as intended, and with someone from the dev team monitoring the defects with respect to dev assignments. Thankfully, the team has been quite cooperative.

This has also pushed me into tinkering a bit with the reporting capabilities of QC. There are built-in reports that I found to be of use, and I’ve also created my own queries for generating my own reports on defects and test case status.

So far, it’s been working out. With the shift to QC, our email is no longer as abused or misused for defect tracking. We can now misuse it for something else (jk). One major advantage is that we can now, if needed and as needed, easily extract defect data. Instead of having to dig through old email, getting the list of open issues across the many applications that we’re handling at a time is now quite easy. Having all bugs logged into QC also allows for easier detection of red flags e.g., if several testers are reporting similar issues at the same time it’s possible that there’s already a global issue; or if devs are deferring a significant number of defects as non-issues, that could be a red flag on the quality of defect reporting, or that valid issues are getting dismissed.

The lesson learned is simple: use things as intended. Email for comms and the defect tracker for defect tracking. They’re there specifically for those purposes so use them accordingly. And, this is a team thing, so even if the testers were so disciplined in logging the defects, it won’t be as efficient if the devs aren’t using the tool as well. Work to have an alignment within your team so that the available tools can be optimized.

How helpful are my bug reports

I reckon that question is something we ought to hardwire into our systems. After all, the main reason why we log bugs is to have them fixed, and helpful bug reports just make it a bit easier for us and our devs to close the bugs. How so?

Helpful bug reports allow the devs and testers to easily replicate the bug. In order to fix a problem, you’d need to be able to see it first. This also goes for testers who need to verify a bug fix (especially when reported by someone else) i.e., we ought to know what was wrong to begin with and what has been changed in order to close the item. If you happen to be a forgetful tester like me, logging enough details on how to replicate the bug and other background info on the bug helps refresh the memory.

Unavoidably, there are cases wherein the bug is very difficult to replicate or it happens intermittently. For such cases, indicate that info in the bug report so that the dev or the tester who’s trying to replicate it knows that the bug would be a challenge and so that the bug won’t immediately be dismissed. Provide as much background info as you can recall when you first encountered the problem, and provide additional details whenever you encounter it again.

Helpful bug reports help isolate the problem. The problem’s easier to address if you know which particular cases trigger it. Say, it’s a problem that occurs across all screens, then it could be a framework bug or a browser-related problem. If it’s a problem that only occurs for particular data or in particular environments, indicate it in the bug report.

Helpful bug reports are written with some consideration on who’s going to read them. It’s easier when you report a problem to your dev since both of you are working on the same function. But sometimes, you have to report a bug to a framework dev who may not have as much background on the domain as you do. Or sometimes you have to report a bug that will be handled by another tester. Or sometimes you have to report a bug that will need to be reviewed and approved by the HK SA. For such cases, you need to provide enough background information for the bug report to be better understood.

Helpful bug reports provide information on the expected results. Say, a tester found a UI bug wherein a particular text that’s supposed to be bold is italicized. The report shouldn’t be something like the ‘text “blah blah” must not be italicized.’ The dev might just remove the italics and not set it to bold. If you know the expected behavior, then provide the information. If you know of a forum topic discussing the bug, then post a link to it and post a summary of the resolution once it’s already available.

Helpful bug reports are concise, without leaving out the details that could be useful. Provide the details without being too wordy about it. Too much text tends to make the eyes glaze over. And with that note plus some links below, I end this post.

How to report bugs effectively
How to write a good bug report

Bug isolation

A friend had shared with me a paper on bug isolation. It’s a 2004 paper by Danny Faught entitled How to Make Your Bugs Lonely: Tips on Bug Isolation. It starts off with the quotation: “A problem well-stated is half-solved.” The paper then got me thinking on whether we cover or emphasize this strongly enough during tester training in our company. Sure, our testers indicate the expected behavior versus the actual behavior in their bug reports, and sometimes they provide the steps to replicate the problem. But giving the steps does not necessarily equate to giving the conditions that trigger the bug report.

In bug isolation, one attempts to really zero in on the bug by reducing the factors that obscure the problem.  For instance, you streamline the steps to get to the bug, or you narrow down the data needed to trigger the bug.  Basically, you refrain from posting shoddy bug descriptions like “It doesn’t work”, and instead you try to find out the specific conditions that trigger the bug.  You might think that bug isolation only benefits the developers since they’ve less effort to render in trying to replicate the bug.  But I think, this will also help the tester in thoroughly verifying bug fixes.  How else would one know if a problem is solved if he doesn’t know the problem to begin with.  Verifying the absence of the previously detected symptoms is not always a sure-fire guarantee.

Some nifty tools – bugshooting, pixie, jruler

Here are some nifty tools. Their usage is not limited to testing though.

Bug Shooting. It’s another screen capture tool which has a more catchy name than what I normally use i.e., Gadwin PrintScreen. Its edge over Gadwin is that if you intend to make some edits (e.g., add some arrows, circles, or some text), it has those functions immediately available after taking the screenshot. Whereas in Gadwin, you’d still need to open some image editing tool like Paint. With Gadwin, your screenshots are automatically saved into your capture folder. With Bug Shooting, you’d still need to save the file manually. But alternatively, Bug Shooting has an option of automatically sending the screenshot to an application. E.g., after taking a screenshot, you can send it to your email application and the image gets automatically attached.

Pixie. This tool acts like a color picker. So while it’s running, you’d just have to point your mouse over any part of the screen and Pixie returns the color you’re pointing at in hex, rgb and other formats. It also gives out the x and y positions.

JR Screen Ruler. If I remember correctly, this one was shared by Tats a long time ago. It offers a better alternative to using your “man-calipers” (as Chry puts it), or resizing some other window to use as a make-shift ruler. With this you can measure the length or width of certain screen objects in pixels, inches or centimeters.

Article: How to report bugs effectively

“… When you report a bug, you are doing so because you want the bug fixed. There is no point in swearing at the programmer or being deliberately unhelpful: it may be their fault and your problem, and you might be right to be angry with them, but the bug will get fixed faster if you help them by supplying all the information they need…”  Read more… How to Report Bugs Effectively.

You can also provide some references to supplement your report. For instance, you can take screenshots using a tool like Gadwin Printscreen. You can also capture the bugs together with the steps to replicate them on video through a tool like CamStudio.

Acknowledgements: Thanks to Roy for sharing the file.