Bug Tracking Done Right

Many companies today track bugs and features in a bug tracking system, this is a good thing. It is necessary to record bugs which I define as things that aren’t working in a software application and that have the potential to affect the performance and output of such applications. When a customer calls and tell us that they couldn’t purchase an item in our website, or that some data does not look correct, or that they got an error and the application is throwing some obscure error message then you know that there is a problem somewhere. It doesn’t necessarily mean that there is a problem with the software itself, it could be a problem related to another system being down, corrupted data in a database, lack of permission, etc… When something like this happens, companies big and small need to have a bug triage system that allows them to sort out these bug reports before they go into a bug tracking system and are exposed to software engineers.

Bug Triage Done Right

In a hospital, triage happens the instant a patient arrives through the emergency room doors. His vital signs are checked, his status assessed, and he gets sorted in amongst all the other patients waiting for treatment.

Bug triage is a lot like that because we assess bugs to determine whether or not they have enough information to be worked on and assign a priority to them as soon as possible.

Triaging bugs consists of several things:

• Responding to new bugs as they are filed.
• Ensuring that new bugs have all the necessary information.
• Assigning bugs to the proper package.
• Confirming bug reports by trying to reproduce them.
• Setting the priority of bugs reports.
• Searching for and marking duplicates in the bug tracking system.
• Sending bugs to their upstream authors, when applicable.
• Cross-referencing bugs from other distributions.
• Expiring old bugs.

If you receive a considerable number of bug reports every day and you don’t have a bug triage system then you are doing it wrong. Remember that each one of these “bugs” will need to go through the steps mentioned above, or at a minimum they will need to be read, assessed, and sorted so that it can be easily reproduced, fixed and tested if they are in fact bugs. I have worked in projects where software users had the ability to add bugs to our bug tracking system causing us (the software developers) to deal with a large numbers of items where only a small percentage of them were real bugs. This is wasteful and it hurts the productivity of a software team, make sure that whatever gets logged in your bug tracking system is in fact a bug that has all the information necessary to be reproduced, fixed, and tested. Have a system that checks for the things listed above and be very careful not to pass bug reports to software developers without confirming them during the bug triage step. You’ll have happier developers and a better product overall.

Below is a flow chart showing you the basic steps of bug triage:

As this flow chart shows, many of the request that come from software users might not even be bugs, they could be a feature request or a support question and that is very common. The bug triage step should be the first thing you do with all bug reports added to your bug tracking system and if you are not doing some sort of triaging you are doing it wrong. If your bug tracking system doesn’t have a way for you to do bug triage, don’t worry that not everything is lost, you could simply create a spreadsheet where all the new bug reports are sent so bug triaging can be done.

Confirmed Bugs

The result from bug triaging are confirmed bugs, and when a bug is confirmed the status should change to “Ready”, “Confirmed” or “Triaged” so it can be made available to the software developer to work on it, after the importance and priority have been set. Before changing the status of a bug report make sure you have answered all the questions below BEFORE you mark it as Ready, Confirmed or Triaged, or whatever status you choose in your system to pass it on to the next level:

• Does the bug report describe a valid bug?
• Does the bug report contain enough details?
• Is the bug report ready to be worked on by any software developer?

Only if ALL of these conditions are satisfied, you can change the status of the bug report and move to the next step which is setting the importance and priority of this bug.

Setting Importance Of Bugs

Below are some examples and the meanings of the most common importance values:

• Undecided: The default for new bugs. Also means that there is insufficient information to determine importance. PLEASE mark all of your bug reports with this status as they are entered into your bug tracking system.
• Wishlist: Missing functionality.
  • These aren’t always bugs, but can be ideas for new features which do not yet exist.
  • These can also be requests for new features.
  • If it is non-trivial to implement, it should rather be written as a feature specification.
  • These can be bugs that affect an experimental extension or non-essential feature of a given application/project.
  • Bugs that would only be fixed on a best-effort or outside-contribution basis might also be considered wishlist.
• Low: Bugs which affect functionality, but to a lesser extent than most bugs, examples are:
  • Bugs that have easy workarounds
  • Bugs that affect unusual end-user configurations or uncommon hardware
  • Bugs that affect a non-essential aspect and limited scope of the application
  • Bugs that have a moderate impact on a non-core application
  • Cosmetic/usability issues that does not limit the functionality of a non-core application
  • Non-ideal default configurations
• Medium: Most bugs are of medium importance, examples are:
  • A bug that has a moderate impact on a core application.
  • A bug that has a severe impact on a non-core application.
  • A bug which impacts accessibility of a non-core application.
  • A usability issue that does not limit the functionality of a core application.
• High: A bug which fulfills one of the following criteria:
  • Has a severe impact on a small portion of your application’s users (estimated)
  • Makes an application generally unusable for some users
    • For example, if a certain area of your application does not work at all, or on a specific browser version, etc…
  • A problem with an essential hardware component (disk controller, built-in networking, video card, keyboard, mouse)
  • Has a moderate impact on a large portion of your application’s users (estimated)
  • Prevents the application or any dependencies from functioning correctly at all
  • Renders essential features or functionality of the application or dependencies broken or ineffective
  • Impacts accessibility of a core application
• Critical: A bug which has a severe impact on a large portion of your application’s users
  • Causes data corruption
  • Crashes the entire application or operating system
  • Renders the system temporarily or permanently unusable
  • Severely affects application’s core functionality

Once the bug reports have been set with the right importance, then it is easier for everyone involved in fixing them to know what bugs are priority.

Software developers are the ones who should set the bug report status to “Dev Complete” or something similar as soon as they have fixed a bug so the QA team can test again and make sure the bug is fixed. Involving your QA team early on in this process is key, you shouldn’t wait until a developer is done “fixing” a bug to get QA involved in the process… it is too late by then and I will write about this in a future post.

Invalidating Bugs

Sometimes, you will have to invalidate a bug report. There are bugs reports that just don’t seem as important or perhaps are lacking a lot of details so they just get added to the “to be triaged” group. Yes, I am talking about those bugs who stay in a backlog with the same status in a bug tracking system for days and even months, these should be invalidated, retired, or removed from your bug tracking system. If you keep these bug reports around because you worry these might be real bugs, then removing them will not make any difference, if it is a real bug it will be reported again, trust me.

You see, when a bug report does not seem to be critical, or perhaps it falls to the bottom of a list because there are so many other bugs with higher priority, then you should consider how you handle these not so important bugs. My advice is to let them expire automatically. You can set up your bug tracking system so if a bug report doesn’t have any status changes, or any new information added to it for 60 days then it automatically expires and it is invalidated from your bug tracking system. If you don’t do this, then is really easy to end up with an unmanageable number of bug reports and as Joel Spolsky wrote a while ago, you’ll be just adding software inventory to your team. It is not a good thing, check it out.

Keep It Simple

As with ANY other system, find what works for you and your team and stick to it. Avoid at all costs the idea of keeping ALL bug reports that get submitted to your bug tracking system. Just like your to-do lists, if it goes unmanaged, the items at the bottom of the list end up just wasting space, are forgotten and it also makes the to-do list incrementally larger than what it really is. Keep it simple, triage all bug reports, fix them and test them. Do not keep bug reports around thinking that one day you’ll get to it, you never will and if you do, it is because you or somebody else made the bad decision about spending time and effort on something that was very low priority and probably not important.

Every time you open your bug tracking application and then you look at the feature or bug backlogs and realize you have just too many items, do not cry, just remember to set an expiration date on all backlog items. I suggest 60 days, and as your process gets better you can then lower this number, IF you make it more than 60 days, don’t be surprised if some of those things never see the light of day.

Feature requests can also become a problem, every time somebody suggests a cool new feature or has an idea… make sure you look at your backlog first and decide if this new feature or idea are worth doing in the first place, this might prevent you from wasting time documenting and designing a crazy idea that might also never see the light of day.

Keep your backlog and entire bug tracking system lean and tidy, and hopefully you’ll spend less effort working on things that never see the light of day. Trust me, “Backlog grooming” sessions is something to be avoided, it just adds to the overhead and more meetings, who wants to do that?!