Am I bugging you? Bug Reports
I love seeing bugs. It’s better for me to see them than for my customers to see them. What I hate seeing are poorly written bug reports—misleading or generic titles, unclear or missing reproduction steps, exaggerated priority, overstated severity, and inappropriate, cowardly, and poorly documented resolutions.
Why can’t people write decent bug reports? It’s not like a decent report is longer or much harder to write than a lame report. It’s not like clear definitions for everything in a bug report don’t exist. Ah, but those definitions do vary and sometimes conflict from team to team. What are the right definitions to use for everything in a bug report? I’m glad you asked.
Eric Aside
Every piece of software ever written has hundreds or thousands of bugs, depending on its size and complexity. Some bugs are innocuous, like “I’d prefer the close button to be wider.” Some bugs are misunderstandings, like “It wouldn’t let me use an obscene name for my gamer tag.” And some bugs are nasty issues that must be fixed no matter what, like exposing personal information. Since bugs are often found by people outside of the development team, bug reports must be written and tracked to closure—typically using a work item tracking database, like Product Studio® (legacy MS tool) or Team Foundation Server®.
Bug dissection
All bug reports have the same basic set of information.
§ Title—a short description of the issue
§ Assignment—who’s taking care of the issue at the moment
§ Repro steps—the steps necessary to reproduce the problem
§ Priority—the urgency and importance of the issue
§ Severity—the fallout from the issue
§ Resolution—how the issue was resolved
There are a bunch of other fields that are helpful in reproducing the issue and understanding the root cause, but the basic set is short and simple. Let’s cut through the controversy and lay out the rules for each field.
Title and assignment
The title of a bug should be a terse one-line description of the issue that is specific enough to identify that issue uniquely, making bug report searches and identification easy. “The screen blanks when you hit the Cancel button” is a poor title. “Blank screen after canceling avatar editor” is a great title. The second version is shorter, yet provides more specific context around where and when the issue occurs.
When you create a new bug report, you must assign it to someone to resolve. However, unless you are part of the development team, you shouldn’t assign the bug to an individual, even if you know the whole team personally. Instead, assign the bug to the team. This is typically done by specifying the area or team in the bug report and accepting the default assignment. The default assignment is typically “Active” or “Triage.” You don’t know better. Trust the team to know who should work on the issue.
Eric Aside
There are some groups who want all bugs assigned to individuals. This ensures no bugs are ignored. However, even those teams must check for bugs assigned to Active or Triage to ensure they aren’t missed. After all, people outside the team don’t know what other values to use.
As a general rule, all bugs should be assigned to individuals or groups who will check them regularly. Since most triage teams meet daily, I’ve always liked the idea of assigning bugs to Active or Triage as a default.
Repro steps
There is nothing more frustrating than a bug report without a decent repro (reproduction steps). It’s like your significant other telling you, “You know what you did!” with no further explanation. Now I know I messed up and have no way to correct it. Terrific.
Repro steps should be short and sweet—the minimal set that triggers the issue. You should also include the build number (typically a separate field), the environment you used (the version of the operating system, browser, and any other relevant details), and any preparation necessary (like signing into Xbox.com with a gold account).
Sometimes you aren’t sure how you triggered the bug because it’s intermittent or associated with a weird state. In this case, provide the build number, environment, and set up, then describe the circumstances, acknowledging that precise repro steps aren’t clear.
After describing the minimal repro, you must indicate what you expected to happen (“Expected”), followed by what actually happened (“Actual”). All repro steps should have these three sections—the setup, the expected results, and the actual results. That way someone reading the bug report knows exactly what went wrong and how to reproduce it.
Often a picture or video tells a thousand words. There are many tools to create screen captures of both stills and compressed videos. Attaching these files to a bug report can be the difference between a properly fixed issue and an elusive one.
Eric Aside
It’s annoying to see a bug report with 15 repro steps when the issue can be reproduced in four steps. Not only are four steps shorter and easier to understand, but they also allow the developer and the tester to close the bug far faster. It takes less time to reproduce the bug, less time to determine the cause (fewer possibilities), and less time to verify the issue has been fixed.
Priority
There are endless arguments over the meaning of the priority field, a value that typically ranges from 0 to 3. Surely you have better ways to spend your time. Instead, let’s lay out a few simple rules then define priority based on those rules.
§ Priority should never have to be adjusted once properly set, unless the bug itself changes character. If priority 1 means “fix this sprint or milestone” and priority 2 means “fix next sprint or milestone,” then you’ve got to change the priority of bugs at the end of every sprint or milestone. Not only is that a waste of time, but it updates the “last changed date” on the bug, an act that causes the loss of important information.
§ Priority should be easy to assign and differentiate. You don’t want the team spending a bunch of time arguing over the priority of every bug. It should be obvious, both when writing the bug report and when reading it.
§ Priority should be memorable and actionable. No one should have to ask, “What was pri 2 again?” No one should have to question what needs to be done for each priority level.
Based on these three rules, here are priority definitions that serve well.
|
Priority |
Description |
Timeframe |
|
Pri 0 |
A CRITICAL failure that requires URGENT attention that doesn’t have a known WORKAROUND. This is a blocking bug. |
You can use the bathroom after you resolve the issue or find a workaround. |
|
Pri 1 |
A CRITICAL failure that requires URGENT attention. |
Must be resolved in the current sprint or milestone. |
|
Pri 2 |
A CRITICAL failure. |
Must be resolved before release. |
|
Pri 3 |
A failure or suggestion. |
Should be resolved before release. |
Pri 0 issues typically block testing, deployment, or some other time-sensitive work. Given the seriousness of pri 0 bugs, you can’t submit them and expect something to happen. You must send mail to the individual or team, and then call or walk over and talk to them until someone is actively working on resolving the issue. If a viable workaround is found, pri 0 bugs should be changed to pri 1.
Eric Aside
Teams do vary their definitions of priority. Some start at pri 1 instead or pri 0. Some break the rules I listed at the start of this section or have a separate field to indicate a blocking bug.
If you open a bug in a different team’s work-item database, be sure to use their definitions. The definitions typically pop up in a tooltip or help screen.
Severity
Severity is even simpler than priority, yet it’s also often misused. Severity refers to the fallout of the issue, NOT how important it is. The definitions are:
§ Severity 1—the issue causes a CRASH or customer data LOSS
§ Severity 2—the issue causes a MALFUNCTION that inhibits action
§ Severity 3—the issue causes an INCONVENIENCE or unfinished LOOK
Please note that severity is independent of priority—in other words, severity has nothing to do with priority. A priority 1 bug is more important than a priority 2 bug, regardless of severity. Displaying offensive content is severity 3 but priority 1. Crashing when a user does a forced reboot is severity 1 but priority 3. Nothing makes you the subject of engineer ridicule like claiming a non-crashing bug is severity 1 just because it’s high priority. You sound like an idiot.
Resolution
One of the most important and most often misused fields in a bug report is “Resolution”—the indicator of what was done to resolve the issue. Resolving a bug means you are no longer concerned about the issue and you don’t plan any further work once the bug originator verifies that the resolution closes the bug.
If the issue requires more work before you release, even if it’s not the responsibility of your team, then the bug should remain active and assigned to one of your team members to track.
Here are the possible values for the resolution field in alphabetical order:
§ By Design—the bug report describes the intended behavior. It works as designed.
§ Duplicate—the bug has the same cause and nearly the same user experience as an earlier reported bug. Never resolve an older bug as a duplicate of a newer bug—regardless of how much nicer the newer bug report is—unless you like making enemies of the originator and losing the “first seen” date.
§ External—the bug is caused by something outside of your control AND you can release without the bug being fixed. If you can't release without having someone outside your group fix the issue, then the bug should remain active and assigned to someone in your group to track, linking to the issue on the other team.
§ Fixed—the bug is fixed. My favorite resolution.
§ Not Repro—you couldn’t get the bug to recur in the build and environment noted. Saying “It works on my machine” doesn’t cut it—check with the originator first whenever possible.
§ Postponed—you won’t fix this bug in this release. Postponed is for the same gutless slackers who say they’ll start writing unit tests tomorrow. Real engineers leave the bug active and use a “Fix By” field in the bug report to indicate a future release when they truly intend to fix the issue.
§ Won't Fix—you won’t fix the bug ever. My second favorite resolution—it shows you have enough experience to know when a bug simply isn’t worth fixing, usually because the fix causes more trouble than the bug itself.
When you resolve a bug, you must provide a description as well as fill in the resolution field. That description is important. You get fewer arguments about resolutions, understand the issue better upon recurrence, and protect yourself and the company if the issue later makes headlines. This happened to an old team of mine once—we saved the company millions in penalties when our resolution description for an offensive content bug proved our lack of malice.
When a bug is resolved, it is automatically assigned to the person who opened it. If that person isn’t on the team, the bug should be assigned to another team member who can verify the resolution with the originator of the bug. You can’t always count on people outside the team to validate a resolution in a thorough and timely manner. Of course, if the resolution isn’t satisfactory, the bug should be reactivated.
Eric Aside
I first defined resolutions for my development team ten years ago. Looking back at that mail, the definitions here still stand.
Keep it simple
There are many other fields in a bug report. I mentioned using the build and environment fields to capture reproduction information and the “Fix By” field to indicate when a bug will be addressed. There are also fields to track root cause, how the bug was discovered, area of the product or service where the bug occurred, potential security impact, and countless other variations of information.
When setting bug report requirements, demand no less than what you need and no more than what you’ll use. Requiring more than necessary will cause people to complain and stop submitting bug reports—neither of which serves you or your customers well.
By keeping bug reports easy to write and easy to read, you encourage people to submit clear bug reports for the issues they find. Using bug templates that prefill some fields also helps. There’s no better gift to our engineers and the customers we care about than a well-written bug report that averts an issue before it ever reaches our users.
