First step in troubleshooting complex issues: Define and scope your issue properly

 最近在查调试相关资料的时候，无意看到Tess的一篇关于如何快速分析复合场景问题的博文，感觉很实用，Mark备忘。

My 9 questions for a pretty thorough problem description

When I call up a customer to start working on an issue I am generally looking for them to answer 9 questions in their own words.

1. What is happening? 
2. What did you expect to happen? 
3. When is it happening? 
4. When did it start happening? 
5. How does this problem affect you? 
6. What do you think the problem is? and what data are you basing this on? 
7. What have you tried so far? 
8. What is the expected resolution? 
9. Is there anything that would prohibit certain troubleshooting steps or solutions?

1. What is happening

tips：准确描述问题

Better answer:  Once or twice a day over the last two weeks we have gotten reports from customers that the login page is not responding.  Eventually if they wait long enough the page will time out.  We have confirmed this behavior by logging in at the time of the failure, but all other pages seem to be working well during the problem period.   The problem persists until we restart IIS.  We have not seen any events in the event log matching the time of the failure.

2. What did you expect to happen

tips：建立基线，收集相关日志

For a performance issue we need to have a baseline response time.  Even if the problem we are trying to resolve is that pages are timing out, it is important to know if the expected response time for the page is 5 seconds or a few milliseconds, and also if the application has been stress tested and shown those results during testing.

In the best of cases we would have a solid set of performance logs, event logs and iis logs from normal operation to compare to once we gather for the faulty state.

3. When is it happening?

tips：详细的收集各类数据

A solid repro scenario is of course optimal, but with production type issues this is seldom the case.   Things that I am looking for here are.

• How often the problem reproduces (eg. once or twice a day in the above example)
• Do we know what actions people take, i.e. it happens when they try to log in?
• Do we know if it is confined to one server, one application, one page, one action?
• Do we know if it is reproducible in test?
• Is it happening only for certain users?
• Is it happening only under load?
• Does it only happen when memory usage is high, or when CPU usage is over 80%?
• Does it always happen at 8 am when the first people in the office start logging in?
• etc. etc.  any and all conditions relating to the issue are interesting.

4. When did it start happening?

Knowing when it started happening is often one of the most important clues to finding root cause.

5. How does this problem affect you?

tips：了解影响范围，根据问题造成的影响选择不同的处理策略

When I work on an issue, I like to know what kind of impact the problem has on the customer and the users of the system.  

The reason I ask about this is because if a problem causes users not to be able to log in for example, and this is a critical application for the business, obviously we should be starting by finding a temporary fix, rather than going through root cause analysis, or maybe starting with a temporary fix first (like recycling when memory reaches 600 MB if the problem is a memory leak, to avoid an unexpected crash), and then preparing for root cause analysis.

It also tells me a little bit about the priority that troubleshooting is going to get.  If the issue is very severe, maybe it is ok to perform some troubleshooting actions that have a lot of impact on the system, if it is going to help us find the problem faster.

6. What do you think the problem is? and what data are you basing this on?

tips：批判精神，不要过度相信问题反馈人提供的信息，不要被带歪了

This is especially interesting if you come in as an external troubleshooter, as you get a lot of insight into known problem areas of the application by asking what they think the problem might be.

There is a temptation to follow down the path of what is already guessed because it often sounds very plausible, but when I look at a problem I try to keep a very open mind to avoid getting stuck in a tunnel.

If someone believes that the problem is “x” then oftentimes, because of how our minds work, we tend to look at the data from that direction, even if it doesn’t fit. Part #2 of this question is what data you are basing your theory on?  I try to look at that data with fresh eyes, to see if it really collaborates the theory or if there are some holes in it.   To be honest, if a person comes to me with an issue and already has a theory, the first thing i try to do is to disprove it.

7. What have you tried so far?

tips：有没有尝试过什么验证

This question serves two purposes.

1. Finding out what has been done so we don’t need to re-invent the wheel.    
2. Finding out the results of those actions as it gives us more data about the problem.

Keep in mind here as well that reliability is important.  If something was presumably done but there is no documentation of it or the results, it is probably worthwhile re-doing it, depending on how complex the task was.

8. What is the expected resolution?

tips：对应的分析思路是什么

This might sound like a repeat of question 2 (What did you expect to happen), but it is very different.

An expected resolution might be to “avoid the crash” which you can do by preemptive recycling, separating the app into a different application pool, moving to 64-bit, reverting back to a previous build etc.

Another expected resolution might be to “find the root cause to avoid that it happens again”.

And a third may be, “if we get the pages to consistently respond in less than 5 seconds under load we are good”.

Defining the expected resolution is crucial as this is what you will measure and verify the solution against to determine when you are done troubleshooting.  If you don’t have this, how will you know when you are done?  

This expected resolution, just like requirements in a software project, should also be agreed upon so that all involved parties work against the same goal.

9. Is there anything that would prohibit certain troubleshooting steps or solutions?

tips：是否可以通过什么方式避免问题的发生

Knowing the limitations of data gathering or certain solutions both determine what actions you can take and change the expectations of what can be achieved. 

On this question I would expect answers like, we can not install any tools on the server without going through a process of 10 change requests, or we can reproduce this on a test machine so you can live debug if you need to.

 

 

then...

Summarizing everything

Once I have all the facts from the questions above I usually sit down and summarize everything in a format that looks something like this:

Problem description: 
=========================== 
… 
Expected resolution: 
=========================== 
… 
Troubleshooting done: 
============================ 
… 
Next steps: 
=========================== 
… 
Timeline for next steps: 
=========================== 
…

 

From: https://blogs.msdn.microsoft.com/tess/2009/09/09/first-step-in-troubleshooting-complex-issues-define-and-scope-your-issue-properly/