Exploratory testing: Addressing misunderstood concepts and common challenges
Exploratory testing is probably the most common testing technique for software. Though widely employed, it is often misused and misunderstood by testing organizations and testers. Exploratory testing can harvest significant returns in terms of detected defects and overall software quality when employed appropriately. Combined with other testing techniques and technologies, it can become a significant testing force multiplier
Table of contents:
What is exploratory testing?
When to employ exploratory testing
Exploratory testing tutorial, part 2
What is exploratory testing?
Exploratory testing is often misconstrued to be the equivalent of undisciplined random testing of the application space, much in the same way that Agile development is often twisted to mean a very loose and undisciplined approach to software development. Neither definition is the intent of the discipline or technique, but it can certainly become the real-world implementation. For our purposes, let us define exploratory testing as structured ad hoc unscripted testing of an application space where the tester simultaneously learns the application space, evolves test cases, performs test execution and publishes defects.
Exploratory testing is really a way of thinking about testing and the application space under test. The definition speaks to a state of mind rather than a formal methodology. Our definition includes two key elements that are not often stressed in discussions on or about exploratory testing: structure and publication of defects.
Exploratory testing should be structured in the sense that it should be context driven, with the context determined by the goals of the current testing effort. The context of any exploratory testing effort is situational, but there are several common points of reference:
Functional area
- Focus on a particular aspect of the application space (i.e., billing).
- Focus on a problematic area of the application identified by production issues.
- Evaluation of the testing abilities or aptitude of a tester through paired testing. This can also be used as a mutual training exercise between testing peers.
- Focus on learning a new functionality while evolving test cases and identifying defects. This is the most common context for exploratory testing.
Defect name
- The name or title is the essence of the defect, including the functional area and nature of the defect.
- The description clearly states what sequence of events leads to the defect, and includes a screen snapshot or transcriptions of any error messages.
- The defect description provides sufficient detail for the triage team and the developer fixing the defect to duplicate the defect.
- The severity assigned to a defect is dependent on the impact of the defect on the testing effort, and the risk the defect presents to the business if the defect is rolled out into production.
- The impacted area can be referenced by the functional component or area of the system.
Exploratory testing is part of a testing continuum that goes from very informal exploratory testing to extremely rigid scripted test cases that capture every aspect of the test. All have their place in the overall testing process. The ability to leverage the appropriate approach at the appropriate time will yield the highest return for your testing investment.
When to employ exploratory testing
If we think of testing as a continuum that goes from completely freestyle exploratory testing to extremely rigid and structured scripted test cases, then where and when should we use structured ad hoc exploratory testing? The answer should be when it derives the most value. But how do you determine when that is? There are several common triggers that would move the focus of your testing efforts toward exploratory testing:
In each case the trigger could come from the team as a whole or from a particular tester whose concerns are based on one or more of these common triggers. (Keep in mind that this is far from an exhaustive list.)
There are two additional triggers for moving toward exploratory testing that are difficult to quantify but equally important: tester fatigue and target fixation.
In a situation where most if not all testing is manual, tester fatigue can set in. Manual testing following a predefined set of scripts with known inputs and outputs does become a very mind-numbing activity. Eventually the tester begins to see what they expect to see, leading to detectable defects reaching production. Moving back and forth between manual scripted testing and unscripted structured exploratory testing will reinvigorate the testers, lead to new defects being discovered and improve the overall quality of the end product.
Once again, in a situation where most if not all testing is predetermined scripted manual test scripts or even automated test scripts, target fixation can set in. Target fixation occurs when the tester becomes fixated on one aspect of the application to the exclusion of all other aspects of the application space. Moving toward exploratory testing and encouraging the exploration of other aspects of the application space will reinvigorate testers, allow them to refocus their efforts, lead to new defects being discovered and improve the overall quality of the end product. In all cases, leveraging exploratory testing in combination with other techniques and tools (i.e., test automation) will increase both job satisfaction and product quality.
