Configuration testing: QA pros discuss 10 things you may not know

Configuration testing is the testing of a system with each of the configurations of software and hardware that are supported. This can be a challenge for testers, especially with today's rate of change. Just ask Marc Nadeau, senior director of QA at Blackboard Inc., a provider of education technology. "In the last 12 months we've seen three versions of browsers, and we're on the verge of seeing two more operating systems this year," says Nadeau. "It's becoming a more daunting task."

To help avoid some of the pitfalls to this task, here are some things you might not know about configuration testing:

1. You may not realize the impact a configuration can have on how an application functions.

"Applications don't move on their own, they have pieces and parts and if those pieces and parts don't move with it …" it can create a problem, said Karen N. Johnson, an independent software test consultant. For example, she said, the root of a problem to a recent Web application she was testing was that RubyGem needed to be updated. RubyGem is the Ruby standard for publishing and managing third-party libraries.

2. There are few standards on the hardware side.

"There are not standards efforts to identify what a given piece of hardware delivers," said Frank Cohen, CEO and founder of PushToTest, "which is ironic because at the chip level there's incredible specifications for what the function of the chip is. But beyond the motherboard level there's almost nothing that's standard. Say you're reading the CPU utilization on Vista, so you have to go through a proprietary API; versus with Windows, XP you have PerfMon."

3. There's a development lifecycle to hardware as well as software.

"As you move from [software] development to production, they might not have the right test data as input to an automated test to know something changed [in the hardware]," said Cohen.

4. It's easy to not know that the environment changed because of background updates that may be turned on/running.

According to Johnson, maintaining environments is becoming increasingly difficult with Windows updates and other background processes that may make changes and updates. "It's a nightmare," she said. "A case in point is a Web application. With so many Windows machines having automatic updates, the environment can be changing at such a rate of speed it's elusive. This week I spent time tracking down how a toolbar added to a browser introduced a bug. It's not uncommon for people to end up with a huge amount of add-ons for their environment."

5. Developers can provide a heads-up to testers.

With issues like add-on and updates, "I don't expect developers to anticipate all of it, but if they can tell me about the pieces that are vulnerable then I can spend time on how the browsers carry something out. It is possible for them to highlight something," Johnson said. For example, she said, an application may have a third-party widget. "If they haven't coded that piece, I might want to look at it with a different eye."

6. Configuration testing can be difficult without automation.

Manual testing "works fine if you've got a small, limited deployment," Cohen said, "but for Internet, SOA, and Rich Internet Applications there's no way to be successful—there are too many moving parts. When a system is talking to another system, say load testing, the results will vary each time because it depends on what the other system is doing."

7. A definition or model to test against is needed.

"Most often organizations get tripped up because they're not 100% representative of what is implemented," Cohen said. "The method for configuration testing is to identify models of use for a particular piece of equipment and test against that model. Say a student at a high school is using a new laptop—they're not going to be needing Gigabit Ethernet connections. So you identify archetypes."

"What we've done is rotate mini test cycles and operations across platforms," said Blackboard's Nadeau, "then we have a subset of certain tests. We've created a suite of browser tests, so we test those parts of the applications that tend to be impacted by the OS or a browser change, so anytime there's a new browser or an OS change we execute that suite of things. The best we can do is create a suite of tests to do rigorous tests in those areas, and then a spattering of tests across platforms when [the application] stabilizes. That's where automation is critical."

8. Configuration testing can be expensive.

Johnson lists several reasons for the high cost of configuration testing, including the following:

• It takes awareness to know what all the aspects of the configuration are that matter.
• You have to take care of the environment to maintain all the factors.
• It may increase the amount of testing needed.
• It could require multiple hardware configurations that generate more costs.

9. It's not well understood at the executive level.

"For the most part the topic bores people," Johnson said. "The technology people would find it interesting, but when it comes up a couple levels in the management line it bores them instantly, but if they don't understand the importance it comes back with a ripple effect of problems. They don't know what needs ten more devices to test on; they don't know the labor involved in the care and feeding of machines. You've got to fight for money if people don't know what the point is."

10. Not everybody does configuration testing.

"They either skip it and don't understand it, or they're tuned into it and overwhelmed by the complexity, or they figure they can't afford it and skip it. In some cases that is the practical solution," said Johnson. "Reports come in from production and you then drill in and figure it out. I don't mean to make that sound alarming. But for the amount of labor it takes to find two or three bugs, it may not be worth it. One of biggest things is to figure out the audience. There are always laggards—are you going to worry about Windows 98 when it's one percent of your audience?"