what Tests Belong in the BVTs?

　　BVTs or Build Verification Tests are standard Microsoft parlance for the tests we run every day to ensure that we didn’t break anything important with our checkins the day before. I’ve previously written about the importance of keeping them clean. Within the range of tests that consistently pass, which ones should be in the BVT? BVT test failures should be something you’re willing to act on immediately. In other words, the failures must be important. Based on that, here are some criteria:

  • Test major scenarios not minor ones. If major features are failing, they will be fixed right away. If a minor feature is failing, it should be noted, but may have to wait until later to be fixed.
  • Test majority use cases, not corner cases. Tests for the interaction of 3 parts shouldn’t be in the BVTs. Tests outside most user scenarios shouldn’t be in the BVTs. While every book on testing says to test the boundary conditions, the BVTs may not be the place to do that. Instead, pick the most likely to be used values and scenarios.
  • Run "positive" not "negative" tests. By that I mean, don’t send out-of-bounds conditions or invalid values. These are valid tests and should definitely be run, but not in the BVTs. An API faulting when sent a null pointer should be fixed, but the fix can wait until next week. BVTs should be a carefully guarded set of tests. They need to run quickly, consistently, and their results should matter. If these rules are followed, the BVTs will be effective because failures will be respected. Restricting the BVTs to the most important scenarios will ensure that the results are given the appropriate respect.

Published Wednesday, March 05, 2008 8:31 AM by SteveRowe

　　BVT（Build Verification Test，构建验证测试）是微软内部的一个标准说法，指的是每天都要运行的测试，以确保前一天入库的内容没有破坏重要功能。先前我曾写过一篇文章说明保持 BVT整洁的重要性（文章稍后翻译）。在那些持续通过的测试中，哪些应放入BVT中呢？BVT失败时，应该是你要立即去处理的。也就是说，BVT中失败的 测试必须是非常重要的。基于这一观点，以下是一些有关BVT的准则：

  • 测试重要场景，而非次要场景。如果一个重要功能失败了，那么应当立即解决。而如果失败的是一个次要场景，那么只是要引起重视，可能需要推到以后再去处理。
  • 测试主要用例，而非次要用例。对3个功能模块相互交互的测试不应放在BVT中。对最常见使用场景之外的测试不应放在BVT中。虽然每本测试书籍都说要测试边界值，但BVT不应用来测这些东西。BVT测试的是最常见的输入值和使用场景。
  • 做“正常(positive)”测试而非“异常”(negative)测试。不要测试边界外的条件和异常值。当然做这些测试都是必要的，但不应在BVT中做。传递一个空指针而导致的API失效当然要进行修复，但可以放到下周嘛。
　　BVT应当是一组仔细选取过的测试。它们应当能够快速、一致地运行，其结果要非常重要。始终坚持这些原则的话，BVT就非常有效果，因为任何失败都会受到足够重视。将BVT限制在最重要的场景将保证测试结果能够得到恰当的处理。。