The Architecture Journal:Test-Driven Development and Continuous Integration for Mobile Applications
Test-Driven Development and Continuous Integration for Mobile Applications
by Munjal Budhabhatti
Summary: This article demonstrates how test-driven development and continuous integration addresses the unique challenges encountered when creating Windows Mobile applications.
Contents
Current State of Mobile Development: Issues and Challenges
Test-Driven Development
Unit testing
Continuous Integration
Source Code Repository
wMobinium.net
Faster Successful Builds
Benefits of Using CI
Conclusion
Resources
About the Author
Current State of Mobile Development: Issues and Challenges
Globally, the number of mobile phone subscribers is approximately 2.5 billion and is expected to grow to 4 billion by 2010. The mobile device is now a richer platform for application delivery due to such an exponential growth and wide spread usage. The critical factor, as always, is the end user experience: application usability, reliability, and performance.
Complicating matters, the software development world is moving from weekly and monthly deployment cycles to continuous deployment. So how can one ensure that a user always has the best experience?
Many that have looked at the agile space will be familiar with two of the core extreme programming practices: The driving of development with automated tests, a style of development introduced by called test-driven development; and a software development practice of frequently integrating builds called continuous integration, as articulated by Matthew Foemmel and Martin Fowler.
These practices are not new to the software world. However, mobile application development has lagged in taking advantage of the test-driven development and continuous integration endowed by the enterprise software community. This is partially a result of limited or unavailable mobile platform support in existing toolsets such as NUnit/MSTest or Cruisecontrol.net/Team Foundation Server.
A few mobile testing tools allow recording user interactions via a graphical representation of the client device but do not provide granular control over the tests. Other tools either demand scripting on a mobile device or expect tests to be executed, manually, on the device. As a result, mobile application testing is inefficient and complex, hindering productivity.
Test-Driven Development
Test-driven development (TDD) is an evolutionary approach where the development of code is driven by first writing an automated test case, followed by writing the code to fulfill the test and then refactoring.
TDD essentially is test-first development + refactoring
Red/Green/Refactor—the TDD mantra—is an order to the task of programming:
- Red: Write a small automated unit test that doesn’t pass, and perhaps doesn’t even compile at first. (See Figure 1.)
- Green: Write the code necessary to pass the failing test. Ensure other tests pass as well, if present, in the suite. Check-in the code in the source code repository. (See Figure 2.)
- Refactor: Making existing code beautiful, in small incremental steps, without changing the intent. (See Figure 3.)
- 红: 写一小段自动不能通过的单元测试，甚至一开始都不能编译。（如图1）
- 绿: 写必须的代码使得测试能够通过。同时保证其他在test suite中的的测试也能通过。（如图2）
- 重构: 再不改变程序意图的前提下通过增量的步骤使代码更加优雅。（如图3）
Figure 1. Sample test cases
Figure 2. Sample code
Figure 3. Sample refactoring
This technique is thus reverse to traditional programming—developing code followed by writing a test, which is executed either manually or automatically. Why embrace such a change, especially when one might tend to think that it’s extra work? In reality, test-driven development is risk-averse programming, investing work in the near term to avoid failures (and even more work) in the late term—Kent Beck has called it “a way of managing fear during programming.”
“Test-driven development is a way of managing fear during programming—fear not in a bad way—but fear in the legitimate. If pain is nature’s way of saying ‘Stop! ’ then fear is nature’s way of saying ‘Be careful.’”—Kent Beck
The benefits of TDD
- Design improvement. Writing a self-contained test case enforces the creation of decoupled code—not tightly integrated with other code—thereby increasing the cohesion of the code while decreasing the coupling.
- Documentation. A well-written unit test case provides a working specification and communicates the intent of the code clearly. In addition, whenever the code changes, the unit test case must be updated to pass the test suite. Hence, a unit test case always stays in sync with the code naturally. This is unlike traditional unit test cases, developed with Microsoft PowerPoint or Microsoft Word. While such documents start off with good intentions, over time, the result often becomes out of sync with the underlying implementation.
- Safe change in the system. TDD provides continuous feedback about whether the changes made in the code worked well with the other parts of the system.
- Fail fast. Unit testing can isolate problems quickly, reducing debugging activities and allowing the system to fail fast.
- Beautiful code. Beautiful code means code which expresses intent clearly, can afford changes to add features, and has no duplication. Refactoring retains the behavioral semantics of the code—functionality is neither added nor removed. It is about improving the code quality which brings business value.
- 设计上的进步. 开发自包含的测试用例强迫代码的解藕－不要和其他代码的绑定－因此同时提升了内聚
- 文档. 一个写得很好的单元测试用例提供了一个工作文档并且清楚的传达了程序的意图。测试驱动开发中的单元测试用例总是于代码自然的同步。因此，不像传统的用Microsoft PowerPoint or Microsoft Word开发的单元测试用例，虽然从很好的意图开始，随着时间的流逝，结果是测试用例和实现脱节。
- 确保改变的安全性. TDD为改动是否与系统的其他部分正常工作提供持续的反馈
- 快速失败. 单元测试可以快速隔离问题，减少debug并允许系统快速失败。
- 漂亮的代码. 漂亮的代码意味着清楚的表达意图，可以修改以增加功能，没有冗余。重构保持了代码语意的行为－不是增加或减少功能，而是提高代码质量从而带来业务上的好处。
Automated unit test execution is one of the vital requirements of TDD. However, because the testing tools are still evolving, the automated execution is not currently viable in mobile application development. Implementing TDD in this environment is therefore quite challenging, if not impossible.
Unit testing
Testing is customarily thought to be as a methodical process of proving the existence or lack of faults in a system. When a test case is written before writing the code, the test case becomes a specification, instead of a mere verification of the feature.
Tests are also a way of documenting found defects. Let’s assume a defect was discovered in quality assurance while testing newly deployed bits. Even if this defect was very trivial to fix, TDD demands a test case. First, write a test case that simulates such a failing behavior, and then write code to pass the test. Such a practice would ensure that defects, no matter how petty, do not creep through the system and regression testing becomes part of the test suite. Automated test execution locally, before committing the changes to the , would further reduce the broken builds phenomenon.
It is important to prepare the test environment on an emulator as close to the target hardware as possible. Developing and testing Windows Mobile applications on an makes little sense when targeting hardware exclusively for ARM architecture, an architecture dominant in low power consumption electronics. Furthermore, in the real world, components often have dependencies on other objects, databases, or network connections. It is very easy to fall into a trap of assuming that these dependencies work flawlessly. Hence if tests are written without taking dependencies into consideration, an incorrect feedback is possible for those tests which fail due to dependency problems.
One way to safeguard against dependencies is to build the object graph or set up the database in a required state before executing the test case. This would solve the issue but would increase test execution time and build time.
A more elegant approach would be to instantiate test objects and replace object dependency by implementing mocks or stubs—objects that imitate the behavior of real objects. This ensures isolated test execution and hence reliable test results. Caution should be taken while faking the real objects with mocks or stubs. It is probable that the entire unit test suite executes faultlessly, yet the product might fail in quality assurance testing. I have found in my own experience that complementing mocks and stubs objects with integration tests provides a true sense of confidence.
Martin Fowler has described continuous integration (CI) as a software development practice of frequently integrating builds, often multiple times a day. A typical CI workflow, as shown in Figure 4, would be:
Developer:
- Writes a new unit test case.
- Executes the unit test case locally on the emulator and confirms that the test case is failing.
- Adds or modifies code to pass the test case.
- Executes the unit test case locally on the emulator and confirms that the test case is passing.
- Commits the code to the SCR.
CI Server:
- Downloads the source code whenever there is a change in the SCR.
- Compiles and inspects the source code, and creates new binaries.
- Sets up external dependencies such as database schemas, and resources (configuration files, satellite assemblies).
- Deploys the new binaries to and executes the tests on the emulator.
- Packages and deploys the product to staging environment.
- Generates feedback based on the results of the build.
Martin Fowler 描述了持续集成（CI)是一个频繁构建的软件开发实践，常常是每天构建几次。一个典型的CI工作流，如图4所示：
Developer:
- 开发新的单元测试用例
- 本地执行单元测试用例，确认测试失败
- 增加或修改代码使测试通过。
- 本地执行单元测试用例，确认测试通过
- 提交代码到SCR
CI Server:
- 只要SCR中有代码改动就下载源代码
- 编译和检查源代码，创建新的binaries
- 安装外部依赖如数据库表，和资源（配置文件，卫星程序集）
- 部署binaries到模拟器并执行之
- 打包并部署产品到staging环境
- 基于构建结果产生反馈信息
Figure 4. Continuous integration in .Net Windows Mobile application
Source Code Repository
All the essential files required to build a product reside in the source code repository (SCR). It plays an important role in the software development life cycle and CI. SCR tools such as Subversion and Visual Studio Team Foundation source control enable teams to work collaboratively—on the same or different artefacts simultaneously, track code changes effortlessly, and work on different versions of files concurrently.
The CI server obtains the latest source code from SCR, locates all required dependencies, and builds the product independently from the previous build output. SCR allows the team to be more productive—a new team member does not need to reconfigure third-party libraries, project structures, or IDE settings for the project. Moreover, it reduces debugging time by allowing the team to remove the current changes which would be small and incremental if using TDD practices. The system could be safely reverted to a previous version of the code.
It is vital to include all dependencies in the SCR: This includes the Windows Mobile SDK, .Net Compact framework installer, Virtual Machine Network service drivers and other third-party components and utilities.
“Continuous integration is a software development practice where members of a team integrate their work frequently—usually each person integrates at least daily—leading to multiple integrations per day. Each integration is verified by an automated build (including test) to detect integration errors as quickly as possible. Many teams find that this approach leads to significantly reduced integration problems and allows a team to develop cohesive software more rapidly.”—Martin Fowler
－Martin Fowler
Automated build
Contradictory to some misconceptions, the automated build in mobile applications is much more than a simple code compilation. It assembles source code from the SCR, compiles code to create binaries and dependencies (such as configuration objects, resource assemblies, and so forth), inspects and deploys compiled binaries to a mobile device or an emulator, loads database schemas, and executes tests remotely on the mobile device.
Build tools such as Nant and MSBuild do not support mobile application deployment, partially because mobile device support in these tools is still immature. In addition, unit testing tools such as NUnit and MSTest have a similar problem of executing tests remotely on a mobile device. To avoid these limitations a new tool is essential.
构建工具入Nant和MSBuild并不支持移动应用程序部署，部分因为这些工具对移动设备的支持不成熟。另外，单元测试工具如Nunit和MSTest在远程执行测试中也有同样问题。为了避免这些限制，一个新的工具是必不可少的。
wMobinium.net
To address these problems, I created a tool wMobinium.net which assists TDD and CI implementation in .Net mobile applications. wMobinium.net is a unit testing tool that supports automated deployment and automated remote unit test execution. It has a Visual Studio add-in to support TDD in .Net mobile applications. It is a freely available tool on the CodePlex open source Website (see Resources).
Automated deployment
Unlike desktop application development, mobile application development faces unique challenges in deployment: first, a deployment is required for unit testing features on the device and second, deployment is necessary after a successful build to deliver the working solution on a staging environment for quality assurance testing.
For every build, the newly compiled binaries and dependencies must be copied to a program folder on a device. When an application entails testing on multiple devices, there will be added complications to the deployment process. To circumvent these problems, wMobinium.net offers a deployment tool which implements Window CE’s Remote Application Programming Interface (RAPI), and facilitates file and folder deployments. This relieves the pain of manual deployments.
对于每次构建，新编译的binaries和依赖必须被拷贝到设备的程序目录中。当一个应用限定测试需要在多个设备上，出现了新的部署的难题。为了克服这些问题，wMobinium.net提高你关了一个部署工具，用Window CE的RAPI实现，轻松文件和目录的部署。
Frequent commits
One of the key components for a successful CI implementation is frequent commits and hence frequent builds. With longer commit intervals, team members tend to work in isolation and the build is more prone to integration issues. When a team member spends more time on a feature and encounters integration problems while committing the code, he tends to be reluctant to remove the changes and revert to a previous version of the code. This reluctance might increase the amount of time and resources spent on debugging activities. In an ideal scenario, team members check-in the code at intervals of 30-60 minutes or less. The check-in duration could be extended by a few hours, but should always be less than a day.
Faster Successful Builds
Once a developer commits code to the SCR, waiting for the feedback from CI slows down the development process. Longer waits result in decreased productivity. Furthermore, some of the subprocesses of the build, such as deployment and testing, are executed on a device/emulator making these processes inherently slower than it would have been on a desktop.
To reduce build times, concentrate on the weakest link—the component that takes the longest time to execute. More often than not, the cause would be an external dependency, such as a database or other objects. Accessing a database and setting up test data for each test case is a resource-intensive operation. As I mentioned earlier, mocks or stubs, should do the trick. If it is impractical to do so, move the test cases to secondary or nightly builds—scheduled builds that execute at night when most of the resources are idle. Test cases targeting tests scenarios on multiple devices should be added to secondary or nightly builds as well.
Failing builds cause the most frustration—as if the entire chain of software churning has been stopped. The code in the SCR is no more reliable and the team is blocked from getting the latest source code. The team needs to resolve the issue quickly by fixing the build. If a test case is causing the failure and fixing the problem might require a longer duration, it is safe to ignore the test case temporarily to allow the build to succeed. However, it is vital to track these ignored test cases on an easily accessible project wall, a physical white board or a virtual board using collaborative software. The ignored tests should be fixed at a later time and added back to the test suite.
为了减少构建时间，集中力量最薄弱的环节－考虑执行时间最长的组件。更多的往往是一个外部的依赖，如数据库或其他物体。访问一个数据库，并建立测试数据为每个测试案例是一个资源密集型操作。正如我刚才所说，mocks或者stubs，是可以采取的技巧。如果这样做是不可行的，把测试用例转移到第二级或者夜间构建－－构建可以被预定在夜间利用大部分资源闲置的时间执行。测试案例中的针对多种设备的测试方案也应该被转移到夜间构建中。
失败的构建最令人沮丧，仿佛软件制造链被停止。SCR中的代码不再可靠和整个team因为不能获取最新的正确代码而被卡住。该小组需要解决问题并迅速修复构建。如果一个测试案例造成失败而解决这个问题可能需要较长的时间，可以暂时的忽视测试案例并允许构建成功。然而，重要的是在项目中容易的追踪这些暂时忽略的测试案例，可以试用白板或一个协同软件（bugzilla）。暂时忽略的测试应在稍后的时间修复并增加回测试suite。
Automated unit testing
It is quite common to see that the same defects resurface after a few builds and we often hear a quality assurance analyst say, “but this defect was already fixed in a previous build.” Boomerang defects reveal the importance of writing a test for each encountered defect before modifying the code base. Once the test case is fixed, the entire test suite must be executed and passed before checking in the modified code to the SCR.
I have been on a few projects where the team executes the test suite manually on a mobile device. Imagine a mobile application developer who spends few minutes changing the functionality, but spends double the time to test it manually. This will not only discourage a developer but will also affect productivity.
wMobinium.net resolves this annoyance by automating the entire unit testing workflow. Unlike traditional unit testing, wMobinium.net presents the test case selection on the desktop, executes tests on the device, and displays results on the desktop. It takes care of some of the complications such as the following:
wMobinium.net通过自动化真个单元测试流程解决了这些问题。不像传统的单元测试，wMobinium.net在本地展示测试用例的选择，在远程设备上执行，并在本地展示测试结果。他解决了如下的复杂问题：
Remote execution of test cases
In order to execute the test cases remotely on a mobile device/emulator, the tool serializes metadata information of the selected test cases, starts a conduit process and executes tests on the device. To provide correct reporting to the CI server, the remote process must be started synchronously and monitored continuously, which is quite a challenge.
为了在移动设备商远程执行测试用例， 这个工具将选择的需要运行的测试用例元数据序列化，启动一个管道进程并在设备上执行测试。为了向CI服务器提供正确的测试报告，远程进程必须同时启动并不停的监控测试过程，这是一个相当的挑战。
Serializing test results to desktop
In the absence of support for remoting in the .Net Compact Framework version 2.0, the device must communicate with the desktop using sockets. The events must be serialized, sent to the desktop through a socket, deserialized, and propagated to the appropriate event listeners.
.Net Compact Framework version 2.0 不报含对remoting的支持，设备必须通过sockets和本地通信。 事件被序列化，通过socket发送到本地，反序列化然后传播到合适的实践侦听程序。
wMobinium.net add-in
The tools described here assist the CI server to continuously build, deploy, and test a .Net mobile application. It would be convenient if the unit testing feature was supported as an integrated tool in Visual Studio.
wMobinium.net add-in, a Visual Studio add-in (Figure 5), is a part of the wMobinium.net toolset. After activating the add-in, all the available tests in the solution are displayed in the tool window. A typical workflow would be:
- wMobinium.net tool displays available test cases in a solution.
- User selects test cases to execute.
- Selected test cases are serialized and sent to the connected device/ emulator.
- Test cases are executed on the device/emulator and results are sent back to the desktop.
- Tool displays results on the desktop.
wMobinium插件，一个Visual Studio 插件（图5），是wMobinium.Net工具集的一部分。在激活插件后，所有解决方案中可用的测试显示在工具窗口。一个典型的工作流：
1. wMobinium.net 工具显示解决方案中的可用测试用例
2. 用户选择测试用例以执行
3. 被选择的测试用例被序列化到链接的设备或者模拟器
4. 测试用例在设备或者模拟器上被执行，测试结果返回本地。
5. 工具在本地显示测试结果
Figure 5. wMobinium.net add-in
Benefits of Using CI
Stakeholders and project sponsors always favor reliable outcomes, clear communication, project visibility, and superior quality of software. Software development, however, seldom offers such qualities without the right processes and practices.
When everything seems to be going well, any defect might suddenly jeopardize the development schedule. Especially during “Big-bang” integrations, even small issues—like missing configuration entries, out-of-sync database, or missing dependencies—could be extremely detrimental when encountered together.
Continuous integration enables faster feedback. At every change—adding new or modifying existing features, no matter how big or small—the CI server would integrate the new parts which would pass through the entire automatic build cycle—compilation, testing, inspection, and deployment. This provides visibility of the progress of the project, enhances the quality of the software developed, and builds the morale of the team.
CI does not provide these functionalities out-of-the-box. It is very possible to implement CI without including automated tests or inspection in the builds process; however, such a setup would be the least beneficial. Many, including me, consider that CI without testing is not CI at all.
当一切看起来运行良好，任何缺陷都可能突然影响开发计划。特别是在大量的整合中，任何小问题－比如丢失配置项，数据库不同步，丢失依赖项－当它们一起出现的时候简直糟透了。
持续集成可以更快的得到反馈。每一次改动－增加或者修改现存的功能，不管是大是小－CI服务器整合这些新的部分，它们应该通过整个自动构建周期－编译，测试，检查和部署。这提供了项目进程的可视化，增强了软件开发的质量，并且提高了整个team的士气。
自动构建并不提供开箱即用的功能。很有可能实现一个不包含测试和检查的CI构建过程；然而如此的一个安装过程也是有用的。很多人包括我都认为没有测试的CI不是一个真正的CI。
Conclusion
From a user perspective, TDD and CI implementation is the same in a traditional desktop application as it is in a mobile application. The user creates a new failing automated test case, writes the code to pass the test, and refactors the code without changing the intent. The CI server polls for the latest source code creates new binaries, executes the tests, and generates the feedback. However, in a mobile application the implementation differs in the remote execution of test cases, notification of test results, and build deployments. These complexities are handled by the wMobinium tool.
In my past experience at one of the biggest microfinance organizations in Africa, my team and I developed a .Net mobile application. In the absence of supporting tools, the development implementing TDD and CI, although arduous, improved overall application design, reliability, and performance.
With the release of .Net Compact Framework version 2.0, the performance of .Net mobile applications has improved radically. The newer version provides improved developer productivity, greater compatibility with the full .Net framework, and increased support for device debugging. Combining .Net Compact Framework with TDD and CI (using wMobinium.net) would bring greater benefits to an organization and take the mobile application platform to the next level.
With the proliferation of new mobile devices today, the mobile application is becoming a crucial part of a broader enterprise product offering. It is pragmatic, more than ever, to bring mobile application development out from isolation and include it in enterprise-wide test-driven development and continuous integration efforts.
之前我在非洲最大的金融组织中的经验中，我的team和我开发了一个.NET移动应用程序。在缺少支持工具的情况下，使用TDD和CI开发虽然艰苦，但是提高了整个应用程序的设计，可靠性和性能。
随着.Net Compact Framework version 2.0的发布，.Net mobile application的性能得到了快速的提升.最新的版本提供了更高的生产率,更多和完整.NET架构的兼容性,也提升了设备debug的支持..Net Compact Framework ,TDD 和CI(使用wMobinum.net) 的结合为组织带来更大的好处并把移动应用开发提高到一个新的层次.
在新的移动设备蓬勃发展的今天,移动应用程序正在变成广泛的企业应用产品提供的一部分.将移动应用程序开发从孤立带入企业级别的测试驱动开发和持续集成比以往任何时候都更具有实际意义.
Resources
- “Continuous Integration,” Martin Fowler and Matthew Foemmel http://www.martinfowler.com/articles/continuousIntegration.html
- Continuous Integration: Improved Software Quality and Reducing Risk, Paul Duvall, Steve Matyas, and Andrew Glover (Addison-Wesley Professional, 2007)
- Test-Driven Development: By Example, Kent Beck (Addison-Wesley Professional, 2002)
- wMobinium.net http://www.codeplex.com/wMobinium
About the Author
Munjal Budhabhatti is a senior solution developer at ThoughtWorks. He possesses over 10 years of experience in designing large-scale enterprise applications and has implemented innovative solutions for some of the largest microfinance, insurance and financial organizations in Africa, Asia, Europe, and North America. He spends most of his time writing well-designed enterprise applications using agile processes.
This article was published in the Architecture Journal, a print and online publication produced by Microsoft. For more articles from this publication, please visit the Architecture Journal Web site.
本文发表于Architecture Journal,微软的可打印的和在线产品.更多细节,请访问Architecture Journal Web site.
