In software engineering, continuous integration means the continuous application of quality control processes — small units of effort, applied frequently.
In this blog we’ll show you how to set up a CI job with Hudson/Jenkins that automatically builds and executes your Ranorex automation as well as automatically sends out the generated test reports for every committed change in a Subversion repository.
- Advantages of Continuous Integration Testing
- Infrastructure
- Plugins
- Add New Job
- Run Job
- Add Repository Hook
- Conclusion
Advantages of Continuous Integration Testing
Continuous integration has many advantages:
- When tests fail or bugs emerge, developers can revert the codebase to a bug-free state without wasting time for debugging
- Developers detect and fix integration problems continuously – and thus avoid last-minute chaos at release dates
- Early warning of broken/incompatible code
- Early warning of conflicting changes
- Immediate testing of all changes
- Constant availability of a “current” build for testing, demo, or release purposes
- Immediate feedback to developers on the quality, functionality, or system-wide impact of their written code
- Frequent code check-ins push developers to create modular, less complex code
Infrastructure
Continuous Integration Tool
You can find a download link and installation description for Hudson and Jenkins here:
In this blog post we are going to use Jenkins as CI tool. There shouldn’t be much of a difference when using Hudson.
As Jenkins or the nodes executing the CI jobs normally are started as Windows services, they do not have sufficient rights to start UI-applications.
Please make sure that Jenkins as master and its slave nodes, where the Ranorex automation should be triggered, are not started as a service.
For the Jenkins master, open the “Services” tool (which is part of the “Administrative Tools” in the control panel), choose “Jenkins” service, stop the service, and set the “Startup type” to disabled:
Use the following command to start Jenkins manually from the installation folder:
java -jar jenkins.war
After starting Jenkins, use this address to access the web interface:
To configure your Jenkins server, navigate to the Jenkins menu and select “Manage Jenkins” -> “Configure System”:
Note: It is necessary to have the Ranorex main components – and a valid Ranorex license – installed on each machine you want to build and execute Ranorex code.
Source Code Management
As mentioned before, we are going to use a Subversion repository as base of our continuous integration process.
In this sample, we have two solutions in our repository: the application under test and as the automated Ranorex tests.
To start the application under test from your test project, simply add a new “Run Application” action to your action table in Ranorex Studio, which starts the application under test, using a relative path to the repository root:
Plugins
As we want to build our code for each committed change within our SVN repository, we need a Subversion as well as a MS Build plugin for Jenkins. An additional mail plugin will make sure that a mail is sent with each build.
Install Plugins
Open the “Manage Plugins” section (“Manage Jenkins” -> “Manage Plugins”), choose the following plugins from the list of available plugins and install them if they are not installed already:
- MSBuild Plugin
- Email Extension Plugin
- Subversion Plugin
Configure Plugins
The installed plugins also need to be configured. To do so
- open the “Configure System” and configure the “Extended E-Mail Notification” plugin. To do so, set the recipients and alter the subject and content (adding the environment variable $BUILD_LOG to the content will add the whole console output of the build and the test to the sent mail),
- configure the “E-mail Notification” plugin by setting the SMTP server.
- and navigate to “Global Tool Configuraiton” and configure your “MSBuild” plugin by choosing the “msbuild.exe” installed on your machine.
Add New Job
Now, as the system is configured, we can add a new Jenkins job, which will update the checked out files from a SVN repository, build both the application under test and the Ranorex automation project, execute the application under test as well as the automation code and send a mail with the report file attached.
Start by creating a new item. Choose “Build free-style software project” as job type and enter a job name:
Configure Source Code Management
Next, we have to check out the source of both the application under test and our test automation project. Start with choosing Subversion as source code management tool. Then, enter the repository holding your application under test as well as your test automation project. Finally, choose “Use ‘svn update’ as much as possible” as check out strategy:
With this configuration, the application under test as well as the test automation project will be checked out and updated locally.
Add Build Steps
Now, as the source code management is configured, we can start with processing the updated files.
First of all, let’s add MSBuild steps for both projects:
Choose your configured MSBuild version and enter the path of the solution file relative to the repository root (which is the workspace folder of the Jenkins job) for both the automated and the automating project:
With adding these two build steps, the executables will be automatically built. Now the newly built application should be tested.
This can be accomplished by adding a new “Windows batch command” that starts the test suite executable:
As you can see, some command line arguments are passed to the test suite executable.
In this sample, the command line arguments “/zr”, which triggers the test suite executable to generate a zipped report file, and “/zrf:.ReportsReport-Build-%BUILD_NUMBER%.rxzlog”, which defines the name and the location of the generated zipped report file, are used.
You can find a list of all available command line arguments in the section “Running Tests without Ranorex Studio” in our user guide.
The test suite executable returns “0” on success and “-1” on failure. Based on this return value, Jenkins will mark the build as successful or failure.
Add Post-Build Action
After building and executing the application under test and the Ranorex test script, we will send an email which informs us about the success of the triggered build.
This email should include the zipped report file, mentioned before, as attachment.
To do so, add the new post-build action “Editable Email Notification”, choose the report file location defined before as attachment, and add triggers for each job status you want to be informed about. In this sample, an email will be sent if a job has failed or succeeded.
Run Job
Once you’ve completed these steps and saved your changes, check if everything works as expected by clicking “Build now”:
After running the generated job, you will see all finished builds within the build hierarchy. Icons indicate the status of the individual builds.
You can view the zipped report files of all builds by opening them in the local workspace (“Workspace/Reports”):
As configured before, an email will be sent to the specified email address(es), including the console output in the email text as well as the generated zipped report file as attachment.
Add Repository Hook
Now we can manually trigger a build. As we are working with Subversion, it would be beneficial to trigger the script for each commit.
To do so, you can add a server side repository hook, which automatically triggers Jenkins to start a new build for each change committed, as described in the subversion plugin documentation.
Alternatively, you can activate polling of the source code management system as build trigger in your Jenkins job configuration.
As shown in following picture, you can define the interval, after which the source code management will be invoked (e.g. 5 minutes after every full hour):
Conclusion
Following the steps above you will be able to easily setup a continuous integration process performing the automated test of the application you develop. Each commit will now trigger an automated test run. Once the test run has finished, you’ll instantly receive a mail with the Ranorex test report.
Note: This blog was originally published in July 2012 and has been revised to reflect recent technical developments.
