Introduce MBT tool robogenerator
What is Tool Robogenerator?
Test Data generator for `Robot Framework <http://www.robotframework.org>`
What is the tool’s purpose?
You know those ugly bugs that users report that somehow were missed by all your
ATDD-, TDD- and exploratory testing?
Those bugs that lurk in the corners of complexity of the system that you are building.
Those bugs that will make users of your application hate you..
This tool is here to help you seek and destroy those bugs before the users will find them.
It gives you the ability to generate a huge number of tests that can go through
a very vast number of similar (or not so similar) test scenarios.
What is it exactly?
If you know Regular Expressions:
A regular expression is a pattern that represents all the possible strings that match that pattern.
Robogenerator works in a similar way. A machine represents a set of tests in a compact way.
This machine can be used to generate all (or part of) the tests that it represents.
If you know Model-Based Testing or automata theory:
With Robogenerator you define an extended finite state machine (in a python grammer) that represents a set of tests.
Robogenerator also contains algorithms that can be used to generate real executable Robot Framework tests from a Robogenerator model.
If you know Combinatorial Testing or like data driven testing:
With Robogenerator you can define sets of data and rules about that data, that can be used to generate data driven tests
(and also keyword driven tests). This allows you to keep your data in compact sets and let the system generate the real
test data vectors from there.
The language used to describe the model
It will use pure python grammar, there are advantage and disadvantage about this.
Advantage:
Very flexible, all python related grammar could be used here
Disadvantage:
For tester who totally has no programming experience or weak competence on it, it is a barrier for them to use this framework and turn their test ideas into test automation case.
Here I would like to introduce this tool by several actual examples
No1: using combinatorial techniques
1. How we describe our test
2. How to filter out invalid test data
3 Currently we support 4 kinds of data generation algorithm.
random ----generate test data randomly
DFS---- deep first search, generate all of possible combinations
pairwise--- generate test data in pairwise algorithm
smart-random-------generate test data randomly, and in next run, it will only select un-tested combination in all possible combination
4. Could indicate how many test case to generate
5. Could indicate the style which is used to display test data
normal---- normal Robot Test Case
atdd--------use Robot Template method, so the input/output model in test case would be more visible
6. The test case name would adapt with actual data combination
7. The output filename could be indicated
5.1the output file could be indicated inside model file, or indicated as a command-line argument
5.2 The output file could be in TXT, HTML,or other valid Robot Case format
8. The case step should be a keyword which has been defined inside Robot Resource file or python test library
9. Could indicate actual test device info, such as ip, could test data could come from actual test device, it could be well adapted to different hardware topology.
You could use your own test library to control how to use this feature
9.How to generate test case:
Assume the model filename is create_and_delete_volume_test.py , and randomly generate 10 test case, output file is show.html
robogenerator - g random –t 10 –o show.html –ip *.*.*.* create_and_delete_volume_test.py
No2: using model-based testing with finite state machine
1. How to describe our MBT test model
2. different strategy to test
ShortestPath ----- Generate case according to China Postman graph algorithm, which could balance test feedback cycle and test coverage
StateCoverage ------- When need to select next step, it will select the minimum state as their next State
ActionNameCoverage ------- Select least covered Action Name To test
DynamicRandom ------------- Randomly select next State but try to avoid already tested
3. Could indicate how many steps inside one test case
4. Examples
generate test case with shortest path cost,assume each step has same weight,and output to show.html
robogenerator SIP_Call_Service.py –-strategy ShortestPath –o show.html
generate test case with ActionName coverage, and generate 1 case, each case will limit test steps to 100
robogenerator SIP_Call_Service.py –-strategy ActionNameCoverage –t 1 –n 100 –o show.html
No3:using both combinatorial and state-machine model techniques
At first it will generate test data according to actual test env, and update parameters, and then generate test case according to specified test strategy.
robogenerator –g dfs –ip *.*.*.* –strategy ShortestPath –o show.html
Other extended Function (Could generate State-machine based graph)
If you have installed tool GraphViz, then you could use robograph (a tool inside robogenerator) to generate state-activity graph by your state-machine model.
1. At first
robograph SIP_Call_Service # without .py suffix
then there will be a file SIP_Call_Service.dot in current directory
Then you use Gvedit.ext to open file SIP_Call_Service.dot .
It will show following picture:
From the picture we could see, there are a lot of possible sequence to execute all possible test case, if we don’t have tool help, it will be difficult for us to design test procedure which could balance test cost and test coverage.