浅议自动化测试框架 --- 之脚本分类

一个足够灵活的测试框架应该支持测试脚本分类/分组，并且在不改变框架代码的情况下可以随着项目的进展任意添加新的分类分组，本文用一种简单方式尝试实现上述目标。

首先，需要明确的是所有测试脚本应该以文本格式呈现，而不应是二进制格式或人眼不可读的某种私有格式。在此基础上，我们可以通过向脚本插入某些测试框架可以理解的元数据配置信息来达到脚本分类的目的。例如, 下面的的示例中，通过添加配置信息 “#group-xxx：[enabled|disabled]", 测试脚本可以声明它属于哪一个分组/类以及状态（enabled|disabled) 

#!/usr/bin/python3

#group-AAA: enabled
#group-C13 : enabled
#group-C16 :enabled
#group-C18 : disabled

# test logic ...

通过上面的配置，当我们想要运行一组测试特定的脚本时，只要把这些脚本加入同一分组 (例如 “whitebox")，使用类似的命令 ”./run --group whitebox", 就可以很方便的达到我们的目的。

 

下面是我们的具体实现：

.
├── run.py
├── tests
│   ├── test1.py
│   ├── test2.py
│   └── test3.sh
└── utils
    └── parser.py

 

run.py 是entry point, 由它指定运行哪一组测试用例:

stephenw@stephenw-devbox1:/local/project/tmp/$ ./run.py -h
usage: run.py [-h] [-t TESTROOT] -g GROUP

run.py is the entry of easy test frameworkt

optional arguments:
  -h, --help            show this help message and exit
  -t TESTROOT, --testroot TESTROOT
                        The root directory of test scripts
  -g GROUP, --group GROUP
                        The test group to be run 

 

utils/parser.py 包含解析测试脚本的工具类，用于逐行扫描并获取脚本所属的测试分组。 

tests 目录作为测试脚本的根目录，当前有三个测试脚本， group C13 包含: test1.py/test2.py/test3.sh； group C16包含 test1.py； gropu C18包含 test2.py, 但是处于disabled状态，无法执行；group AAA 包含test2.py

stephenw@stephenw-devbox1:/local/project/tmp$ cat tests/test1.py  
#!/usr/bin/python3

#group-C13 : enabled
#group-C16 :enabled

import sys

print('This is test1')
sys.exit(0)

stephenw@stephenw-devbox1:/local/project/tmp$ cat tests/test2.py 
#!/usr/bin/python3

#group-C13 : enabled
#group-C18 : disabled
#group-AAA: enabled

import sys

print('This is test2')
sys.exit(0)

stephenw@stephenw-devbox1:/local/project/tmp$ cat tests/test3.sh
#!/bin/sh

#group-C13 : enabled

echo "This is test 3"
exit 0
stephenw@stephenw-devbox1:/local/project/tmp$ 　　

基于上述配置，当我们运行group C13时，三个脚本应该全部运行； 当我们运行group C16时，应该只有test1.py运行，当我们运行group C18时，没有脚本可以运行；当我们运行group AAA时，只有test2.py运行。实际结果也验证了上述推论：

stephenw@stephenw-devbox1:/local/project/tmp$ ./run.py --group C13
Below tests will be run:
	 /local/project/tmp/easytest/tests/test3.sh
	 /local/project/tmp/easytest/tests/test2.py
	 /local/project/tmp/easytest/tests/test1.py
stephenw@stephenw-devbox1:/local/project/tmp$ ./run.py --group C16
Below tests will be run:
	 /local/project/tmp/easytest/tests/test1.py
stephenw@stephenw-devbox1:/local/project/tmp$ ./run.py --group C18
Below tests will be run:
	None
stephenw@stephenw-devbox1:/local/project/tmp$ ./run.py --group AAA
Below tests will be run:
	 /local/project/tmp/easytest/tests/test2.py
stephenw@stephenw-devbox1:/local/project/tmp$

　　

具体代码如下：

1) run.py:

#!/usr/bin/python3 -u

import argparse
from os import path
import sys
from utils.parser import Parser


sys.path.append(path.dirname(__file__))


def get_argparser():
    """Construct and return supported arguments"""

    desc = 'run.py is the entry of easy test frameworkt'
    argparser = argparse.ArgumentParser(description=desc)

    argparser.add_argument('-t', '--testroot', help='The root directory of test '\
                           'scripts')
    argparser.add_argument('-g', '--group', help='The test group to be run',
                           action='append', required=True)
                           
    return argparser 


if __name__ == '__main__':
    args = get_argparser().parse_args()
    if not args.testroot:
        args.testroot = path.join(path.dirname(path.abspath(__file__)), 'tests')

    tests = Parser.getTestsFromGroups(args.testroot, args.group)
    print('Below tests will be run:')
    if not tests:
        print('\tNone')
    else:
        for test in tests:
            print('\t', test)

2)  utils/parser.py

#!/usr/bin/python3

import glob
from os import path
import re
import sys


TEST_GROUP_ENABLED = 'enabled' 
TEST_GROUP_DISABLED = 'disabled' 


class MalformedScript(Exception):
    pass


class TestProperty(object):
    def __init__(self):
        self.groups = []
        self.parallel = False

    def addGroup(self, group):
        self.groups.append(group)

    def setParallel(self, parallelRun):
        self.parallel = parallelRun
        

class Parser(object):

    @staticmethod
    def getTestsFromGroups(testRoot, testGroups):
        """ Search under directory 'testRoot' and return all test scripts
            belonging to groups 'testGroups'
        """

        tests = []
        for testScript in glob.glob(path.join(testRoot, '**/*.*'), recursive=True):
            #print('check script ', testScript)
            scriptGroups, _ = Parser.getTestProperty(testScript)
            for targetGroup in testGroups:
                if targetGroup in scriptGroups:
                    tests.append(testScript)
                    break
        return tests


    @staticmethod
    def getTestProperty(testScript):
        """Parse test script and return group list to which the test belongs and
           parallelism state (if the test can run with others in parallel).

           Note: 1) a test can belong to one or multiple groups by below
                    declarations:
                        #group-A : enabled
                        #group-B : enabled
                        #group-C : disabled
                
                    Above statments make the test belong to group 'A', 'B', 'C'
                    at same time, but the test is disabled in group 'C'.

                 2) A test can declare to be parallel (can run with other tests
                    in parallel) or not by below statement
                        #parallel : true  #or false

        """

        # Set patterns for parsing group and parallel properties
        regexGroup = re.compile('#group-(.*)(\s*):(\s*)(.+)')
        regexParallel = re.compile('#parallel(\s*):(\s*)(.+)')

        # Start to parse test script line by line
        testProperty = TestProperty()
        with open(testScript) as f:
            alreadySetParallel = False
            for line in f:
                # Parse groups
                if line.startswith('#group-'):
                    m = regexGroup.match(line)
                    if m:
                        group = m.group(1).strip()
                        state = m.group(4).strip()
                        if state == TEST_GROUP_ENABLED:
                            testProperty.addGroup(group)
                    else:
                        msgFmt = 'Invlaid setting "{}" in {}'
                        raise MalformedScript(msgFmt.format(line, testScript))

                # Parse parallelism, we should do this at most once
                elif line.startswith('#parallel'):
                    if alReadySetParallel:
                        msgFmt = 'More than one "#parallel: xxx" exist in {}'
                        raise MalformedScript(msgFmt.format(testScript))

                    alreadySetParallel = True
                    m = regexParallel.match(line)
                    if m:
                        parallelRun = (m.group(3).strip().tolower() == 'true')
                        testProperty.setParallel(parallelRun)

        return testProperty.groups, testProperty.parallel