Discussion on the state of cloud computing and open source software that helps build, manage, and deliver everything-as-a-service.
Data Driven Tests in Marvin
Data driven tests (DDT) is useful for generating more tests with less code. If you have lot of assertions to be repeated with related data sets or parameters, Data driven tests are a way to go - saves a lot of redundant code. The data sets provided as input will be used iteratively in the test scripts in effect, running different tests cases. I’ve recently tried to implement this with Marvin, CloudStack’s Integration Test framework which is written in Python.
Before beginning to code test suites in data driven fashion, it’s best to understand the entities which drive the tests. For example, we may want to test the same set of VM life cycle tests for different network offerings such as a simple isolated offering, an offering with persistent network feature and an offering with Netscaler as a service provider for LB. These offerings then become the data sets which are supplied as input to the test script. And the assertions for VM life cycle are repeated for each of these network offerings.
DDT in Python
Python provides a powerful and efficient way to achieve this with the ddt library. It comes with a class decorator @ddt and method decorators @data and @file_data. Use the class decorator with the TestCase class and specify the data sets with which the tests need to be run with the @data or @file_data method decorator. While @data takes in the arguments directly to be passed to the test, @file_data will load the data set from a JSON file.
CloudStack Test Case Example
Here’s an example:
Consider a case where we need to verify various network operations in a VPC with 2 different VPC offerings: Default VPC Offering and Default VPC Offering with NS as LB.
First, Install DDT (pip install ddt) if not already done.
More info here: http://ddt.readthedocs.org/en/latest/example.html
Then, start by defining the class decorator with the test class:
Then define the data sets using the method decorator:
def test_01_create_network(self, value):
(where the network offerings are already defined prior to starting the tests)
Result: test_01_create_network will be iterated for both the network offerings.
The complete test suite can be found here:
There’s another way of writing data driven tests in python viz. Generative tests. Nose (we use nose as the test runner with marvin) supports test generators. Detailed here: https://nose.readthedocs.org/en/latest/writing_tests.html#test-generators Using this, we can write test functions which are generators. But however, it doesn’t work if we are using unittest.TestCase subclasses, which is the case with marvin.
So, for now, I am sticking with ddt and hoping to implement this for all VPC tests and explore the possibility of using this with other test suites as well.