In this post I will show how coverage.py can help you write complete tests and how to write a test for a case that should not ever occur.
Meanwhile I am adding more bytecodes to my interpreter, so if you are interested in my implementation of python bytecodes take a look at https://github.com/leovt/interp/blob/master/interp.py.
In my last post i described why and how I switched to unittest testing. Running the tests after every code change makes it much more comfortable and I can be reasonably confident that the cange did not break what was working before.
But there is a nagging question: Do I really test all possible paths in my code? Did I forget to test some exceptional code path? The answer to this question is given by Ned Batchelders coverage.py
This script will record which lines of code are executed and then produce a nice report.
Installation
I am using ubuntu, so I could install it simply with
sudo apt-get install python-coverage
Running the coverage tool
First clear the results of previous runs
leonhard@leovaio:~/interp$ python-coverage erase
Run the unittest with the coverage tool
leonhard@leovaio:~/interp$ python-coverage run interp_test.py -b ...s........ ---------------------------------------------------------------------- Ran 12 tests in 0.019s OK (skipped=1)
Then run coverage again to produce a report.
leonhard@leovaio:~/interp$ python-coverage report -m Name Stmts Miss Cover Missing ---------------------------------------------------------------------- /usr/share/pyshared/coverage/collector 132 127 4% 3-229, 236-244, 248-292 /usr/share/pyshared/coverage/control 236 235 1% 3-355, 358-624 /usr/share/pyshared/coverage/execfile 35 14 60% 3-17, 42-43, 48, 58-65 interp 182 1 99% 269 interp_test 103 8 92% 44-52, 122 ---------------------------------------------------------------------- TOTAL 688 384 44%
The line that interests me is the one for interp.py
interp 182 1 99% 269
Apparently my tests miss one line, line 269 in interp.py. This line is throwing the exception, when an unknown bytecode is found. How can I write a test that executes this line?
I could simply run my interpreter with bytecodes that I have not yet implemented, but this would invalidate the test when I add those bytecodes later. Therefore I want to test with a bytecode that does not exist in Python, e.g. bytecode 255.
Obvisually I cannot create a code object with this code in it just by definig a function. I use a technique called mocking which takes advantage of duck-typing in Python. I don’t really need to pass a code object to my interpreter, any object with the right attributes will do. There are libraries that help creating mock objects, but for this simple use case I will just roll my own:
class Mock(object):
co_code = '\xff\x00\x00'
co_nlocals = 0
I can then verify that this illegal bytecode is acutally causing line 269 to be executed and the exeption being raised by adding the unittest
def test_unknown_opcode(self):
class Mock(object):
co_code = '\xff\x00\x00'
co_nlocals = 0
with self.assertRaises(interp.InterpError):
interp.execute(Mock(), {})
By the way the coverage tool can also produce a very nice html report hilighting with colors the covered and not covered lines in a python file.
