Using Coverage to generate reports (Python)

A couple of tips so I don't have to look it up again. A few months back I wanted to generate coverage data for a stand-alone application, with a few tweaks.

Combining the output of Trial and Python unit tests

$ coverage run -p tests/my_python_tests.py
$ coverage run -p tests/my_trial_tests.py
$ coverage combine
$ coverage html -d tests/coverage_html  --include=./*py --omit='lib/*.py,tests/*.py'

Helpful resources to get used to combine:

Getting coverage report for a simple project

$ coverage run tests/python_tests.py
$ coverage report --include=./*py --omit='lib/*.py,tests/*.py' # For a text summary
$ coverage html -d tests/reports  --include=./*py --omit='lib/*.py,tests/*.py' # For a nice HTML report

Installing the latest version

To get access to the command line tools, I had to install Coverage in a virtualenv to get around some other constraints on the system. I thought I'd build it from master while I was at it :)

$ virtualenv coverage
$ cd coverage
$ source bin/activate
$ pip install hg+http://bitbucket.org/ned/coveragepy

Coverage is a lovely library and a wonderful way to encourage yourself to think more about your unit tests -- give it a try if you haven't already. (PS: For Django, see django-coverage)

Leave a comment

Bees with machine guns Load testing

Another interesting PyCon talk to watch was Best practices for impossible deadlines. One of the tools mentioned in the talk is Bees with machine guns, a load testing tool that uses the power of EC2 and with a name like that I was gleeful thinking about the day when I would have an opportunity to use it.

This week the day came :D And despite the useful README I had to tweak a bit to get things working; here are my notes for later, and for anyone else who might ask themselves the same questions.

Bringing up the swarm

The documentation mentions some of the defaults you should override, but not all of them and you might find yourself scratching your head wondering why things are timing out.

Pre-requisites:

  • Creates a 'bees' (or whatever) security group, with SSH/port 22 open (-g)
  • Make sure the AMI you're trying to find is available with that name in the zone you're in (-i)
  • Make sure the region in your Boto config file matches the region you're trying to connect to (-z)
  • Mind the default username! If you don't specify it, it will default to newsapp and the bees won't be able to connect (-l)
  • For additional paranoia I also created a specific key pair just for the bees, to make sure there's no way in hell I can bring down an instance with something important on it (-k)
./bees up -s 4 -g bees -k bees -i ami-12345678 -z eu-west-1a -l root

Still timing out!

Now... The bees use the apache benchmark tool to hammer your site. If the tool doesn't produce any output the script assumes there was a time out. Which means, if the tool isn't installed on the AMI the bees will always time out! Without an appropriate error message. There are a few options. You could create a custom AMI that has ab installed. This is a bit overkill. Or, if the AMI you're using supports it (not a given!) you can launch a user data script by tweaking the bees.py script around line 100 and adding something like this to the run_instances call. (Debian-based system, use sudo if necessary.)

   user_data = '#!bin/bash\napt-get -y install apache2-utils'

thus becoming:

reservation = ec2_connection.run_instances(
        image_id=image_id,
        min_count=count,
        max_count=count,
        key_name=key_name,
        security_groups=[group],
        instance_type=EC2_INSTANCE_TYPE,
        placement=zone,
        user_data = '#!bin/bash\napt-get -y install apache2-utils')

Other tips

If testing say a Django-based site, don't forget to add the ending slash at the end of the URL. ab doesn't follow 301 codes.

If changing regions to test from elsewhere:

  • Find the new AMI id for the image. It has to be EBS backed.
  • You may encounter errors if the zone doesn't offer t1.micro images, but the error message will clearly indicate so.
  • ~/.boto must be updated:
[Boto]
#ec2_region_name = eu-west-1
#ec2_region_endpoint = eu-west-1.ec2.amazonaws.com
ec2_region_name = us-east-1
ec2_region_endpoint = us-east-1.ec2.amazonaws.com

[Credentials]                                                        
aws_access_key_id = ......
aws_secret_access_key = .....
Leave a comment | 2 so far

A Quick and Incomplete Intro to Jasmine Javascript unit testing

Recently I needed to find some sort of Javascript framework to start unit testing new code, some of which was using jQuery. I ended up deciding on Jasmine.

Why Jasmine?

  • Primarily and essentially, it's maintained, it seems fairly active and has a community around it: looking at the pull requests, github network, issues, code changes and glowing reviews all over the web
  • It can test asynchronous calls
  • It can integrate with a continuous integration system (though that seems to require an external plug-in, listed on the website? -- to be investigated more deeply when I need to set this up!)
  • It attempts to be platform/language/etc agnostic, though you do feel a strong tie to Ruby, rummaging around
  • Other nice things
  • And I've heard the name mentioned in local tech meetups and mailing lists, which is always good (it's good enough to be picked up by real people, who can potentially become a source of help in the future if needed :))

Getting started (~10 minutes)

Download

You should download Jasmine from this link: http://pivotal.github.com/jasmine/download.html. It's actually a bit unclear from reading the wiki, I initially downloaded the latest version from its associated tag and had trouble because the directory structure didn't seem to match the documentation.

Read the instructions

The main wiki page really serves as the user guide. Here're a few helpful links to get started, all short and useful:

There's a chapter on mocking as well, which I skipped until I need it -- it's enough to try out already!

Interlude: Vocabulary

I'm assuming this is because Jasmine is "BDD" (Behaviour Driven Development) and I come from TDD (Test Driven Development), but there was a bit of new vocabulary to get used to. Here's a short BDD to TDD dictionary.

  • Spec: Test
  • beforeEach: setUp
  • afterEach: tearDown
  • Spies: Mocks, mocking
  • .toBeTruthy: .assertTrue (it sounded like it could be approximately or almostEquals to my untrained ears)

Setting up

Have a look at A simple project for an extremely brief overview, and open the example specs, especially PlayerSpec.js which I found really useful. That should be enough to get you going!

Don't forget to include jQuery or whichever, in your spec runner file if using such a framework.

I'm not sure yet what's the best structure for organising Javascript tests. I'd like to find out what people do. At the moment I've put my spec runner file and tests in the directory that makes the most sense semantically, but this means that by default the file would end up accessible on prod after deployment, which does not sound ideal. Any reading suggestions for me or best practices to share?

Leave a comment

Testing django admin customisations

In preparation for an upgrade, I've been writing unit tests for a Django app with the help of a fantastic book -- Django 1.1 Testing and Debugging by Karen M. Tracey, there'll be a review coming up when I finish it.

I had some issues with the code to test admin customisations (Chapter 4), I want to document the changes I made for future reference.

Error 403

Despite using client.login() in the setUp() method, response returned with a status_code of 403 (Forbidden) when creating a new item.

class AdminCustomisationTest(TestCase):
    
    def setUp(self):
        username = 'test_user'
        pwd = 'secret'

        self.u = User.objects.create_user(username, '', pwd)
        self.u.is_staff = True
        self.u.is_superuser = True
        self.u.save()

        self.assertTrue(self.client.login(username=username, password=pwd),
            "Logging in user %s, pwd %s failed." % (username, pwd))

        Survey.objects.all().delete()

    def tearDown(self):
        self.client.logout()
        self.u.delete()

    def test_add_survey_ok(self):
        self.assertEquals(Survey.objects.count(), 0)

        post_data = { 'title': u'Test OK',
                      'open_date': datetime.now(),
                    }

        response = self.client.post(reverse('admin:survey_survey_add'), post_data)

        self.assertRedirects(response, reverse('admin:survey_survey_changelist'))
        self.assertEquals(Survey.objects.count(), 1)

At first I blamed some customisations I did to the authentication backends to allow OpenID, however printing the response.content revealed otherwise:

<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en"><body><h1>403 Forbidden</h1><p>Cross Site Request Forgery detected. Request aborted.</p></body></html>

The Django CSRF protection system prevented the tests from passing. Note that this app runs on Django 1.1, Django 1.2 overhauled the CSRF system and would likely work without problems.

I'm sure there are more elegant ways to solve this. Here's a method that works:

    def get_csrf_token(self, response):
        csrf = "name='csrfmiddlewaretoken' value='"
        start = response.content.find(csrf) + len(csrf)
        end = response.content.find("'", start)

        return response.content[start:end]

    def test_add_survey_ok(self):
        self.assertEquals(Survey.objects.count(), 0)

        response = self.client.get(reverse('admin:survey_survey_add'))
        csrf = self.get_csrf_token(response)
        post_data = { 'title': u'Test OK',
                      'open_date': datetime.now(),
                      'csrfmiddlewaretoken': csrf,
                    }

        response = self.client.post(reverse('admin:survey_survey_add'), post_data)

        self.assertRedirects(response, reverse('admin:survey_survey_changelist'))
        self.assertEquals(Survey.objects.count(), 1)

(Note/Update: If I had waited until the next chapter, I would have found out how to integrate Twill with Django apps and none of this would have been necessary -- haha!)

This field is required (datetime)

After this, although the response.status_code was finally 200, the article still wasn't created. Peering through response.content showed that the datetime field was considered to be missing. This is what the postdata should look like instead for a datetime field:

        post_data = { 'title': u'Test OK',
                      'open_date_0': '2011-03-17',
                      'open_date_1': '9:50:00',
                      'csrfmiddlewaretoken': csrf,
                    }

Test passes!

Leave a comment

Python Ireland February meet-up Introduction, unit testing

A delayed summary of Python Ireland's February meet-up last Wednesday -- another great event, after which I promptly fell sick (getting ready to kick ass again now though :)). In January, the group had a simple and nice "pub meet-up", where a few Python beginners and novices turned up (including one who's now my student *waves*), and it was thus decided it would be good to have some introductory talks in the following months. Hence February's first talk:

Introduction to Python

In this talk, Sean gave a quick introduction to Python by comparing it with other programming languages ; namely C, Java and Ruby.

I must say I took notes in preparation for my crash course later on :o) Nice ideas on how to present a new language to programmers. The comparison wasn't only on the syntax or verbosity, and general programming paradigms espoused by the languages but also on the different cultures encouraged by their respective community. Very interesting! Though unfortunately, probably difficult to follow for beginners new to programming, of which there was at least 1 in the room -- but it's difficult (impossible) to create a talk that caters to everyone in the whole world.

Somehow post-talk questions mostly resolved around high performance parallel programming :)

Advanced unit testing

Rory enthusiastically introduced us to the latest additions in the unit test framework for Python 2.7 (backported from 3.x into unittest2), and after this talk I can't wait to enthusiastically switch my programs to Python 2.7. Exciting stuff. For instance...

  • more Asserts! With cool names, like assertAlmostEquals ;) assertMultiLineEquals sounds fairly cool too, it gives you a diff as opposed to the whole strings that didn't match.
  • when you have a slow setUp() or you simply do reads without writes... I introduce setUpClass()! Yes! There's even a setUpModule if all the tests for your module share a setup step. Of course the same exist for tearDown().
  • "discover" will find and run your tests without you having to add the __main__ stanza to all your test files, yay. It works like this:
python -m unittest discover -s [path]

You can specify a class or function if you like.

The rest of the talk was about mocks. Mocks. Mocks. They're not in the main unit test framework and voidspace's seems to be the recommended library. It was interesting to see how they're used. Mock patch enables doing all the mocking stuff with fancy decorators, thus avoiding you the pain of manually storing and restoring the original value of something before you replace it with a mock. Overall veeery interesting.

AGM & Pub

A short and sweet AGM followed where the committee ended up re-elected (go committee, y'all rock!). We then moved on to the pub where general Python and other techie discussions happily prevailed for the rest of the evening.

Leave a comment