Training at EuroPython 2014 Making your first contribution to OpenStack

OpenStack logo

Last week I ran a 3-hour training on how to get started contributing to OpenStack at EuroPython. The aim was to give a high-level overview of how the contribution process works in the project and guide people through making their first contribution, from account creation to submitting a patch.

Overview

The session starts with an extremely fast overview of OpenStack, geared toward giving the participants an idea of the different components and possible areas for contribution. We then go through creating the accounts, why they're all needed, and how to work with DevStack for the people who have installed it. From there we finally start talking about the contribution process itself, some general points on open-source and OpenStack culture then go through a number of ideas for small tasks suitable for a first contribution. After that it's down to the participants to work on something and prepare a patch. Some people chose to file and triage/confirm bugs. The last part is about making sure the patch matches the community standards, submitting it, and talking about what happens next both to the patch and to the participant as a new member of the community.

Preparing

During the weeks preceding the event, I ran two pilot workshops with small groups (less than 10 people) in my local hackerspace, in preparation for the big one in Berlin. That was absolutely invaluable in terms of making the material more understandable and refining the content for items I didn't think of covering initially (e.g. screen, openrc files) and topics that could use more in-depth explanations (e.g. how to find your first task), timings, and generally getting a feel for what's reasonably achievable within a 3-hour intro workshop.

Delivering

I think it went well, despite some issues at the beginning due to lack of Internet connectivity (always a problem during hands-on workshops!). About 70 people had signed up to attend (a.k.a. about 7 times too many), thankfully other members of the OpenStack community stepped up and offered their help as mentors - thanks again everyone! In the end, about half the participants showed up in the morning, and we lost another dozen to the Internet woes. The people who stayed were mostly enthusiastic and seemed happy with the experience. According to the session etherpad, at least 5 new contributors uploaded a first patch :) Three are merged so far.

Distributing the slides early proved popular and useful. For an interactive workshop with lots of links and references it's really helpful for people to go back on something they missed or want to check again.

Issues

The start of the workshop is a bit lecture-heavy and could be titled "Things I Desperately Wish I Knew When Starting Out," and although there's some quizzes/discussions/demoing I'd love to make it more interactive in the future.

The information requested in order to join the Foundation tends to surprise people, I think because people come at it from the perspective of "I want to submit a patch" rather than "I am preparing to join a Foundation." At the hackerspace sessions in particular (maybe because it was easier to have candid discussions in such a small group), people weren't impressed with being forced to state an affiliation. The lack of obvious answer for volunteers gave the impression that the project cares more about contributions from companies. "Tog" might make an appearance in the company stats in the future :-)

On the sign-up form, the "Statement of Interest" is intimidating and confusing for some people (I certainly remember being uncertain over mine and what was appropriate, back when I was new and joining the Foundation was optional!). I worked around this after the initial session by offering suggestions/tips for both these fields, and spoke a bit more about their purpose.

A few people suggested I simply tell people to sign up for all these accounts in advance so there's more time during the workshop to work on the contribution itself. It's an option, though a number of people still hit non-obvious issues with Gerrit that are difficult to debug (some we opened bugs for, others we added to the etherpad). During one of the pilot sessions at the hackerspace, 6 of the 7 participants had errors when running git review -s  - I'm still not sure why, as it Worked On My Machine (tm) just fine at the same time.


Overall, I'm glad I did this! It was interesting to extract all this information from my brain, wiki pages and docs and attempt to make it as consumable as possible. It's really exciting when people come back later to say they've made a contribution and that the session helped to make the process less scary and more comprehensible. Thanks to all the participants who took the time to offer feedback as well! I hope I can continue to improve on this.

Leave a comment

Testing in Horizon Unit testing for the Openstack Dashboard

Belatedly, here are the notes for the design session/tutorial I gave about testing in Horizon at the OpenStack Summit in Portland, back in April. The etherpad is available over there. Session description:

The main aspect: the Horizon unit tests can be quite complex for new contributors and people extending Horizon to wrap their head around. Mocking with mox, the django unit testing framework,the openstack specific parts of the testing framework, selenium, fixtures/test data handling, qunit...This session could work as a tutorial/tips and tricks on the different testing components. Common errors being thrown and how to debug them. If people could bring up their pain points, that would also be useful.

If there is time, it would be interesting to also address the issue from another angle and think on how to improve what we have, particularly on the Selenium front which has been quite unstable.

General structure

There are 3 main parts to Horizon testing (4 if you include the bits that come from the Python unit testing framework, but we won't get into it here. If you've done unit testing before, it's the usual set of assertions and scaffolding that come with any unit testing framework).

As an example to map to what this is all talking about, I recommend keeping InstancesTest.test_index in the background.

Django unit testing

Docs: https://docs.djangoproject.com/en/1.4/topics/testing/

At the moment Horizon is compatible with 1.4 onwards. The django documentation is excellent and I recommend having a look. Thanks to django we get a lot of goodies for free to help with testing a web application. Among other things:

  • A test client, which mimics a very simple browser (no Javascript) to do GET and POST requests, and built-in tooling to have interesting interactions with the responses.
  • A bunch of additional assertions, to check the HTML, templates, etc., all documented in the link above.

If you're familiar with django already, or while you're reading the django docs, there are a couple of things to watch out for:

  • Horizon does not use models.py, and does not have a database
  • Horizon doesn't use fixtures either (actually it does, but they're very different since they're not done the django way - cf. no models)

Horizon unit testing

Docs: http://docs.openstack.org/developer/horizon/topics/testing.html

Helper classes: https://github.com/openstack/horizon/blob/d4b0ab4aa395bf4df2964efcc358100117efdaa0/horizon/test/helpers.py#L65

There are some docs for testing in Horizon, which contain useful advice for writing good tests in general. A few sections only are specific to Horizon:

Now let's have a look at helpers.py, where the TestCase classes we extend in Horizon tests are defined.

The setUp() and tearDown() methods do the housekeeping for mox/mocking so that we don't have to worry about it when writing tests. The aforementioned Horizon-specific assertions are also defined in this class. It extends the django TestCase class thus all of the django unit test goodness is available.

In general, this class is the best documentation available of what happens in the tests and how they are set up.

Openstack Dashboard unit testing

APIs: https://github.com/openstack/horizon/tree/d4b0ab4aa395bf4df2964efcc358100117efdaa0/openstack_dashboard/api

Test data: https://github.com/openstack/horizon/tree/d4b0ab4aa395bf4df2964efcc358100117efdaa0/openstack_dashboard/test/test_data

Helper classes: https://github.com/openstack/horizon/blob/d4b0ab4aa395bf4df2964efcc358100117efdaa0/openstack_dashboard/test/helpers.py#L98

The Horizon tips and tricks mentioned earlier also apply, but there are no specific documentation page about the topic.

A quick overview of openstack_dashboard/ and the sections that matter to us in the context of unit testing:

  • APIs

The API directory is the only place that talks directly with the outside world, that is, the various openstack clients. This is why Horizon doesn't have a database, because it doesn't store any data itself.

  • Test Data

The test data is also stored in a single directory, and contains the fixtures, that are used to represent (mock) the data returned by the different clients.

  • Helper classes

Like in the "framework" part of Horizon, a helpers.py file defines the TestCases we extend later in the unit tests. This is where a lot of the magic happens: the TestCase extends the Horizon TestCase helper class described earlier, loads the test data, sets up mox, creates a fake user to log in. There's also a couple of useful assertions defined that are used all over the place.

There are other TestCase classes in there, for tests that may require an Admin user, testing the APIs, or Selenium.

A quick look at the example

The flavours returned by self.flavors.list() come from the test data.

We'll look at the mocking stuff in the Tools section. The APIs being mocked all live in the API directory, so this is the only place that needs to be mocked.

self.client() is the default django client, reverse() and assertTemplateUsed() also come from django.

self.assertItemsEquals() is a Python assertion.

Tools

Mox

In Horizon, mocks are used a lot, everywhere or otherwise running the unit tests would require a fully set up, running Openstack environment.

I found mox a bit difficult to get used to. There's a specific terminology, that translates to a different set of steps than is common in other mocking tools like mock.

First you record. That's the part in the tests where you create the stubs (in a decorator in the example) and "record" what you expect will happen (that's the place in the example that says: "when api.nova.flavour_list() is called with these exact arguments as described, return self.flavors.list()").

Then you replay, with self.mox.ReplayAll() which will make sure the rest of the test will get the data it expects, that you just mocked.

Finally, the reverify step is done in the parent TestCase class' tearDown() function, which calls self.mox.VerifyAll() and ensures the functions recorded were all called, and in the order defined.

There are lots of catches in mox, it's quite strict. Order matters. By default it assumes the mocked function will only be called once and fails otherwise (that's a big one that can be difficult to track down). MultipleTimes() will save you if a function needs to be called more than a couple of times.

Stubbing can be done via a decorator (which is the favoured way going forward) or a StubOutWithMox function, which can still be found in places.

Mox errors can be confusing and I recommend reading the Horizon docs about understanding mox output, which have a couple of paragraphs explaining different errors that may be encountered, the dreaded Expected and Unexpected calls.

Selenium

Helper classes: https://github.com/openstack/horizon/blob/d4b0ab4aa395bf4df2964efcc358100117efdaa0/openstack_dashboard/test/helpers.py#L326

We use Selenium for testing Javascript interactions. It's a bit heavyweight since it requires starting a browser, so Python unit tests are preferred when possible.

It's more stable now (thanks Kieran), so hopefully we can write a few more tests for the places where it's needed.

qUnit (briefly)

qUnit is used for some of the pure Javascript tests.

It's not used a lot in Horizon. The handmade fixtures take a lot of effort to make so maybe it's better to use Selenium in most cases.

Tips and tricks

  • See the Tips and tricks from the Horizon testing topic
  • Use pdb to check the environment status
  • Anything else? From the session:
    • Mock everything, and if it doesn't work mock it again.
    • Selenium tests: having a flag to turn off/on mocking? So we can run them as integration tests when needed and make sure we still match the correct APIs - cf. blueprint
    • Using Selenium tests as integration tests: build more tests (start a VM, ssh into it)

Discussion

Unfortunately, the day was running late (and I was speaking at the very next session) therefore the discussion part didn't have time to happen.

I'm disappointed about that and would welcome people discussing their experience and pain points, particular from a newcomer's perspective.

Fortunately when it comes to discussing the Selenium issues, Kieran Spear had successfully fixed it right before the Summit :)

Leave a comment