An overdue post on the latest course I taught, which ended about 2 weeks ago. I taught a group of 9 people (8 adults, 1 teenager) of various backgrounds the basics of programming, using Python. It went well overall, and the class atmosphere was enjoyable.

For...

Unfortunately I still fail to explain some concepts clearly -- namely the dreaded for loop. I kept telling myself I needed to change the way I introduce it and finally did so, and wow. It went even worse than before. I could see in my student's eyes how they were trying to understand, until eventually shrugging it off when I couldn't find any different way to explain it. It may have come across a bit better eventually when we started working with files and used for loops to go through every line, but introducing the concept went very blergh. Definitely need to do it differently next time. I think I will try to introduce the for loop over 2 sessions: one where we only see and do exercises related to "for i in range(0, x)" and separately introduce "for item in my_list".

I don't know if I just forgot before, but failing to get a concept across is really, really depressing. I was bummed out for a while afterwards, wondering if I'm just making more damage than good and messing up my goal of making programming less intimidating.

...Yet another reason to do better next time!

Exercises

I should make sure to have at least one "Fill in the blanks" and "Change this existing program so that..." type of exercises for every concept. I think they encourage students to read and try to understand existing code -- an important skill to develop, and it also makes the first attempt at using a new structure more manageable.

In general, I'd really like if I could find a neat program that could be cut down into exercises relating to each concept, that'd build up to something cool and awesome as the course goes on. It would help get across that complex programs are really made up of simpler pieces ; if you can't figure out how to do something, break it down further. I haven't really figured out a program that fits the bill yet.

Overall, material-wise I'm getting there (except for For :|), it'd be worthwhile focusing on improving the exercises for the next course.

Homework

OMG some people actually did the exercises as homework this time :O Wow. I tip my hat to them, it's great and it paid dividends for them. Unfortunately it also meant they had nothing to do while the others were catching up, so... I should probably give away only a portion of the exercises as homework, and save a few for the actual session. (The idea with the exercises is that everybody should get at least 1 done before we move on ; having many gives the faster students something to do while others don't feel too rushed.)

Duration

I taught over 5 weeks this time (2h30 in the evening with a 20-30mins break), and I will do 6 weeks next time. For real beginners it's still reasonable and it will let me take more time on concepts like For, and maybe an opportunity to show at the end what else there is to programming, and give ideas on which direction they could go on from here (GUIs, web development, robots...).

Speaking of robots

At the last session I brought a Finch, which I still haven't had much time to play with :') But with the Jython bindings it was possible to show a 5 lines Python program that has an effect in the real world. The Finch was very popular!

I want to play with the robot more, and see if perhaps it would make sense to use it to teach the whole beginner course at some point in the future (which would bring its own set of issues, $$$-wise).

Student feedback

As usual I asked if students could fill in a feedback form.

Everybody agreed they learnt a lot. "Fun" and "interesting" were in about everyone's "3 words to describe the course", with"hard" and "challenging" occasionally thrown in there :) "Useful" came up in half the forms too, which makes me happy.

The projector wasn't great with yellow which unfortunately was the default colour for strings in the IDE. It would have been worthwhile spending the time on fixing that rather than let people squint (sorry :().

A couple of requests to link more clearly exercises with the real world. So very difficult when we're still introducing new concepts! But definitely something to strive for.

Someone suggested writing a sample program where every line is explained (e.g. if x:    <-- this will do...). I think it's a good idea, I will try for next time. It would be useful as a reference.

I should be teaching this course again around March. Any pointers to beginner exercises, general comments, teaching advice, teaching programming tips all very much welcome!

As mentioned before, I'm teaching programming again, to a group of beginners :) (Although as usual, even in a group of complete beginners there are variations in people's levels and comfort with the subject.)

Initial advertising

I'm really happy with how the PR worked out this time. The first time I taught this course in Tog, I made the mistake of putting "Python" in the name of the course and ended up wasting a lot of time screening out people who wouldn't be a good fit. This time I only mentioned it in passing, and the amount and type of answers I got was much more manageable and appropriate. We still managed to fill up the course and to have a waiting list (hope to see you next time!), so it's a success. I'm pasting the announcement here for future reference:

Class: Introduction to Programming

Are you curious about programming? Ever wished you could write your own scripts to automate repetitive tasks, or for fun? Or been interested in understanding how an application works (and why there are always bugs in the software you use??). Tog is running a new class on learning how to program, for complete beginners with no previous programming knowledge.

The course will cover basic programming skills, which will also give you a better understanding of how computers work. We will be using Python, an excellent learning language because it is simple yet powerful, and extremely readable. At the end of the course you will know how to write simple programs, and you should have enough understanding of the basics to move on to either more complex programming tasks in Python or learn another language if you wish to.

The course will begin on October 20th and will run for 5 weeks every Thursday from 7.00pm to 9.30pm. The cost is €40 for non-members, and free for members. The class size is limited to about 8 people, for a relaxed atmosphere conducive to learning.

If you are interested in attending, or have any question about the course, please use the form below or leave a comment — we’re happy to answer any query! You must sign up if you want to attend, space is limited.

Please note that this is a course for complete beginners, who are not yet familiar with the usual syntax for programming. If you’re a programmer interested in picking up Python, you should keep an eye out for our crash courses!

A fair percentage of the students heard about the course from Twitter even though I'm not on it, which is interesting. I should explore how to use the tool better. As I mostly hang out with fellow nerds, reaching out to interested beginners is always tough, so anything that can help...

Pace

The pace of the course is manageable as well -- which doesn't mean preparing and teaching isn't time-consuming still! But I felt quite burnt out last time after the pace at which I taught the PhD students (mostly through my own fault), so it's refreshing to come back to teaching pure beginners and not go crazy from week 2.

Surprisingly enough and to my utter delight, a huge portion of the students took on the exercises as homework! I'm so happy when I get questions about the exercises or lessons via email, to see people interested enough to poke at problems outside of the classroom. If we keep it up we should be able to cover more as well, which will be interesting for everyone.

As for feedback, I found a couple of positive tweets after the first session, which is always encouraging. Woohoo \o/

Congratulations to Python Ireland and the PyCon Ireland committee for the successful 2nd edition of the conference! If you weren't there, you should feel sad. I would be.

Lots of interesting talks and just like last year, a whole bunch of very friendly and knowledgeable attendees to talk to and share a meal with (or a drink, for lucky people who won several raffles... :o)) (hehe) (I did!)

I sadly missed the first keynote, which I look forward to catching up on on video. I was busy helping out at the registration desk, and discovered I really enjoyed welcoming attendees to the conference. A new hobby!

I don't know how productive in general Sunday was, sprint-wise and code-output wise. The open space format seemed to work well on the other hand! I attended the RSI talk and the buildout tutorial (must look more into Buildout) and spent the rest of the day PRing for Tog, distributing Berocca and talking shop. On Sunday afternoon, together with a fellow Tog member we stealthily stole away a dozen attendees to go and visit the hackerspace, conveniently located right behind the venue. Delegates were returned to the conference unharmed and inspired (I hope!).

And because one cannot ever have enough Python, the usual monthly meetup is on this Wednesday!

PyCon Ireland 2011 starts tomorrow! If you haven't got your ticket yet, that's tough because we sold out a few days ago! Congratulations, organisers, for what promises to be a fantastic event.

I also seem to have inherited a yellow staff tee-shirt so... Feel free to find me and say hi if you're lost (or not!). I'll hand wave and do my best to be helpful. :)

Curious about programming? Ever wished you could write your own scripts to automate repetitive tasks, or for fun? Or been interested in understanding how an application works (and why there are always bugs in the software you use??). I'm teaching a new class in Tog on learning how to program, for complete beginners with no previous programming knowledge. We'll be using Python, and you can find more information including how to register over at Tog's website: Class - Introduction to Programming.

A couple of tips so I don't have to look it up again. A few months back I wanted to generate coverage data for a stand-alone application, with a few tweaks.

Combining the output of Trial and Python unit tests

$ coverage run -p tests/my_python_tests.py
$ coverage run -p tests/my_trial_tests.py
$ coverage combine
$ coverage html -d tests/coverage_html  --include=./*py --omit='lib/*.py,tests/*.py'

Helpful resources to get used to combine:

• coverage in python
• Coverage command line usage (Manual)

Getting coverage report for a simple project

$ coverage run tests/python_tests.py
$ coverage report --include=./*py --omit='lib/*.py,tests/*.py' # For a text summary
$ coverage html -d tests/reports  --include=./*py --omit='lib/*.py,tests/*.py' # For a nice HTML report

Installing the latest version

To get access to the command line tools, I had to install Coverage in a virtualenv to get around some other constraints on the system. I thought I'd build it from master while I was at it :)

$ virtualenv coverage
$ cd coverage
$ source bin/activate
$ pip install hg+http://bitbucket.org/ned/coveragepy

Coverage is a lovely library and a wonderful way to encourage yourself to think more about your unit tests -- give it a try if you haven't already. (PS: For Django, see django-coverage)

Are you a programmer interested in picking up Python and Django?

Rory is running a crash course in Tog starting this Thursday. There'll be two 2-hours sessions -- the first one about Python, next week about Django.

It's short notice but if you've been meaning to learn Python, now's your chance!

More information, including how to register, over here.

Link: Talk description and video

The Python Code Dojo is a community organised monthly meeting.

Dojo

A dojo is a place where you go to practice stuff, learning is a continuous process. It's based on the idea of deliberate practice.

Paris

Codingdojo.org was started in Paris, where it follows a very structured format.

Katas are forms that you practice to prepare yourself. You learn how to solve a problem using baby steps. In Paris they do this in silence, unless you really don't understand and have to ask a question. "Randori kata" is public pair programming, with a pilot and a co-pilot that solve a problem on stage.

London

The London Dojo works more like a seminar and attendees are encouraged to interrupt. Participation is expected. They do team dojo where the team must solve a problem within a timeframe. Problems are written on a blackboard, people vote for one and then everyone works at solving it in a team of 5 or 6 people over 1h30. Finally there is a show, tell, review and question event where each team presents their solution/approach.

Why participate in a dojo?

• The educational benefit, of learning by doing
• You can fail safely in a sympathetic environment, and experiment
• People teach one another, all levels can attend
• You build a community: in London, that's relaxing with pizza and beer

What's a good dojo?

From the attendee's perspective: it's fun, you get to solve problems, it's safe to make mistakes, show and tell is encouraged which is good to get feedback.

From the organiser's perspective: it self-organises, mostly.

To see if it's going well: see that there is a positive aim, something is done to reach this aim, with some sort of feedback at the end.

Personal observations

Beware of systems and gurus. Ignore systems if something else works for you, you can actually do damage otherwise. Learn to practice learning!

Q&A tidbits

When they (or another dojo?) started using meetup.com they doubled their numbers! Or EventBrite, the idea is to have a centralised system, with tickets to predict attendance.

The lightning talks were very fast paced (5 minutes) so I only jolted down some project names I want to check out and interesting tidbits, and missed speaker names just about all the time. Sorry!

To easily create diagrams, check out blockdiag. It includes different shapes that make me hopeful it might be a less painful way to do nice topologies.

Someone's project to remove the GIL in Pypy (future):

global_lock.acquire() / .release()
object.acquire()
with Transaction():

Learning a language in 60 hours: http://sotospeak.se/ (English homepage). It's a piece of software for your mobile phone, that encourages you to learn a language like children do. It's written in Python.

shlex for simple lexical analysis.

Python Edinburgh are a bunch of cool folks with their own conference :)

DjangoZoom, effortless deployment for Django (like Heroku?)

pip install null, if a need to use the Null object pattern arises.

Link: Talk description and video

2 years and a half ago, SourceForge was all PHP except for one little Python service. SourceForge was originally written in 1999, when Python wasn't so great at the web. After 10 years though, it all started to atrophy.

So SourceForge decided to do a little experiment and assigned 2 people to create FossFor.us. They chose a web framework, thinking that Python was good and impressed with the Django documentation: there was no real learning curve. Couchdb was a bit slow, though.

The experiment was a success (although the site is now gone, due to managerial reasons if I understood correctly) and so they were then given 6 weeks to redo the download page/files of SourceForge. It had to be dynamic, to offer you a download relevant to your platform: they use the user agent string to figure out the operating system, and borrowed code from setuptools to figure out which release is the latest.

Their admins loved Apache so they went with mod_wsgi, which worked out well.

The whole system worked fine on a laptop handling all the traffic from SourceForge: that was the only load testing! And unfortunately when the system went live it took about 8 seconds to load a page. There was no CPU or memory usage. It turned out they saturated their gigabit Ethernet card, by loading all the releases, which some projects have a lot of (e.g. JBoss). Memcached turned out to be slower than MongoDB, because of pickling, the 4 megabytes objects and CPU.

Finally they updated the list to not include all releases and the project was deemed a success! Thus everything shall become Python. They are developing their new platform openly, Allura. They use a FUSE filesystem in Python to control permissions.

From now on, they have an internal mandate that everything should be written using Python unless there is a good reason not to. And now SourceForge can compete again! As well as explore new directions.

Link: Talk description and video

This talk was about non-relational databases. I didn't take a lot of notes :o) The most important morale is probably: don't keep the mind altering substances and the tools in the same shed.

With 2 decades of relational databases, they are pretty robust by now. They cover different spectrum of ACID compliance ; for instance MySQL is faster, Postgres is more reliable (though becoming faster... if you tick off the reliability options!). Relational databases are supposed to be normalised, except they are not really: there is also a spectrum here as databases tend to get denormalised for performance reasons.

Amazon uses an "eventually consistent" system, which they can pull off by charging at shipping time only. Conflicts are rare, if 2 orders are placed and there is only 1 item available, someone might get a gift certificate instead.

The NoSQL taxonomy includes wildly different tools that don't have much in common except for the fact that they don't use SQL: key-value stores, document stores, ....

CAP: Consistency, Availability, Partition tolerance. You can have 1 or 2, not all 3. (Brewer's Theorem)

There was only 1 Postgres database for all of SourceForge for a long time, while they were in the top 100 sites. Don't obsess about scale you'll never achieve.

One of the question was about how difficult it is to convert from a relational database to NoSQL. The answer is, from something like Postgres to MongoDB, it wouldn't be that much work (he did suggest 4 people 6 weeks though, which doesn't sound that trivial to me). Changing to Cassandra on the other hand would be a huge effort.

Link: Talk description and video

I couldn't find the slides online but please do link me if you find them, they were a treasure trove of awesome tips and very well laid out!

The talk touched on many things, the following tips are unrelated to each other and in no particular order.

Beware the corner cases of negative slicing -- if you use a variable for slicing there's likely a bug lurking in your code!

mytuple = 1, 2, 3 # Tuple declaration also works without parenthesis

Learn Python the way Guido intended it and indented it :P

for x in reversed(seq): # Better than negative slicing because it's clearer, there's no need to do a double take

for i, x in enumerate(seq): # That's Pythonic! Forget about i in range, seq[i]

for x, y in zip(seqx, seqy)

for/else: a for loop with an else, the else executes if there was no break in the loop, that is, when the loop runs to completion. It's only useful when there is a break (the searching use case).

None is always smaller than everything else, in comparisons.

Sets aren't guaranteed order after being sorted, because __lt__ has been overridden to indicate subsets and supersets.

If you intend to define __lt__ you must also implement the 6 ordering functions.

# Key function:
sorted(s, key=str.lower) # (awesome!)

If a class takes in an iterable, and emits iterables lots of awesome may occur, with the prebuilt tools and other surprising uses (kind of works like pipes and filters).

d = defaultdict(dict)
d[stuff][otherstuff] = "blah"
# as opposed to
d = dict()
d[stuff] = dict()
# etc...

d.update(dict(d))

Links: Talk description, video and slides

Know the environment

Use Linux with the "Terminator" shell.

Use a version control system.

Use virtualenv, for managing different versions of Python and dependencies, e.g.

virtualenv path --python=python2.7 --no-sites-packages
system libs: ./configure --prefix=envpath ; export LD_LIBRARY_PATH=envpath/lib

Use yolk, handy tool to query pypi and status of pypi installed packages

yolk -l (to see installed packages)
yolk -U (to see if there are updates on pypi)
yolk -a

Use the bash autocomplete script.

Use djangodevtools (PyPi/site with description of the new commands), for lots of useful things such as adding test coverage (./manage.py cover test myapp).

Continuous integration

Use Buildbot and Django-Buildbot (note: there was a configuration example on the speaker slides).

In settings.py:

try:
    from setting_local import *

setting_local.py shouldn't be shared or checked in.

Thanks to alias, from djangodevtools, you can simply create commands to do whatever you want, e.g. clean up rabbit queues. The commands are stored in a manage.cfg file that is shared.

uWSGI is an application server with many options. --auto-snapshot sounds quite interesting. It supports clustering. It'll be in the official Django deployment documentation from the next release (ticket 16057).

Coding

get_profile() tends to be a problem. It's possible to monkey patch the User.get_profile() method (to make sure a new profile is created if it doesn't exist) but you have to be careful where it's loaded. It's also possible to use a Meta proxy together with a new middleware (set up after the authentication middleware)

Django model form factory (django.forms.models.modelform_factory) sounds interesting to create forms more quickly.

uWSGI

There was an on-the-fly short talk on uWSGI after the talk, by someone whose name I didn't pick up. It can talk to many protocols, it has lots of plugins so you can only use what you need. It's not the fastest but speed isn't the main factor that should make you decide to use it.

Links: Talk description and video and slides.

This talk will be about 3 tools, that can be considered secret weapons: they offer great payoffs, for low efforts.

(Note: the slides enhance most of these concepts with lots of code examples, have a look!)

Haystack

Haystack does full text search, and is available as modular search for Django. It's very easy to get a nice search interface if you already use the Django ORM, and the queryset can also be defined to limit search queries to what you want (e.g. only published entries).

You can have different templates/html bits depending on the type of objects returned by the search.

Scaling/Backend

• Woosh (Python) - Good if you have no write traffic, and not a lot of traffic in geenral
• Xapian (C)
• Solr (Java) tends to be the default choice. It has an HTTP interface, and there are tons of things that are already baked into Solr, like filtering down by object type. Objects can be associated with a template, although it sounds like it's more about relevance than display: the speaker mentioned showing the title twice in the template to increase its weight in search results. It can scale with SearchQuerySet, and works faster than complicated crazy SQL.

Search indexes usually don't like being updated much. Haystack offers several solutions. Sites with low write traffic can update the index in real time at every change. Or changes can be batched every 6 hours. At a higher scale, you have to roll your own solution. For Lanyrd they have added a "needs_indexing" boolean to their models that defaults to True and is also set in the save() hook. Then using a management command or something else, it's possible to look at what needs to be indexed, process it and set the flag to False.

Solr has master/slaves capabilities and knows how to balance the reads between slaves, the writes should be sent to master. Haystack only knows how to talk to one url, but using nginx it's possible to balance and to set up different proxies depending on the URLs to make sure the writes go where they should -- remember, Solr speaks HTTP.

Celery

Celery is a fully featured, distributed task queue manager. Any task that would take more than 200ms should go on the queue! For instance...

• Updating the search index
• Resizing images
• Hitting external APIs
• Generating reports

Using the @task decorator, the method works normally if called directly, but also gains a delay() that adds the method to the queue to be picked up by workers.

For tasks launched by users (such as uploading a picture or figuring out what's at a url):

• To deal with people using Javascript or not: if 'ajax' in request.POST, show an inline loading indicator, otherwise redirect.
• Use memcached for automatic house keeping, in case the user closes the browser and doesn't come back, don't keep the task around forever. The oldest will get dropped out automatically after a few hours.

Use celerybeat for scheduling, celerymon for monitoring the worker cluster, celerycam to take snapshots -- this helps figuring out when/where things go wrong.

The activity stream pattern gives everyone an "inbox" when everyone needs to receive something, like a tweet: it gets written to everyone's stream. Redis can handle 100,000 writes/second and is a handy tool to deal with this; this is also the kind of tasks that's a great candidate for queueing.

Fabric

Fabric is great for automated and repeatable deployments, it also makes it easier to roll back. You could use chef and puppet, which are ridiculously powerful but quite complex to set up. Fabric fits the developer mental model better generally, it kind of wraps your current processes into Python.

For instance, you can create a clear_cache() that calls flush_all() on the cache. Then, to clear the cache on your server, call from your machine:

fab -H host1,host2 clear_cache

The file (fabfile.py) is version controlled therefore documenting your process -- so you don't forget how to do it 6 months from now!

env is a global variable used by Fabric, you can add your own variables to it that can be reused in other commands, for instance env.deploy_date to store the deployment date and time and make it easier to roll back and roll forward (he uses symlinks).

They use a servers.json configuration file that documents the instance id, public dns, private dns, names and roles (solr, memcached, etc). Fabric can use this to deploy, nginx can use it to load balance, Django can import it in the settings to know what to talk to.

Taking a break from writing up my notes into blog posts to share a few general thoughts about the conference.

General feeling: Woohooooo! This was a fantastic week, I learnt a ton and met an amazing amount of awesome people. If you're here because I gave you one of my cards, hi! o/ It was lovely to meet you. (If you asked about the card and forgot, these are MooCards from moo.com. Get yourself some! People fought over these at EuroPython, I'll have you know. They're that good!)

The conference was wonderfully well organised, including the evening events. I fondly recommend the bistecca alla Florentina from Zaza! Everyone was incredibly friendly, like at PyCon Ireland last year it was common to strike up interesting conversations with a random stranger besides you and impromptu dinner plans would be shared between groups.

I was humbled by how egoless most people I spoke with were. They seem to know that no one knows absolutely everything about Python (and there were funny anecdotes about this, such as famous names requesting new features that are already in the language!). I was incredibly surprised when one of the keynote speakers sat at my table during lunch on the first day -- I had assumed well-know people would have solid cliques and no time or desire to meet new faces. And of course they ended up being just as nice as everyone else.

Some things I grumbled over: the constant strikes in Italy, first at the airport when I landed then the trains when I (tried to) leave! I was disappointed to miss out on the training I was hoping to attend as well, the rooms were a bit small and filled up long before the training starting time. Lesson learnt for next year!

...And indeed I am much looking forward to going again next year. In the meantime I welcome all Pythonistas to PyCon Ireland in Dublin this October! :D