jpichon.net - python

Extracting reviews from Goodreads into Markdown pages

2023-05-07T11:00:00+01:00

In early 2020, it became really difficult to read books. I'm slowly starting to get past that at last, but while Goodreads was super fun for a few years, right now tracking and counting and rating just makes reading feel like a chore and Not Fun as a hobby.

I still would like to keep a record of what I read, though. And I used to write book reviews on this blog before getting onto Goodreads. It seems like a nice way to keep a record of what I read without it becoming a number thing. I'll probably still write Goodreads reviews for small authors since I know it can help, but I feel less pressed about having everything there.

I also wanted to save the reviews I wrote only on Goodreads here, and wrote a script to migrate my reviews into Pelican-friendly Markdown pages since that's what powers this blog now. I decided to keep to a single entry per year rather than one entry per book, since I had a few good reading years in there (and others will only a single review!)

Step 1: CSV export of the books

First, if you go to 'My books' and find the 'Tools' menu at the bottom of the leftside menu on Goodreads, you'll find a page to export your library. You may have to try a couple of times: my first export only had a handful a books, the second one looks more comprehensive although the number of books was off by two, but what can you do.

Step 2: Python script to create the Markdown pages

This is the script I wrote to extract only the books I've actually read. I don't really care about DNF (did not finish) and to-read, right now. I also hardcoded the years relevant to me. May someone find something helpful in here!

import csv
from dataclasses import dataclass
from datetime import date, datetime


@dataclass
class Review:
    title: str
    author: str
    date_read: date
    review: str
    rating: int


def get_reviews(year):
    reviews = []

    with open('goodreads_library_export.csv') as csvfile:
        reader = csv.DictReader(csvfile)

        for row in reader:
            try:
                r = Review(row['Title'],
                           row['Author'],
                           row['Date Read'],
                           row['My Review'],
                           int(row['My Rating']))
            except ValueError:
                # When the int() cast fails, usually it means the CSV is
                # corrupted for that line. In my case, it was for a few to-read
                # records so I ignore them rather than attempt to fix the
                # original CSV. You can print the row here if you want to check
                # what's failing.
                pass

            if year is None or year in r.date_read:
                reviews.append(r)

    return reviews


def rating_or_review(review):
    # If I wrote a review, return that
    if review.review:
        return review.review

    # Otherwise, make the rating into words.
    if review.rating >= 4:
        return "I really enjoyed it."
    elif review.rating == 3:
        return "It was fine."
    else:
        return "Wasn't for me."


def format_reviews(reviews, year):
    with open(f'book-reviews-{year}.md', 'w') as f:
        f.write(f"Title: Book reviews: Year {year}\n")
        f.write(f"Date: {datetime.now().isoformat()}\n")
        f.write("tags: book review\n\n")

        for r in reviews:
            # A couple of abandoned books sneaked in with a '0' rating, and I'm
            # not interested in preserving those
            if r.rating != 0:
                f.write(f"## {r.title} by {r.author}\n\n")
                f.write(f"{rating_or_review(r)}\n")
                f.write("\n")


for year in range(2013, 2023):
    reviews = get_reviews(str(year))
    # Chronological order
    reviews = sorted(reviews, key=lambda r: r.date_read)
    format_reviews(reviews, year)

The hardest part was probably to decide what text to convert a rating into, since I didn't want to keep numbers!

Step 3: Checking the output looks right and recalling fond memories

I used the 'Year in Books' pages on Goodreads to compare the results. There was some funkiness sometimes, like a book read in 2011 showing in year in books but without any shelves and a date read showing as 2020, even though I don't remember messing with it. The review also shows as Jan 2020 on the Goodreads UI despite appearing in the correct 'Year in books'. 2020 turned out to be date_added (which is definitely false) while the date_read field is empty. Maybe some data migration funkiness on the Goodreads side at some point during the last 12 years. Otherwise, a duplicate once, and a couple of intra-Goodreads links that didn't work.

I still have to clean up the file for 2022. I was getting annoyed with tracking myself so I didn't write reviews, but if it's for the blog I wouldn't mind adding a few notes. And I need to decide if I want to post my 2023 reviews as I go, or batch them in some way!

PyCon Ireland 2014

2014-11-20T08:56:00+00:00

Together with a couple of colleagues, I wrote a short report about PyCon Ireland 2014. Once again the conference was a lot of fun and I'm looking forward to the 2015 edition! :)

Training at EuroPython 2014 | Making your first contribution to OpenStack

2014-08-01T11:18:00+01:00

Last week I ran a 3-hour training on how to get started contributing to OpenStack at EuroPython. The aim was to give a high-level overview of how the contribution process works in the project and guide people through making their first contribution, from account creation to submitting a patch.

Overview

The session starts with an extremely fast overview of OpenStack, geared toward giving the participants an idea of the different components and possible areas for contribution. We then go through creating the accounts, why they're all needed, and how to work with DevStack for the people who have installed it. From there we finally start talking about the contribution process itself, some general points on open-source and OpenStack culture then go through a number of ideas for small tasks suitable for a first contribution. After that it's down to the participants to work on something and prepare a patch. Some people chose to file and triage/confirm bugs. The last part is about making sure the patch matches the community standards, submitting it, and talking about what happens next both to the patch and to the participant as a new member of the community.

Preparing

During the weeks preceding the event, I ran two pilot workshops with small groups (less than 10 people) in my local hackerspace, in preparation for the big one in Berlin. That was absolutely invaluable in terms of making the material more understandable and refining the content for items I didn't think of covering initially (e.g. screen, openrc files) and topics that could use more in-depth explanations (e.g. how to find your first task), timings, and generally getting a feel for what's reasonably achievable within a 3-hour intro workshop.

Delivering

I think it went well, despite some issues at the beginning due to lack of Internet connectivity (always a problem during hands-on workshops!). About 70 people had signed up to attend (a.k.a. about 7 times too many), thankfully other members of the OpenStack community stepped up and offered their help as mentors - thanks again everyone! In the end, about half the participants showed up in the morning, and we lost another dozen to the Internet woes. The people who stayed were mostly enthusiastic and seemed happy with the experience. According to the session etherpad, at least 5 new contributors uploaded a first patch :) Three are merged so far.

Distributing the slides early proved popular and useful. For an interactive workshop with lots of links and references it's really helpful for people to go back on something they missed or want to check again.

Issues

The start of the workshop is a bit lecture-heavy and could be titled "Things I Desperately Wish I Knew When Starting Out," and although there's some quizzes/discussions/demoing I'd love to make it more interactive in the future.

The information requested in order to join the Foundation tends to surprise people, I think because people come at it from the perspective of "I want to submit a patch" rather than "I am preparing to join a Foundation." At the hackerspace sessions in particular (maybe because it was easier to have candid discussions in such a small group), people weren't impressed with being forced to state an affiliation. The lack of obvious answer for volunteers gave the impression that the project cares more about contributions from companies. "Tog" might make an appearance in the company stats in the future :-)

On the sign-up form, the "Statement of Interest" is intimidating and confusing for some people (I certainly remember being uncertain over mine and what was appropriate, back when I was new and joining the Foundation was optional!). I worked around this after the initial session by offering suggestions/tips for both these fields, and spoke a bit more about their purpose.

A few people suggested I simply tell people to sign up for all these accounts in advance so there's more time during the workshop to work on the contribution itself. It's an option, though a number of people still hit non-obvious issues with Gerrit that are difficult to debug (some we opened bugs for, others we added to the etherpad). During one of the pilot sessions at the hackerspace, 6 of the 7 participants had errors when running git review -s - I'm still not sure why, as it Worked On My Machine (tm) just fine at the same time.

Overall, I'm glad I did this! It was interesting to extract all this information from my brain, wiki pages and docs and attempt to make it as consumable as possible. It's really exciting when people come back later to say they've made a contribution and that the session helped to make the process less scary and more comprehensible. Thanks to all the participants who took the time to offer feedback as well! I hope I can continue to improve on this.

Python meet-up, November 2012: DevOps 101 | DevOps 101

2012-11-27T20:19:00+00:00

Shame on me for not blogging about PyCon Ireland this year, another success where I happily spent a big chunk of time greeting attendees at the registration desk and attended some interesting talks.

The stylish volunteers:

In the meantime, I took better notes at the November Python Ireland meet-up, where Ulrich took us through DevOps 101, going from the basics of what it is and all sort of related tools to help, mostly Python-based.

I thought interesting that Ulrich worked from a specific definition of DevOps that differs from the ones I heard. There was a time when I used to call myself "a devop" because in addition to development work I was also maintaining production and test servers and handling deployments, etc. However at the time I learnt from other sources that "a devop" was actually a system admin who also writes code to make their job easier... so I just stopped using the word ;) Ulrich based his talk on the premise of development and operations working together.

Development tends to move fast while operations are more stable and conservative, why would you put these groups together?

The need to scale is an important one. You need the knowledge that the developers have of the application itself, but you also need to understand infrastructure. For instance if you deploy something and suddenly the number of requests spikes, you want devs and ops to be close together to solve the problem.

Following a "release early, release often" principle where you might deploy several times a day makes that collaboration important.

Environments are complicated and it's important for people to integrate ; if the departments remain separate people tend to stick only with the same group of people.

Devops enable better communication and understanding. Developers can move fast and deploy faster as they know what ops need. This also helps mitigate risk thanks to the regular deployments: fewer changes are deployed at once (helpful to determine what breaks) and it becomes easier to do.

The talk then moved on to tools, including advice on how to create a simple django app to keep track of which versions of software are deployed where -- very simple and yet usually enough.

Some of the projects that were mentioned:

Using Sword instead of puppet (...which is still best, apparently) if you want to stick to Python, though it's a very young project. Using such tools you can start versioning your configuration and environment, and store reasons for changes in commit messages.
Fabric for ssh the Python way :)
Virtualenv to isolate Python and manage dependencies
Buildout is like Maven for Python projects. It's used to maintain dependencies and deploy, though it can be overkill for many projects...
Augeas is not Python specific and supports many configuration formats

Ulrich then went through some of Martin Fowler's principles for refactoring among other things, and also talked about Jenkins, not Python but known as the "Wordpress of Continuous Integration" because it has so many plugins. Sonar can be used for code quality.

We tried something different for the second part of the evening and watched a video of a Pyramid talk. I don't think it quite worked, even at 30 minutes it felt too long. Perhaps we should try again with a video more likely to appeal to a wider range of people, or simply shorter talks. Fair play to Diarmuid for standing up and taking the heat of answering questions afterwards ;)

EuroPython 2012 | Let's go

2012-06-30T20:55:00+01:00

I'm very happy to be going to EuroPython again this year :) Once more the talks I'm most looking forward to are mainly related to scalability, perhaps with a dash of internationalisation/encoding on the side. The tutorial on Django testing with Selenium should be interesting too.

Say hi if you see me! :)