Bad News Bears

Here's a depiction of its "queue length" or backlog -- the number of messages it has received, but has yet to process (We want that number to be as low as possible, such that it handles messages as soon as they arrive.)

You read it right--that's hundreds of thousands of messages still-to-process. You wouldn't get your messages until the day-after and besides which, FMN would never get it under control. The backlog would just continue to pile-up until we restarted it, dropping all buffered messages.

Optimizing

We introduced a couple improvements to the back-end code which have produced favorable results.

Firstly, we rewrote the pkgdb caching mechanism twice over. It is typical for users to have a rule in their preferences like "notify me of any event having anything to do with a package that I have commit access on". That commit access data is stored in pkgdb (now pkgdb2). In order for that to work, the FMN back-end has to query pkgdb for what packages you have commit access on.

In the first revision, we would cache the ACLs per-user. Our call to pkgdb2 to get all that information per-user took around 15 seconds to execute and we would cache the results for around 5 minutes. This became untenable quickly: the closer we grew to 20 test users, we ended up just querying pkgdb all day. Math it out: 20 users, 15 seconds per query. 5 minutes to cache all 20 users, and by that time, the cache of the first user had expired so we had to start over.

We moved temporarily to caching per-package, which was at least tenable.

Secondly, we (smartly, I think) reorganized the way the FMN back-end manages its own database. In the first revision of the stack, every time a new message would arrive FMN would:

• query the database for every rule and preference for every user,
• load those into memory,
• figure out where it should forward the message,
• do that, and then drop the rules and preferences (only to load them again a moment later).

Nowadays, we cache the entire ruleset in memory (forever). The front-end is instrumented to publish fedmsg messages whenever a user edits something in their profile -- when the back-end receives such a message, it invalidates its cache and reloads from the database (i.e., infrequently and as-needed).

We ended up applying the same scheme to the pkgdb caching, such that whenever a message arrives indicating that someone's ACLs have changed somewhere in pkgdb, we delete that portion of our local cache for that user, which is then refreshed the next time it is needed.

Things get better

Take a look now:

It still spikes when major events happen, but it quickly gets things under control again. More to come... (and, if you love optimizing stuff and looking for hotspots, please come get involved; there are too many things to hack on).

Join the Fedora Infrastructure Apps team in #fedora-apps on freenode!

By the numbers

It was among the most productive sprints I've ever been a part of. The commit activity graph on github tells us:

• 25 commits on Sunday
• 47 commits on Monday
• 43 commits on Tuesday
• 55 commits on Wednesday
• 6 commits on Thursday (we were packing up to leave).

None of those stats count commits to other repos related to bodhi2, like this feature and that feature.

To Do

At the FAD, we decided on a schedule of sorts: we'll try to get bodhi2 into staging before the alpha feature freeze in July, but we won't try to get Bodhi2 into production until a few weeks after F21 is released (Novemeber). I had made some previous "predictions" that we'd see it before Flock, but I hadn't factored in any interactions with the release process. We shouldn't rush it and disturb that.

Here are some things left that need doing:

• Performance - Some pages (notably the user profile page) take far too long to load. We need to work on some optimizations
• Finish the masher rewrite - The rewrite is awesome now (it has 70%+ test coverage. the bodhi1 masher had 0% coverage). It's just going to take more time to finish working through all the things that process is responsible for.
• Long-form templates from fedmsg.meta - We hope to remove all the email code from Bodhi2 and just use Fedora Notifications (fmn) for notifications. One thing missing is the ability in fmn to do "long form" templates (long emails with lots of information). We have a plan to do that by adding a msg2long_form(msg, **config) function to fedmsg.meta. It just needs to get written, released, and leveraged at this point.
• Candidates Deps would be a cool thing to auto-populate in the new update form. Currently, you type in a package and it auto-populates a checkbox list with all the candidate builds it could find. It would be extra neat if it could also find candidate dependencies and add them to that list as well.
• Karma Policy Control - we have fine-grained karma feedback now, but it would be nice for maintainers to set fine-grained policy based on that feedback. Right now, it looks just like Bodhi1 (we just didn't have time to work through this corner at the FAD).
• Taskotron feedback - We talked it over and submitted a patch, but there's still some work to be done to display taskotron results in the Bodhi2 UI.

Framing

Jessica McKellar's keynote was amazing. It was good in its own right, but was also important in so far as it framed the process of change ongoing in both the python community and its major conference. People know the state of women in FLOSS is abysmal. The effort to turn things around was clear and front and center this year: "outreach works".

Guid Van Rossum's keynote featured a "no to 2.8" image meaning, there is not nor will there be a python-2.8 release. For Fedora and its downstreams, it serves as a reminder that we need to beef up on python3 preparedness. We're making progress. Do pay attention to the proposed change to make python 3 the default in Fedora 22 and help clear the way where you can.

Diversion into ML

I attended a number of talks on machine learning and big data and got all inspired. I wrote this little throwaway program in between sessions. It reads in a bunch of newgroups and then tries to predict which newsgroup each posting belongs to (and after lots of tweaking, it got a 100% success rate which is crazy).

Ultimately, what I want to do is write a plugin for mailman3. In the mailman3/hyperkitty UI you can tag/categorize threads. Each community I imagine would end up tagging threads with their own jargon: "this thread is a flamewar", "this is a feature request", "this one's a bug report", "this is a moonshot".. or whatever. We could build a plugin that reads in all those tags for each community and then builds a predictive model that can be applied to new messages as they come in.

We could proactively tag messages using the community's own jargon! My little script seems to work well enough and we know how to build mailman3 plugins. At this point we just have to wait for the community to accumulate a set of real world tags we could train and test against.

Anyways...

Sprints

Just being there puts us in a position to build relationships with other upstreams and the rest of the community but the PyCon sprints were concretely productive for us this year. A couple people from my team were present.

• Aurélien got his changes merged into mailman proper(!) and they prepared to cut a beta release of the whole suite(!!).
• Toshio put work in on warehouse, the next generation of PyPI.
• Luke tore it up on Bodhi2 (crystal ball -- we'll have it deployed before Flock this year).
• I got a barrel's worth of python3 porting done (fedmsg is now python3 ready!).

Shout out

My good friend Remy Decausemaker got the spot to give the very very last lightning talk at pycon entitled "Adventures in Hackademia" where he talked about the the first open source collegiate minor in the US that some of us have been helping to get started. Very exciting.