It's been a while coming but this past week I released and deployed a new cut of FMN, Fedora's Fedmsg Notification system (it's a neat webapp that can send you notifications of events as they happen throughout Fedora Infrastructure). The release is full of bug fixes and little feature enhancements, I'll talk about a few here:
New defaults - this is the big one. The default settings were the biggest issue raised by early adopters. The complaints fell along two main axes: 1) the defaults deliver lots of messages that most users won't want and 2) the defaults are hard to understand and therefore hard to modify.
- In that first respect, the settings have been narrowed down greatly such that now you don't receive notifications about the bulk of your own activity. One of the most absurd cases was IRC meetings. If you were participating in an IRC meeting, you would get emails about topic changes in the IRC meeting you were already participating in. Clearly, you don't need an email about that.
- With the revised defaults, there are two main filters built: "Events on packages I own, except this kind, this kind, and that kind" and "Events involving my username, except this kind, this kind, and that kind." These are hopefully more intuitive to read and modify. Initial feedback from testers has been positive.
- There's also a new @mention filter that comes with the defaults which will notify you if someone references @your_fas_username anywhere in a fedmsg message, i.e. if someone adds @ralph in a fedorahosted trac ticket, I'll get a ping by IRC about it.
There's a button in the web interface you have to press to discard your current preferences and load in the new defaults. Try it out! Here's where you find it:
On to other new features, there is now a sensible digest mode for IRC that will collapse some collections of messages into one. It looks something like this:
- mschwendt and hobbes1069 commented on freecad-0.14-5.fc20,smesh-220.127.116.11-1.fc20
- limb changed sagitter's commit, watchcommits, and 2 others permissions on engauge-digitizer (epel7) to Approved
- kevin requested 2 xfdesktop stable updates for F21 and F20
- limb changed group::kde-sig's commit, watchcommits, and watchbugzilla permissions on ksysguard (f21, f20, and master) to Approved
- sagitter submitted 4 engauge-digitizer testing updates for F21, F20, and 2 others
- remi submitted 4 php-phpseclib-file-asn1 testing updates for F21, F20, and 2 others
There is now a long form format for some messages over email. Specifically, git pushes to dist-git will include the patch in the email. If you have requests for other kinds of information you want forwarded to you, please file an RFE so we know what to prioritize.
Lastly, I'll say that making the most of FMN requires a little bit of imagination (which can be hard to do if you aren't already familiar with it). Here are some for custom filters you could build for your account:
I get IRC notifications about all my github events now through FMN. I built a filter to forward messages that:
- are a github message
- match the regex (ralphbean|fedora-infra) (which simply means the message must contain either ralphbean or fedora-infra)
- are not continuous integration updates from Travis-CI.org (because those are spammy)
- are not deletions of tags or branches, since I don't care too much about those
And I use that to follow Fedora Infrastructure app development -- new pull requests, new issues, etc..
You could apply a similar scheme and build a filter to follow something in Fedora you're an expert at or want to learn more on. You could for instance build a filter for "systemd" like this:
- include messages that match the regex systemd
- ignore All messages from the Koji build system
- ignore All messages from dist-git SCM.
Anyways, please give the latest FMN a test and report any issues as you find them. Cheers!
The good news is, most everything was smooth sailing.. with one exception. FMN has been having a terrible time staying on top of its workload. After we brought the ARM Koji hub onto the fedmsg bus, FMN just started to drown.
Bad News Bears
Here's a depiction of its "queue length" or backlog -- the number of messages it has received, but has yet to process (We want that number to be as low as possible, such that it handles messages as soon as they arrive.)
You read it right--that's hundreds of thousands of messages still-to-process. You wouldn't get your messages until the day-after and besides which, FMN would never get it under control. The backlog would just continue to pile-up until we restarted it, dropping all buffered messages.
We introduced a couple improvements to the back-end code which have produced favorable results.
Firstly, we rewrote the pkgdb caching mechanism twice over. It is typical for users to have a rule in their preferences like "notify me of any event having anything to do with a package that I have commit access on". That commit access data is stored in pkgdb (now pkgdb2). In order for that to work, the FMN back-end has to query pkgdb for what packages you have commit access on.
In the first revision, we would cache the ACLs per-user. Our call to pkgdb2 to get all that information per-user took around 15 seconds to execute and we would cache the results for around 5 minutes. This became untenable quickly: the closer we grew to 20 test users, we ended up just querying pkgdb all day. Math it out: 20 users, 15 seconds per query. 5 minutes to cache all 20 users, and by that time, the cache of the first user had expired so we had to start over.
We moved temporarily to caching per-package, which was at least tenable.
Secondly, we (smartly, I think) reorganized the way the FMN back-end manages its own database. In the first revision of the stack, every time a new message would arrive FMN would:
- query the database for every rule and preference for every user,
- load those into memory,
- figure out where it should forward the message,
- do that, and then drop the rules and preferences (only to load them again a moment later).
Nowadays, we cache the entire ruleset in memory (forever). The front-end is instrumented to publish fedmsg messages whenever a user edits something in their profile -- when the back-end receives such a message, it invalidates its cache and reloads from the database (i.e., infrequently and as-needed).
We ended up applying the same scheme to the pkgdb caching, such that whenever a message arrives indicating that someone's ACLs have changed somewhere in pkgdb, we delete that portion of our local cache for that user, which is then refreshed the next time it is needed.
Things get better
Take a look now:
It still spikes when major events happen, but it quickly gets things under control again. More to come... (and, if you love optimizing stuff and looking for hotspots, please come get involved; there are too many things to hack on).
Join the Fedora Infrastructure Apps team in #fedora-apps on freenode!