fedmsg

The Fedora Infrastructure Message Bus

FLOCK13, Charleston SC, August 9th, 2013

Go sit in #fedora-fedmsg on irc.freenode.net.

http://threebean.org/presentations/fedmsg-flock13/

fedmsg

what it is

The Fedora Infrastructure Message Bus is a python package and API used around Fedora Infrastructure to send and receive messages to and from applications.

It's not my idea! In 2009, jkeating started making noise about a message bus and how cool it would be.

We wanted to hook all the services in Fedora Infrastructure up to send messages to one another over a message bus instead of communicating with each other in the heterogeneous, “Rube-Goldberg” ways they do now.

It uses X.509 certificates to sign messages (like "HTTPS" certs).

It's meant to be deployed in an open infrastructure. Anyone can read. Anyone can write. Only some messages are trusted.

It is built on top of ØMQ.

There is no central broker and as far as we can tell, no single point of failure.

It is publicly subscribable -- hit up tcp://hub.fedoraproject.org:9940 with a zmq.SUB socket.

It has Fedora in the name, but Debian Infrastructure is picking it up this summer. They're making progress too. We now need to change the name to mean the FEDerated Message Bus instead.

fedmsg

how to do it

first

you have to get it

$ sudo yum -y install fedmsg

sending messages

from the shell

$ echo "Hello World." | fedmsg-logger --modname=git --topic=repo.update

$ echo '{"a": 1}' | fedmsg-logger --json-input

$ fedmsg-logger --message="This is a message."

$ fedmsg-logger --message='{"a": 1}' --json-input

sending messages

from python

import fedmsg

fedmsg.publish(
    topic='testing',
    msg={
        'test': 'Hello World',
        'foo': jsonifiable_objects,
        'bar': a_sqlalchemy_object,
    }
)

consuming messages

at the command line

$ fedmsg-tail --really-pretty
{
    "i": 1,
    "timestamp": 1344344053.2337201,
    "topic": "org.fedoraproject.prod.bodhi.update.comment",
    "msg": {
        "comment": {
            "update_title": "nethack4-4.0.0-1.fc20",
            "group": None,
            "author": "ralph",
            "text": "I'm so pumped to pwn those minotaurs!",
            "karma": 1,
            "anonymous": False,
            "timestamp": 1344344050.0
        }
    }
}

consuming messages

from python

import fedmsg

for name, endpoint, topic, msg in fedmsg.tail_messages():
    print topic, msg

consuming messages

with a daemon

fedmsg-hub is a daemon that can make writing your own long-running consumers simpler. There are docs on fedmsg.com for writing plugins, but they look like this:

import pprint
import fedmsg.consumers


class MyConsumer(fedmsg.consumers.FedmsgConsumer):
    topic = "org.fedoraproject.*"
    config_key = 'myconsumer.enabled'

    def consume(self, message):
        pprint.pprint(message)

lmacken wrote an example consumer with everything you need. Clone it and use it as a starting point as you please.

consuming messages

at the command line... an aside

There are lots of fun options to fedmsg-tail.

(Aside: there is a plugin system to provide domain-specific metadata about messages. Debian is working on their own. Install the Fedora one!)

$ sudo yum -y install python-fedmsg-meta-fedora-infrastructure

With that, you can use the more fantastic options:

$ fedmsg-tail --terse
buildsys.build.state.change -- ausil's tncfhh-0.8.3-14.fc20 completed
http://koji.fedoraproject.org/koji/buildinfo?buildID=439734
trac.ticket.update -- kevin closed a ticket on the Fedora Infrastructure trac instance as 'fixed'
https://fedorahosted.org/fedora-infrastructure/ticket/3904
bodhi.update.request.testing -- mmckinst submitted nawk-20121220-1.fc18 to testing
https://admin.fedoraproject.org/updates/nawk-20121220-1.fc18
wiki.article.edit -- Hguemar made a wiki edit to "Flock:Rideshare"
https://fedoraproject.org/w/index.php?title=Flock:Rideshare&diff=prev&oldid=347430

consuming messages

like you're living in the future

You can run:

$ fedmsg-tail --gource | gource --log-format custom -

Which makes something like this.

topics

what messages?

topics

Full list at http://fedmsg.com/en/latest/topics/, including:

things that use fedmsg

there's a lot of them at this point

koji

stalk

David Aquilina's (dwa's) koji stalk monitors koji over fedmsg and rebuilds packages for arm and ppc.

FAS2Trac (ftl)

(fama updater)

herlo's FAS2Trac fama updater (ftl) listens to messages indicating that a user has applied for membership in the ambassadors group -- it then files a ticket in the ambassadors' trac instance for a potential sponsor via XMLRPC.

compose

downloader

p3ck's fedmsg-download listens for messages that the daily branched and rawhide compose process has finished -- it then downloads the latest builds from rsync://dl.fedoraproject.org/fedora-linux-development

notifications

on the desktop

lmacken's fedmsg-notify listens for messages and displays a filtered stream on your desktop with libnotify.

notifications

on the desktop

For kicks, there's a gnome-shell extension.

notifications

on the desktop

You can turn on and off all kinds of message types.

notifications

on the desktop

Some more advanced message filtration.

monitoring the bus

in the browser with websockets

Fedora Bus Monitor

...and gource,

of course

Fedmsg piped live to gource

nom

all the data

datanommer was originally conceived of by Ian Weller. It's a fedmsg-hub plugin that sits listening on the bus and simply throws every message into a database.

It has a partner: datagrepper! Datagrepper is an HTTP JSON API for the whole history of messages kept in datanommer.

reports

10 ways from sunday

pingou has been really busy writing tools against datagrepper...

reports

10 ways from sunday

pingou's fedora-news is a nice HTML5 mobile-ready app that gives you access to all sorts of the latest information from the Fedora Community.

reports

10 ways from sunday

pingou's this-week-in-fedora weekly posts stats about the most active contributors.

reports

10 ways from sunday

Every week, pingou's owner changes report tool emails the devel list with a report of what packages were orphaned, unorphaned and retired.

fedora badges

for you, and you, and you

Fedora badges is a new webapp launching this weekend that awards "badges" to Fedora contributors for their activity.

Check out the talk on it tomorrow!

future

stuff

future

stuff

Debian deployment! See the latest post from @olasd.

To listen to debian messages on your box, add this

to /etc/fedmsg.d/endpoints.py:

# You can get messages from here too!
"debian-infrastructure": [
    "tcp://fedmsg.olasd.eu:9940",
],

future

stuff

The debian developers are submitting lots of new features to be included in the 0.7.0 release.

fedora

mobile

See Ricky Elrod's landing page.

There's a hackfest on it this weekend!

future

stuff

fedmsg-notifications. -- Problem: all of our applications carry their own email code. With that comes further baggage and maintenance.

With fedmsg notifications for interesting infrastructure events, we can put all that code in one place where it can be more easily maintained.

Benefit to the end-user: manage notification preferences in one place instead of per-app.

What about notifications to different contexts? Email? IRC privmsg? Android? RSS?

A repo has been created for this but contains nothing more than notes at this point.

future

stuff

Mirror pushing. -- Problem: We have over 200 mirrors that help serve Fedora releases. You can read more about them here. As it stands they all run rsync on some interval to poll for new content.

There was some discussion of pushing the data years ago, but understandably, mirror admins are reluctant to allow someone access to push content onto their machines. With a fedmsg solution, we only push a notification; the pulling is still within the admin's control.

There is a pull request waiting for review that will add a fedmsg-trigger command to fedmsg core. We can use that to kick off rsync jobs when messages matching certain criteria are received.

get on the bus!

Get the source:

Presented by:

http://threebean.org/presentations/fedmsg-flock13/

Development discussed in #fedora-apps.

Join #fedora-fedmsg for the firehose.

SpaceForward
Left, Down, Page DownNext slide
Right, Up, Page UpPrevious slide
POpen presenter console
HToggle this help