Make tools with fedmsg!

Workshop at FLOCK14, Prague CZ, August 8th, 2014

Go sit in #fedora-fedmsg on irc.freenode.net.

http://threebean.org/presentations/fedmsg-flock14/

fedmsg

what it is

The Fedora Infrastructure Message Bus is a python package and API used around Fedora Infrastructure to send and receive messages to and from applications.

It is publicly subscribable -- hit up tcp://hub.fedoraproject.org:9940 with a zmq.SUB socket.

It has Fedora in the name, but Debian Infrastructure started picking it up last summer. They've made progress to the point that we had to change the name to mean the FEDerated Message Bus instead.

data.gouv.fr is using it too. Maybe others? We get questions and clarifications on the deployment docs from time to time.

fedmsg

what do?

There are two aspects to this workshop:

Do you want me to cover?

first

you should get it

sudo yum install fedmsg

There's also a plugin that let's us render Fedora Infrastructure messages nicely. You should install that too:

sudo yum install python-fedmsg-meta-fedora-infrastructure

A taste

of the bus

Clone the repo from https://github.com/ralphbean/fedmsg2gource

Run:

python fedmsg2gource.py --days 14 > testing.log
cat testing.log | \
    gource -i 10 \
        --user-image-dir ~/.cache/avatars/ \
        --log-format custom \
        --viewport 1024x730 \
        -

Explore

the datagrepper API

https://apps.fedoraproject.org/datagrepper

say

you wanted your own local bus

sudo yum install fedmsg-relay
sudo systemctl start fedmsg-relay
echo "Hello World." | fedmsg-logger --modname=git --topic=repo.update
echo '{"a": 1}' | fedmsg-logger --json-input
fedmsg-logger --message="This is a message."
fedmsg-logger --message='{"a": 1}' --json-input

or from python:

import fedmsg

fedmsg.publish(
    topic='testing',
    msg={
        'test': 'Hello World',
        'foo': jsonifiable_objects,
        'bar': a_sqlalchemy_object,
    }
)

if

you want to consume

fedmsg-tail --really-pretty
{
    "i": 1,
    "timestamp": 1344344053.2337201,
    "topic": "org.fedoraproject.prod.bodhi.update.comment",
    "msg": {
        "comment": {
            "update_title": "nethack4-4.0.0-1.fc20",
            "group": None,
            "author": "ralph",
            "text": "I'm so pumped to pwn those minotaurs!",
            "karma": 1,
            "anonymous": False,
            "timestamp": 1344344050.0
        }
    }
}

consuming messages

from python

import fedmsg

for name, endpoint, topic, msg in fedmsg.tail_messages():
    print topic, msg

consuming messages

with a daemon

fedmsg-hub is a daemon that can make writing your own long-running consumers simpler. There are docs on fedmsg.com for writing plugins, but they look like this:

import pprint
import fedmsg.consumers


class MyConsumer(fedmsg.consumers.FedmsgConsumer):
    topic = "org.fedoraproject.*"
    config_key = 'myconsumer.enabled'

    def consume(self, message):
        pprint.pprint(message)

consuming messages

at the command line... an aside

There are lots of fun options to fedmsg-tail like --terse.

fedmsg-tail --terse
buildsys.build.state.change -- ausil's tncfhh-0.8.3-14.fc20 completed
http://koji.fedoraproject.org/koji/buildinfo?buildID=439734
trac.ticket.update -- kevin closed a ticket on the Fedora Infrastructure trac instance as 'fixed'
https://fedorahosted.org/fedora-infrastructure/ticket/3904
bodhi.update.request.testing -- mmckinst submitted nawk-20121220-1.fc18 to testing
https://admin.fedoraproject.org/updates/nawk-20121220-1.fc18
wiki.article.edit -- Hguemar made a wiki edit to "Flock:Rideshare"
https://fedoraproject.org/w/index.php?title=Flock:Rideshare&diff=prev&oldid=347430

things that use fedmsg

there's a lot of them at this point

koji

stalk

David Aquilina's (dwa's) koji stalk monitors koji over fedmsg and rebuilds packages for arm and ppc.

FAS2Trac (ftl)

(fama updater)

herlo's FAS2Trac fama updater (ftl) listens to messages indicating that a user has applied for membership in the ambassadors group -- it then files a ticket in the ambassadors' trac instance for a potential sponsor via XMLRPC.

compose

downloader

p3ck's fedmsg-download listens for messages that the daily branched and rawhide compose process has finished -- it then downloads the latest builds from rsync://dl.fedoraproject.org/fedora-linux-development

synchronization

of package ACLs

So, it used to be that when someone was granted commit access to a package in the Fedora PackageDB (pkgdb), the webapp simply wrote to a database table indicating the new relationship. Every hour, a cronjob would run that queried the state of that database and then re-wrote out the ACLs for gitolite -- the software that manages access to our package repositories.

Consequently, we had lots of waiting: you would request commit access to a repository, then wait for an owner to grant you rights, then wait for that cronjob to run before you could actually push.

With a new fedmsg consumer that we have in place, those gitolite ACLs are re-written in response to fedmsg messages from the pkgdb. It is much faster.

notifications

to email, irc, the desktop, and android

There's the new FMN system that can deliver notifications to you via irc, email, and android.

There's also lmacken's fedmsg-notify which listens for messages and displays a filtered stream on your desktop with libnotify.

reports

10 ways from sunday

Every week, pingou's owner changes report tool emails the devel list with a report of what packages were orphaned, unorphaned and retired.

reports

10 ways from sunday

There's also the Release Engineering Dashboard which grabs data from datagrepper on all the latest updates syncs, composes, image builds, etc.. and puts their status all in one place. Pure HTML/javascript -- there's no server-side app here.

fedora badges

for you, and you, and you

Fedora badges launched last year at Flock13. It awards "badges" to Fedora contributors for their activity.

Pretty fun. :)

To sum that up

The assimilation of message producing services is nearly complete.

There are many message consuming services already in place.. but we can likely make many more. Which is why you're here, no?

Today's

task

Surprise! We're going to make a Twitter Bot!

How

it's going to work

Take this

It's dangerous out there

sudo yum install fedmsg
sudo yum install python-fedmsg-meta-fedora-infrastructure
sudo yum install python-fabulous
sudo yum install tweepy

Your first

fedmsg script

import fedmsg
import pprint

print "Posting up to listen on the fedmsg bus.  Waiting for a message..."
for name, endpoint, topic, msg in fedmsg.tail_messages():
    pprint.pprint(msg)

Give it a run.

It's like a million voices cried out

and then were silent

#topic_filter = 'fedbadges'     # We really want this, but its rare
topic_filter = 'fedoratagger'   # This is much easier to test with

for name, endpoint, topic, msg in fedmsg.tail_messages():
    if topic_filter not in topic:
        # Bail out if the topic doesn't match
        continue

    pprint.pprint(msg)

See http://fedmsg.com/en/latest/topics for more

Some config

at the top

import fedmsg.config
import logging.config

# First, load the fedmsg config from fedmsg.d/
config = fedmsg.config.load_config()

# Then, configure the python stdlib logging to use fedmsg's logging config
logging.config.dictConfig(config.get('logging'))

So

meta

import fedmsg.meta

# Initialize fedmsg's "meta" module if you have the fedora infra plugin
fedmsg.meta.make_processors(**config)

for name, endpoint, topic, msg in fedmsg.tail_messages():
    if topic_filter not in topic:
        continue

    # Only act on your own messages -- things that *you* did.
    if 'YOUR_FAS_USERNAME' not in fedmsg.meta.msg2usernames(msg, **config):
        continue

    # Use it to make nice text and other things
    # See also: msg2icon, msg2link, msg2usernames, msg2packages...
    subtitle = fedmsg.meta.msg2subtitle(msg, **config)
    print subtitle

A picture

is worth a thousand words

import tempfile
import urllib
import os

import fabulous.image

for name, endpoint, topic, msg in fedmsg.tail_messages():
    # This returns a URL (most of the time)
    icon = fedmsg.meta.msg2icon(msg, **config)

    _, filename = tempfile.mkstemp(suffix='.png')
    print "Downloading", icon, "to", filename
    urllib.urlretrieve(icon, filename)

    print fabulous.image.Image(filename)

    print "Cleaning up %r" % filename
    os.remove(filename)

Intermezzo

We have a neat working script that gets fedmsg messages pushed to it. It can extract neato stuff and print it.

But... if we want to move to the next step, we have to take a break from our happy hacking to go and deal with Twitter, its API, and API keys.

The Twitter API

We're going to have to:

  1. Create our own "app". Visit https://apps.twitter.com/app/new
  2. Modify that app's permission to include "Read and Write".
  3. Authorize that app with our own account, which yields oauth tokens. To do this, click the "Create my access token" button at the bottom of your app's detail page.

We will keep those tokens a secret and our little bot will use them to login and tweet on our behalf. You'll get four secret strings.

Storing

those secrets

First, add a directory called fedmsg.d/ to your current working directory.

In it, put a file called fedmsg.d/twitter-secrets.py that looks like this:

config = dict(
    consumer_key        = "your api key goes here",
    consumer_secret     = "your api secret goes here",
    access_token_key    = "your access token goes here",
    access_token_secret = "your access token secret goes here",
)

Test that fedmsg can read in that new config file by looking for them in:

fedmsg-config | less

Using

those secrets

Go back to badgebot.py and add the following:

import tweepy

consumer_key        = config['consumer_key']
consumer_secret     = config['consumer_secret']
access_token_key    = config['access_token_key']
access_token_secret = config['access_token_secret']

auth_handler = tweepy.OAuthHandler(consumer_key, consumer_secret)
auth_handler.set_access_token(access_token_key, access_token_secret)
twitter_api = tweepy.API(auth_handler)

And

further down

for name, endpoint, topic, msg in fedmsg.tail_messages():

    subtitle = fedmsg.meta.msg2subtitle(msg, **config)
    link = fedmsg.meta.msg2link(msg, **config)
    icon = fedmsg.meta.msg2icon(msg, **config)

    _, filename = tempfile.mkstemp(suffix='.png')
    print "Downloading", icon, "to", filename
    urllib.urlretrieve(icon, filename)

    # Construct and post our tweet.
    #print fabulous.image.Image(filename)
    content = subtitle + " " + link
    print "Tweeting %r" % content
    twitter_api.update_with_media(filename, content)

    print "Cleaning up %r" % filename
    os.remove(filename)

Does it work?

systemd

for real

Make a new file called badgebot.service with these contents:

[Unit]
Description=A Twitter bot for your Fedora Badges.  Wow.
After=network.target
Documentation=http://fedmsg.com

[Service]
ExecStart=/usr/local/bin/badgebot.py
Type=simple
User=fedmsg
Group=fedmsg

[Install]
WantedBy=multi-user.target

install.sh

#!/bin/bash -x
# install.sh - (re)install and (re)start the badgebot

# Install our script
cp badgebot.py /usr/local/bin/badgebot.py

# Make sure no one else can read our secrets.
cp fedmsg.d/twitter-secrets.py /etc/fedmsg.d/.
chown fedmsg:fedmsg /etc/fedmsg.d/twitter-secrets.py
chmod o-r /etc/fedmsg.d/twitter-secrets.py

# Copy in service file for systemd
cp /home/threebean/devel/badgebot/badgebot.service /usr/lib/systemd/system/badgebot.service
systemctl daemon-reload
systemctl restart badgebot

Watch the journal:

sudo journalctl -u badgebot --follow

Does it work? Debug!

Open Hacking Time

SpaceForward
Left, Down, Page DownNext slide
Right, Up, Page UpPrevious slide
POpen presenter console
HToggle this help