[three]Bean

Speeding up that nose test suite.

Jun 23, 2015 | categories: python, fedmsg, fedora, nose View Comments

Short post: I just discovered the --with-prof option to the nosetests command. It profiles your test suite by using the hotshot module from the Python standard library and it found a huge sore spot in my most frequently run suite. In this pull request we got the fedmsg_meta running 31x faster.

Compare before:

(fedmsg_meta)❯ time $(which nosetests) -x
Ran 3822 tests in 270.822s

OK (SKIP=1638)
----------------------------------------------------------------------
Success!
$(which nosetests) -x  267.30s user 1.32s system 98% cpu 4:33.53 total

And after:

(fedmsg_meta)❯ time $(which nosetests) -x
Ran 3822 tests in 5.982s

OK (SKIP=1638)
----------------------------------------------------------------------
Success!
$(which nosetests) -x  3.87s user 0.71s system 52% cpu 8.700 total

That test suite used to take forever. It's the whole reason I wrote nose-audio in the first place!

View Comments

PyCon 2015 (Part II)

Apr 21, 2015 | categories: python, fedora, pycon View Comments

I wrote last week about how a few of us from Fedora were at PyCon US in Montreal for the week. It's all over and done with and we're back home now (I got a flu-like bug for the second time this season on the way home...) So, these are just some quick notes on what I did at the sprints!

  • Early on, I ported python-fedora to python3 and afterwards bkabrda picked up the torch and ported fedmsg and the expansive fedmsg_meta module. The one thing standing in the way of full-on python3 fedmsg is the M2Crypto library which will probably not see python3 compatibility anytime soon. Slavek courageously ported half of fedmsg's crypto stack to the python3-compatible cryptography library only to find that it didn't support the other half of the equation. We're keeping those changes in a branch until that gets caught up.

  • The most exciting bit was helping Nolski with his tool that puts fedmsg notifications on the OSX desktop. It totally works. Crazy, right?

  • Later in the week, I helped decause a bit with his new cardsite app. Load it up and let it run for a while. It's yet another neat way to visualize the activity of the Fedora community in realtime.

  • I started a prototype of fedora-hubs which doesn't do much but display little dummy widgets, but it is useful for reflecting on how the architecture ought to work.

  • I wrote some code to get the fedmsg-notify desktop tool to pull its preferences from the FMN service. The changes work, but they required some server-side patches to FMN that are done, but haven't yet been rolled out to production (and we're in freeze for the Beta release anyways..).

    In order to use your FMN preferences, you currently have to set a gsettings value by hand which is unacceptable and gross, but I'm not sure how to present it in the config UI. We can't just go all-in with FMN because there are other distros out there (Debian) which use fedmsg-notify but which don't run their own FMN service. We'll have to think on it and let it sit for a while.

  • Lastly, Bodhi2 saw some good work. (We fixed some bugs that needed to be hammered out before release and we actually have an RPM and installed it on a cloud node! Staging will be coming next once some el7 compat deps get sorted out.)

  • ncohglan introduced us to the authors of kallithea which led to some conversations about pagure and where we can collaborate.

  • I was really glad to meet sijis for fun-time late-night hackery in the hotel lobby.

That's all I can remember. It was a whirlwind, as always. Happy Hacking!

View Comments

PyCon 2015 (Part I)

Apr 12, 2015 | categories: python, fedora, pycon View Comments

A few of us from Fedora are at PyCon US in Montreal for the week. The conference portion is almost over and the sprints start tomorrow, but in the meantime here are some highlights from the best sessions I sat in on:

Some hacking happened in the interstitial periods!

  • I wrote a prototype of a system to calculate and store statistics about fedmsg activity and some plugins for it. This will hopefully turn out to be quite useful for building community dashboards in the future (like a revamped releng dashboard or the nascent fedora-hubs).
  • We ported python-fedora to python3! Hooray!
  • The GSOC deadline really snuck up on us, so Pierre-Yves Chibon and I carved out some time to sit down and go over all the pending applications.

I'm really looking forwards to the sprints and the chance to work and connect with all our upstreams. We'll be holding a "Live From Pycon" video cast at some point. Details forthcoming. Happy Hacking!

View Comments

Fedora Photowall

Sep 04, 2014 | categories: python, fedora View Comments

If you didn't already know it, there is a Fedora Badge for associating a libravatar with your Fedora account.

https://badges.fedoraproject.org/pngs/mugshot.png

A fun by-product of having such a thing is that we have a list of FAS usernames of people who have public avatars with predictable URLs. With that, I wrote a script that pulls down that list and assembles a "photo wall" of a random subset. Check it out:

http://threebean.org/blog/static/images/montage2.png

I wrote it as a part of another project that I ended up junking, but the script is neat on its own.

Perhaps we can use it as a splash image for a Fedora website someday (say, next year's Flock site or a future iteration of FAS?). It might make a fun desktop wallpaper, too.

Here's the script if you'd like to re-use and modify. You can tweak the dimensions variable to change the number of rows and columns in the output.

#!/usr/bin/env python
""" fedora-photowall.py - Make a photo montage of Fedora contributors.

Dependencies:
 $ sudo yum install python-sh python-beautifulsoup4 python-requests ImageMagick

Usage:   python fedora-photowall.py
Author:  Ralph Bean
License: LGPLv2+

"""

import hashlib
import os
import random
import urllib

import sh
import bs4
import requests

dimensions = (12, 5)

datadir = './data'
avatar_dir = datadir + '/avatars'
montage_dir = datadir + '/montage'


def make_directories():
    try:
        os.makedirs(datadir)
    except OSError:
        pass

    try:
        os.makedirs(avatar_dir)
    except OSError:
        pass

    try:
        os.makedirs(montage_dir)
    except OSError:
        pass


def avatars(N):
    url = 'https://badges.fedoraproject.org/badge/mugshot/full'
    response = requests.get(url)
    soup = bs4.BeautifulSoup(response.text)
    last_pane = soup.findAll(attrs={'class': 'grid-100'})[-1]
    persons = last_pane.findAll('a')

    persons = random.sample(persons, N)

    for person in persons:
        name = person.text.strip()
        openid = 'http://%s.id.fedoraproject.org/' % name
        hash = hashlib.sha256(openid).hexdigest()
        url = "https://seccdn.libravatar.org/avatar/%s" % hash
        yield (name, url)


def make_montage(candidates):
    """ Pull down avatars to disk and stich with imagemagick """

    filenames = []
    for name, url in candidates:
        filename = os.path.join(avatar_dir, name)
        if not os.path.exists(filename):
            print "Grabbing", name, "at", url
            urllib.urlretrieve(url, filename=filename)
        else:
            print "Already have", name, "at", filename
        filenames.append(filename)

    args = filenames + [montage_dir + '/montage.png']
    sh.montage('-tile', '%ix%i' % dimensions, '-geometry', '+0+0', *args)
    print "Output in", montage_dir


def main():
    make_directories()
    N = dimensions[0] * dimensions[1]
    candidates = avatars(N)
    make_montage(candidates)

if __name__ == '__main__':
    main()

And another example of output:

http://threebean.org/blog/static/images/montage4.png

Cheers!

View Comments

A tiny optimization

Sep 03, 2013 | categories: python, nitpicking, fedora View Comments

Talking over a pull request with @pypingou, we found that this one method of constructing a set from a list of stripped strings was slightly faster than another:

#!/usr/bin/env python
""" Timing stuff.

::
    $ python timeittest.py
    set(map(str.strip, ['wat '] * 200))
    30.9805839062
    set([s.strip() for s in ['wat '] * 200])
    31.884624958
"""

import timeit


def measure(stmt):
    print stmt
    results = timeit.timeit(stmt)
    print results


measure("set(map(str.strip, ['wat '] * 200))")
measure("set([s.strip() for s in ['wat '] * 200])")

Admission: Pierre bullied me into blogging about this!


UPDATE: Folks in the comments recommended using a generator or itertools.imap. The results are significantly better. Here they are:

import itertools; [s.strip() for s in ['wat '] * 200]
28.2224271297
import itertools; (s.strip() for s in ['wat '] * 200)
3.0280148983
import itertools; map(str.strip, ['wat '] * 200)
25.7294211388
import itertools; itertools.imap(str.strip, ['wat '] * 200)
2.3925549984

UPDATE (again): Comments further reveal that the update above is misleading -- the generators aren't actually doing any work there. If we force them to spin out, we get results like these:

import itertools; set([s.strip() for s in ['wat '] * 200])
33.4951019287
import itertools; set((s.strip() for s in ['wat '] * 200))
35.5591659546
import itertools; set(map(str.strip, ['wat '] * 200))
33.7568879128
import itertools; set(itertools.imap(str.strip, ['wat '] * 200))
35.9931280613

No clear benefit for use of imap or generators.

View Comments

Next Page »