[three]Bean

Originally posted at http://threebean.wordpress.com/2011/04/06/a-cry-of-resistance-against-cuomos-cuts/.

I just wrote and published blityclip. Install it and you can bitly-ify anything in your clipboard.

I have my xmonad config set to rock it:

    ,("C-b", spawn "bitlyclip")

Do you ever write code that becomes way nested and totally out of hand? Being smart with itertools can help un-indent your biz.

Below follow two functions definitions: nested_generator and itertools_generator. Put them in a module named itertools_blogpost.py.

from itertools import product

def nested_generator(carnivores, herbivores, plants):
    """ You do this every day. """
    for carnivore in carnivores:
        for herbivore in herbivores:
            for plant in plants:
                yield "%s eat %s eat %s" % (carnivore, herbivore, plant)

def itertools_generator(carnivores, herbivores, plants):
    """ This can 'flatten' your code,
    make it less nested, more pythonic. """
    for carnivore, herbivore, plant in product(carnivores, herbivores, plants):
        yield "%s eat %s eat %s" % (carnivore, herbivore, plant)

Let's test it with another script.

#!/usr/bin/env python

from itertools_blogpost import nested_generator, itertools_generator

if __name__ == '__main__':
    # Operate on these lists
    carnivores = ["lions", "tigers", "bears"]
    herbivores = ["hippies"]
    plants = ["carrots", "beets", "garlic"]

    # Test to show that the two functions are `equivalent`
    result1 = list(nested_generator(carnivores, herbivores, plants))
    result2 = list(itertools_generator(carnivores, herbivores, plants))
    are_they_equal = (result1 == result2)
    print "Gravey? -->", are_they_equal # Gravey!

    # Test to show that they run in an equivalent amount of time
    from timeit import Timer
    nested_timer = Timer(
        """
        carnivores = ["lions", "tigers", "bears"]
        herbivores = ["hippies"]
        plants = ["carrots", "beets", "garlic"]
        nested_generator(carnivores, herbivores, plants)
        """, "from itertools_blogpost import nested_generator")

    itertools_timer = Timer(
        """
        carnivores = ["lions", "tigers", "bears"]
        herbivores = ["hippies"]
        plants = ["carrots", "beets", "garlic"]
        itertools_generator(carnivores, herbivores, plants)
        """, "from itertools_blogpost import itertools_generator")

    nested_times = nested_timer.repeat()
    itertools_times =  itertools_timer.repeat()
    n = len(nested_times)

    print "          Nested  Iter"
    print "Maximum   {0:2.3f}   {1:2.3f}".format(
        max(nested_times), max(itertools_times))
    print "Minimum   {0:2.3f}   {1:2.3f}".format(
        min(nested_times), min(itertools_times))
    print "Average   {0:2.3f}   {1:2.3f}".format(
        sum(nested_times)/n, sum(itertools_times)/n)

Their output is equivalent and (run it for yourself) the times are roughly equivalent. Sometimes one comes out on top of the other, sometimes vice versa. No statistically significant difference.

The upshot: with itertools.product you can say goodbye to awfully nested for loops.

I spent Monday through Thursday of this week at the Open Science Grid (OSG) all-hands meeting in Boston, MA. Work sent me to get me up to speed on what's going on.

The OSG is cool. It is a national, distributed computing grid for data-intensive research that is really bent on an open approach to high-throughput computing. On average, it serves 1.2 million cpu-hours of computation per day.

Unlike Blue Waters (which you can't get onto) and unlike the TeraGrid (which requires you apply for and wait for allocations), the OSG makes it simple for you to run as much research code as you need to. The whole project appears to be really driven by the Compact Muon Solenoid (CMS) and the A Toroidal LHC ApparatuS (ATLAS) experiments from the LHC. There are a ton of different disciplines computing on the OSG, but CMS and ATLAS dominate the usage.

The OSG is really complex and the conference was total acronym overload but here are the two coolest things:

1. XRootD-- Think 'the bittorrent of file systems'. XRootD is a real necessity for the CMS and ATLAS projects that need to move tremendous amounts of data very quickly.

Before XRootD, compute submissions to the OSG would have to have their jobs directed only to sites where the data was known to already exist or users would have to copy the data there directly (with globus-copy-url or something equivalent).

With XRootD, a running job simply asks for a file and and the local redirector asks its peers, who ask their peers until all the instances that have the file respond and simultaneously feed the data to the requester. It scales linearly!

2. Cloud Computing(!)-- A couple different projects are going on to introduce 'dynamic, on-demand cloud infrastructures'. The team at Clemson is building a piece of software called kestrel that submits a number of 'pilot' jobs to condor that then spin up each of their own VMs and phone home for more work. Argonne Labs and the NERSC Magellan team have had some luck running super-physics software in a different by similar setup.

Heading home, I've got a lot of new ideas and things to try and play with. Connecting our resources to the OSG seems a priority, but first we'll need to write a GRAM plugin for the Globus Toolkit to add SLURM support (acronym overload, right?).

tw2.jit.SQLARadialGraph is a python class that encapsulates the awesome RGraph widget from the Javascript InfoVis Toolkit and automatically sets it up with JSON data from a sqlalchemy-mapped database. This post is a tutorial on how to make use of it in an application built on the Pyramid web framework.

1. Create a fresh pyramid app from the pyramid_alchemy paster template (instructions).

2. Create some sqlalchemy models with one-to-many and/or many-to-many relationships and populate the DB along the lines described in my SQLARadialGraph in a Turbogears 2.1 app tutorial.

3. Include tw2.jit and formencode in the requires list in your setup.py file and re-run python setup.py develop with your virtualenv active.

4. Include the Toscawidgets 2 middleware in the Pyramid WSGI stack by specifying egg:tw2.core#middleware just before your application's line in the pipeline setting inside development.ini.

5. Create a module /widgets.py with the following contents:

from tw2jitpyramiddemo import models
from tw2.jit import SQLARadialGraph

class UserGraph(SQLARadialGraph):
    id = 'whatever'
    base_url = '/jit_data'
    entities = [models.MyModel, models.MyGroup]
    width = '750'
    height = '533'

6. Expose the UserGraph widget in one of your views.

Specifically, modify /views.py and change the view_model(context, request) view to look something like:

from widgets import UserGraph

def view_model(context, request):
    return {'item':context, 'project':'tw2.jit-pyramid-demo',
            'jitwidget':UserGraph(rootObject=context)}

7. Render the widget in the corresponding template.

If you're using Chameleon templates (/templates/model.pt) this looks like:

<div tal:content="structure jitwidget.display()"></div>

If you're using Mako templates (/templates/model.mak) this looks like:

${jitwidget.display() | n}

8. Open a route to the widget's own controller.

Add the following three lines to /__init__.py just before the return config.make_wsgi_app() line:

from widgets import UserGraph
jit_view = lambda c, r : UserGraph.request(r)
config.add_route('jit_data', '/jit_data', view=jit_view, xhr=True)

And we're done. Restart the app and checkout http://localhost:6543/1 and it might look something like this.

You can, of course, tweak all kinds of style and functional parameters on SQLARadialGraph like those outlined near the end of my post on using SQLARadialGraph in a Turbogears 2.1 app.

Maybe you disagree, but the Pyramid implementation above went much more smoothly than the TG 2.1 setup. I think I like it.