[three]Bean http://threebean.org/blog Creates a Blog Tue, 14 May 2013 18:03:38 GMT Blogofile hourly 1 In which I avoid the inverse unicode sandwich http://threebean.org/blog/sieve Fri, 22 Jun 2012 18:00:00 EDT http://threebean.org/blog/sieve In which I avoid the inverse unicode sandwich

Problem #1 - I need to test tons of HTML output for correctness (because I maintain toscawidgets2). That output varies slightly because tw2 supports five different templating languages (mako, genshi, jinja2, kajiki, and chameleon). Using double-equals (==) just won't do it.

Solution #1 - We used strainer. It works!

Problem #2 - Imagine porting this to Python 3. Yes, that's right. The encoding is sniffed by hand and then used to encode regular expressions; these are in turn applied to parse XML. Think "inverse unicode sandwich with a side of Cthulhu."

Solution #2 - I wrote sieve: a baby module child of one corner of FormEncode and another corner of strainer. It works on pythons 2.6, 2.7, and 3.2. If you like, you may use it:

>>> from sieve.operators import eq_xml, in_xml
>>> a = "<foo><bar>Value</bar></foo>"
>>> b = """
... <foo>
...     <bar>
...         Value
...     </bar>
... </foo>
... """
>>> eq_xml(a, b)
True
>>> c = "<html><body><foo><bar>Value</bar></foo></body></html"
>>> in_xml(a, c)  # 'needle' in a 'haystack'
True

p.s. -- I looked into xmldiff. Awesome!

]]> Teaching Open Source, Web-Based, Game Programming at RIT http://threebean.org/blog/floss-seminar-rundown Tue, 06 Mar 2012 18:00:00 EST http://threebean.org/blog/floss-seminar-rundown Teaching Open Source, Web-Based, Game Programming at RIT

This last quarter at RIT I taught (for the first time) a course on open source, web-based, game programming. The quarter just came to an end and I really want to brag on my students. But before I do that, here's a glimmer of what we covered:

  • How to use bash and vim
  • How to use git and github
  • How to submit patches to projects you know nothing about
  • What makes a good, casual game
  • How to make games with HTML5
  • How to program in JavaScript, CoffeeScript, and Python
  • Serverside programming with Turbogears2
  • How to get your code running on Red Hat's Openshift cloud.
  • How to give a lightning talk(!)

Really cool stuff was built into the class; it was about making open source software, so we open sourced the syllabus! Really cool stuff happened along the way; we hit ugly problems with openshift, so we patched the quickstarter script!

You can endure my self-indulgent drivel yes, but you get the best picture from reading the students' blog posts themselves. Enough of this! Les projets de cours!

#1 - Lazorz teamed up with the Boston Museum of Science to make "[An] educational game about the physics of light. In it's current incarnation it helps demonstrate the concepts of reflection and color filtering. The development team has plans to include other concepts such as refraction and prisms in future releases."

http://lazorz-fossrit.rhcloud.com/image/lazorzscreenshot.png

They got a nasty front-end built, by hand, with javascript and HTML5 that works on every mobile device we could test it on. It is intuitive and fun. They have almost-working Facebook auth and they have an almost-working Turbogears2 backed JSON store. (really, this close to completion!)

#2 - Gold Rush is visually and game-ly amazing. If I had to pick which of the three project was going to make one million dollars, it'd be this one.

http://i.imgur.com/XJCshh.jpg

The front-end is built in Unity (closed source, but compiles to a ton of platforms) and the back-end JSON store is built in TurboGears2. These guys completely flew through development, mastering skills they'd never heard of in a day. Its 3D with a moving camera and an incredibly fun game. The original game idea belongs to team member Eric Heaney. It's fun to play as a normal old card game (I've played it at a number of parties since he pitched it to the class).

#3 -- WebBotWar (the python web robot fighting game) wins the prize for master hack. They:

  • Forked pybotwar, ripped it's UI off and made it export JSON.
  • Reimplemented the front-end with jCanvas (and cooked up all their own art assets!).
  • Wrote a TurboGears2 app that spins up serverside instances of their pybotwar fork.
    • Those instances dump their state into mongodb (or memcached, long story).
    • Their javascript client polls the TG2 app for the game state and voilà.

They have (awesome) plans to:

  • Allow you to upload your own scripts (they have it working, just not secured).
  • Make a built-in script editor.
  • Tighter facebook integration (challenge your friends!)

BTW, it works on android and iPad. It even works on Epiphany (wat?) They also want you to know that python box2d is a pain in the ass.

https://github.com/ralphbean/WebBot/raw/c127a15b0c5f1d5683c7619676fc7aec4970e061/pywebbot-screenshot.png ]]> ToscaWidgets2 Bugsprint Report http://threebean.org/blog/bugsprint-report Sun, 04 Mar 2012 15:00:00 EST http://threebean.org/blog/bugsprint-report ToscaWidgets2 Bugsprint Report

Here's the rundown of the pre-PyCon tw2 bugsprint. The unstoppable Greg Jurman and I coordinated in google+ and IRC (with lots of ad-hoc visitors in both) and did all the damage we could do to the bug list. Our goal was to close enough that we could justify a solid 2.0 non-beta release but we didn't quite get there. We settled on 2.0 release candidate 1.

tl;dr - Big progress. We'll seal the deal on 2.0 at PyCon US next week.

Infrastructure:

Tickets closed:

Tickets worked on, but not complete:

Tickets created:

]]> TW2 Bugsprint; the Final Countdown http://threebean.org/blog/the-final-countdown Fri, 02 Mar 2012 11:00:00 EST http://threebean.org/blog/the-final-countdown TW2 Bugsprint; the Final Countdown

Tomorrow marks the start of the toscawidgets2 24 hour bugsprint and I couldn't be more stoked. We haven't even started yet but tw2 development pushed ahead anyways.

We have:

Goal: (solidly) close as many bugs as possible and push as many libraries from beta to release as we can.

Here's to winning that game.

]]> Hacking tw2 resource injection http://threebean.org/blog/2011/12/13/hacking-tw2-resource-injection/ Tue, 13 Dec 2011 05:44:53 EST http://threebean.org/blog/2011/12/13/hacking-tw2-resource-injection/ Hacking tw2 resource injection Tonight, VooDooNOFX was asking in IRC in #turbogears how to disable the injection of jquery.js by tw2.jquery into her/his TG2 app. Using the inject_resources=False middleware config value wouldn't cut it, since she/he wanted tw2 to inject all other resources, they were loading jQuery via google CDN beforehand and tw2's injection was clobbering their code.

I came up with the following hack to myapp/lib/base.py which will remove tw2.jquery.jquery_js from the list of resources tw2 would inject into each page served by a TG2.1 app.

At the top of myapp/lib/base.py import:

import tw2.core.core
import tw2.jquery

and then replace:

        return TGController.__call__(self, environ, start_response)

with the following:

        stream = TGController.__call__(self, environ, start_response)

        # Disable the injection of tw2.jquery
        offending_link = tw2.jquery.jquery_js.req().link
        local = tw2.core.core.request_local()
        local['resources'] = [
            r for r in local.get('resources', list()) if r.link != offending_link
        ]

        return stream

The two tricks to this are

  • Simply knowing that tw2 resources register themselves with the 'request_local' object and that during the return-phase of the WSGI pipeline, the tw2 middleware refers to that list when injecting resources
  • Figuring out where in a TG2 app's request flow to place the call to alter that object after all widgets that might register jquery have declared their resources but before the resources list is injected into the output stream.

We came out of it with a bug filed in the tw2 issue tracker so we can take care of it properly in the future.

]]> raptorizemw - Fact: Every WSGI app is better with a raptor. http://threebean.org/blog/2011/10/03/raptorizemw-fact-every-wsgi-app-is-better-with-a-raptor/ Mon, 03 Oct 2011 20:35:34 EDT http://threebean.org/blog/2011/10/03/raptorizemw-fact-every-wsgi-app-is-better-with-a-raptor/ raptorizemw - Fact: Every WSGI app is better with a raptor. It's done. An over-engineered WSGI middleware component that adds a velociraptor to every page served. Fact: Every WSGI app is better with a raptor.

It's called raptorizemw (pronounced "awesome") and the only way to use it is in production.

]]> TurboGears 2.1 and Foreclosures (more empty houses than homeless people) http://threebean.org/blog/2011/09/24/turbogears-2-1-and-foreclosures-more-empty-houses-than-homeless-people/ Sat, 24 Sep 2011 16:01:21 EDT http://threebean.org/blog/2011/09/24/turbogears-2-1-and-foreclosures-more-empty-houses-than-homeless-people/ TurboGears 2.1 and Foreclosures (more empty houses than homeless people) I wrote an app that scrapes foreclosure data from my county of residence and plots it six ways from Sunday in a TurboGears2.1 app. You can find it at http://monroe-threebean.rhcloud.com/, hosted on redhat's openshift cloud.

It's used by activists with Take Back the Land, Rochester and my local branch of the ISO to find upcoming evictions before they happen and organize the neighborhoods to stop the shuttering of homes. Get a hundred people at the door of the house before the cops come, and no-one is getting evicted (we've had some successes).

We're living in some absurd times where banks got bailed out by the trillions yet still get to collect on our student debt and mortgages. Most of us are being ruined. If you're not, then your neighbor is.

If you're in Boston, check out Vida Urbana or if you're in Chicago, check out the Chicago Anti-Eviction Campaign. Anywhere you go, check out the ISO.

Fork my code, port it to your home town, and start organizing!

]]> Using repoze.who.plugins.ldap in a TurboGears 2.1 app http://threebean.org/blog/2011/07/19/using-repoze-who-plugins-ldap-in-a-turbogears-2-1-app/ Tue, 19 Jul 2011 16:24:24 EDT http://threebean.org/blog/2011/07/19/using-repoze-who-plugins-ldap-in-a-turbogears-2-1-app/ Using repoze.who.plugins.ldap in a TurboGears 2.1 app Often, you will need to authenticate against ldap in your webapp. Here's how to make that happen in a freshly quickstarted TurboGears 2.1 app.

Setting up your environment

mkvirtualenv --no-site-packages repoze-ldap-app
pip install tg.devtools
paster quickstart   # call the app repoze-ldap-app, yes to mako and auth
cd repoze-ldap-app
python setup.py develop
pip install genshi  # This is a workaround.
paster setup-app development.ini
paster serve development.ini  # To test if the basic app works.

Point your browser at http://localhost:8080 just to make sure everything is cool.

Setting up repoze.who.plugins.ldap

Add the following line to the install_requires list in setup.py:

    "repoze.who.plugins.ldap",

Run python setup.py develop to install the newly listed repoze plugin.

Add the following four lines to development.ini which reference an as yet unwritten secondary configuration file. Place them just above the sqlalchemy.url=... lines:

# Repoze.who stuff
who.config_file = %(here)s/who.ini
who.log_level = INFO
who.log_stream = stdout

Create a new file who.ini with the following contents:

# This file is adapted from:
# http://threebean.org/blog/2011/07/19/using-repoze-who-plugins-ldap-in-a-turbogears-2-1-app/
# which has been adapted from:
# http://static.repoze.org/whodocs/#middleware-configuration-via-config-file
# which has been adapted from:
# http://code.gustavonarea.net/repoze.who.plugins.ldap/Using.html

[plugin:friendlyform]
use = repoze.who.plugins.friendlyform:FriendlyFormPlugin
login_form_url= /login
login_handler_path = /login_handler
logout_handler_path = /logout_handler
rememberer_name = auth_tkt
post_login_url = /post_login
post_logout_url = /post_logout

[plugin:auth_tkt]
use = repoze.who.plugins.auth_tkt:make_plugin
secret = omg_this_is_so_secret_lololololol_2938485#butts

[plugin:ldap_auth]
# Here I use my own ldap_auth, since by default ldap allows affirmative
# authentication with *no password specified*.  That is lame; I override it.
use = repozeldapapp.lib.auth:ReconnectingAuthenticatorPlugin

# This is the URI of wherever you want to connect to.  I work at RIT.
ldap_connection = ldap://ldap.rit.edu

# This is the base of the 'distinguished names' (DNs) of persons in your
# particular LDAP instance.  It will vary from server to server.
base_dn = ou=People,dc=rit,dc=edu

[plugin:ldap_attributes]
# I also do some overriding for more security in how I get attributes for
# users.
use = repozeldapapp.lib.auth:ReconnectingLDAPAttrsPlugin
ldap_connection = ldap://ldap.rit.edu

[general]
request_classifier = repoze.who.classifiers:default_request_classifier
challenge_decider = repoze.who.classifiers:default_challenge_decider

[mdproviders]
plugins =
    ldap_attributes

[identifiers]
plugins =
    friendlyform;browser
    auth_tkt

[authenticators]
plugins =
    ldap_auth

[challengers]
plugins =
    friendlyform;browser

Create another new file repozeldapapp/lib/auth.py with the following contents:

from repoze.who.plugins.ldap import (
    LDAPAttributesPlugin, LDAPAuthenticatorPlugin
)
import ldap


class URISaver(object):
    """ Saves the ldap_connection str given to repoze authn and authz """
    def __init__(self, *args, **kw):
        self.uri = kw['ldap_connection']
        super(URISaver, self).__init__(*args, **kw)


class ReconnectingLDAPAttrsPlugin(LDAPAttributesPlugin, URISaver):
    """ Gets attributes from LDAP.  Refreshes connection if stale. """

    def add_metadata(self, environ, identity):
        """ Add ldap attributes to the `identity` entry. """

        try:
            return super(ReconnectingLDAPAttrsPlugin, self).add_metadata(
                environ, identity)
        except Exception, e:
            print "FAILED TO CONNECT TO LDAP 1 : " + str(e)
            print "Retrying..."
            self.ldap_connection = ldap.initialize(self.uri)
            return super(ReconnectingLDAPAttrsPlugin, self).add_metadata(
                environ, identity)


class ReconnectingAuthenticatorPlugin(LDAPAuthenticatorPlugin, URISaver):
    """ Authenticates against LDAP.

    - Refreshes connection if stale.
    - Denies anonymously-authenticated users

    """

    def authenticate(self, environ, identity):
        """ Extending the repoze.who.plugins.ldap plugin to make it much
        more secure. """

        res = None

        try:
            # This is unbelievable.  Without this, ldap will
            #   let you bind anonymously
            if not identity.get('password', None):
                return None
            try:
                dn = self._get_dn(environ, identity)
            except (KeyError, TypeError, ValueError):
                return None

            res = super(ReconnectingAuthenticatorPlugin, self).authenticate(
                environ, identity)

            # Sanity check here (for the same reason as the above check)
            if "dn:%s" % dn != self.ldap_connection.whoami_s():
                return None

        except ldap.LDAPError, e:
            print "FAILED TO CONNECT TO LDAP 2 : " + str(e)
            print "Retrying..."
            self.ldap_connection = ldap.initialize(self.uri)

        return res

Finally, do two things to repozeldapapp/config/middleware.py.

Edit it and at the top of the file add:

from repoze.who.config import make_middleware_with_config

Add the following inside the make_app(...) function, just below the comment line about Wrap your base TurboGears 2..., like so:

    # Wrap your base TurboGears 2 application with custom middleware here
    app = make_middleware_with_config(
        app, global_conf,
        app_conf['who.config_file'],
        app_conf['who.log_stream'],
        app_conf['who.log_level'])

Give it a test

Restart the paster server and reload http://localhost:8080. Try logging in as a user in your ldap instance and you should be all gravy.

You should be all gravy. ]]> Tutorial -- melting your face off with tw2 and TurboGears2.1 http://threebean.org/blog/2011/04/30/tutorial-melting-your-face-off-with-tw2-and-turbogears2-1/ Sat, 30 Apr 2011 06:39:36 EDT http://threebean.org/blog/2011/04/30/tutorial-melting-your-face-off-with-tw2-and-turbogears2-1/ Tutorial -- melting your face off with tw2 and TurboGears2.1

Maunsell Fort tower (Hywel Williams) / CC BY-SA 2.0

Get the source: if you don't want to read through this, you can get the entire source for this tutorial here on my github account.

---

Today it's an epic data widgets tutorial! I've been working on a lot of stuff recently, including trying to pull some of the more quiet tw2 developers back together; I hope that this fits into that scheme and to at some point link directly to this tutorial from the tw2 documentation.

This tutorial won't introduce you to any of tw2.core fundamentals directly, but it will show off some of the flashier and data-driven widgets. If it gets you real hot, check out some of my other tw2+TurboGears2.1 tutorials here and here and my tw2+Pyramid tutorial here, but enough of that, let's cut to the chase:

Agenda

We're going to build an TurboGears2.1 application from start to finish that logs the IP of every request made of it and displays those hits with a couple of fancy-ass tw2 widgets. Here's what it's going to take:

  • Getting TurboGears2.1 installed and running
  • Setting up a data backend
  • "Automatically" listing db entries with tw2.jqplugins.jqgrid:SQLAjqGridWidget
  • Plotting server history with tw2.jqplugins.jqplot:JQPlotWidget
  • Making the layout look like http://google.com/ig with tw2.jqplugins.portlets

1. Getting TurboGears2.1 installed and running

Install (if you haven't already) virtualenvwrapper. It's awesome and you should use it always.

Next, Open up your favorite terminal and do the following:

% mkdir tw2-facemelt-tg2.1 &amp;&amp; cd tw2-facemelt-tg2.1
% mkvirtualenv --no-site-packages --distribute tw2-facemelt-tg2.1
% pip install --use-mirrors tg.devtools
% paster quickstart
        Enter project name: tw2-facemelt-tg2.1
        Enter package name [tw2facemelttg21]:
        Would you prefer mako templates? (yes/[no]): yes
        Do you need authentication and authorization in this project? ([yes]/no): yes
% cd tw2-facemelt-tg2.1

At this point we need to modify the freshly quickstarted TurboGears 2.1 project. There's a bug there! We said want to use mako templates, so TG2.1 isn't configured to install genshi but it does include references to the tgext.admin.controller:AdminController which references genshi but doesn't list its dependency and therefore doesn't install it. That's fine, we'll just remove the references to the AdminController since we won't be using it, anyways.

Remove the following three lines from tw2facemelttg21/controllers/root.py:

from tgext.admin.tgadminconfig import TGAdminConfig
from tgext.admin.controller import AdminController

...

    admin = AdminController(model, DBSession, config_type=TGAdminConfig)

While we're at it, let's turn on the tw2 WSGI middleware.

Edit tw2facemelttg21/config/app_cfg.py and add the following two lines at the bottom:

base_config.use_toscawidgets = False
base_config.use_toscawidgets2 = True

It should be noted that toscawidgets1 (setup by TG2.1 by default) and toscawidgets2 can peacefully coexist in a WSGI app but I'm a tw2 purist. After all, tw2 is faster.

At this point you should be able to run your TG2.1 quickstarted app. Run the following:

% pip install --use-mirrors -e .
% paster setup-app development.ini
% paster serve development.ini

And visit http://localhost:8080 to verify that all this worked.

2. Setting up a data backend

Create a file at tw2facemelttg21/models/bloglog.py with the following contents:

# -*- coding: utf-8 -*-
""" Logs of Bob Loblaw's Law Blog """

from sqlalchemy import *
from sqlalchemy.orm import mapper, relation
from sqlalchemy import Table, ForeignKey, Column
from sqlalchemy.types import Integer, Unicode
#from sqlalchemy.orm import relation, backref

from datetime import datetime

from tw2facemelttg21.model import DeclarativeBase, metadata, DBSession


class ServerHit(DeclarativeBase):
    __tablename__ = 'server_hit'

    id = Column(Integer, primary_key=True)
    timestamp = Column(DateTime, nullable=False, default=datetime.now)

    remote_addr = Column(Unicode(15), nullable=False)

    path_info = Column(Unicode(1024), nullable=False)
    query_string = Column(Unicode(1024), nullable=False)

Edit the module-level file tw2facemelttg21/models/__init__.py and uncomment the following line:

DeclarativeBase.query = DBSession.query_property()

Add the following line to the very bottom of the same file (tw2facemelttg21/models/__init__.py):

from bloglog import ServerHit

The data model should be good to go now, but let's add one little piece of code --- a hook --- that will populate the server_hit table as the app runs.

Edit tw2facemelttg21/lib/base.py and add the following seven-line chunk just inside the __call__(...) method of your BaseController 

    def __call__(self, environ, start_response):
        """Invoke the Controller"""
        # TGController.__call__ dispatches to the Controller method
        # the request is routed to. This routing information is
        # available in environ['pylons.routes_dict']

        entry = model.ServerHit(
            remote_addr=environ['REMOTE_ADDR'],
            path_info=environ['PATH_INFO'],
            query_string=environ['QUERY_STRING'],
        )
        model.DBSession.add(entry)
        model.DBSession.flush()

        request.identity = request.environ.get('repoze.who.identity')
        tmpl_context.identity = request.identity
        return TGController.__call__(self, environ, start_response)

Now blow away and recreate your old sqlite database by typing the following into your trusty terminal:

% rm devdata.db
% paster setup-app development.ini
% paster serve development.ini

Pray for no bugs as you revisit http://localhost:8080. Reload the page a few times and just double-check that your database has entries in it now by running sqlite3 devdata.db. Issue the command select * from server_hit; and you should see all your page requests listed.

Cool? Cool.

(Note to the brave: If we wanted to be really awesome, we would write WSGI middleware to do our request-logging.)

3. "Automatically" listing db entries with tw2.jqplugins.jqgrid:SQLAjqGridWidget

Create a new file tw2facemelttg21/widgets.py with the following content:

import tw2facemelttg21.model as model
from tw2.jqplugins.jqgrid import SQLAjqGridWidget

class LogGrid(SQLAjqGridWidget):
    id = 'awesome-loggrid'
    entity = model.ServerHit
    excluded_columns = ['id']
    datetime_format = "%x %X"

    prmFilter = {'stringResult': True, 'searchOnEnter': False}

    options = {
        'pager': 'awesome-loggrid_pager',
        'url': '/jqgrid/',
        'rowNum':15,
        'rowList':[15,150, 1500],
        'viewrecords':True,
        'imgpath': 'scripts/jqGrid/themes/green/images',
        'shrinkToFit': True,
        'height': 'auto',
    }

Pull the LogGrid widget into your controller by importing it at the top of tw2facemelttg21/controllers/root.py with:

from tw2facemelttg21.widgets import LogGrid

Modify the index method of your RootController in tw2facemelttg21/controllers/root.py so that it looks like:

@expose('tw2facemelttg21.templates.index')
def index(self):
    """Handle the front-page."""
    return dict(page='index', gridwidget=LogGrid)

This will make the LogGrid available in your index template under the name gridwidget. We still need to display it there.

Edit tw2facemelttg21/templates/index.mak and wipe out all of the content. Replace it with only the following:

<%inherit file="local:templates.master"/>
${gridwidget.display() | n}

Cool.

The SQLAjqGridWidget has a request(...) method that does all of its magic (interrogating your sqlalchemy model for its properties and values). We still need to wire up our TurboGears app to forward the ajax<->json requests to the right place.

To do this, add another method to your RootController back in tw2facemelttg21/controllers/root.py that looks like this:

@expose('json')
def jqgrid(self, *args, **kwargs):
    return LogGrid.request(request).body

Lastly, edit your setup.py file and add all the new widget dependencies we'll need (not just the jqgrid), like so:

    install_requires=[
        "TurboGears2 >= 2.1",
        "Mako",
        "zope.sqlalchemy >= 0.4",
        "repoze.tm2 >= 1.0a5",
        "repoze.what-quickstart",
        "repoze.what >= 1.0.8",
        "repoze.what-quickstart",
        "repoze.who-friendlyform >= 1.0.4",
        "repoze.what-pylons >= 1.0",
        "repoze.what.plugins.sql",
        "repoze.who == 1.0.18",
        "tgext.admin >= 0.3.9",
        "tw.forms",
        "tw2.jqplugins.jqgrid",
        "tw2.jqplugins.jqplot",
        "tw2.jqplugins.portlets",
        ],
    setup_requires=["PasteScript >= 1.7"],

Install the newly listed dependencies by running:

% python setup.py develop
% paster serve development.ini

...and revisit http://localhost:8080.

At this point, your app should look something like the following:

We should really (probably) do things as fancy as we can, and make use of jquery-ui's themes.

Edit tw2facemelttg21/lib/base.py do the following two things:

Add this import statement to the top of the file.

from tw2.jqplugins.ui import set_ui_theme_name

And invoke it from inside the __call__(...) method of your BaseController like so:

    def __call__(self, environ, start_response):
        """Invoke the Controller"""
        # TGController.__call__ dispatches to the Controller method
        # the request is routed to. This routing information is
        # available in environ['pylons.routes_dict']

        set_ui_theme_name('hot-sneaks')

        entry = model.ServerHit(
            remote_addr=environ['REMOTE_ADDR'],
            path_info=environ['PATH_INFO'],
            query_string=environ['QUERY_STRING'],
        )
        model.DBSession.add(entry)

        request.identity = request.environ.get('repoze.who.identity')
        tmpl_context.identity = request.identity
        return TGController.__call__(self, environ, start_response)

Nice! But hot-sneaks isn't the only available theme -- you can see a list of all of them right here.

4. Plotting server history with tw2.jqplugins.jqplot:JQPlotWidget

This is going to be awesome.

Add a new widget definition to tw2facemelttg21/widgets.py:

from tw2.jqplugins.jqplot import JQPlotWidget
from tw2.jqplugins.jqplot.base import categoryAxisRenderer_js, barRenderer_js
from tw2.core import JSSymbol

class LogPlot(JQPlotWidget):
    id = 'awesome-logplot'
    interval = 2000
    resources = JQPlotWidget.resources + [
        categoryAxisRenderer_js,
        barRenderer_js,
    ]

    options = {
        'seriesDefaults' : {
            'renderer': JSSymbol('$.jqplot.BarRenderer'),
            'rendererOptions': { 'barPadding': 4, 'barMargin': 10 }
        },
        'axes' : {
            'xaxis': {
                'renderer': JSSymbol(src="$.jqplot.CategoryAxisRenderer"),
            },
            'yaxis': {'min': 0, },
        }
    }

Now we're going to go to town on our RootController. We need to:

  • Make the new widget available in our template
  • Produce data for it
  • Render it in the template

First, add the following imports to the top of tw2facemelttg21/controllers/root.py:

from tw2facemelttg21.models import ServerHit
from tw2facemelttg21.widgets import LogPlot

import sqlalchemy
import datetime
import time

We're also going to need this little recursive_update utility to merge the options dicts. Just add it at the top of tw2facemelttg21/controllers/root.py as a function. It should not be a method of RootController.

def recursive_update(d1, d2):
    """ Little helper function that does what d1.update(d2) does,
    but works nice and recursively with dicts of dicts of dicts.

    It's not necessarily very efficient.
    """

    for k in d1.keys():
        if k not in d2:
            continue

        if isinstance(d1[k], dict) and isinstance(d2[k], dict):
            d1[k] = recursive_update(d1[k], d2[k])
        else:
            d1[k] = d2[k]

    for k in d2.keys():
        if k not in d1:
            d1[k] = d2[k]

    return d1

Add a new method to the RootController class that looks like the following. This will do all of the heavy lifty --- producing the data for the jqplot.

    def jqplot(self, days=1/(24.0)):
        n_buckets = 15
        now = datetime.datetime.now()
        then = now - datetime.timedelta(days)
        delta = datetime.timedelta(days) / n_buckets

        entries = ServerHit.query.filter(ServerHit.timestamp>then).all()

        t_bounds = [(then+delta*i, then+delta*(i+1)) for i in range(n_buckets)]

        # Accumulate into buckets!  This is how I like to do it.
        buckets = dict([(lower, 0) for lower, upper in t_bounds])
        for entry in entries:
            for lower, upper in t_bounds:
                if entry.timestamp >= lower and entry.timestamp < upper:
                    buckets[lower] += 1

        # Only one series for now.. but you could do other stuff!
        series = [buckets[lower] for lower, upper in t_bounds]
        data = [
            series,
            # You could add another series here...
        ]

        options = { 'axes' : { 'xaxis': {
            'ticks': [u.strftime("%I:%M:%S") for l, u in t_bounds],
        }}}

        return dict(data=data, options=options)

Rewrite the index(...) method of your RootController to look like the following:

    @expose('tw2facemelttg21.templates.index')
    def index(self):
        """Handle the front-page."""
        jqplot_params = self.jqplot()
        plotwidget = LogPlot(data=jqplot_params['data'])
        plotwidget.options = recursive_update(
            plotwidget.options, jqplot_params['options'])
        return dict(page='index', gridwidget=LogGrid, plotwidget=plotwidget)

Now the jqplot widget should pull its data from the (perhaps poorly named) jqplot method of your RootController and should merge new options nicely with the predefined ones. It should also be available in your index template under the name plotwidget. Let's use it there!

Edit tw2facemelttg21/templates/index.mak and add the following line:

<%inherit file="local:templates.master"/>
${plotwidget.display() | n}
${gridwidget.display() | n}

Revisit http://localhost:8080/ and you should get something like this.

5. Making the layout look like http://google.com/ig with tw2.jqplugins.portlets

Add the following import to the top of tw2facemelttg21/controllers/root.py:

import tw2.jqplugins.portlets as p

Rewrite the index(...) method of your RootController to look like this:

    @expose('tw2facemelttg21.templates.index')
    def index(self):
        """Handle the front-page."""
        jqplot_params = self.jqplot()
        plotwidget = LogPlot(data=jqplot_params['data'])
        plotwidget.options = recursive_update(
            plotwidget.options, jqplot_params['options'])

        colwidth = '50%'
        class LayoutWidget(p.ColumnLayout):
            id = 'awesome-layout'
            class col1(p.Column):
                width = colwidth
                class por1(p.Portlet):
                    title = 'DB Entries of Server Hits'
                    widget = LogGrid

            class col2(p.Column):
                width = colwidth
                class por2(p.Portlet):
                    title = 'Hits over the last hour'
                    widget = plotwidget

        return dict(page='index', layoutwidget=LayoutWidget)

Now that only the layoutwidget is available to your index template, you'll need to rewrite your tw2facemelttg21/templates/index.mak.

<%inherit file="local:templates.master"/>
${layoutwidget.display() | n}

To simplify all the styling, you'll also need to clear out all the clutter from the TG2.1 quickstart install. Rewrite (for the first time) tw2facemelttg21/templates/master.mak to look like this:

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
                 "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html>
<head></head>
<body>${self.body()}</body>
</html>

Rerun your paster server and you should see something like this.

The portlet windows are movable and collapsible and they automatically retain their state between page loads (with jquery.cookie.js).

---

I hope you enjoyed the tutorial. Once again, you can get the entire source here on my github account. If you have any comments, feedback, or questions --- post 'em!

]]> sqlalchemy + the JavaScript InfoVis Toolkit (jit) http://threebean.org/blog/2011/03/06/sqlalchemy-the-javascript-infovis-toolkit-jit/ Sun, 06 Mar 2011 05:27:24 EST http://threebean.org/blog/2011/03/06/sqlalchemy-the-javascript-infovis-toolkit-jit/ sqlalchemy + the JavaScript InfoVis Toolkit (jit) I've been working on some new ToscaWidgets2 widgetry. Today's post is a tutorial on how to use tw2.jit.SQLARadialGraph with the Turbogears 2.1 framework. You can check out the entire source tree for this tutorial from my github page.

I really like tw2.jit.SQLARadialGraph and have been getting overly-excited about it: foaming at the mouth, rambling... I've tried to cut all that out of this post and just give you the details on how to get started using it and Turbogears 2.1. In short, SQLARadialGraph itself is a python web component that produces an interactive graph of your database via javascript and asynchronous requests for JSON data. Most all of its real rendering work is accomplished with the awesome JavaScript InfoVis Toolkit by Nicolas Garcia Belmonte.

For this post, first we'll configure Turbogears 2.1 to use Toscawidgets 2 middleware. Second we'll modify our new app's bootstrapping process to insert some random data into its development sqlite database. Third, we'll configure and display an instance of tw2.jit.SQLARadialGraph to visualize the data we inserted.

1. Setting up Turbogears

Start with a fresh TG 2.1 quickstarted app by running $ paster quickstart tw2.jit-tg2.1-demo. It will ask you some questions -- for this tutorial we are using mako templates and we are using authentication.

virtualenv

Jump into your newly quickstarted app's directory and run $ virtualenv --no-site-packages virtualenv and $ source virtualenv/bin/activate to cordon off your app's dependencies from your system-wide python site-packages directory.

Configure TG 2.1 for Toscawidgets 2

Add the following three lines to the install_requires argument in your setup.py file.

        "repoze.what-pylons",
        "tg.devtools",
        "tw2.jit",

A fresh Turbogears 2.1 quickstarted app depends on Toscawidgets 1 by default. We'll be using Toscawidgets 2 so we need to remove all the old references as well as tell TG2.1 to use the tw2 middleware instead of tw1.

Remove the following three lines from tw2jittg21demo/controllers/root.py:

from tgext.admin.tgadminconfig import TGAdminConfig
from tgext.admin.controller import AdminController
 ... < snip > ...
    admin = AdminController(model, DBSession, config_type=TGAdminConfig)

Remove the following one line from tw2jittg21demo/lib/base.py:

from tw.api import WidgetBunch

Add the following two lines to the bottom of tw2jittg21demo/config/app_cfg.py:

base_config.use_toscawidgets = False
base_config.use_toscawidgets2 = True

You will also need to enable the query property on all models.

Uncomment the following line in tw2jittg21demo/model/__init__.py:

DeclarativeBase.query = DBSession.query_property()

Test it all with the following commands.

$ python setup.py develop
$ paster setup-app development.ini
$ paster serve --reload development.ini

And point your browser at http://localhost:8080. If all has gone well, you should see the default index page for your newly quickstarted TG2.1 web-app.

2. Add some `interesting` data

We'll use the existing models defined in tw2jittg21demo/model/ to make this tutorial shorter and smoother.

Edit tw2jittg21demo/websetup/bootstrap.py (which is the code that gets run when you issue $ paster setup-app development.ini) and add the following two new function definitions just above the definition of the bootstrap function.

from random import choice, randint

def add_random_users():
    """ Add 9 random users """
    import string
    chars = string.letters
    for first in [u'Sally', u'John', u'Tim']:
        for last in [u'Anderson', u'Flanderson', u'Block']:
            user = model.User()
            user.user_name = unicode((first[0] + last).lower())
            user.display_name = u'%s %s' % (first, last)
            user.email_address = u'%s@socialistworker.org' % user.user_name
            user.password = u''.join([choice(chars) for i in range(12)])
            model.DBSession.add(user)

    model.DBSession.flush()

def add_random_groups():
    """ Generate a number of random groups and add users to them """
    for name in ['developer', 'system admin', 'shmeveloper', 'crispin gladbin']:
        group = model.Group()
        group.group_name = name
        group.display_name = (u"%ss group" % name).title()
        model.DBSession.add(group)

        all_users = model.User.query.all()
        for i in range(randint(0, len(all_users)-2)):
            user = choice(all_users)
            while user in group.users:
                user = choice(all_users)
            
            group.users.append(user)

    model.DBSession.flush()

These new functions still need to be called during the boostrap process, so add two invocation lines just above the transaction.commit() line at the end of the boostrap function.

        model.DBSession.add(u1)
        model.DBSession.flush()

        add_random_users()
        add_random_groups()

        transaction.commit()
    except IntegrityError:
        print 'Warning, there was a problem adding your auth data, it may have already been added:'

Blow away and re-create your database with the following commands:

$ rm devdata.db
$ paster setup-app development.ini

Run $ paster serve --reload development.ini and point your browser at http://localhost:8080 again to make sure nothing is broken.

3. Visualize the database with tw2.jit.SQLARadialGraph

Make the widget available in your root controller

Create a module tw2jittg21demo/widgets.py with the following content:

from tw2jittg21demo import model
from tw2.jit import SQLARadialGraph

def makeUserGraph():
    class UserGraph(SQLARadialGraph):
        id = 'whatever'
        base_url = '/jit_data'
        entities = [model.User, model.Group, model.Permission]
        excluded_columns = ['_password', 'password',
                            'user_id', 'group_id', 'permission_id']
        width = '920'
        height = '525'
        rootObject = model.User.query.first()
    
    return UserGraph

Modify your root controller in tw2jittg21demo/controllers/root.py to make use of the new widget.

Add the following import above the RootController definition:

from tw2jittg21demo.widgets import makeUserGraph

Modify the index(self) method so that the return statement looks like:

        return dict(page='index', widget=makeUserGraph())

And add a new method to serve json data to the jit widget.

    @expose('json')
    def jit_data(self, *args, **kw):
        """ Serve data from the tw2.jit built-in controller """
        return makeUserGraph().request(request).body

Display the widget in your root template

Replace all of the contents of tw2jittg21demo/templates/index.mak with the following two lines:

<%inherit file="local:templates.master"/>
${widget.display() | n }

Give it a spin

Point your browser once again at http://localhost:8080 and you should be greeted by something like the following:

Screenshot of the tw2.jit.SQLARadialGraph in action

Getting fancy

tw2.jit.SQLARadialGraph can make use of various method defined on the sqlalchemy entities of which its been made aware. One is the __unicode__(self) method which is already present on our models from the TG2.1 quickstart process. Another is __jit_data__(self) which must return a json-ifiable python dict.

In the future, we expect to support more attribute, but at the time of this writing the only attribute that is actually used on the client side is the value of an hover_html key if it is present at all.

Add the following two methods to your tw2jittg21demo.model.User class which is defined in tw2jittg21demo/model/auth.py:

    @property
    def gravatar_url(self):
        """ Return a link to the gravatar image for this email addy """
        import hashlib
        hsh = hashlib.md5(self.email_address).hexdigest()
        base = "http://www.gravatar.com/avatar/{hsh}?d=monsterid"
        return base.format(hsh=hsh)

    def __jit_data__(self):
        """ 'hover_html' is the only supported key at present """
        return {
            'hover_html' : """
            <h2>{display_name}</h2>
            <img src="{gravatar_url}" />
            <ul>
                <li>{user_name}</li>
                <li>{created}</li>
            </ul>
            """.format(gravatar_url=self.gravatar_url, **self.__dict__)
        }

Restart your webapp and re-visit http://localhost:8080. You should see fancy pop-ups now when you mouseover any entity for which a __jit_data__ method returns a dict containing a hover_html key.

sqlalchemy entities can provide tw2.jit.SQLARadialGraph with content to be displayed on mouseover

Style

Lastly, you may want to re-style the jit graph to fit your page. Modify tw2jittg21demo/widgets.py by adding the following import statement at the top:

from tw2.core import JSSymbol

and by adding the following extra specifications to the UserGraph class:

        # Try to match colors to the TG banner
        backgroundcolor = '#FFFFFF'
        background = { 'CanvasStyles': { 'strokeStyle' : '#FFFFFF' } }
        Node = { 'color' : '#ffcb2f' }
        Edge = { 'color' : '#307e8a', 'lineWidth':1.5, }

        # Override the label style
        onPlaceLabel = JSSymbol(src="""
            (function(domElement, node){
                domElement.style.display = "none";
                domElement.innerHTML = node.name;
                domElement.style.display = "";
                var left = parseInt(domElement.style.left);
                domElement.style.width = '120px';
                domElement.style.height = '';
                var w = domElement.offsetWidth;
                domElement.style.left = (left - w /2) + 'px';

                domElement.style.cursor = 'pointer';
                if ( node._depth <= 1 )
                    domElement.style.color = 'black';
                else
                    domElement.style.color = 'grey';
            })""")

Once again restart your webapp and reload the page to get the following:

tw2.jit.SQLARadialGraph styled to fit more nicely with the TG2.1 default appearance

Outties

I hope this post was helpful and got you interested enough to check out the code and improve on it. Comments, questions, and patches are all appreciated.

]]>