[three]Bean

Tutorial -- melting your face off with tw2 and TurboGears2.1

Apr 30, 2011 | categories: python, toscawidgets, turbogears View Comments

Maunsell Fort tower (Hywel Williams) / CC BY-SA 2.0

Get the source: if you don't want to read through this, you can get the entire source for this tutorial here on my github account.

---

Today it's an epic data widgets tutorial! I've been working on a lot of stuff recently, including trying to pull some of the more quiet tw2 developers back together; I hope that this fits into that scheme and to at some point link directly to this tutorial from the tw2 documentation.

This tutorial won't introduce you to any of tw2.core fundamentals directly, but it will show off some of the flashier and data-driven widgets. If it gets you real hot, check out some of my other tw2+TurboGears2.1 tutorials here and here and my tw2+Pyramid tutorial here, but enough of that, let's cut to the chase:

Agenda

We're going to build an TurboGears2.1 application from start to finish that logs the IP of every request made of it and displays those hits with a couple of fancy-ass tw2 widgets. Here's what it's going to take:

  • Getting TurboGears2.1 installed and running
  • Setting up a data backend
  • "Automatically" listing db entries with tw2.jqplugins.jqgrid:SQLAjqGridWidget
  • Plotting server history with tw2.jqplugins.jqplot:JQPlotWidget
  • Making the layout look like http://google.com/ig with tw2.jqplugins.portlets

1. Getting TurboGears2.1 installed and running

Install (if you haven't already) virtualenvwrapper. It's awesome and you should use it always.

Next, Open up your favorite terminal and do the following:

% mkdir tw2-facemelt-tg2.1 && cd tw2-facemelt-tg2.1
% mkvirtualenv --no-site-packages --distribute tw2-facemelt-tg2.1
% pip install --use-mirrors tg.devtools
% paster quickstart
        Enter project name: tw2-facemelt-tg2.1
        Enter package name [tw2facemelttg21]:
        Would you prefer mako templates? (yes/[no]): yes
        Do you need authentication and authorization in this project? ([yes]/no): yes
% cd tw2-facemelt-tg2.1

At this point we need to modify the freshly quickstarted TurboGears 2.1 project. There's a bug there! We said want to use mako templates, so TG2.1 isn't configured to install genshi but it does include references to the tgext.admin.controller:AdminController which references genshi but doesn't list its dependency and therefore doesn't install it. That's fine, we'll just remove the references to the AdminController since we won't be using it, anyways.

Remove the following three lines from tw2facemelttg21/controllers/root.py:

from tgext.admin.tgadminconfig import TGAdminConfig
from tgext.admin.controller import AdminController

...

    admin = AdminController(model, DBSession, config_type=TGAdminConfig)

While we're at it, let's turn on the tw2 WSGI middleware.

Edit tw2facemelttg21/config/app_cfg.py and add the following two lines at the bottom:

base_config.use_toscawidgets = False
base_config.use_toscawidgets2 = True

It should be noted that toscawidgets1 (setup by TG2.1 by default) and toscawidgets2 can peacefully coexist in a WSGI app but I'm a tw2 purist. After all, tw2 is faster.

At this point you should be able to run your TG2.1 quickstarted app. Run the following:

% pip install --use-mirrors -e .
% paster setup-app development.ini
% paster serve development.ini

And visit http://localhost:8080 to verify that all this worked.

2. Setting up a data backend

Create a file at tw2facemelttg21/models/bloglog.py with the following contents:

# -*- coding: utf-8 -*-
""" Logs of Bob Loblaw's Law Blog """

from sqlalchemy import *
from sqlalchemy.orm import mapper, relation
from sqlalchemy import Table, ForeignKey, Column
from sqlalchemy.types import Integer, Unicode
#from sqlalchemy.orm import relation, backref

from datetime import datetime

from tw2facemelttg21.model import DeclarativeBase, metadata, DBSession


class ServerHit(DeclarativeBase):
    __tablename__ = 'server_hit'

    id = Column(Integer, primary_key=True)
    timestamp = Column(DateTime, nullable=False, default=datetime.now)

    remote_addr = Column(Unicode(15), nullable=False)

    path_info = Column(Unicode(1024), nullable=False)
    query_string = Column(Unicode(1024), nullable=False)

Edit the module-level file tw2facemelttg21/models/__init__.py and uncomment the following line:

DeclarativeBase.query = DBSession.query_property()

Add the following line to the very bottom of the same file (tw2facemelttg21/models/__init__.py):

from bloglog import ServerHit

The data model should be good to go now, but let's add one little piece of code --- a hook --- that will populate the server_hit table as the app runs.

Edit tw2facemelttg21/lib/base.py and add the following seven-line chunk just inside the __call__(...) method of your BaseController 

    def __call__(self, environ, start_response):
        """Invoke the Controller"""
        # TGController.__call__ dispatches to the Controller method
        # the request is routed to. This routing information is
        # available in environ['pylons.routes_dict']

        entry = model.ServerHit(
            remote_addr=environ['REMOTE_ADDR'],
            path_info=environ['PATH_INFO'],
            query_string=environ['QUERY_STRING'],
        )
        model.DBSession.add(entry)
        model.DBSession.flush()

        request.identity = request.environ.get('repoze.who.identity')
        tmpl_context.identity = request.identity
        return TGController.__call__(self, environ, start_response)

Now blow away and recreate your old sqlite database by typing the following into your trusty terminal:

% rm devdata.db
% paster setup-app development.ini
% paster serve development.ini

Pray for no bugs as you revisit http://localhost:8080. Reload the page a few times and just double-check that your database has entries in it now by running sqlite3 devdata.db. Issue the command select * from server_hit; and you should see all your page requests listed.

Cool? Cool.

(Note to the brave: If we wanted to be really awesome, we would write WSGI middleware to do our request-logging.)

3. "Automatically" listing db entries with tw2.jqplugins.jqgrid:SQLAjqGridWidget

Create a new file tw2facemelttg21/widgets.py with the following content:

import tw2facemelttg21.model as model
from tw2.jqplugins.jqgrid import SQLAjqGridWidget

class LogGrid(SQLAjqGridWidget):
    id = 'awesome-loggrid'
    entity = model.ServerHit
    excluded_columns = ['id']
    datetime_format = "%x %X"

    prmFilter = {'stringResult': True, 'searchOnEnter': False}

    options = {
        'pager': 'awesome-loggrid_pager',
        'url': '/jqgrid/',
        'rowNum':15,
        'rowList':[15,150, 1500],
        'viewrecords':True,
        'imgpath': 'scripts/jqGrid/themes/green/images',
        'shrinkToFit': True,
        'height': 'auto',
    }

Pull the LogGrid widget into your controller by importing it at the top of tw2facemelttg21/controllers/root.py with:

from tw2facemelttg21.widgets import LogGrid

Modify the index method of your RootController in tw2facemelttg21/controllers/root.py so that it looks like:

@expose('tw2facemelttg21.templates.index')
def index(self):
    """Handle the front-page."""
    return dict(page='index', gridwidget=LogGrid)

This will make the LogGrid available in your index template under the name gridwidget. We still need to display it there.

Edit tw2facemelttg21/templates/index.mak and wipe out all of the content. Replace it with only the following:

<%inherit file="local:templates.master"/>
${gridwidget.display() | n}

Cool.

The SQLAjqGridWidget has a request(...) method that does all of its magic (interrogating your sqlalchemy model for its properties and values). We still need to wire up our TurboGears app to forward the ajax<->json requests to the right place.

To do this, add another method to your RootController back in tw2facemelttg21/controllers/root.py that looks like this:

@expose('json')
def jqgrid(self, *args, **kwargs):
    return LogGrid.request(request).body

Lastly, edit your setup.py file and add all the new widget dependencies we'll need (not just the jqgrid), like so:

    install_requires=[
        "TurboGears2 >= 2.1",
        "Mako",
        "zope.sqlalchemy >= 0.4",
        "repoze.tm2 >= 1.0a5",
        "repoze.what-quickstart",
        "repoze.what >= 1.0.8",
        "repoze.what-quickstart",
        "repoze.who-friendlyform >= 1.0.4",
        "repoze.what-pylons >= 1.0",
        "repoze.what.plugins.sql",
        "repoze.who == 1.0.18",
        "tgext.admin >= 0.3.9",
        "tw.forms",
        "tw2.jqplugins.jqgrid",
        "tw2.jqplugins.jqplot",
        "tw2.jqplugins.portlets",
        ],
    setup_requires=["PasteScript >= 1.7"],

Install the newly listed dependencies by running:

% python setup.py develop
% paster serve development.ini

...and revisit http://localhost:8080.

At this point, your app should look something like the following:

We should really (probably) do things as fancy as we can, and make use of jquery-ui's themes.

Edit tw2facemelttg21/lib/base.py do the following two things:

Add this import statement to the top of the file.

from tw2.jqplugins.ui import set_ui_theme_name

And invoke it from inside the __call__(...) method of your BaseController like so:

    def __call__(self, environ, start_response):
        """Invoke the Controller"""
        # TGController.__call__ dispatches to the Controller method
        # the request is routed to. This routing information is
        # available in environ['pylons.routes_dict']

        set_ui_theme_name('hot-sneaks')

        entry = model.ServerHit(
            remote_addr=environ['REMOTE_ADDR'],
            path_info=environ['PATH_INFO'],
            query_string=environ['QUERY_STRING'],
        )
        model.DBSession.add(entry)

        request.identity = request.environ.get('repoze.who.identity')
        tmpl_context.identity = request.identity
        return TGController.__call__(self, environ, start_response)

Nice! But hot-sneaks isn't the only available theme -- you can see a list of all of them right here.

4. Plotting server history with tw2.jqplugins.jqplot:JQPlotWidget

This is going to be awesome.

Add a new widget definition to tw2facemelttg21/widgets.py:

from tw2.jqplugins.jqplot import JQPlotWidget
from tw2.jqplugins.jqplot.base import categoryAxisRenderer_js, barRenderer_js
from tw2.core import JSSymbol

class LogPlot(JQPlotWidget):
    id = 'awesome-logplot'
    interval = 2000
    resources = JQPlotWidget.resources + [
        categoryAxisRenderer_js,
        barRenderer_js,
    ]

    options = {
        'seriesDefaults' : {
            'renderer': JSSymbol('$.jqplot.BarRenderer'),
            'rendererOptions': { 'barPadding': 4, 'barMargin': 10 }
        },
        'axes' : {
            'xaxis': {
                'renderer': JSSymbol(src="$.jqplot.CategoryAxisRenderer"),
            },
            'yaxis': {'min': 0, },
        }
    }

Now we're going to go to town on our RootController. We need to:

  • Make the new widget available in our template
  • Produce data for it
  • Render it in the template

First, add the following imports to the top of tw2facemelttg21/controllers/root.py:

from tw2facemelttg21.models import ServerHit
from tw2facemelttg21.widgets import LogPlot

import sqlalchemy
import datetime
import time

We're also going to need this little recursive_update utility to merge the options dicts. Just add it at the top of tw2facemelttg21/controllers/root.py as a function. It should not be a method of RootController.

def recursive_update(d1, d2):
    """ Little helper function that does what d1.update(d2) does,
    but works nice and recursively with dicts of dicts of dicts.

    It's not necessarily very efficient.
    """

    for k in d1.keys():
        if k not in d2:
            continue

        if isinstance(d1[k], dict) and isinstance(d2[k], dict):
            d1[k] = recursive_update(d1[k], d2[k])
        else:
            d1[k] = d2[k]

    for k in d2.keys():
        if k not in d1:
            d1[k] = d2[k]

    return d1

Add a new method to the RootController class that looks like the following. This will do all of the heavy lifty --- producing the data for the jqplot.

    def jqplot(self, days=1/(24.0)):
        n_buckets = 15
        now = datetime.datetime.now()
        then = now - datetime.timedelta(days)
        delta = datetime.timedelta(days) / n_buckets

        entries = ServerHit.query.filter(ServerHit.timestamp>then).all()

        t_bounds = [(then+delta*i, then+delta*(i+1)) for i in range(n_buckets)]

        # Accumulate into buckets!  This is how I like to do it.
        buckets = dict([(lower, 0) for lower, upper in t_bounds])
        for entry in entries:
            for lower, upper in t_bounds:
                if entry.timestamp >= lower and entry.timestamp < upper:
                    buckets[lower] += 1

        # Only one series for now.. but you could do other stuff!
        series = [buckets[lower] for lower, upper in t_bounds]
        data = [
            series,
            # You could add another series here...
        ]

        options = { 'axes' : { 'xaxis': {
            'ticks': [u.strftime("%I:%M:%S") for l, u in t_bounds],
        }}}

        return dict(data=data, options=options)

Rewrite the index(...) method of your RootController to look like the following:

    @expose('tw2facemelttg21.templates.index')
    def index(self):
        """Handle the front-page."""
        jqplot_params = self.jqplot()
        plotwidget = LogPlot(data=jqplot_params['data'])
        plotwidget.options = recursive_update(
            plotwidget.options, jqplot_params['options'])
        return dict(page='index', gridwidget=LogGrid, plotwidget=plotwidget)

Now the jqplot widget should pull its data from the (perhaps poorly named) jqplot method of your RootController and should merge new options nicely with the predefined ones. It should also be available in your index template under the name plotwidget. Let's use it there!

Edit tw2facemelttg21/templates/index.mak and add the following line:

<%inherit file="local:templates.master"/>
${plotwidget.display() | n}
${gridwidget.display() | n}

Revisit http://localhost:8080/ and you should get something like this.

5. Making the layout look like http://google.com/ig with tw2.jqplugins.portlets

Add the following import to the top of tw2facemelttg21/controllers/root.py:

import tw2.jqplugins.portlets as p

Rewrite the index(...) method of your RootController to look like this:

    @expose('tw2facemelttg21.templates.index')
    def index(self):
        """Handle the front-page."""
        jqplot_params = self.jqplot()
        plotwidget = LogPlot(data=jqplot_params['data'])
        plotwidget.options = recursive_update(
            plotwidget.options, jqplot_params['options'])

        colwidth = '50%'
        class LayoutWidget(p.ColumnLayout):
            id = 'awesome-layout'
            class col1(p.Column):
                width = colwidth
                class por1(p.Portlet):
                    title = 'DB Entries of Server Hits'
                    widget = LogGrid

            class col2(p.Column):
                width = colwidth
                class por2(p.Portlet):
                    title = 'Hits over the last hour'
                    widget = plotwidget

        return dict(page='index', layoutwidget=LayoutWidget)

Now that only the layoutwidget is available to your index template, you'll need to rewrite your tw2facemelttg21/templates/index.mak.

<%inherit file="local:templates.master"/>
${layoutwidget.display() | n}

To simplify all the styling, you'll also need to clear out all the clutter from the TG2.1 quickstart install. Rewrite (for the first time) tw2facemelttg21/templates/master.mak to look like this:

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
                 "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html>
<head></head>
<body>${self.body()}</body>
</html>

Rerun your paster server and you should see something like this.

The portlet windows are movable and collapsible and they automatically retain their state between page loads (with jquery.cookie.js).

---

I hope you enjoyed the tutorial. Once again, you can get the entire source here on my github account. If you have any comments, feedback, or questions --- post 'em!

View Comments
blog comments powered by Disqus

[three]Bean.org