[three]Bean

In #toscawidgets this morning, zykes- was asking about how to do the jquery removal hack I wrote about a month back but in Pyramid instead of TurboGears2. First I'll summarize the problem, then show you our solution.

toscawidgets2 tw2.* provides lots of handy web ui widgets including a wrapper around jquery-ui, jqgrid, and lots of fancy data widgets. One of the problems it solves for you is de-duplicating resources. Say you're including two fancy jquery-related widgets. Both of them require that jquery.js be included on the page. tw2 includes a piece of middleware that tracks all the resources required by the widgets being rendered on a given request, reduces that list to a set with no duplicates, orders it by dependency, and injects it into the tag.

Usually this is fine, but in some cases you want an exception to the rule. Say, like in my previous post, you want to include jquery.js yourself manually and you'd rather not have tw2 stomping all over your javascript namespace. You could disable tw2 injection of all resources, but you want all the others included -- just not jquery.

There is no automatic detection and filtration flag implemented in tw2 and it would be tough to do in the general case. tw2 can't guarantee that it's own jquery.js and your jquery.js are the same, or different, or included, or not included, or anything -- it doesn't even try.

To get tw2 to not do what you don't want, you need to un-register tw2's jquery resource from the middleware yourself (on each request). Previously we came up with a working hack that does this in the context of a TurboGears2 app. Here's the same concept applied to a Pyramid app using a Pyramid "tween".

import tw2.core.core
import tw2.jquery

def remove_jq_factory(handler, registry):
    """ Remove tw2 jquery_js from tw2's middleware registry.

    In order to use this, you need to add the following to your
    myapp/__init__.py file:

        config.add_tween('myapp.tween.remove_jq_factory')

    """

    def remove_jq_tween(request):
        # Send the request on through to other tweens and to our app
        response = handler(request)

        # Before the response is modified by the tw2 middleware, let's remove
        # jquery_js from its registry.
        offending_links = [r.req().link for r in [
            tw2.jquery.jquery_js,
        ]]
        local = tw2.core.core.request_local()
        local['resources'] = [
            r for r in local.get('resources', list())
            if r.link not in offending_links
        ]

        return response

    return remove_jq_tween

(First published to the toscawidgets-discuss google group)

It came up in IRC the other day that tw2 has been through many, many beta iterations.  So many in fact, that a full on tw2 2.0 release is painfully overdue.

That said, there are still plenty of out-standing issues in our issue tracker:
    http://bitbucket.org/paj/tw2core

Before we tag and name the 2.0 final product, I'm proposing -- no, announcing -- a 24 hour tw2 sprint the weekend before PyCon.  We'll write docs, we'll automate tests, we'll settle the dispatch dispute and wax philosophic.

I invite you to join me Saturday, March 3rd at 18:00:00 UTC through Sunday, March 4th 18:00:00 UTC.  Hop into #toscawidgets on freenode or join me in a hangout on google plus.  We'll nail every last bug on deck and still have time for a beer.

Yours in widgetry-

 - threebean

Tonight, VooDooNOFX was asking in IRC in #turbogears how to disable the injection of jquery.js by tw2.jquery into her/his TG2 app. Using the inject_resources=False middleware config value wouldn't cut it, since she/he wanted tw2 to inject all other resources, they were loading jQuery via google CDN beforehand and tw2's injection was clobbering their code.

I came up with the following hack to myapp/lib/base.py which will remove tw2.jquery.jquery_js from the list of resources tw2 would inject into each page served by a TG2.1 app.

At the top of myapp/lib/base.py import:

import tw2.core.core
import tw2.jquery

and then replace:

        return TGController.__call__(self, environ, start_response)

with the following:

        stream = TGController.__call__(self, environ, start_response)

        # Disable the injection of tw2.jquery
        offending_link = tw2.jquery.jquery_js.req().link
        local = tw2.core.core.request_local()
        local['resources'] = [
            r for r in local.get('resources', list()) if r.link != offending_link
        ]

        return stream

The two tricks to this are

• Simply knowing that tw2 resources register themselves with the 'request_local' object and that during the return-phase of the WSGI pipeline, the tw2 middleware refers to that list when injecting resources
• Figuring out where in a TG2 app's request flow to place the call to alter that object after all widgets that might register jquery have declared their resources but before the resources list is injected into the output stream.

We came out of it with a bug filed in the tw2 issue tracker so we can take care of it properly in the future.

I pumped out two new tw2.devtools WidgetBrowser features this afternoon -- you can see them live at http://tw2-demos.threebean.org.

• A view source link that pops open a jquery-ui dialog with the nicely formatted sourcecode of the demo widget rendered by the widget browser (hg commit).
• Metadata pulled from pypi including the latest release available and the total number of downloads across all releases (hg commit).

The 'view source' link stuff is pretty cool. I got the idea and the details both from the moksha demo dashboard using inspect and pygments. Here's the relevant piece of code:

import warnings
import inspect
import pygments
import pygments.lexers
import pygments.formatters

def prepare_source(s):
    try:
        source = inspect.getsource(s)
    except IOError as e:
        warnings.warn(repr(s) + " : " + str(e))
        return ""

    html_args = {'full': False}
    code = pygments.highlight(
        source,
        pygments.lexers.PythonLexer(),
        pygments.formatters.HtmlFormatter(**html_args)
    )

    return code

The metadata part was trickier but to get the same information yourself, run:

import xmlrpclib

module = 'tw2.core'

pypi = xmlrpclib.ServerProxy('http://pypi.python.org/pypi')

total = sum([
    sum([
        d['downloads'] for d in pypi.release_urls(module, version)
    ])
    for version in pypi.package_releases(module, True)
])

print module, total

Maunsell Fort tower (Hywel Williams) / CC BY-SA 2.0

Get the source: if you don't want to read through this, you can get the entire source for this tutorial here on my github account.

Today it's an epic data widgets tutorial! I've been working on a lot of stuff recently, including trying to pull some of the more quiet tw2 developers back together; I hope that this fits into that scheme and to at some point link directly to this tutorial from the tw2 documentation.

This tutorial won't introduce you to any of tw2.core fundamentals directly, but it will show off some of the flashier and data-driven widgets. If it gets you real hot, check out some of my other tw2+TurboGears2.1 tutorials here and here and my tw2+Pyramid tutorial here, but enough of that, let's cut to the chase:

Agenda

We're going to build an TurboGears2.1 application from start to finish that logs the IP of every request made of it and displays those hits with a couple of fancy-ass tw2 widgets. Here's what it's going to take:

• Getting TurboGears2.1 installed and running
• Setting up a data backend
• "Automatically" listing db entries with tw2.jqplugins.jqgrid:SQLAjqGridWidget
• Plotting server history with tw2.jqplugins.jqplot:JQPlotWidget
• Making the layout look like http://google.com/ig with tw2.jqplugins.portlets

1. Getting TurboGears2.1 installed and running

Install (if you haven't already) virtualenvwrapper. It's awesome and you should use it always.

Next, Open up your favorite terminal and do the following:

% mkdir tw2-facemelt-tg2.1 && cd tw2-facemelt-tg2.1
% mkvirtualenv --no-site-packages --distribute tw2-facemelt-tg2.1
% pip install --use-mirrors tg.devtools
% paster quickstart
        Enter project name: tw2-facemelt-tg2.1
        Enter package name [tw2facemelttg21]:
        Would you prefer mako templates? (yes/[no]): yes
        Do you need authentication and authorization in this project? ([yes]/no): yes
% cd tw2-facemelt-tg2.1

At this point we need to modify the freshly quickstarted TurboGears 2.1 project. There's a bug there! We said want to use mako templates, so TG2.1 isn't configured to install genshi but it does include references to the tgext.admin.controller:AdminController which references genshi but doesn't list its dependency and therefore doesn't install it. That's fine, we'll just remove the references to the AdminController since we won't be using it, anyways.

Remove the following three lines from tw2facemelttg21/controllers/root.py:

from tgext.admin.tgadminconfig import TGAdminConfig
from tgext.admin.controller import AdminController

...

    admin = AdminController(model, DBSession, config_type=TGAdminConfig)

While we're at it, let's turn on the tw2 WSGI middleware.

Edit tw2facemelttg21/config/app_cfg.py and add the following two lines at the bottom:

base_config.use_toscawidgets = False
base_config.use_toscawidgets2 = True

It should be noted that toscawidgets1 (setup by TG2.1 by default) and toscawidgets2 can peacefully coexist in a WSGI app but I'm a tw2 purist. After all, tw2 is faster.

At this point you should be able to run your TG2.1 quickstarted app. Run the following:

% pip install --use-mirrors -e .
% paster setup-app development.ini
% paster serve development.ini

And visit http://localhost:8080 to verify that all this worked.

2. Setting up a data backend

Create a file at tw2facemelttg21/models/bloglog.py with the following contents:

# -*- coding: utf-8 -*-
""" Logs of Bob Loblaw's Law Blog """

from sqlalchemy import *
from sqlalchemy.orm import mapper, relation
from sqlalchemy import Table, ForeignKey, Column
from sqlalchemy.types import Integer, Unicode
#from sqlalchemy.orm import relation, backref

from datetime import datetime

from tw2facemelttg21.model import DeclarativeBase, metadata, DBSession


class ServerHit(DeclarativeBase):
    __tablename__ = 'server_hit'

    id = Column(Integer, primary_key=True)
    timestamp = Column(DateTime, nullable=False, default=datetime.now)

    remote_addr = Column(Unicode(15), nullable=False)

    path_info = Column(Unicode(1024), nullable=False)
    query_string = Column(Unicode(1024), nullable=False)

Edit the module-level file tw2facemelttg21/models/__init__.py and uncomment the following line:

DeclarativeBase.query = DBSession.query_property()

Add the following line to the very bottom of the same file (tw2facemelttg21/models/__init__.py):

from bloglog import ServerHit

The data model should be good to go now, but let's add one little piece of code --- a hook --- that will populate the server_hit table as the app runs.

Edit tw2facemelttg21/lib/base.py and add the following seven-line chunk just inside the __call__(...) method of your BaseController

    def __call__(self, environ, start_response):
        """Invoke the Controller"""
        # TGController.__call__ dispatches to the Controller method
        # the request is routed to. This routing information is
        # available in environ['pylons.routes_dict']

        entry = model.ServerHit(
            remote_addr=environ['REMOTE_ADDR'],
            path_info=environ['PATH_INFO'],
            query_string=environ['QUERY_STRING'],
        )
        model.DBSession.add(entry)
        model.DBSession.flush()

        request.identity = request.environ.get('repoze.who.identity')
        tmpl_context.identity = request.identity
        return TGController.__call__(self, environ, start_response)

Now blow away and recreate your old sqlite database by typing the following into your trusty terminal:

% rm devdata.db
% paster setup-app development.ini
% paster serve development.ini

Pray for no bugs as you revisit http://localhost:8080. Reload the page a few times and just double-check that your database has entries in it now by running sqlite3 devdata.db. Issue the command select * from server_hit; and you should see all your page requests listed.

Cool? Cool.

(Note to the brave: If we wanted to be really awesome, we would write WSGI middleware to do our request-logging.)

3. "Automatically" listing db entries with tw2.jqplugins.jqgrid:SQLAjqGridWidget

Create a new file tw2facemelttg21/widgets.py with the following content:

import tw2facemelttg21.model as model
from tw2.jqplugins.jqgrid import SQLAjqGridWidget

class LogGrid(SQLAjqGridWidget):
    id = 'awesome-loggrid'
    entity = model.ServerHit
    excluded_columns = ['id']
    datetime_format = "%x %X"

    prmFilter = {'stringResult': True, 'searchOnEnter': False}

    options = {
        'pager': 'awesome-loggrid_pager',
        'url': '/jqgrid/',
        'rowNum':15,
        'rowList':[15,150, 1500],
        'viewrecords':True,
        'imgpath': 'scripts/jqGrid/themes/green/images',
        'shrinkToFit': True,
        'height': 'auto',
    }

Pull the LogGrid widget into your controller by importing it at the top of tw2facemelttg21/controllers/root.py with:

from tw2facemelttg21.widgets import LogGrid

Modify the index method of your RootController in tw2facemelttg21/controllers/root.py so that it looks like:

@expose('tw2facemelttg21.templates.index')
def index(self):
    """Handle the front-page."""
    return dict(page='index', gridwidget=LogGrid)

This will make the LogGrid available in your index template under the name gridwidget. We still need to display it there.

Edit tw2facemelttg21/templates/index.mak and wipe out all of the content. Replace it with only the following:

<%inherit file="local:templates.master"/>
${gridwidget.display() | n}

Cool.

The SQLAjqGridWidget has a request(...) method that does all of its magic (interrogating your sqlalchemy model for its properties and values). We still need to wire up our TurboGears app to forward the ajax<->json requests to the right place.

To do this, add another method to your RootController back in tw2facemelttg21/controllers/root.py that looks like this:

@expose('json')
def jqgrid(self, *args, **kwargs):
    return LogGrid.request(request).body

Lastly, edit your setup.py file and add all the new widget dependencies we'll need (not just the jqgrid), like so:

    install_requires=[
        "TurboGears2 >= 2.1",
        "Mako",
        "zope.sqlalchemy >= 0.4",
        "repoze.tm2 >= 1.0a5",
        "repoze.what-quickstart",
        "repoze.what >= 1.0.8",
        "repoze.what-quickstart",
        "repoze.who-friendlyform >= 1.0.4",
        "repoze.what-pylons >= 1.0",
        "repoze.what.plugins.sql",
        "repoze.who == 1.0.18",
        "tgext.admin >= 0.3.9",
        "tw.forms",
        "tw2.jqplugins.jqgrid",
        "tw2.jqplugins.jqplot",
        "tw2.jqplugins.portlets",
        ],
    setup_requires=["PasteScript >= 1.7"],

Install the newly listed dependencies by running:

% python setup.py develop
% paster serve development.ini

...and revisit http://localhost:8080.

At this point, your app should look something like the following:

We should really (probably) do things as fancy as we can, and make use of jquery-ui's themes.

Edit tw2facemelttg21/lib/base.py do the following two things:

Add this import statement to the top of the file.

from tw2.jqplugins.ui import set_ui_theme_name

And invoke it from inside the __call__(...) method of your BaseController like so:

    def __call__(self, environ, start_response):
        """Invoke the Controller"""
        # TGController.__call__ dispatches to the Controller method
        # the request is routed to. This routing information is
        # available in environ['pylons.routes_dict']

        set_ui_theme_name('hot-sneaks')

        entry = model.ServerHit(
            remote_addr=environ['REMOTE_ADDR'],
            path_info=environ['PATH_INFO'],
            query_string=environ['QUERY_STRING'],
        )
        model.DBSession.add(entry)

        request.identity = request.environ.get('repoze.who.identity')
        tmpl_context.identity = request.identity
        return TGController.__call__(self, environ, start_response)

Nice! But hot-sneaks isn't the only available theme -- you can see a list of all of them right here.

4. Plotting server history with tw2.jqplugins.jqplot:JQPlotWidget

This is going to be awesome.

Add a new widget definition to tw2facemelttg21/widgets.py:

from tw2.jqplugins.jqplot import JQPlotWidget
from tw2.jqplugins.jqplot.base import categoryAxisRenderer_js, barRenderer_js
from tw2.core import JSSymbol

class LogPlot(JQPlotWidget):
    id = 'awesome-logplot'
    interval = 2000
    resources = JQPlotWidget.resources + [
        categoryAxisRenderer_js,
        barRenderer_js,
    ]

    options = {
        'seriesDefaults' : {
            'renderer': JSSymbol('$.jqplot.BarRenderer'),
            'rendererOptions': { 'barPadding': 4, 'barMargin': 10 }
        },
        'axes' : {
            'xaxis': {
                'renderer': JSSymbol(src="$.jqplot.CategoryAxisRenderer"),
            },
            'yaxis': {'min': 0, },
        }
    }

Now we're going to go to town on our RootController. We need to:

• Make the new widget available in our template
• Produce data for it
• Render it in the template

First, add the following imports to the top of tw2facemelttg21/controllers/root.py:

from tw2facemelttg21.models import ServerHit
from tw2facemelttg21.widgets import LogPlot

import sqlalchemy
import datetime
import time

We're also going to need this little recursive_update utility to merge the options dicts. Just add it at the top of tw2facemelttg21/controllers/root.py as a function. It should not be a method of RootController.

def recursive_update(d1, d2):
    """ Little helper function that does what d1.update(d2) does,
    but works nice and recursively with dicts of dicts of dicts.

    It's not necessarily very efficient.
    """

    for k in d1.keys():
        if k not in d2:
            continue

        if isinstance(d1[k], dict) and isinstance(d2[k], dict):
            d1[k] = recursive_update(d1[k], d2[k])
        else:
            d1[k] = d2[k]

    for k in d2.keys():
        if k not in d1:
            d1[k] = d2[k]

    return d1

Add a new method to the RootController class that looks like the following. This will do all of the heavy lifty --- producing the data for the jqplot.

    def jqplot(self, days=1/(24.0)):
        n_buckets = 15
        now = datetime.datetime.now()
        then = now - datetime.timedelta(days)
        delta = datetime.timedelta(days) / n_buckets

        entries = ServerHit.query.filter(ServerHit.timestamp>then).all()

        t_bounds = [(then+delta*i, then+delta*(i+1)) for i in range(n_buckets)]

        # Accumulate into buckets!  This is how I like to do it.
        buckets = dict([(lower, 0) for lower, upper in t_bounds])
        for entry in entries:
            for lower, upper in t_bounds:
                if entry.timestamp >= lower and entry.timestamp < upper:
                    buckets[lower] += 1

        # Only one series for now.. but you could do other stuff!
        series = [buckets[lower] for lower, upper in t_bounds]
        data = [
            series,
            # You could add another series here...
        ]

        options = { 'axes' : { 'xaxis': {
            'ticks': [u.strftime("%I:%M:%S") for l, u in t_bounds],
        }}}

        return dict(data=data, options=options)

Rewrite the index(...) method of your RootController to look like the following:

    @expose('tw2facemelttg21.templates.index')
    def index(self):
        """Handle the front-page."""
        jqplot_params = self.jqplot()
        plotwidget = LogPlot(data=jqplot_params['data'])
        plotwidget.options = recursive_update(
            plotwidget.options, jqplot_params['options'])
        return dict(page='index', gridwidget=LogGrid, plotwidget=plotwidget)

Now the jqplot widget should pull its data from the (perhaps poorly named) jqplot method of your RootController and should merge new options nicely with the predefined ones. It should also be available in your index template under the name plotwidget. Let's use it there!

Edit tw2facemelttg21/templates/index.mak and add the following line:

<%inherit file="local:templates.master"/>
${plotwidget.display() | n}
${gridwidget.display() | n}

Revisit http://localhost:8080/ and you should get something like this.

5. Making the layout look like http://google.com/ig with tw2.jqplugins.portlets

Add the following import to the top of tw2facemelttg21/controllers/root.py:

import tw2.jqplugins.portlets as p

Rewrite the index(...) method of your RootController to look like this:

    @expose('tw2facemelttg21.templates.index')
    def index(self):
        """Handle the front-page."""
        jqplot_params = self.jqplot()
        plotwidget = LogPlot(data=jqplot_params['data'])
        plotwidget.options = recursive_update(
            plotwidget.options, jqplot_params['options'])

        colwidth = '50%'
        class LayoutWidget(p.ColumnLayout):
            id = 'awesome-layout'
            class col1(p.Column):
                width = colwidth
                class por1(p.Portlet):
                    title = 'DB Entries of Server Hits'
                    widget = LogGrid

            class col2(p.Column):
                width = colwidth
                class por2(p.Portlet):
                    title = 'Hits over the last hour'
                    widget = plotwidget

        return dict(page='index', layoutwidget=LayoutWidget)

Now that only the layoutwidget is available to your index template, you'll need to rewrite your tw2facemelttg21/templates/index.mak.

<%inherit file="local:templates.master"/>
${layoutwidget.display() | n}

To simplify all the styling, you'll also need to clear out all the clutter from the TG2.1 quickstart install. Rewrite (for the first time) tw2facemelttg21/templates/master.mak to look like this:

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
                 "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html>
<head></head>
<body>${self.body()}</body>
</html>

Rerun your paster server and you should see something like this.

The portlet windows are movable and collapsible and they automatically retain their state between page loads (with jquery.cookie.js).

---

I hope you enjoyed the tutorial. Once again, you can get the entire source here on my github account. If you have any comments, feedback, or questions --- post 'em!