[three]Bean

fabulous is definitely the python module of the week

May 04, 2011 | categories: python, lulz View Comments

This is more important than a tweet.

Check out fabulous and then include it in every project you've ever worked on.

Here's what I did with it.

New tw2.devtools WidgetBrowser features

May 02, 2011 | categories: python, toscawidgets View Comments

I pumped out two new tw2.devtools WidgetBrowser features this afternoon -- you can see them live at http://tw2-demos.threebean.org.

A view source link that pops open a jquery-ui dialog with the nicely formatted sourcecode of the demo widget rendered by the widget browser (hg commit).
Metadata pulled from pypi including the latest release available and the total number of downloads across all releases (hg commit).

---

The 'view source' link stuff is pretty cool. I got the idea and the details both from the moksha demo dashboard using inspect and pygments. Here's the relevant piece of code:

import warnings
import inspect
import pygments
import pygments.lexers
import pygments.formatters

def prepare_source(s):
    try:
        source = inspect.getsource(s)
    except IOError as e:
        warnings.warn(repr(s) + " : " + str(e))
        return ""

    html_args = {'full': False}
    code = pygments.highlight(
        source,
        pygments.lexers.PythonLexer(),
        pygments.formatters.HtmlFormatter(**html_args)
    )

    return code

---

The metadata part was trickier but to get the same information yourself, run:

import xmlrpclib

module = 'tw2.core'

pypi = xmlrpclib.ServerProxy('http://pypi.python.org/pypi')

total = sum([
    sum([
        d['downloads'] for d in pypi.release_urls(module, version)
    ])
    for version in pypi.package_releases(module, True)
])

print module, total

View Comments

Tutorial -- melting your face off with tw2 and TurboGears2.1

Apr 30, 2011 | categories: python, toscawidgets, turbogears View Comments

Maunsell Fort tower (Hywel Williams) / CC BY-SA 2.0

Get the source: if you don't want to read through this, you can get the entire source for this tutorial here on my github account.

---

Today it's an epic data widgets tutorial! I've been working on a lot of stuff recently, including trying to pull some of the more quiet tw2 developers back together; I hope that this fits into that scheme and to at some point link directly to this tutorial from the tw2 documentation.

This tutorial won't introduce you to any of tw2.core fundamentals directly, but it will show off some of the flashier and data-driven widgets. If it gets you real hot, check out some of my other tw2+TurboGears2.1 tutorials here and here and my tw2+Pyramid tutorial here, but enough of that, let's cut to the chase:

Agenda

We're going to build an TurboGears2.1 application from start to finish that logs the IP of every request made of it and displays those hits with a couple of fancy-ass tw2 widgets. Here's what it's going to take:

Getting TurboGears2.1 installed and running
Setting up a data backend
"Automatically" listing db entries with tw2.jqplugins.jqgrid:SQLAjqGridWidget
Plotting server history with tw2.jqplugins.jqplot:JQPlotWidget
Making the layout look like http://google.com/ig with tw2.jqplugins.portlets

1. Getting TurboGears2.1 installed and running

Install (if you haven't already) virtualenvwrapper. It's awesome and you should use it always.

Next, Open up your favorite terminal and do the following:

% mkdir tw2-facemelt-tg2.1 &amp;&amp; cd tw2-facemelt-tg2.1
% mkvirtualenv --no-site-packages --distribute tw2-facemelt-tg2.1
% pip install --use-mirrors tg.devtools
% paster quickstart
        Enter project name: tw2-facemelt-tg2.1
        Enter package name [tw2facemelttg21]:
        Would you prefer mako templates? (yes/[no]): yes
        Do you need authentication and authorization in this project? ([yes]/no): yes
% cd tw2-facemelt-tg2.1

At this point we need to modify the freshly quickstarted TurboGears 2.1 project. There's a bug there! We said want to use mako templates, so TG2.1 isn't configured to install genshi but it does include references to the tgext.admin.controller:AdminController which references genshi but doesn't list its dependency and therefore doesn't install it. That's fine, we'll just remove the references to the AdminController since we won't be using it, anyways.

Remove the following three lines from tw2facemelttg21/controllers/root.py:

from tgext.admin.tgadminconfig import TGAdminConfig
from tgext.admin.controller import AdminController

...

    admin = AdminController(model, DBSession, config_type=TGAdminConfig)

While we're at it, let's turn on the tw2 WSGI middleware.

Edit tw2facemelttg21/config/app_cfg.py and add the following two lines at the bottom:

base_config.use_toscawidgets = False
base_config.use_toscawidgets2 = True

It should be noted that toscawidgets1 (setup by TG2.1 by default) and toscawidgets2 can peacefully coexist in a WSGI app but I'm a tw2 purist. After all, tw2 is faster.

At this point you should be able to run your TG2.1 quickstarted app. Run the following:

% pip install --use-mirrors -e .
% paster setup-app development.ini
% paster serve development.ini

And visit http://localhost:8080 to verify that all this worked.

2. Setting up a data backend

Create a file at tw2facemelttg21/models/bloglog.py with the following contents:

# -*- coding: utf-8 -*-
""" Logs of Bob Loblaw's Law Blog """

from sqlalchemy import *
from sqlalchemy.orm import mapper, relation
from sqlalchemy import Table, ForeignKey, Column
from sqlalchemy.types import Integer, Unicode
#from sqlalchemy.orm import relation, backref

from datetime import datetime

from tw2facemelttg21.model import DeclarativeBase, metadata, DBSession


class ServerHit(DeclarativeBase):
    __tablename__ = 'server_hit'

    id = Column(Integer, primary_key=True)
    timestamp = Column(DateTime, nullable=False, default=datetime.now)

    remote_addr = Column(Unicode(15), nullable=False)

    path_info = Column(Unicode(1024), nullable=False)
    query_string = Column(Unicode(1024), nullable=False)

Edit the module-level file tw2facemelttg21/models/__init__.py and uncomment the following line:

DeclarativeBase.query = DBSession.query_property()

Add the following line to the very bottom of the same file (tw2facemelttg21/models/__init__.py):

from bloglog import ServerHit

The data model should be good to go now, but let's add one little piece of code --- a hook --- that will populate the server_hit table as the app runs.

Edit tw2facemelttg21/lib/base.py and add the following seven-line chunk just inside the __call__(...) method of your BaseController

    def __call__(self, environ, start_response):
        """Invoke the Controller"""
        # TGController.__call__ dispatches to the Controller method
        # the request is routed to. This routing information is
        # available in environ['pylons.routes_dict']

        entry = model.ServerHit(
            remote_addr=environ['REMOTE_ADDR'],
            path_info=environ['PATH_INFO'],
            query_string=environ['QUERY_STRING'],
        )
        model.DBSession.add(entry)
        model.DBSession.flush()

        request.identity = request.environ.get('repoze.who.identity')
        tmpl_context.identity = request.identity
        return TGController.__call__(self, environ, start_response)

Now blow away and recreate your old sqlite database by typing the following into your trusty terminal:

% rm devdata.db
% paster setup-app development.ini
% paster serve development.ini

Pray for no bugs as you revisit http://localhost:8080. Reload the page a few times and just double-check that your database has entries in it now by running sqlite3 devdata.db. Issue the command select * from server_hit; and you should see all your page requests listed.

Cool? Cool.

(Note to the brave: If we wanted to be really awesome, we would write WSGI middleware to do our request-logging.)

3. "Automatically" listing db entries with `tw2.jqplugins.jqgrid:SQLAjqGridWidget`

Create a new file tw2facemelttg21/widgets.py with the following content:

import tw2facemelttg21.model as model
from tw2.jqplugins.jqgrid import SQLAjqGridWidget

class LogGrid(SQLAjqGridWidget):
    id = 'awesome-loggrid'
    entity = model.ServerHit
    excluded_columns = ['id']
    datetime_format = "%x %X"

    prmFilter = {'stringResult': True, 'searchOnEnter': False}

    options = {
        'pager': 'awesome-loggrid_pager',
        'url': '/jqgrid/',
        'rowNum':15,
        'rowList':[15,150, 1500],
        'viewrecords':True,
        'imgpath': 'scripts/jqGrid/themes/green/images',
        'shrinkToFit': True,
        'height': 'auto',
    }

Pull the LogGrid widget into your controller by importing it at the top of tw2facemelttg21/controllers/root.py with:

from tw2facemelttg21.widgets import LogGrid

Modify the index method of your RootController in tw2facemelttg21/controllers/root.py so that it looks like:

@expose('tw2facemelttg21.templates.index')
def index(self):
    """Handle the front-page."""
    return dict(page='index', gridwidget=LogGrid)

This will make the LogGrid available in your index template under the name gridwidget. We still need to display it there.

Edit tw2facemelttg21/templates/index.mak and wipe out all of the content. Replace it with only the following:

<%inherit file="local:templates.master"/>
${gridwidget.display() | n}

Cool.

The SQLAjqGridWidget has a request(...) method that does all of its magic (interrogating your sqlalchemy model for its properties and values). We still need to wire up our TurboGears app to forward the ajax<->json requests to the right place.

To do this, add another method to your RootController back in tw2facemelttg21/controllers/root.py that looks like this:

@expose('json')
def jqgrid(self, *args, **kwargs):
    return LogGrid.request(request).body

Lastly, edit your setup.py file and add all the new widget dependencies we'll need (not just the jqgrid), like so:

    install_requires=[
        "TurboGears2 >= 2.1",
        "Mako",
        "zope.sqlalchemy >= 0.4",
        "repoze.tm2 >= 1.0a5",
        "repoze.what-quickstart",
        "repoze.what >= 1.0.8",
        "repoze.what-quickstart",
        "repoze.who-friendlyform >= 1.0.4",
        "repoze.what-pylons >= 1.0",
        "repoze.what.plugins.sql",
        "repoze.who == 1.0.18",
        "tgext.admin >= 0.3.9",
        "tw.forms",
        "tw2.jqplugins.jqgrid",
        "tw2.jqplugins.jqplot",
        "tw2.jqplugins.portlets",
        ],
    setup_requires=["PasteScript >= 1.7"],

Install the newly listed dependencies by running:

% python setup.py develop
% paster serve development.ini

...and revisit http://localhost:8080.

At this point, your app should look something like the following:

We should really (probably) do things as fancy as we can, and make use of jquery-ui's themes.

Edit tw2facemelttg21/lib/base.py do the following two things:

Add this import statement to the top of the file.

from tw2.jqplugins.ui import set_ui_theme_name

And invoke it from inside the __call__(...) method of your BaseController like so:

    def __call__(self, environ, start_response):
        """Invoke the Controller"""
        # TGController.__call__ dispatches to the Controller method
        # the request is routed to. This routing information is
        # available in environ['pylons.routes_dict']

        set_ui_theme_name('hot-sneaks')

        entry = model.ServerHit(
            remote_addr=environ['REMOTE_ADDR'],
            path_info=environ['PATH_INFO'],
            query_string=environ['QUERY_STRING'],
        )
        model.DBSession.add(entry)

        request.identity = request.environ.get('repoze.who.identity')
        tmpl_context.identity = request.identity
        return TGController.__call__(self, environ, start_response)

Nice! But hot-sneaks isn't the only available theme -- you can see a list of all of them right here.

4. Plotting server history with `tw2.jqplugins.jqplot:JQPlotWidget`

This is going to be awesome.

Add a new widget definition to tw2facemelttg21/widgets.py:

from tw2.jqplugins.jqplot import JQPlotWidget
from tw2.jqplugins.jqplot.base import categoryAxisRenderer_js, barRenderer_js
from tw2.core import JSSymbol

class LogPlot(JQPlotWidget):
    id = 'awesome-logplot'
    interval = 2000
    resources = JQPlotWidget.resources + [
        categoryAxisRenderer_js,
        barRenderer_js,
    ]

    options = {
        'seriesDefaults' : {
            'renderer': JSSymbol('$.jqplot.BarRenderer'),
            'rendererOptions': { 'barPadding': 4, 'barMargin': 10 }
        },
        'axes' : {
            'xaxis': {
                'renderer': JSSymbol(src="$.jqplot.CategoryAxisRenderer"),
            },
            'yaxis': {'min': 0, },
        }
    }

Now we're going to go to town on our RootController. We need to:

Make the new widget available in our template
Produce data for it
Render it in the template

First, add the following imports to the top of tw2facemelttg21/controllers/root.py:

from tw2facemelttg21.models import ServerHit
from tw2facemelttg21.widgets import LogPlot

import sqlalchemy
import datetime
import time

We're also going to need this little recursive_update utility to merge the options dicts. Just add it at the top of tw2facemelttg21/controllers/root.py as a function. It should not be a method of RootController.




def recursive_update(d1, d2):
    """ Little helper function that does what d1.update(d2) does,
    but works nice and recursively with dicts of dicts of dicts.

    It's not necessarily very efficient.
    """

    for k in d1.keys():
        if k not in d2:
            continue

        if isinstance(d1[k], dict) and isinstance(d2[k], dict):
            d1[k] = recursive_update(d1[k], d2[k])
        else:
            d1[k] = d2[k]

    for k in d2.keys():
        if k not in d1:
            d1[k] = d2[k]

    return d1




Add a new method to the RootController class
that looks like the following.  This will do all of the heavy lifty ---
producing the data for the jqplot.


    def jqplot(self, days=1/(24.0)):
        n_buckets = 15
        now = datetime.datetime.now()
        then = now - datetime.timedelta(days)
        delta = datetime.timedelta(days) / n_buckets

        entries = ServerHit.query.filter(ServerHit.timestamp>then).all()

        t_bounds = [(then+delta*i, then+delta*(i+1)) for i in range(n_buckets)]

        # Accumulate into buckets!  This is how I like to do it.
        buckets = dict([(lower, 0) for lower, upper in t_bounds])
        for entry in entries:
            for lower, upper in t_bounds:
                if entry.timestamp >= lower and entry.timestamp < upper:
                    buckets[lower] += 1

        # Only one series for now.. but you could do other stuff!
        series = [buckets[lower] for lower, upper in t_bounds]
        data = [
            series,
            # You could add another series here...
        ]

        options = { 'axes' : { 'xaxis': {
            'ticks': [u.strftime("%I:%M:%S") for l, u in t_bounds],
        }}}

        return dict(data=data, options=options)





Rewrite the index(...) method of your
RootController to look like the following:


    @expose('tw2facemelttg21.templates.index')
    def index(self):
        """Handle the front-page."""
        jqplot_params = self.jqplot()
        plotwidget = LogPlot(data=jqplot_params['data'])
        plotwidget.options = recursive_update(
            plotwidget.options, jqplot_params['options'])
        return dict(page='index', gridwidget=LogGrid, plotwidget=plotwidget)





Now the jqplot widget should pull its data from the (perhaps poorly named)
jqplot method of your RootController and should merge
new options nicely with the predefined ones.  It should also be
available in your index template under the name
plotwidget.  Let's use it there!

Edit tw2facemelttg21/templates/index.mak and
add the following line:


<%inherit file="local:templates.master"/>
${plotwidget.display() | n}
${gridwidget.display() | n}




Revisit http://localhost:8080/ and you should get something like
this.



5.  Making the layout look like http://google.com/ig with tw2.jqplugins.portlets

Add the following import to the top of
tw2facemelttg21/controllers/root.py:


import tw2.jqplugins.portlets as p





Rewrite the index(...) method of your
RootController to look like this:


    @expose('tw2facemelttg21.templates.index')
    def index(self):
        """Handle the front-page."""
        jqplot_params = self.jqplot()
        plotwidget = LogPlot(data=jqplot_params['data'])
        plotwidget.options = recursive_update(
            plotwidget.options, jqplot_params['options'])

        colwidth = '50%'
        class LayoutWidget(p.ColumnLayout):
            id = 'awesome-layout'
            class col1(p.Column):
                width = colwidth
                class por1(p.Portlet):
                    title = 'DB Entries of Server Hits'
                    widget = LogGrid

            class col2(p.Column):
                width = colwidth
                class por2(p.Portlet):
                    title = 'Hits over the last hour'
                    widget = plotwidget

        return dict(page='index', layoutwidget=LayoutWidget)





Now that only the layoutwidget is available to your
index template, you'll need to rewrite your
tw2facemelttg21/templates/index.mak.


<%inherit file="local:templates.master"/>
${layoutwidget.display() | n}





To simplify all the styling, you'll also need to clear out all the clutter
from the TG2.1 quickstart install.  Rewrite (for the first
time) tw2facemelttg21/templates/master.mak to look like this:



<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
                 "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html>
<head></head>
<body>${self.body()}</body>
</html>





Rerun your paster server and you should see something like
this.



The portlet windows are movable and collapsible and they automatically retain their state between page loads (with
jquery.cookie.js).

---

I hope you enjoyed the tutorial.  Once again, you can get the entire source
here
  on my github account.  If you have any comments, feedback, or questions
--- post 'em!




  View Comments






  
  

  
  Ceiling Lenin
  Apr 29, 2011 | categories: 

politics, lulz
View Comments


  

    
  



  


  View Comments

 





  
  

  
  python's dir() is a (big) bottleneck
  Apr 29, 2011 | categories: 

python
View Comments


  

    
  
Wrote this up due to this commit to tw2.core.

(EDIT: Thanks to lmacken for the tip on iteritems().)


#!/usr/bin/env python
""" Experiment showing that :func:`dir` is a bottleneck.

Expected output:
    1.384130 [getattr(obj, k) for k in dir(obj)]
    0.343161 [v for k, v in obj.__dict__.items()]
    0.278336 [v for k, v in obj.__dict__.iteritems()]
"""

from timeit import Timer

statements = [
    "[getattr(obj, k) for k in dir(obj)]",
    "[v for k, v in obj.__dict__.items()]",
    "[v for k, v in obj.__dict__.iteritems()]",
]

for stmt in statements:
    print "%.6f" % Timer(stmt, "import os as obj").timeit(10000), stmt






  


  View Comments

 





  
 « Previous Page
  --  
 Next Page »


      


 
[three]Bean.org

[three]Bean

fabulous is definitely the python module of the week

New tw2.devtools WidgetBrowser features

Tutorial -- melting your face off with tw2 and TurboGears2.1

Agenda

1. Getting TurboGears2.1 installed and running

2. Setting up a data backend

3. "Automatically" listing db entries with tw2.jqplugins.jqgrid:SQLAjqGridWidget

4. Plotting server history with tw2.jqplugins.jqplot:JQPlotWidget

5. Making the layout look like http://google.com/ig with tw2.jqplugins.portlets

Ceiling Lenin

python's dir() is a (big) bottleneck

3. "Automatically" listing db entries with `tw2.jqplugins.jqgrid:SQLAjqGridWidget`

4. Plotting server history with `tw2.jqplugins.jqplot:JQPlotWidget`

5. Making the layout look like http://google.com/ig with `tw2.jqplugins.portlets`