[three]Bean

Cached function calls with expiration in python with shelve and decorator

Jun 08, 2011 | categories: python View Comments

Cacheing with decorators is nice. Sometimes you don't want to use something super heavyweight, but just a little something you rolled on your own.

import time
import random

@scached(cache_file='shelvecache.db', expiry=datetime.timedelta(seconds=5))
def f1(foo, bar='baz'):
    """ Example of using the cache decorator """

    print " ** starting in f1.. sleepy time"
    time.sleep(5)
    # The result of all my hard work
    result = random.random()
    print " ** woke up with", result
    return result

if __name__ == '__main__':
    print f1('hai') # slow
    print f1('hai') # fast
    print f1(foo='hai') # fast

    print "okay.. sleeping on the outside"
    time.sleep(5)

    print f1('hai') # slow again
    print f1('hai') # fast again

Here's the code that provides the @scached decorator.

import datetime
import decorator
import shelve
from hashlib import md5

def scached(cache_file, expiry):
    """ Decorator setup """

    def scached_closure(func, *args, **kw):
        """ The actual decorator """
        key = md5(':'.join([func.__name__, str(args), str(kw)])).hexdigest()
        d = shelve.open(cache_file)

        # Expire old data if we have to
        if key in d:
            if d[key]['expires_on'] < datetime.datetime.now():
                del d[key]

        # Get new data if we have to
        if key not in d:
            data = func(*args, **kw)
            d[key] = {
                'expires_on' : datetime.datetime.now() + expiry,
                'data': data,
            }

        # Return what we got
        result = d[key]['data']
        d.close()

        return result

    return decorator.decorator(scached_closure)

For extra cool points, combine the above with my post on shelve and context managers.

View Comments

Switching virtualenvs with a python context manager

Jun 06, 2011 | categories: pip, python, virtualenv, pypi View Comments

EDIT: I released this. You can find it on pypi.

---

Got it! Spent the last day working on a control script for moksha to replace my mistake of choosing fabric.

With this little nugget, you can do cool in-python context switching of virtualenvs with VirtualenvContext like this:

#!/usr/bin/python

from blogcopypaste import VirtualenvContext

try:
    import kitchen
except ImportError as e:
    print "kitchen is definitely not installed in system-python"

with VirtaulenvContext("my-venv"):
    import kitchen
    print "But it *is* installed in my virtualenv"

try:
    import kitchen
except ImportError as e:
    print "But once I exit that block, I lose my powers again..."

kitchen could be any non-standard-library python package you choose. (Although kitchen itself is pretty cool).

I learned a ton about ihooks and the python __import__ built-in... PEP 302 was an eye-opener.

Here's the code that makes that fancy VirtualenvContext happen:

""" Virtualenv context management! """

import os
import sys
import ihooks
import warnings
import imp

def _silent_load_source(name, filename, file=None):
    """ Helper function.  Overrides a import hook.  Suppresses warnings. """
    with warnings.catch_warnings():
        warnings.simplefilter("ignore")
        return imp.load_source(name, filename, file)

class VenvModuleLoader(ihooks.ModuleLoader):
    """ Overridden ModuleLoader.

    Checks for a virtualenv first and remembers imports.
    """

    remembered = []

    def __init__(self, venv, verbose=0):
        self.venv = venv
        ihooks.ModuleLoader.__init__(self, verbose=verbose)
        self.hooks.load_source = _silent_load_source

    def default_path(self):
        workon = os.getenv("WORKON_HOME", None)
        venv_location = "/".join([
            workon, self.venv, 'lib/python2.7/site-packages'])
        full = lambda i : "/".join([venv_location, i])
        venv_path = [venv_location] + [
            full(item) for item in os.listdir(venv_location)
            if os.path.isdir(full(item))] + sys.path
        return venv_path + sys.path

    def load_module(self, name, stuff):
        """ Overloaded just to remember what we load """
        self.remembered.append(name)
        return ihooks.ModuleLoader.load_module(self, name, stuff)

class VirtualenvContext(object):
    """ Context manager for entering a virtualenv """

    def __init__(self, venv_name):
        self.venv = venv_name
        self.loader = VenvModuleLoader(venv=self.venv)
        self.importer = ihooks.ModuleImporter(loader=self.loader)

    def __enter__(self):
        # Install our custom importer
        self.importer.install()

        # Pretend like our exectuable is really somewhere else
        self.old_exe = sys.executable
        workon = os.getenv("WORKON_HOME", None)
        sys.executable = "/".join([workon, self.venv, 'bin/python'])

    def __exit__(self, exc_type, exc_value, traceback):
        # Uninstall our custom importer
        self.importer.uninstall()

        # Reset our executable
        sys.exectuable = self.old_exe

        # Unload anything loaded while inside the context
        for name in self.importer.loader.remembered:
            if not name in sys.modules:
                continue
            del sys.modules[name]
        self.importer.loader.remembered = []
        sys.path_importer_cache.clear()

Fun fact: you can combine this with the install_distributions function in my previous post to do:

with VirtualenvContext('some-environment'):
    install_distributions(['Markdown'])
View Comments

Installing from pip inside python or, a simple pip API

Jun 06, 2011 | categories: pip, python View Comments

So, it's a long story, but my bid to convert moksha's development tools to fabric was a terrible idea: mandatory sshd on dev boxes, passwords in the clear when chaining commands between systems, simple failure to work. It was the wrong tool for the job. Now I have my work cut out for me to replace it with something worthwhile.

I have a need to invoke pip from inside python and to do it within the current virtualenv. I googled and found this somewhat old thread on the virtualenv list which indicates that there's not much help out there.

Here's my stab at it.

import pip.commands.install

def install_distributions(distributions):
    command = pip.commands.install.InstallCommand()
    opts, args = command.parser.parse_args()
    # TBD, why do we have to run the next part here twice before actual install
    requirement_set = command.run(opts, distributions)
    requirement_set = command.run(opts, distributions)
    requirement_set.install(opts)

And a test to see if it works.

#!/usr/bin/env python

from blogmodule import install_distributions

def run_test():
    try:
        import markdown
        print "Markdown is installed!  Aborting."
        assert(False)
    except ImportError as e:
        print "Markdown isn't yet installed.  That's good."

    install_distributions(["Markdown"])

    try:
        import markdown
        print "Markdown is now installed.  That's good!"
    except ImportError as e:
        print "Markdown never got installed.  That's bad."
        assert(False)

if __name__ == '__main__':
    run_test()

Looking forward to doing something cool with this and python context managers to manage virtualenvs.

View Comments

Context manager for python shelve module

Jun 06, 2011 | categories: python View Comments

Getting in the flow of using context managers is great.

# This feels really old.
f = open('foo.txt')
handle_file(f)
f.close()

# This feels really great.
with open('foo.txt') as f:
    handle_file(f)

Python's shelve module doesn't seem to be up to date; I get so frustrated whenever I try to use it inside with syntax and am refused.

I fixed the problem!

import shelve

class cmshelve(object):
    """ Context manager for shelve """

    def __init__(self, filename):
        self.filename = filename

    def __enter__(self):
        self.obj = shelve.open(self.filename)
        return self.obj

    def __exit__(self, exc_type, exc_value, traceback):
        self.obj.close()

You can use it a little something like this

>>> with cmshelve('foo.db') as d:
...     d['foo'] = "bar"
...     print d
{'foo': 'bar'}

>>> # This proves that the shelve was actually closed
>>> print d
Traceback (most recent call last):
File "<stdin>", line 1, in ?
ValueError: invalid operation on closed shelf

>>> # And as you'd expect, you can go back and re-open it just fine
>>> with cmshelve('foo.db') as d:
...     print d
{'foo': 'bar'}
View Comments

``didit`` -- lightweight CLI task recording

May 05, 2011 | categories: python, gtd View Comments

I wrote and published didit this afternoon. I hope you find it useful for the same reasons I wrote it.

% didit-remember -c work -m 'Wrote `diddit`.  Thank god.'
% didit-remember --message 'Helped L. User parallelize his ``Mathematica`` code.'
% didit-remember -c personal       # <-- This launches `vim` for me!

% didit-report --categories=work,general,personal
Category 'work, general, personal' over timespan 'week'
-------------------------------------------------------

----

2011-05-05:

  - Wrote `diddit`.  Thank god.
  - Helped L. User parallelize his ``Mathematica`` code.
  - Drank a beer.

One of the upshots of using .rst:

% didit-report --category=work > thisweek.rst &amp;&amp; rst2pdf thisweek.rst
View Comments

« Previous Page -- Next Page »