[three]Bean
Cached function calls with expiration in python with shelve and decorator
Jun 08, 2011 | categories: python View CommentsCacheing with decorators is nice. Sometimes you don't want to use something super heavyweight, but just a little something you rolled on your own.
import time import random @scached(cache_file='shelvecache.db', expiry=datetime.timedelta(seconds=5)) def f1(foo, bar='baz'): """ Example of using the cache decorator """ print " ** starting in f1.. sleepy time" time.sleep(5) # The result of all my hard work result = random.random() print " ** woke up with", result return result if __name__ == '__main__': print f1('hai') # slow print f1('hai') # fast print f1(foo='hai') # fast print "okay.. sleeping on the outside" time.sleep(5) print f1('hai') # slow again print f1('hai') # fast again
Here's the code that provides the @scached decorator.
import datetime import decorator import shelve from hashlib import md5 def scached(cache_file, expiry): """ Decorator setup """ def scached_closure(func, *args, **kw): """ The actual decorator """ key = md5(':'.join([func.__name__, str(args), str(kw)])).hexdigest() d = shelve.open(cache_file) # Expire old data if we have to if key in d: if d[key]['expires_on'] < datetime.datetime.now(): del d[key] # Get new data if we have to if key not in d: data = func(*args, **kw) d[key] = { 'expires_on' : datetime.datetime.now() + expiry, 'data': data, } # Return what we got result = d[key]['data'] d.close() return result return decorator.decorator(scached_closure)
For extra cool points, combine the above with my post on shelve and context managers.
Switching virtualenvs with a python context manager
Jun 06, 2011 | categories: pip, python, virtualenv, pypi View CommentsEDIT: I released this. You can find it on pypi.
---Got it! Spent the last day working on a control script for moksha to replace my mistake of choosing fabric.
With this little nugget, you can do cool in-python context switching
of virtualenv
s with VirtualenvContext
like this:
#!/usr/bin/python from blogcopypaste import VirtualenvContext try: import kitchen except ImportError as e: print "kitchen is definitely not installed in system-python" with VirtaulenvContext("my-venv"): import kitchen print "But it *is* installed in my virtualenv" try: import kitchen except ImportError as e: print "But once I exit that block, I lose my powers again..."
kitchen
could be any non-standard-library python package you
choose. (Although kitchen
itself is pretty cool).
I learned a ton about ihooks
and the python
__import__
built-in... PEP 302 was an
eye-opener.
Here's the code that makes that fancy VirtualenvContext
happen:
""" Virtualenv context management! """ import os import sys import ihooks import warnings import imp def _silent_load_source(name, filename, file=None): """ Helper function. Overrides a import hook. Suppresses warnings. """ with warnings.catch_warnings(): warnings.simplefilter("ignore") return imp.load_source(name, filename, file) class VenvModuleLoader(ihooks.ModuleLoader): """ Overridden ModuleLoader. Checks for a virtualenv first and remembers imports. """ remembered = [] def __init__(self, venv, verbose=0): self.venv = venv ihooks.ModuleLoader.__init__(self, verbose=verbose) self.hooks.load_source = _silent_load_source def default_path(self): workon = os.getenv("WORKON_HOME", None) venv_location = "/".join([ workon, self.venv, 'lib/python2.7/site-packages']) full = lambda i : "/".join([venv_location, i]) venv_path = [venv_location] + [ full(item) for item in os.listdir(venv_location) if os.path.isdir(full(item))] + sys.path return venv_path + sys.path def load_module(self, name, stuff): """ Overloaded just to remember what we load """ self.remembered.append(name) return ihooks.ModuleLoader.load_module(self, name, stuff) class VirtualenvContext(object): """ Context manager for entering a virtualenv """ def __init__(self, venv_name): self.venv = venv_name self.loader = VenvModuleLoader(venv=self.venv) self.importer = ihooks.ModuleImporter(loader=self.loader) def __enter__(self): # Install our custom importer self.importer.install() # Pretend like our exectuable is really somewhere else self.old_exe = sys.executable workon = os.getenv("WORKON_HOME", None) sys.executable = "/".join([workon, self.venv, 'bin/python']) def __exit__(self, exc_type, exc_value, traceback): # Uninstall our custom importer self.importer.uninstall() # Reset our executable sys.exectuable = self.old_exe # Unload anything loaded while inside the context for name in self.importer.loader.remembered: if not name in sys.modules: continue del sys.modules[name] self.importer.loader.remembered = [] sys.path_importer_cache.clear()
Fun fact: you can combine this with the install_distributions function in my previous post to do:
with VirtualenvContext('some-environment'): install_distributions(['Markdown'])
Installing from pip inside python or, a simple pip API
Jun 06, 2011 | categories: pip, python View CommentsSo, it's a long story, but my bid to convert
moksha's development tools to fabric was a terrible idea: mandatory
sshd
on dev boxes, passwords in the clear when chaining commands
between systems, simple failure to work. It was the wrong tool for the job.
Now I have my work cut out for me to replace it with something worthwhile.
I have a need to invoke pip from inside python and to do it within the current virtualenv. I googled and found this somewhat old thread on the virtualenv list which indicates that there's not much help out there.
Here's my stab at it.
import pip.commands.install def install_distributions(distributions): command = pip.commands.install.InstallCommand() opts, args = command.parser.parse_args() # TBD, why do we have to run the next part here twice before actual install requirement_set = command.run(opts, distributions) requirement_set = command.run(opts, distributions) requirement_set.install(opts)
And a test to see if it works.
#!/usr/bin/env python from blogmodule import install_distributions def run_test(): try: import markdown print "Markdown is installed! Aborting." assert(False) except ImportError as e: print "Markdown isn't yet installed. That's good." install_distributions(["Markdown"]) try: import markdown print "Markdown is now installed. That's good!" except ImportError as e: print "Markdown never got installed. That's bad." assert(False) if __name__ == '__main__': run_test()
Looking forward to doing something cool with this and python context
managers to manage virtualenv
s.
Context manager for python shelve module
Jun 06, 2011 | categories: python View CommentsGetting in the flow of using context managers is great.
# This feels really old. f = open('foo.txt') handle_file(f) f.close() # This feels really great. with open('foo.txt') as f: handle_file(f)
Python's shelve
module doesn't seem to be up to date; I get so
frustrated whenever I try to use it inside with
syntax and am
refused.
I fixed the problem!
import shelve class cmshelve(object): """ Context manager for shelve """ def __init__(self, filename): self.filename = filename def __enter__(self): self.obj = shelve.open(self.filename) return self.obj def __exit__(self, exc_type, exc_value, traceback): self.obj.close()
You can use it a little something like this
>>> with cmshelve('foo.db') as d: ... d['foo'] = "bar" ... print d {'foo': 'bar'} >>> # This proves that the shelve was actually closed >>> print d Traceback (most recent call last): File "<stdin>", line 1, in ? ValueError: invalid operation on closed shelf >>> # And as you'd expect, you can go back and re-open it just fine >>> with cmshelve('foo.db') as d: ... print d {'foo': 'bar'}
``didit`` -- lightweight CLI task recording
May 05, 2011 | categories: python, gtd View CommentsI wrote and published didit this afternoon. I hope you find it useful for the same reasons I wrote it.
% didit-remember -c work -m 'Wrote `diddit`. Thank god.' % didit-remember --message 'Helped L. User parallelize his ``Mathematica`` code.' % didit-remember -c personal # <-- This launches `vim` for me! % didit-report --categories=work,general,personal Category 'work, general, personal' over timespan 'week' ------------------------------------------------------- ---- 2011-05-05: - Wrote `diddit`. Thank god. - Helped L. User parallelize his ``Mathematica`` code. - Drank a beer.
One of the upshots of using .rst
:
% didit-report --category=work > thisweek.rst && rst2pdf thisweek.rst
« Previous Page -- Next Page »