[three]Bean

Fork me on GitHub

WebSockets on OpenShift (Moksha in the Cloud)

Jan 07, 2013 | categories: python, fedora, moksha, openshift View Comments

I've been waiting on the OpenShift team to crack the WebSocket nut for a while and they finally got it back in December. To try it out, I tried to set up the Moksha Demo Dashboard on two different gears.

It wasn't too tricky. I created two OpenShift "DIY"-type apps, one for the WSGI app and another for the WebSocket server (the moksha-hub). All the work in those two repos is done in the .openshift/action_hooks directories (the code is actually just installed from PyPI). Additionally, the diy/development.ini files hold all the configuration.

It's live now at http://mokshademo-threebean.rhcloud.com/ but it's our same demo as before. Other apps still in the development pipeline should be more interesting when they arrive.

http://threebean.org/moksha-screenshot-2012-10-25.png
View Comments

The Moksha Demo Dashboard

Oct 25, 2012 | categories: python, moksha, fedora View Comments

Just writing to show off how easy it is to stand up the moksha demo dashboard these days (it used to be kind of tricky).

http://threebean.org/moksha-screenshot-2012-10-25.png

First, install some system dependencies if you don't already have them:

sudo yum install zeromq zeromq-devel python-virtualenvwrapper

Open two terminals. In the first one run:

mkvirtualenv demo-dashboard
pip install mdemos.server mdemos.menus mdemos.metrics
wget https://raw.github.com/mokshaproject/mdemos.server/master/development.ini
paster serve --reload development.ini

And in the second one run:

workon demo-dashboard
moksha-hub

"Easy." Point your browser at http://localhost:8080/ for action.

p.s. -- In other news, I got fedmsg working with zeromq-3.2 in our staging infrastructure yesterday. It required this patch to python-txzmq That one aside, python-zmq and php-zmq "just worked" in epel-test. If you're writing zeromq code, you probably want to read this porting guide.

View Comments

Moksha Tutorial - Live Graph of User Activity

Sep 17, 2012 | categories: python, pyramid, fedora, moksha View Comments

I've been working recently on sprucing up the fedmsg docs and the underlying Moksha docs. It seemed a natural extension of that work to put together some Moksha tutorials like this one.

Here I'll be showing you how to add a websocket-powered d3 graph of user activity to a Pyramid app using, as I said, Moksha.

Note

You can find the source for this tutorial on github.

Bootstrapping

Note

Bootstrapping here is almost exactly the same as in the Hello World tutorial. So if you've gone through that, this should be simple.

The exception is the new addition of a tw2.d3 dependency.

Set up a virtualenv and install Moksha and Pyramid (install virtualenvwrapper if you haven't already):

$ mkvirtualenv tutorial
$ pip install pyramid
$ pip install moksha.wsgi moksha.hub

$ # tw2.d3 for our frontend component.
$ pip install tw2.d3

$ # Also, install weberror for kicks.
$ pip install weberror

Use pcreate to setup a Pyramid scaffold, install dependencies, and verify that its working. I like the alchemy scaffold, so we'll use that one. Its kind of silly, though: we won't be using a database or sqlalchemy at all for this tutorial:

$ pcreate -t alchemy tutorial
$ cd tutorial/
$ rm production.ini  # moksha-hub gets confused when this is present.
$ python setup.py develop
$ initialize_tutorial_db development.ini
$ pserve --reload development.ini

Visit http://localhost:6543 to check it out. Success.

Enable ToscaWidgets2 and Moksha Middlewares

Note

Enabling the middleware here is identical to the Hello World tutorial.

Moksha is framework-agnostic, meaning that you can use it with TurboGears2, Pyramid, or Flask. It integrates with apps written against those frameworks by way of a layer of WSGI middleware you need to install. Moksha is pretty highly-dependent on ToscaWidgets2 which has its own middleware layer. You'll need to enable both, and in a particular order!

Go and edit development.ini. There should be a section at the top named [app:main]. Change that to [app:tutorial]. Then, just above the [server:main] section add the following three blocks:

[pipeline:main]
pipeline =
    egg:WebError#evalerror
    tw2
    moksha
    tutorial

[filter:tw2]
use = egg:tw2.core#middleware

[filter:moksha]
use = egg:moksha.wsgi#middleware

You now have three new pieces of WSGI middleware floating under your pyramid app. Neat! Restart pserve and check http://localhost:6543 to make sure its not crashing.

Provide some configuration for Moksha

Warning

This is where things begin to diverge from the Hello World tutorial.

We're going to configure moksha to communicate with zeromq and WebSocket. As an aside, one of Moksha's goals is to provide an abstraction over different messaging transports. It can speak zeromq, AMQP, and STOMP on the backend, and WebSocket or COMET emulated-AMQP and/or STOMP on the frontend.

We need to configure a number of things:

  • Your app needs to know how to talk to the moksha-hub with zeromq.
  • Your clients need to know where to find their websocket server (its housed inside the moksha-hub).

Edit development.ini and add the following lines in the [app:tutorial] section. Do it just under the sqlalchemy.url line:

##moksha.domain = live.example.com
moksha.domain = localhost

moksha.notifications = True
moksha.socket.notify = True

moksha.livesocket = True
moksha.livesocket.backend = websocket
#moksha.livesocket.reconnect_interval = 5000
moksha.livesocket.websocket.port = 9998

zmq_enabled = True
#zmq_strict = True
zmq_publish_endpoints = tcp://*:3001
zmq_subscribe_endpoints = tcp://127.0.0.1:3000,tcp://127.0.0.1:3001

Also, add a new hub-config.ini file with the following (nearly identical) content. Notice that the only real different is the value of zmq_publish_endpoints:

[app:tutorial]
##moksha.domain = live.example.com
moksha.domain = localhost

moksha.livesocket = True
moksha.livesocket.backend = websocket
moksha.livesocket.websocket.port = 9998

zmq_enabled = True
#zmq_strict = True
zmq_publish_endpoints = tcp://*:3000
zmq_subscribe_endpoints = tcp://127.0.0.1:3000,tcp://127.0.0.1:3001

Emitting events when users make requests

This is the one tiny little nugget of "business logic" we're going to add. When a user anywhere makes a Request on our app, we want to emit a message that can then be viewed in graphs by other users. Pretty simple: we'll just emit a message on a topic we hardcode that has an empty dict for its body.

Add a new file, tutorial/events.py with the following content:

from pyramid.events import NewRequest
from pyramid.events import subscriber

from moksha.hub.hub import MokshaHub

hub = None

def hub_factory(config):
    global hub
    if not hub:
        hub = MokshaHub(config)
    return hub

@subscriber(NewRequest)
def emit_message(event):
    """ For every request made of our app, emit a message to the moksha-hub.
    Given the config from the tutorial, this will go out on port 3001.
    """

    hub = hub_factory(event.request.registry.settings)
    hub.send_message(topic="tutorial.newrequest", message={})

Combining components to make a live widget

With those messages now being emitted to the "tutorial.newrequest" topic, we can construct a frontend widget with ToscaWidgets2 that listens to that topic (using a Moksha LiveWidget mixin). When a message is received on the client the javascript contained in onmessage will be executed (and passed a json object of the message body). We'll ignore that since its empty, and just increment a counter provided by tw2.d3.

Add a new file tutorial/widgets.py with the following content:

from tw2.d3 import TimeSeriesChart
from moksha.wsgi.widgets.api.live import LiveWidget


class UsersChart(TimeSeriesChart, LiveWidget):
    id = 'users-chart'
    topic = "tutorial.newrequest"
    onmessage = """
    tw2.store['${id}'].value++;
    """

    width = 800
    height = 150

    # Keep this many data points
    n = 200
    # Initialize to n zeros
    data = [0] * n


def get_time_series_widget(config):
    return UsersChart(
        backend=config.get('moksha.livesocket.backend', 'websocket')
    )

Rendering Moksha Frontend Components

With our widget defined, we'll need to expose it to our chameleon template and render it. Instead of doing this per-view like you might normally, we're going to flex Pyramid's events system some more and inject it (and the requisite moksha_socket widget) on every page.

Go back to tutorial/events.py and add the following new handler:

from pyramid.events import BeforeRender
from pyramid.threadlocal import get_current_request

from moksha.wsgi.widgets.api import get_moksha_socket

from tutorial.widgets import get_time_series_widget


@subscriber(BeforeRender)
def inject_globals(event):
    """ Before templates are rendered, make moksha front-end resources
    available in the template context.
    """
    request = get_current_request()

    # Expose these as global attrs for our templates
    event['users_widget'] = get_time_series_widget(request.registry.settings)
    event['moksha_socket'] = get_moksha_socket(request.registry.settings)

And lastly, go edit tutorial/templates/mytemplate.pt so that it displays users_widget and moksha_socket on the page:

<div id="bottom">
  <div class="bottom">
    <div tal:content="structure users_widget.display()"></div>
    <div tal:content="structure moksha_socket.display()"></div>
  </div>
</div>

Running the Hub alongside pserve

When the moksha-hub process starts up, it will begin handling your messages. It also houses a small websocket server that the moksha_socket will try to connect back to.

Open up two terminals and activate your virtualenv in both with workon tutorial. In one of them, run:

$ moksha-hub -v hub-config.ini

And in the other run:

$ pserve --reload development.ini

Now open up two browsers, (say.. one chrome, the other firefox) and visit http://localhost:6543/ in both. In one of them, reload the page over and over again.. you should see the graph in the other one "spike" showing a count of all the requests issued.

View Comments

Scheduling meetings for the Fedora Messaging SIG

Mar 05, 2012 | categories: zeromq, fedmsg, fedora, moksha View Comments

So I'm spinning up the Fedora Messaging SIG. It's been sitting idle for years; this time we're really going to do it. We're going to put 0mq messaging hooks into bodhi, koji, pkgdb, fas, you name it.

I'm building a python module called fedmsg to wrap it all and it comes with a proposal-in-development. And if you're not excited yet, we've already 0mq enabled the Moksha Hub and achieved a ~100 times speedup over AMQP/qpid.

The point of this post is to ask about times for an IRC meeting of the Messaging SIG. Right now I'm thinking Tuesdays at 16:00-17:00 UTC in #fedora-meeting. If you're interested in helping with the effort but that's a bad time for you, let me know and we'll work it out.

I'll be out at PyCon until next Wednesday; so the first meeting won't be until March 20th at the earliest.


http://threebean.org/blog/static/images/0mq-enable-all-the-things.jpg
View Comments

Using fabric to manage moksha development

Apr 13, 2011 | categories: python, fabric, moksha View Comments

Now I know that fabric is really meant for managing deployment, but to try and teach myself what it can do, I rewrote some monster bash-scripts that manage the development environment for Luke Macken's awesome project moksha.

I came up with:

  • A master fabfile to manage moksha-proper development.
  • A template fabfile that gets placed in the base directory of new apps created from moksha's paster templates.

Here's an example of how to use them to start up a new moksha app.
#!/bin/bash

rm -rf threebean
mkdir -p threebean
pushd threebean

git clone git://github.com/ralphbean/moksha.git
pushd moksha
fab -H localhost bootstrap
popd

# If you don't already have it sourced in your .bashrc!
source /usr/bin/virtualenvwrapper.sh

workon moksha
paster moksha -l -c -u -C -P << EOF
testapp
moksha.test
EOF
deactivate

pushd testapp
fab -H localhost install
popd


pushd moksha
fab -H localhost develop
fab -H localhost start
popd

popd

# You should see the testapp widget show up under the 'widgets' menu.
firefox http://localhost:8080/ &amp;
View Comments