[three]Bean

A Release Engineering Dashboard?

Aug 30, 2013 | categories: releng, fedmsg, fedora, datagrepper View Comments

Messing around, I came up with a read-only dashboard for Fedora Release Engineering. You can check it out live on threebean.org.

Screenshot of an alpha dashboard for Release Engineering.

Right now all it does is try to show the status of the compose process which runs once a day for rawhide and the "branched" pre-release (that's F20 right now). If any one of the components of that process (say, the pungify part) is happening right now, it'll tell you that and have it show up in yellow in stead of blue. If the compose hasn't happened in over 20 hours, that text will show up in gray to indicate that it is "stale" or "out of date" (those words might be too strong, but whatever).

The point of this post is to ask for feedback -- should this thing do anything more? If so, then what? Should we make it live under fedoraproject.org?

(details: It's just HTML, CSS, and js and it makes its queries against datagrepper)

View Comments

Querying all package history from the command line

Jul 16, 2013 | categories: pkgwat, fedmsg, datanommer, fedora, datagrepper View Comments

After my last post on querying datagrepper, I wrote it into my pkgwat tool as the second-coolest subcommand: $ pkgwat history PKGNAME:

$ sudo yum -y install pkgwat

$ pkgwat history firefox
+------------+------------------------------------------------+---------------------+
| date       | event                                          | link                |
+------------+------------------------------------------------+---------------------+
| 2013/07/15 | stransky commented on bodhi update firefox-22. | http://ur1.ca/enmvg |
| 2013/07/15 | suren commented on bodhi update firefox-22.0-1 | http://ur1.ca/enmvg |
| 2013/07/10 | stransky's firefox-21.0-1.fc17 untagged from f | http://ur1.ca/eauo1 |
| 2013/07/10 | xhorak's firefox-21.0-4.fc19 tagged into trash | http://ur1.ca/ea995 |
| 2013/07/10 | xhorak's firefox-21.0-4.fc18 tagged into trash | http://ur1.ca/ea995 |
| 2013/07/05 | stransky's firefox-20.0-3.fc20 tagged into tra | http://ur1.ca/ea995 |
| 2013/07/04 | xhorak's firefox-21.0-4.fc19 untagged from f19 | http://ur1.ca/ea9a4 |
| 2013/07/04 | xhorak's firefox-21.0-4.fc18 untagged from f18 | http://ur1.ca/ea9a1 |
| 2013/07/03 | stransky's firefox-20.0-4.fc20 untagged from f | http://ur1.ca/ea9ac |
| 2013/07/02 | pbrobinson's firefox-22.0-2.fc18 tagged into f | http://ur1.ca/ea9a1 |
| 2013/07/02 | pbrobinson's firefox-22.0-2.fc18 completed     | http://ur1.ca/enmvh |
| 2013/07/02 | firefox-22.0-2.fc18 started building           | http://ur1.ca/enmvh |
+------------+------------------------------------------------+---------------------+

So the next time you're hacking on something and you say to yourself: "Wat wat wat. What just happened?" $ pkgwat history PKGNAME is another resource to draw on for clarity.

You can get the full help with $ pkgwat help history:

usage: pkgwat history [-h] [-f {csv,table}] [-c COLUMN]
                      [--quote {all,minimal,none,nonnumeric}]
                      [--rows-per-page ROWS_PER_PAGE] [--start-page PAGE]
                      package

Show the fedmsg history of a package. This command queries
https://apps.fedoraproject.org/datagrepper/

positional arguments:
  package

optional arguments:
  -h, --help            show this help message and exit
  --rows-per-page ROWS_PER_PAGE
  --start-page PAGE

output formatters:
  output formatter options

  -f {csv,table}, --format {csv,table}
                        the output format, defaults to table
  -c COLUMN, --column COLUMN
                        specify the column(s) to include, can be repeated

CSV Formatter:
  --quote {all,minimal,none,nonnumeric}
                        when to include quotes, defaults to nonnumeric
View Comments

Querying fedmsg history for package details by example

Jun 11, 2013 | categories: fedmsg, datanommer, fedora, datagrepper View Comments

In case you missed it, you can query fedmsg history now with the datagrepper API.

I wrote up an example here to show how you might use it in a script. This will print out the whole history of the hovercraft package (at least everything that is published via fedmsg anyways). In time, I hope to add it to python-pkgwat-api and as a subcommand of the pkgwat command line tool. Until then, you can use:

#!/usr/bin/env python
""" Query the history of a package using datagrepper!

Check out the api at https://apps.fedoraproject.org/datagrepper/

:Author: Ralph Bean <rbean@redhat.com>
:License: LGPLv2+

"""

import datetime
import re
import requests

regexp = re.compile(
    r'<p class="success">Your ur1 is: '
    '<a href="(?P<shorturl>.+)">(?P=shorturl)</a></p>')


def shorten_url(longurl):
    response = requests.post("http://ur1.ca/", data=dict(longurl=longurl))
    return regexp.search(response.text).groupdict()['shorturl']


def get_data(package, rows=20):
    url = "https://apps.fedoraproject.org/datagrepper/raw/"
    response = requests.get(
        url,
        params=dict(
            package=package,
            delta=9999999,
            meta=['subtitle', 'link'],
            rows_per_page=rows,
            order='desc',
        ),
    )
    data = response.json()
    return data.get('raw_messages', [])


def print_data(package, links=False):
    for message in get_data(package, 40):
        dt = datetime.datetime.fromtimestamp(message['timestamp'])
        print dt.strftime("%Y/%m/%d"),
        print message['meta']['subtitle'],
        if links:
            print shorten_url(message['meta']['link'])
        else:
            print


if __name__ == '__main__':
    print_data(package='hovercraft', links=False)

And here's the output:

View Comments

Querying fedmsg history

May 14, 2013 | categories: fedmsg, datanommer, fedora, datagrepper View Comments

Yesterday, Ian Weller and I got the first iteration of "datagrepper" into production. It is a JSON api that lets you query the history of the fedmsg bus. In case you're confused.. it is related to, but not the same thing as "datanommer". You can check out the datagrepper docs on its index page as well as on its reference page.

It is another building block. With it, we can now:

  • Use it as a reliability resource for other fedmsg projects.
    • Say you have a daemon that listens for fedmsg messages to act on... but it crashes. When it gets restarted it can ask datagrepper "What messages did I miss since this morning at 4:03am?" and catch up on those as it can.
  • Build apps that query it to show "the latest so many messages that meet such and such criteria."
    • Imagine an HTML5 mobile app that shows you the latest of anything and everything in Fedora. (pingou is at work on this).
    • Imagine package-centric UI widgets that show the latest Fedora-wide events pertaining to a certain package. We could embed them in the Fedora Packages app.
    • Imagine user-centric UI widgets that show the latest activity of developers. You could embed yours in your blog or wiki page.
  • Statistics! The whole dataset is available to you and updated in real time. Can you tell any cool stories with it?

It is, like I mentioned, an initial release so please be gentle. We have a big list of plans and bugs to crack on. If you run into issues or simply have questions, feel free to file a bug or ping us in #fedora-apps on freenode.

View Comments
« Previous Page

[three]Bean.org