Yesterday, Ian Weller and I got the first iteration of "datagrepper" into production. It is a JSON api that lets you query the history of the fedmsg bus. In case you're confused.. it is related to, but not the same thing as "datanommer". You can check out the datagrepper docs on its index page as well as on its reference page.
It is another building block. With it, we can now:
- Use it as a reliability resource for other fedmsg projects.
- Say you have a daemon that listens for fedmsg messages to act on... but it crashes. When it gets restarted it can ask datagrepper "What messages did I miss since this morning at 4:03am?" and catch up on those as it can.
- Build apps that query it to show "the latest so many messages
that meet such and such criteria."
- Imagine an HTML5 mobile app that shows you the latest of anything and everything in Fedora. (pingou is at work on this).
- Imagine package-centric UI widgets that show the latest Fedora-wide events pertaining to a certain package. We could embed them in the Fedora Packages app.
- Imagine user-centric UI widgets that show the latest activity of developers. You could embed yours in your blog or wiki page.
- Statistics! The whole dataset is available to you and updated in real time. Can you tell any cool stories with it?
It is, like I mentioned, an initial release so please be gentle. We have a big list of plans and bugs to crack on. If you run into issues or simply have questions, feel free to file a bug or ping us in #fedora-apps on freenode.
Crazy idea: Take some data and mangle it into a git log. Run gource on that git log.
You may have seen
a previous post of mine on
datanommer. Well, it's been running for almost two months now. I took the
datanommer-dump and wrote
a script to transform it into a fake git log. The purpose? To run gource on it:
If you have problems with the video, you can download it directly here.
It's like the "Fedora Movie". Just to be clear, the visualization is not of the development of fedmsg itself, but of Fedora in general. You can see wiki edits, IRC meetings, updates to packages, etc.. Can you find yourself in there?
I used gravatar.com to grab the avatar
Last week we finally got datanommer working in our production environment. Originally ianweller's idea, it is a consumer that sits listening to the fedmsg bus and logs every event to a postgresql database.
It's nice to have in place now. With the data we can make more confident statements about what's happening on the bus... we can record a series of events from our production environment... play those back in staging for testing scenarios... and most importantly, we can make pretty graphs.
I made the following with the output of the datanommer-dump command and these scripts:
You can see the ups and downs of the day/night cycle and you can see activity dip on the weekend, too. Neat!