[three]Bean

Fedmsg Reliability -- No More Dropped Messages?

Feb 28, 2013 | categories: fedmsg, zeromq, fedora View Comments

Quick Timeline:

  • February, 2012 - I started thinking about fedmsg.
  • March, 2012 - Development gets under way.
  • July, 2012 - Stuff gets working in staging.
  • August-September, 2012 - The first messages hit production. We notice that some messages are getting dropped. We chalk it up to programming/configuration oversights. We are correct about that in some cases.
  • October-November, 2012 - Some messages are still getting dropped. Theory develops that it is due to zeromq2's lack of TCP_KEEPALIVE. It takes time to package zeromq3 and test this, integrate it with Twisted bindings, and deploy. More message sources come online during this period. Other stuff like externally consuming messages [1] [2], datanommer [3] [4], and identica [5] happen too.
  • January, 2013 - I enabled the TCP_KEEPALIVE bits in production at FUDCon Lawrence.

I haven't noticed any dropped messages since then: a marked improvement. However, our volume of messages is so much higher now that it is difficult to eyeball. If you notice dropped messages, please let me know in #fedora-apps on freenode.

View Comments
blog comments powered by Disqus