[three]Bean

Pungi Performance, Profiling?

Sep 01, 2017 | categories: factory2, fedora View Comments

Happy Flockery!

This might not be new to others, but I wanted to identify which parts of the compose process were slow, so I wrote a little script. If you didn't know, our compose process is broken up into phases some of which run in serial, others of which run in parallel. The composes take a long time, and speeding them up would be great.

#!/usr/bin/env python3
""" Usage:  python3 analyze-compose.py Fedora-27-20170901.n.1 """

import operator
import sys

import requests
import dogpile.cache

cache = dogpile.cache.make_region().configure(
        'dogpile.cache.dbm',
        expiration_time=500,
        arguments=dict(filename='/var/tmp/datagrepper-cache.dbm')
)

# Fedora-27-20170901.n.1
target = sys.argv[-1]

@cache.cache_on_arguments()
def get_messages(target):
        url = 'https://apps.fedoraproject.org/datagrepper/raw'
        params = dict(
                category='pungi',
                rows_per_page=100,
                contains=target,
                delta=1278000,
        )
        print("Querying %r %r" % (url, params))
        response = requests.get(url, params=params)
        return [m for m in response.json()['raw_messages']]


if __name__ == '__main__':
        print('Analyzing compose %r' % target)
        messages = get_messages(target)
        starts, stops = {}, {}
        for message in messages:
                if message['topic'].endswith('compose.status.change'):
                        continue
                if message['topic'].endswith('compose.phase.start'):
                        starts[message['msg']['phase_name']] = message['timestamp']
                if message['topic'].endswith('compose.phase.stop'):
                        stops[message['msg']['phase_name']] = message['timestamp']

        deltas = {}
        for phase in starts:
                if phase not in stops:
                        continue
                deltas[phase] = stops[phase] - starts[phase]

        deltas = list(deltas.items())
        deltas.sort(key=operator.itemgetter(1), reverse=True)
        for phase, delta in deltas:
                print("{0:20} {1:.2f} hours".format(phase, delta / 60.0 / 60.0))

This script prints out stats about all of the phases. Take a look at this compose from a few days ago and focus in on just one phase:

❯ ./analyze-compose.py Fedora-27-20170830.n.0
Analyzing compose 'Fedora-27-20170830.n.0'
[snip]
createrepo           8.50 hours
[snip]

That createrepo phase is gigantic!

After one or two patches we were able to get it down a little tighter the next day:

./analyze-compose.py Fedora-27-20170901.n.1
Analyzing compose 'Fedora-27-20170901.n.1'
[snip]
createrepo           0.33 hours
[snip]

createrepo phase -- greatly improved.

The next one that jumps out at me is the pkgset phase. What's going on there?

I'm not sure yet but have posted a patch introducing more logging so we can get a feel for it.

View Comments

Factory 2, Sprint 14

Apr 11, 2017 | categories: sprint, factory2, fedora View Comments

Hello all. Bi-weekly update from the Factory 2 team on our work here. We have three videos this sprint.

Two are related to the module-build-service: both on submitting builds. We have some new client tooling to show which should make manual submission and monitoring of module-builds much simpler. The second is a demo of an early prototype of our continuous rebuild system.

Additionally, we have a demo of some early prototype work for the arbitrary branching proposal that we talked about at DevConf back in February.

coco-module-rebuild, by jkaluza

This demo shows initial prototype of Continuos Compose Service and first use-case: Rebuild of module in Module Build Service as a result of update of module definition in dist-git.

mbs-build, by jkaluza

This demo shows new 'mbs-build' command which can be used to submit and manage Module Build Service builds.

pdc-sla-apis, by mprahl

This demo shows an API developed to store SLA and EOL information on package branches in Product Definition Center (PDC). The API would act as a gold source for other tools that need to make decisions based on this information. The code shown in this demo is in review and could change before it makes it to master.

View Comments

Factory 2, Sprint 13

Mar 24, 2017 | categories: sprint, factory2, fedora View Comments

The Factory 2.0 team is plugging along on a design phase for our next set of projects in the F27 timeframe.

  • The WaiverDB prototype is done (a key component in our vision for CI/CD). Next steps here are to have our technical fellows review the prototype and make sure it will meet our needs. We'll be aiming for production deployment soon after that (so we can iterate as needed).
  • The design work for Arbitrary Branching and Automated Rebuilds is nearing completion. We'll be planning out prototype work for those in the coming sprints.
  • Our design for PolicyEngine and Module Updates are both still cooking. Expect to see conclusions to their design phases in the next few sprints.

arbitrary-branches-fedora-focus-doc, by mprahl

This demo discusses the concept of “Arbitrary Branching" (package dist-git branches) and how that is a requirement for Fedora 27 to be a fully modularized operating system. It also elaborates on the high-level changes that will need to take place for this to become a reality.

coco-focus-doc, by jkaluza

In this demo, I briefly describe the status of Continus Compose Service focus document.

waiverdb-prototype, by mjia

In this demo I describe the background of WaiverDB, what problems it solved, showing how to use the HTTP APIs to manage the waivers.

View Comments

Factory 2, Sprint 12

Mar 24, 2017 | categories: sprint, factory2, fedora View Comments

Late report here. This sprint actually concluded two weeks ago. We did a combined demo with the Modularity and base-runtime teams, but never published our stuff independently. Better late than never; here it is!

mbs-in-production, by threebean

In this demo I show the Module Build Service in its initial deployment to production, showing how to query the API and how to submit a new module build.

At the request of FESCo, use of the MBS is locked down to only members of the Modularity Working Group until we have an FPC approved module review process in place in the F27 timeframe.

multiple_submit_same_nvr, by fivaldi

When someone tries to submit the module build and some components are submitted for a build and the module build fails but the submitted components are not removed from the Koji, there can be a situation where the same module and component is submitted for a build for second time, but the first Koji build of this component finishes before the second one. This video presents how this is handled via requests to Koji API.

pungi_modular_compose, by jkaluza

View Comments

Factory 2, Sprint 10

Feb 23, 2017 | categories: sprint, factory2, fedora View Comments

The Factory 2.0 team is back from Brno and DevConf. We had two talks to look for, one on Factory 2.0 current work and another done in conjunction with the Modularity team on Modularity itself. Since returning, we've been working with other teams to set our plans for F27 while simultaneously getting the module build service ready for production for F26.

For the MBS we have all the pieces in staging, and we're now working with Patrick Uiterwijk (the Fedora Infra Security Officer) on an audit of the code. At the time of this writing, we have answers and patches to all of the issues. We'll be working with Patrick in the coming days to finish this out.

The broad strokes of our plans for F27 are described in the devconf talk. We have a draft of a more focused, bullet-list of subprojects slated for F27, which we'll be publishing in about a month after sorting out some CI details with Fedora Infrastructure, Fedora QA, and the Atomic folks.

mbs-reuse-component-builds, by mprahl

This demo shows a feature for the Module Build Service which reuses component builds from previous builds of the module if the component and the buildroot haven't changed.

module-lint-on-commit, by threebean

In this demo, I show the check_modulemd[1] check (developed by the base-runtime team) being automatically run in the online taskotron environment[2].

A commit to a module in dist-git is pushed and linting errors are produced in resultsdb (which in turn can be consumed by other systems).

pdc-upgrade, by threebean

Here we show the latest upgraded instance of the Product Definition Center in Fedora[1] with the new /unreleasedvariants/ endpoint[2] for the Module Build Service[3].

View Comments
Next Page »

[three]Bean.org