Factory 2, Sprint 10

Feb 23, 2017 | categories: sprint, factory2, fedora View Comments

The Factory 2.0 team is back from Brno and DevConf. We had two talks to look for, one on Factory 2.0 current work and another done in conjunction with the Modularity team on Modularity itself. Since returning, we've been working with other teams to set our plans for F27 while simultaneously getting the module build service ready for production for F26.

For the MBS we have all the pieces in staging, and we're now working with Patrick Uiterwijk (the Fedora Infra Security Officer) on an audit of the code. At the time of this writing, we have answers and patches to all of the issues. We'll be working with Patrick in the coming days to finish this out.

The broad strokes of our plans for F27 are described in the devconf talk. We have a draft of a more focused, bullet-list of subprojects slated for F27, which we'll be publishing in about a month after sorting out some CI details with Fedora Infrastructure, Fedora QA, and the Atomic folks.

mbs-reuse-component-builds, by mprahl

This demo shows a feature for the Module Build Service which reuses component builds from previous builds of the module if the component and the buildroot haven't changed.

module-lint-on-commit, by threebean

In this demo, I show the check_modulemd[1] check (developed by the base-runtime team) being automatically run in the online taskotron environment[2].

A commit to a module in dist-git is pushed and linting errors are produced in resultsdb (which in turn can be consumed by other systems).

pdc-upgrade, by threebean

Here we show the latest upgraded instance of the Product Definition Center in Fedora[1] with the new /unreleasedvariants/ endpoint[2] for the Module Build Service[3].

View Comments

Factory 2 at DevConf

Feb 08, 2017 | categories: devconf, factory2, fedora View Comments

Just posting a link to share my talk at DevConf.

You can find the slides here.

View Comments

Factory 2, Sprint 8 Report

Jan 16, 2017 | categories: sprint, factory2, fedora View Comments

Happy New Year from the Factory 2.0 team.

Here's a reminder of our current priorities. We are:

  • Preparing elementary build infrastructure for the Fedora 26 Alpha release.
  • Deserializing pipeline processes that could be done more quickly in parallel.
  • Building a dependency chain database, so that we can build smarter rebuild automation and pipeline analytics.
  • Monitoring pipeline performance metrics, so that as we later improve things we can be sure we had an effect.

We are on track with respect to three of the four priorities: module build infrastructure will be ready before the F26 Alpha freeze. Our VMs are provisioned, we're working through the packaging rituals, and we'll be ready for an initial deployment shortly after devconf. Internally, our MvP of resultsdb and resultsdb-updater are working and pulling data from some early-adopter Platform Jenkins masters and our internal performance measurement work is bearing fruit slowly but steadily: we have two key metrics updating automatically on our kibana dashboard, with two more in progress to be completed in the coming sprints.

We have made a conscious decision to put our work on the internal depedency chain database on hold. We're going to defer our deployment to production for a few months to ensure that our efforts don't collide with a separate release engineering project ongoing now.

Tangentially, we're glad to be assisting with the adoption of robosignatory for automatic rpm signing. It's an excellent example of upstream/downstream cooperation between the Fedora and RHEL services teams.

mbs-optimization, by jkaluza

This demo shows optimizations of module build in module build service comparing the diagrams from old and new version of MBS.

resultsdb-updater-pdc-updater-updates, by mprahl

This demo shows the changes in ResultsDB-Updater and how it reflects in ResultsDB. Additionally, progress is shown on pdc-updater working internally.

developers-instance, by fivaldi

In this video, I am presenting local devleoper's instance of MBS using docker-compose. The aim is to provide the simplest way of a custom MBS instance for dev/testing purposes. At the end, I'm showing how to submit a testing module build.

fedpkg-pdc-modulemd, by jkaluza

This demo shows the newly implemented "fedpkg module-build" command workflow and improved storage of module metadata in the PDC.

mbs-scheduler-and-resultsdb-prod, by mprahl

This demo briefly explains the changes in the Module Build Service scheduler to use fedmsg's "Hub-Consumer" approach. Additionally, ResultsDB is briefly shown in production with results populated from ResultsDB-Updater.

module-checks-in-taskotron, by threebean

In this demo, I show how we've wrapped elementary tests produced by the base-runtime team so that they can be executed and managed by the taskotron CI system in place in Fedora Infrastructure. Benefits include:

  • No lock-in to taskotron. Jenkins-job-builder could wrap the core test in a similar way.
  • An avocado-to-resultsdb translator is written which will be generally useful in future sprints.

Work on taskotron-trigger to automatically respond to dist-git events was implemented and merged upstream, but is pending a release and deployment.

View Comments

Factory 2, Sprint 4 Report

Nov 22, 2016 | categories: sprint, factory2, fedora View Comments

Work continues on Factory 2.0...

Recall that we have 1000 different problems we're trying to solve, but we're attempting to focus on an isolated subset for now: problems we've picked so that their solutions can enable higher-level problem solving in the coming months. The work currently includes a focus on:

  • Preparing elementary build infrastructure for the Fedora 26 Alpha release.
  • Deserializing pipeline processes that could be done more quickly in parallel.
  • Building a dependency chain database, so that we can build smarter rebuild automation and pipeline analytics.
  • Monitoring pipeline performance metrics, so that as we later improve things we can be sure we had an effect.

Overall, I'm happy with the work output of this sprint, but there are complications.

  • We made significant progress on the module build infrastructure. The stack is maturing - the number of new features will be slowing as we start to prepare it for production for F26. Our proposals were approved by FESCo last week. We still have a long way to go, but we're currently on target to build an F26 Edition out of the base modules. Status: Green.
  • Our deserialization effort hit some roadbumps as we negotiate how to interface with existing test-execution environments. We're not blocked at the moment and are in dialogue with our partners. We will slip a few weeks on our MvP deliverable here. Status: Yellow.
  • The dependency chain MvP is ready for deployment efforts in Sprint 5. We underestimated how this work would depend on other message bus enablement pre-requisites, which has pushed our delivery date back a few weeks. Status: Yellow.
  • Our performance metrics work is progressing, but much more slowly than we had initially expected. We promised a delivery date of Dec. 1st for this, but we will miss it by a wide margin and need to adjust expectations. The primary factor is that the metrics are just more complex than we anticipated. The pipeline is complex, therefore exacting meaningful measurements is complex. A separate factor is some unpredictability with write-access to the integration lab's ELK instance we are depending on. The delay in work here does not block our other efforts. Status: Red.

mbs-build-profiles, by threebean

Here we show how we're re-using the "installation profiles" feature of modulemd to define the buildsystem "build groups" for a module. It's a natural extension of the install profiles metadata which furthmore helps unblock the base-runtime team in their quest to produce the first generational core!

mbs-config, by fivaldi

In this video I'm trying to explain changes in configuration internals of Module Build Service. This aims to split configuration data/logic so that it will be simplier to use and understand for contributors/users.

modulemd1-mock, by jkaluza

In this demo I inform about the support of modulemd-1.0 format in Module Build Service, describes the build-order feature and shows how the module using this feature builds using the mock builder.

resultsdb-updater, by mprahl

This video shows the new microservice ResultsDB-Updater which listens on the CI message bus for test results and adds them to ResultsDB.

umb-brew, by mikeb

A demo of message publication from Brew to the Unified Message Bus. Explains the topic hierarchy and the message format.

umb-dist-git-demo, by mikeb

A demo showing the publication of messages from a dist-git repo to the Unified Message Bus. Shows how the repos are configured for publication on the server side, and explains a little bit about message format.

View Comments

Factory 2.0, Sprint 3 Report

Nov 08, 2016 | categories: sprint, factory2, fedora View Comments

This was our first full sprint with the new team! Welcome, Jan Kaluza, Courtney Pacheco, Vera Karas, and Stanislav Ochotnicky. We're glad to have Filip Valder join us in sprint 4 starting today.

Our top priority in sprint 3 was making sure that the base runtime team isn't blocked. They have a big job ahead of themselves to curate and build a collection of base modules at the core of the distro, and they need to use our prototype build tooling to do it. Anytime they're blocked, the Factory 2.0 team is trying to chase down the solution -- fixing tracebacks and developing new features. Cheers to Matt Prahl and Jan Kaluza for staying on top of this.

Meanwhile, we're continuing apace with the Dependency Chain and Deserialization epics that we originally scheduled for work this quarter. Mike Bonnet has been chasing down difficult technical pre-requisites for the later (message bus enablement), Matt Prahl demoed his dependency chain web UI, and Courtney Pacheco is giving shape to our metrics project (so we can have some confidence that future pipeline changes we make actually improve the state of affairs).

As always, we're double-tasked with laying the groundwork for work in future quarters. Thanks to Stanislav Ochotnicky for starting the conversation with Platform representatives about workflow changes and continuous integration, and thanks to all those who participated in this round of resultsdb/CI discussions.

f26-changes, by threebean

Here I talk about the Change proposals we've filed with FESCo for the Fedora 26 release.

I talk about this in the video, but the changes are designed specifically to limit the amount of risk we impose on the rest of the release process and the Modularity initiative. We can use the F26 release to assess the viability of Modularity and decide then how far we want to go in the Fedora 27 timeframe.

mbs-mock-backend, by jkaluza

In this demo I describe what is Mock Builder and why it is useful to Module Build Service. I also show it in action briefly.

pdc-tangle-web, by mprahl

In this demo I talk about an Angular2 web app I wrote called PDC Tangle Web. The app queries PDC based on the user's search criteria to show an artifact's dependencies. This is a beta version, so please keep in mind more features will come, but feel free to make suggestions on GitHub for what you'd like to see added.

resultsdb, by threebean

This one's just pointing you to a blog post that is a transcription of an internal document on resultsdb and our plans for it.

I know that showing off a blog post isn't all that exciting, but I'm glad we've gotten to this point. It is the product of a long process of discussion and eventual agreement.

View Comments

Next Page ยป