[three]Bean

Hello all. Bi-weekly update from the Factory 2 team on our work here. We have three videos this sprint.

Two are related to the module-build-service: both on submitting builds. We have some new client tooling to show which should make manual submission and monitoring of module-builds much simpler. The second is a demo of an early prototype of our continuous rebuild system.

Additionally, we have a demo of some early prototype work for the arbitrary branching proposal that we talked about at DevConf back in February.

coco-module-rebuild, by jkaluza

This demo shows initial prototype of Continuos Compose Service and first use-case: Rebuild of module in Module Build Service as a result of update of module definition in dist-git.

mbs-build, by jkaluza

This demo shows new 'mbs-build' command which can be used to submit and manage Module Build Service builds.

pdc-sla-apis, by mprahl

This demo shows an API developed to store SLA and EOL information on package branches in Product Definition Center (PDC). The API would act as a gold source for other tools that need to make decisions based on this information. The code shown in this demo is in review and could change before it makes it to master.

The Factory 2.0 team is plugging along on a design phase for our next set of projects in the F27 timeframe.

• The WaiverDB prototype is done (a key component in our vision for CI/CD). Next steps here are to have our technical fellows review the prototype and make sure it will meet our needs. We'll be aiming for production deployment soon after that (so we can iterate as needed).
• The design work for Arbitrary Branching and Automated Rebuilds is nearing completion. We'll be planning out prototype work for those in the coming sprints.
• Our design for PolicyEngine and Module Updates are both still cooking. Expect to see conclusions to their design phases in the next few sprints.

arbitrary-branches-fedora-focus-doc, by mprahl

This demo discusses the concept of “Arbitrary Branching" (package dist-git branches) and how that is a requirement for Fedora 27 to be a fully modularized operating system. It also elaborates on the high-level changes that will need to take place for this to become a reality.

coco-focus-doc, by jkaluza

In this demo, I briefly describe the status of Continus Compose Service focus document.

waiverdb-prototype, by mjia

In this demo I describe the background of WaiverDB, what problems it solved, showing how to use the HTTP APIs to manage the waivers.

Late report here. This sprint actually concluded two weeks ago. We did a combined demo with the Modularity and base-runtime teams, but never published our stuff independently. Better late than never; here it is!

mbs-in-production, by threebean

In this demo I show the Module Build Service in its initial deployment to production, showing how to query the API and how to submit a new module build.

At the request of FESCo, use of the MBS is locked down to only members of the Modularity Working Group until we have an FPC approved module review process in place in the F27 timeframe.

multiple_submit_same_nvr, by fivaldi

When someone tries to submit the module build and some components are submitted for a build and the module build fails but the submitted components are not removed from the Koji, there can be a situation where the same module and component is submitted for a build for second time, but the first Koji build of this component finishes before the second one. This video presents how this is handled via requests to Koji API.

pungi_modular_compose, by jkaluza

The Factory 2.0 team is back from Brno and DevConf. We had two talks to look for, one on Factory 2.0 current work and another done in conjunction with the Modularity team on Modularity itself. Since returning, we've been working with other teams to set our plans for F27 while simultaneously getting the module build service ready for production for F26.

For the MBS we have all the pieces in staging, and we're now working with Patrick Uiterwijk (the Fedora Infra Security Officer) on an audit of the code. At the time of this writing, we have answers and patches to all of the issues. We'll be working with Patrick in the coming days to finish this out.

The broad strokes of our plans for F27 are described in the devconf talk. We have a draft of a more focused, bullet-list of subprojects slated for F27, which we'll be publishing in about a month after sorting out some CI details with Fedora Infrastructure, Fedora QA, and the Atomic folks.

mbs-reuse-component-builds, by mprahl

This demo shows a feature for the Module Build Service which reuses component builds from previous builds of the module if the component and the buildroot haven't changed.

module-lint-on-commit, by threebean

In this demo, I show the check_modulemd[1] check (developed by the base-runtime team) being automatically run in the online taskotron environment[2].

A commit to a module in dist-git is pushed and linting errors are produced in resultsdb (which in turn can be consumed by other systems).

• [1] https://github.com/fedora-modularity/check_modulemd/
• [2] https://taskotron-dev.fedoraproject.org

pdc-upgrade, by threebean

Here we show the latest upgraded instance of the Product Definition Center in Fedora[1] with the new /unreleasedvariants/ endpoint[2] for the Module Build Service[3].

• [1] https://pdc.fedoraproject.org/
• [2] https://pdc.fedoraproject.org/rest_api/v1/unreleasedvariants/
• [3] https://fedoraproject.org/wiki/Changes/ModuleBuildService

Happy New Year from the Factory 2.0 team.

Here's a reminder of our current priorities. We are:

• Preparing elementary build infrastructure for the Fedora 26 Alpha release.
• Deserializing pipeline processes that could be done more quickly in parallel.
• Building a dependency chain database, so that we can build smarter rebuild automation and pipeline analytics.
• Monitoring pipeline performance metrics, so that as we later improve things we can be sure we had an effect.

We are on track with respect to three of the four priorities: module build infrastructure will be ready before the F26 Alpha freeze. Our VMs are provisioned, we're working through the packaging rituals, and we'll be ready for an initial deployment shortly after devconf. Internally, our MvP of resultsdb and resultsdb-updater are working and pulling data from some early-adopter Platform Jenkins masters and our internal performance measurement work is bearing fruit slowly but steadily: we have two key metrics updating automatically on our kibana dashboard, with two more in progress to be completed in the coming sprints.

We have made a conscious decision to put our work on the internal depedency chain database on hold. We're going to defer our deployment to production for a few months to ensure that our efforts don't collide with a separate release engineering project ongoing now.

Tangentially, we're glad to be assisting with the adoption of robosignatory for automatic rpm signing. It's an excellent example of upstream/downstream cooperation between the Fedora and RHEL services teams.

mbs-optimization, by jkaluza

This demo shows optimizations of module build in module build service comparing the diagrams from old and new version of MBS.

resultsdb-updater-pdc-updater-updates, by mprahl

This demo shows the changes in ResultsDB-Updater and how it reflects in ResultsDB. Additionally, progress is shown on pdc-updater working internally.

developers-instance, by fivaldi

In this video, I am presenting local devleoper's instance of MBS using docker-compose. The aim is to provide the simplest way of a custom MBS instance for dev/testing purposes. At the end, I'm showing how to submit a testing module build.

fedpkg-pdc-modulemd, by jkaluza

This demo shows the newly implemented "fedpkg module-build" command workflow and improved storage of module metadata in the PDC.

mbs-scheduler-and-resultsdb-prod, by mprahl

This demo briefly explains the changes in the Module Build Service scheduler to use fedmsg's "Hub-Consumer" approach. Additionally, ResultsDB is briefly shown in production with results populated from ResultsDB-Updater.

module-checks-in-taskotron, by threebean

In this demo, I show how we've wrapped elementary tests produced by the base-runtime team so that they can be executed and managed by the taskotron CI system in place in Fedora Infrastructure. Benefits include:

• No lock-in to taskotron. Jenkins-job-builder could wrap the core test in a similar way.
• An avocado-to-resultsdb translator is written which will be generally useful in future sprints.

Work on taskotron-trigger to automatically respond to dist-git events was implemented and merged upstream, but is pending a release and deployment.