How continuous integration can help you keep pace with the Linux kernel

Almost all of Collabora's customers use the Linux kernel on their products. Often they will use the exact code as delivered by the SBC vendors and we'll work with them in other parts of their software stack. But it's becoming increasingly common for our customers to adapt the kernel sources to the specific needs of their particular products.

A very big problem most of them have is that the kernel version they based on isn't getting security updates any more because it's already several years old. And the reason why companies are shipping kernels so old is that they have been so heavily modified compared to the upstream versions, that rebasing their trees on top of newer mainline releases is so expensive that is very hard to budget and plan for it.

To avoid that, we always recommend our customers to stay close to their upstreams, which implies rebasing often on top of new releases (typically LTS releases, with long term support). For the budgeting of that work to become possible, the size of the delta between mainline and downstream sources needs to be manageable, which is why we recommend contributing back any changes that aren't strictly specific to their products.

But even for those few companies that already have processes in place for upstreaming their changes and are rebasing regularly on top of new LTS releases, keeping up with mainline can be a substantial disruption of their production schedules. This is in part because new bugs will be in the new mainline release, and new bugs will be in the downstream changes as they get applied to the new version.

Those companies that are already keeping close to their upstreams typically have advanced QA infrastructure that will detect those bugs long before production, but a long stabilization phase after every rebase can significantly slow product development.

To improve this situation and encourage more companies to keep their efforts close to upstream we at Collabora have been working for a few years already in continuous integration of FOSS components across a diverse array of hardware. The initial work was sponsored by Bosch for one of their automotive projects, and since the start of 2016 Google has been sponsoring work on continuous integration of the mainline kernel.

One of the major efforts to continuously integrate the mainline Linux kernel codebase is kernelci.org, which builds several configurations of different trees and submits boot jobs to several labs around the world, collating the results. This is being of great help already in detecting at a very early stage any changes that either break the builds, or prevent a specific piece of hardware from completing the boot stage.

Though kernelci.org can easily detect when an update to a source code repository has introduced a bug, such updates can have several dozens of new commits, and without knowing which specific commit introduced the bug, we cannot identify culprits to notify of the problem. This means that either someone needs to monitor the dashboard for problems, or email notifications are sent to the owners of the repositories who then have to manually look for suspicious commits before getting in contact with their author.

To address this limitation, Google has asked us to look into improving the existing code for automatic bisection so it can be used right away when a regression is detected, so the possible culprits are notified right away without any manual intervention.

Another area in which kernelci.org is currently lacking is in the coverage of the testing. Build and boot regressions are very annoying for developers because they impact negatively everybody who work in the affected configurations and hardware, but the consequences of regressions in peripheral support or other subsystems that aren't involved critically during boot can still make rebases much costlier.

At Collabora we have had a strong interest in having the DRM subsystem under continuous integration and some time ago started a R&D project for making the test suite in IGT generically useful for all the DRM drivers. IGT started out being i915-specific, but as most of the tests exercise the generic DRM ABI, they could as well test other drivers with a moderate amount of effort. Early in 2016 Google started sponsoring this work and as of today submitters of new drivers are using it to validate their code.

Another related effort has been the addition to DRM of a generic ABI for retrieving CRCs of frames from different components in the graphics pipeline, so two frames can be compared when we know that they should match. And another one is adding support to IGT for the Chamelium board, which can simulate several display connections and hotplug events.

A side-effect of having continuous integration of changes in mainline is that when downstreams are sending back changes to reduce their delta, the risk of introducing regressions is much smaller and their contributions can be accepted faster and with less effort.

We believe that improved QA of FOSS components will expand the base of companies that can benefit from involvement in development upstream and are very excited by the changes that this will bring to the industry. If you are an engineer who cares about QA and FOSS, and would like to work with us on projects such as kernelci.org, LAVA, IGT and Chamelium, get in touch!

More acceleration support for video in webkit-clutter

The last few days I have been working on reworking how video gets rendered inside a web page in the Clutter port of WebKit, using Clutter-GStreamer. This work is contributed by Collabora and sponsored by Bosch.

Previously, we were doing the same as the Gtk+ port, that is, rendering each frame to a Cairo texture (with whatever format conversion that was needed) which would be then composited along the rest of the page (with Cairo as well). On the Clutter port that big surface would be uploaded to a texture in the GPU side of things and composed along the rest of the stage.

Joone and Gustavo have been working lately on adding accelerated compositing support to the Clutter port (and both will be blogging in the next few days about this work), which means that the page gets split in layers that get sent individually to the GPU where they get composited. Layers can be cached so only the ones that changed need to be re-rendered and re-uploaded.

This reduces dramatically the amount of work that happens on the CPU, so that animations with complex pages are feasible, even those involving video elements during playback, such as this:



Because page composition happens now in the GPU instead of in Cairo, we don't need to copy each frame to a Cairo surface (saving a copy) and don't need either to change the pixel format from whatever GStreamer gave us to whatever Cairo wants.

Another benefit of using Clutter-GStreamer is that alpha pre-multiplication can happen in the GPU (implemented as a shader), saving more CPU.

Furthermore (and more importantly), we can use standard GStreamer elements that use the hardware for decoding. Given the proper hardware and driver support, frames could be placed directly where the GPU can find them when compositing the page.

Hopefully this will make the Clutter port of WebKit more appealing for mobile platforms, where low-power GPUs are commonly available but the CPU is often taxed by other tasks.

All the code is here: https://gitorious.org/webkit-clutter/webkit-clutter/commits/ac2

Thanks to Bosch and Collabora for allowing me working on this. To know more about Collabora's involvement in WebKit: http://www.collabora.com/projects/webkit/

WebKit, Clutter and companies giving back

Time passes by and another project nears its end. For the past several months I have worked on a project for Bosch involving Clutter, Mx and WebKit and have had the pleasure of interacting daily with the upstream communities.

I feel I have been lucky because many of the companies that are starting to base their products on FOSS fail to understand the dynamics that will allow them to make the most from the components they are reusing. In this particular case, Bosch has understood that a big part of the value of FOSS resides in the possibility of adapting the software to their specific needs, and also in sharing the burden of maintaining the software with other organizations.

For that to be practical, you have to resist the urge to settle at the beginning of the project on specific stable releases, because if you do, you not only will miss new features and many bug fixes that will land only in the unstable branch, but you will have a much harder time integrating upstream whatever modifications you make. This situation often leads to companies having to maintain forks of the components they had to modify, and that's when the costs start spiraling up. David Neary wrote recently about in The Cost of Going it Alone.

Additionally, by working within the upstream community you will be doing so side-by-side with the people that are most knowledgeable about that specific component, either because they have been working on it for a long time, or quite often, because they are its creators. This is going to massively increase the chances that whatever modifications you end up doing are done in the best way and also that they will be integrated upstream and become maintained by other parties.

Besides technological soundness and reduced maintenance costs, there was an additional factor that made even more important the collaboration with upstream: Bosch had devised a new user experience that pushed the limits of the UI toolkit because of being innovative and because of scalability reasons. And understandably, they didn't want to compromise there because that innovative UI brought a lot of value to their product.

Getting into a FOSS community can be tough sometimes (specially if it's your first time), but in this case I have had the luck of finding a very open community in Clutter, well resourced and with a clear idea of how to deal with those contributions that can bring the most to the project. I take this opportunity to congratulate and thank the colleagues at the Clutter team at Intel OTC: Emmanuele Bassi, Robert Bragg, Neil Roberts, Thomas Wood, Brian Peel and Chris Lord (now at Mozilla).

Here comes a summary of the contributions sponsored by Collabora on behalf of Bosch:

- 16 commits to Clutter, most notably adding the base for gesture-based actions and several improvements to ClutterEffect subclasses.

- 16 commits to Mx, mainly on MxKineticScrollView

- 5 commits to WebKit's Clutter port, the most notable change was allowing the user to explicitly load images

- Filed 30 bugs in total

Besides thanking the individuals mentioned above, I would like to send big kudos to Bosch, Collabora and Intel for the courage and insight shown when investing in this new way of working together that we call Open Source.

You can read more about the Clutter port of WebKit here and about Collabora's involvement in WebKit here.