A Panfrost milestone

The video

Below you can see glmark2 running as a Wayland client in Weston, on a NanoPC -T4 (so a RK3399 SoC with a Mali T-864 GPU)). It's much smoother than on the video, which is limited to 5FPS by the webcam.

Weston is running with the DRM backend and the GL renderer.

The history behind it

For more than 10 years, at Collabora we have been happily helping our customers to make the most of their hardware by running free software.

One area some of us have specially enjoyed working on has been open drivers for GPUs, which for a long time have been considered the next frontier in the quest to have a full software platform that companies and individuals can understand, improve and fix without having to ask for permission first.

Something that has saddened me a bit has been our reduced ability to help those customers that for one reason or another had chosen a hardware platform with ARM Mali GPUs, as no open driver was available for those.

While our biggest customers were able to get a high level of support from the vendors in order to have the Mali graphics stack well integrated with the rest of their product, the smaller ones had a much harder time in achieving that level of integration, which manifested in reduced performance, increased power consumption and slipped milestones.

That's why we have been following with great interest the several efforts that aimed to come up with an open driver for GPUs in the Mali family, one similar to those already existing for Qualcomm, NVIDIA and Vivante.

At XDC last year we had the chance of meeting the people involved in the latest effort to develop such a driver: Panfrost. And in the months that followed I made some room in my backlog to come up with a plan to give the effort a boost.

At that point, Panfrost was only able to get its bits in the screen by an elaborate hack that involved copying each frame into a X11 SHM buffer, which besides making the setup of the development environment much more cumbersome, invalidated any performance analysis. It also limited testing to demos such as glmark2.

Due to my previous work on Etnaviv I was already familiar with the abstractions in Mesa for setups in which the display of buffers is performed by a device different from the GPU so it was just a matter of seeing how we could get the kernel driver for the Mali GPU to play well with the rest of the stack.

So during the past month or so I have come up with a proper implementation of the winsys abstraction that makes use of ARM's kernel driver. The result is that now developers have a better base on which to work on the rendering side of things.

By properly creating, exporting and importing buffers, we can now run applications on GBM, from demos such as kmscube and glmark2 to compositors such as Weston, but also big applications such as Kodi. We are also supporting zero-copy display of GPU-rendered clients in Weston.

This should make it much easier to work on the rendering side of things, and work on a proper DRM driver in the mainline kernel can proceed in parallel.

For those interested in joining to the effort, Alyssa has graciously taken the time to update the instructions to build and test Panfrost. You can join us at #panfrost in Freenode and can start sending merge requests to Gitlab.

Thanks to Collabora for sponsoring this work and to Alyssa Rosenzweig and Lyude Paul for their previous work and for answering my questions.

Slides on the Chamelium board

Yesterday I gave a short talk about the Chamelium board from the ChromeOS team, and thought that the slides could be useful for others as this board gets used more and more outside of Google.

If you are interested in how this board can help you automate the testing of your display (and not only!) code and hardware, a new mailing list has been created to discuss its uses. We at Collabora will be happy to help you integrate this board in your CI lab as well.

Thanks go to Intel for sponsoring the preparation of these slides and for allowing me to share them under an open license.

And of course, thanks to Google's ChromeOS team for releasing the hardware design with an open hardware license along with the code they are running on it and with it.

How continuous integration can help you keep pace with the Linux kernel

Almost all of Collabora's customers use the Linux kernel on their products. Often they will use the exact code as delivered by the SBC vendors and we'll work with them in other parts of their software stack. But it's becoming increasingly common for our customers to adapt the kernel sources to the specific needs of their particular products.

A very big problem most of them have is that the kernel version they based on isn't getting security updates any more because it's already several years old. And the reason why companies are shipping kernels so old is that they have been so heavily modified compared to the upstream versions, that rebasing their trees on top of newer mainline releases is so expensive that is very hard to budget and plan for it.

To avoid that, we always recommend our customers to stay close to their upstreams, which implies rebasing often on top of new releases (typically LTS releases, with long term support). For the budgeting of that work to become possible, the size of the delta between mainline and downstream sources needs to be manageable, which is why we recommend contributing back any changes that aren't strictly specific to their products.

But even for those few companies that already have processes in place for upstreaming their changes and are rebasing regularly on top of new LTS releases, keeping up with mainline can be a substantial disruption of their production schedules. This is in part because new bugs will be in the new mainline release, and new bugs will be in the downstream changes as they get applied to the new version.

Those companies that are already keeping close to their upstreams typically have advanced QA infrastructure that will detect those bugs long before production, but a long stabilization phase after every rebase can significantly slow product development.

To improve this situation and encourage more companies to keep their efforts close to upstream we at Collabora have been working for a few years already in continuous integration of FOSS components across a diverse array of hardware. The initial work was sponsored by Bosch for one of their automotive projects, and since the start of 2016 Google has been sponsoring work on continuous integration of the mainline kernel.

One of the major efforts to continuously integrate the mainline Linux kernel codebase is kernelci.org, which builds several configurations of different trees and submits boot jobs to several labs around the world, collating the results. This is being of great help already in detecting at a very early stage any changes that either break the builds, or prevent a specific piece of hardware from completing the boot stage.

Though kernelci.org can easily detect when an update to a source code repository has introduced a bug, such updates can have several dozens of new commits, and without knowing which specific commit introduced the bug, we cannot identify culprits to notify of the problem. This means that either someone needs to monitor the dashboard for problems, or email notifications are sent to the owners of the repositories who then have to manually look for suspicious commits before getting in contact with their author.

To address this limitation, Google has asked us to look into improving the existing code for automatic bisection so it can be used right away when a regression is detected, so the possible culprits are notified right away without any manual intervention.

Another area in which kernelci.org is currently lacking is in the coverage of the testing. Build and boot regressions are very annoying for developers because they impact negatively everybody who work in the affected configurations and hardware, but the consequences of regressions in peripheral support or other subsystems that aren't involved critically during boot can still make rebases much costlier.

At Collabora we have had a strong interest in having the DRM subsystem under continuous integration and some time ago started a R&D project for making the test suite in IGT generically useful for all the DRM drivers. IGT started out being i915-specific, but as most of the tests exercise the generic DRM ABI, they could as well test other drivers with a moderate amount of effort. Early in 2016 Google started sponsoring this work and as of today submitters of new drivers are using it to validate their code.

Another related effort has been the addition to DRM of a generic ABI for retrieving CRCs of frames from different components in the graphics pipeline, so two frames can be compared when we know that they should match. And another one is adding support to IGT for the Chamelium board, which can simulate several display connections and hotplug events.

A side-effect of having continuous integration of changes in mainline is that when downstreams are sending back changes to reduce their delta, the risk of introducing regressions is much smaller and their contributions can be accepted faster and with less effort.

We believe that improved QA of FOSS components will expand the base of companies that can benefit from involvement in development upstream and are very excited by the changes that this will bring to the industry. If you are an engineer who cares about QA and FOSS, and would like to work with us on projects such as kernelci.org, LAVA, IGT and Chamelium, get in touch!

Validating changes to KMS drivers with IGT

New DRM drivers are being added to almost each new kernel release, and because the mode setting API is so rich and complex, bugs do slip in that translate to differences in behaviour between drivers.

There have been previous attempts at writing test suites for validating changes and preventing regressions, but they have typically happened downstream and focused on the specific needs of specific products and limited to one or at most a few of different hardware platforms.

Writing these tests from scratch would have been an enormous amount of work, and gathering previous efforts and joining them wouldn't be much worth it because they were written using different test frameworks and in different programming languages. Also, there would be great overlap on the basic tests, and little would remain of the trickier stuff.

Of the existing test suites, the one with most coverage is intel-gpu-tools, used by the Intel graphics team. Though a big part is specific to the i915 driver, what uses the generic APIs is pretty much driver-independent and can be made to work with the other drivers without much effort. Also, Broadcom's Eric Anholt has already started adding tests for IOCTLs specific to the VideoCore-IV driver.

Collabora's Micah Fedke and Daniel Stone had added a facility for selecting DRM device files other than i915's and I improved the abstraction for creating buffers so it works for drivers without GEM buffers. Next I removed a bunch of superfluous dependencies on i915-only stuff and got a useful subset of tests to run on a Radxa Rock2 board (with the Rockchip 3288 SoC). Around half of these patches have been merged already and the other half are awaiting review. Meanwhile, Collabora's Robert Foss is running the ported tests on a Raspberry Pi 2 and has started sending patches to account for its peculiarities.

The next two big chunks of work are abstracting CRC checksums of frames (on drivers other than i915 this could be done with Google's Chamelium or with a board similar to Numato Opsis), and the buffer management API from libdrm that is currently i915-only (bufmgr). Something that will have to be dealt with in the future is abstracting the submittal of specific loads on the GPU as that's currently very much driver-specific.

Additionally, I will be scheduling jobs in our LAVA instance to run these tests on the boards we have in there.

Thanks to Google for sponsoring my time, to the Intel OTC folks for their support and reviews, and to Collabora for sponsoring Robert's, Micah's and Daniel's time.

Lucid sleep in the free desktop

For the past year I have been working on the kernel side to bring some ChromeOS features to upstream.

One of the areas I'm currently working on is what Google calls Lucid Sleep, which is basically the ability of performing work while the machine is in a low power state such as suspend. I'm writing this blog post because there has been interest on this in different communities and the discussion is currently a bit dispersed.

Small mobile devices have been able to do that since basically always and this feature brings it to bigger devices that traditionally have been either on or off. It's similar to what Microsoft calls InstantGo (previously Connected Standby).

Childhood Dream by theSong / CC BY-NC-ND 3.0

A few examples of tasks that the system could perform while apparently sleeping are:

• Checking if the battery level is so low that it would be better to completely power down the machine
• Starting a network backup if the present connectivity allows it (a known access point may have become accessible)
• Downloading email
• Checking for new instant messages

With regards to functionality and leaving performance considerations aside, userspace could implement this without requiring any new support in the kernel as illustrated in this scenario:

• We assume that a video is currently playing in YouTube
• User closes the lid
• PM daemon notifies userspace of an impending sleep
• Browser pauses playback
• Compositor switches off the screen
• Kernel freezes userspace, suspends devices and puts the CPUs to idle
• Time passes...
• RTC alarm fires off
• Kernel resumes devices and unfreezes userspace
• Userspace realizes there hasn't been any user activity since it went to sleep last, so stays in "dark resume" mode
• Userspace does any lucid tasks it wants, then goes back to sleep again
• Kernel freezes userspace, suspends devices and puts the CPUs to idle
• Time passes...
• User opens lid
• Kernel resumes devices and unfreezes userspace
• PM daemon notices the SW_LID event, so notifies userspace that this is a full-on resume
• Compositor switches screen on
• Browser resumes playback

No changes needed in the kernel is always good news, but there's two issues.

Lost input events

Sometimes the event from the input device that woke the system up gets lost before it reaches userspace, so we don't know if we can stay dark and do our lucid stuff, or if the user expects the machine to power completely on.

This is in any case a bug, but if it needs to be fixed in the firmware, we may not be able to do much about it. At most we could get the kernel to synthesize an input event, but sometimes it may not have enough information to do so.

Performance

When the system wakes up, there tends to be a lot to do in the kernel and userspace, so it could take several seconds for the screen to come up from the moment the user opened the lid in the scenario presented above.

For ChromeOS this isn't acceptable so they are carrying some patches in their kernel that make some shortcuts possible (the screen is left on at suspend time, and the kernel knows at resume time whether it has to power it on based on which was the wakeup source, thus not having to wait for userspace).

Fortunately, there have been some changes recently in the kernel PM subsystem that can speed up resumes quite a bit and we can make use of them to offset the penalty of dropping those shortcuts.

The first is idling the CPUs instead of suspending to firmware, which on modern SoCs should be quite efficient and much faster, by a few tenths of seconds.

The other is to leave idle devices that are already in a low power state alone when suspending, which means that we don't have to wait for them to resume when the system wakes up. In every system I have seen there's always a few devices that take a long time to resume, so this can shave several tenths of seconds from the total resume time.

Both need some amount of support in either the platform or in device drivers, and that's what I'm currently working on for the Tegra-based Chromebooks.

Dynamic scaling of the memory bus

The problem

These days there's quite good support for CPU scaling in the mainline kernel, and many ARM SoCs are making use of it already. But in modern hardware with lots of very fast external memory, running the memory bus at its maximum frequency drastically reduces the amount of time that the device can run when on battery.

A problem that many teams are finding when trying to upstream their power management code is that there's currently no way for several clock consumers to influence the frequency of the memory bus. There has been a few tries to upstream the solutions currently in vendor trees, but so far no acceptable solution has been found.

I'm helping to upstream some of the stuff in the ChromeOS tree, and this issue is currently blocking very interesting work from reaching mainline.

The past

In the vendor tree for Tegra this is addressed by creating virtual clocks that are child of the clock that wants to be influenced. Depending on the type of the virtual clock, setting its rate will influence the rate of its parent clock by setting a floor or ceiling value.

In Qualcomm's vendor tree for the Snapdragon family of SoCs, the concept of a voter clock is introduced. Drivers can vote on the rate of a given clock by "voting" through a child clock, so not that different to how Tegra does it.

Both approaches have the critical disadvantage of adding clk instances for things that aren't real clocks, thus making the API considerably more confusing for relatively little gain.

Both vendor trees have additional API for registering bandwidth needs: tegra_isomgr and msm_bus_scale. They bear quite some resemblance with each other and with pm_qos_interface, but both are tightly tied to specificities of their platforms.

The discussion was brought back to life a couple of months ago when a patch was posted for allowing the tegra-drm driver to set the frequency rate of the external memory controller based on the amount of bandwidth that was needed by the display controller for refreshing the display. Of course, that patch was rejected because there are other components that need to have a say in the frequency rate of the memory bus.

But in that discussion some kind of plan took form and I have been working on making something from it that can be merged upstream.

A possible future

There's so far two main additions to existing frameworks, with the rationale being explained further below:

• Add per-user floor and ceiling constraints to the Common Clock Framework, so drivers can set maximum and minimum frequency rates that the clock should respect. Patchset here.
• Add a PM_QOS_MEMORY_BANDWIDTH class to pm_qos, for drivers to register their expected bandwidth needs. Patchset here.

The idea is for the following agents to be able to influence the current frequency of the memory bus:

• Thermal: a cooling device would call clk_set_ceiling_rate to cap the memory bus to a frequency based on the current temperature.
• Power: a battery driver would set a ceiling in the same way, based on the remaining capacity.
• Devfreq: a devfreq driver wrapping a power management unit such as the ACTMON on Tegra or the PPMU on Exynos would set a floor frequency based on the current load stats.
• Cpufreq: a cpufreq driver would set a floor frequency based on the current CPU frequency.
• Devices that can anticipate how much memory bandwidth will need (such as the display controller, the camera, multimedia codecs, an ISP, USB, etc) would register their requirements in the PM_QOS_MEMORY_BANDWIDTH class. The EMC driver would be listening for notifications and setting a floor frequency based on the aggregated bandwidth that is needed.

The impression so far is that this approach matches the needs of the Tegra and Exynos SoCs, and people working on Rockchip upstreaming are evaluating it. Others working on other SoCs are very welcome to look at it and comment, so the result is also useful to them and they can improve their power management in mainline without having to refactor things later.

Multi-touch gestures in Mutter-based compositors

A customer has asked for documentation on handling multi-touch gestures in their Mutter-based compositor (see my previous post) and I thought that it could be a good idea to have it in the GNOME wiki, in case it helps when they are added to GNOME Shell:

Multi-touch gestures in Mutter-based compositors

I'm not sure of what would be the best use for multi-touch gestures in the Shell, probably for resizing windows (with a 4-finger pinch gesture) or for switching desktops (with a 3 or 4 finger swipe). Probably some ideas can be taken from the multi-tasking gestures in recent versions of iOS, such as using pinch gestures to activate hidden panels or to switch to another views.

Something I feel strongly is about restricting system-wide gestures to more than 3 fingers, because the user experience and the implementation gets quite complicated if the compositor and the applications need to compete for touch sequences in similar gestures.

It's currently a bit convoluted due to zero support in Mutter for touch events, but once the Shell starts using touch events, I think it will make sense to move some of the setup and boilerplate into Mutter.

Once more, thanks to my employer Collabora for sponsoring this work:

Multi-touch gestures in Gnome Shell

I'm keeping this small branch in which touch event support is added to Mutter. Plug-ins can register to get touch events before any other client and then accept or reject touch sequences depending on whether the shell is interested on the gesture or not.

As can be seen in this example of a mutter plug-in, any subclass of ClutterGestureAction can be used, which includes gesture recognizers for pan, zoom and rotate actions, but creating new recognizers is pretty easy.

The mutter branch is up for reviewing in bugzilla and any comments on the approach will be very welcome. And if anybody wants to play with multi-touch gestures in Gnome Shell, please link to your work from the wiki so we can track it.

If anybody from the design team has already started thinking about this, I would be very glad to hear their thoughts on this.

As always, I'm grateful to my employer Collabora for sponsoring this work, and I hope GNOME benefits from it.

Multi-touch in WebKit-Clutter

Following my past work on multi-touch support in Clutter, have been playing lately in implementing the W3C Touch Events API in the Clutter port of WebKit.

A lot of code can be reused from WebCore without problems, but we'll need to do some mildly complex event translation because the W3C API and the one in Clutter (and in XInput and in Gtk+) are very different.

But for now, a quick demo of a web page drawing the touch events that it receives, limited to 2 touch points because that's the maximum supported by the hardware I have here:


This is still early work, but once event translation is done, this should be very close to be feature-complete. And a nice side-effect is that given that the touch API in Clutter is so similar to Gtk+'s, it should be pretty straightforward to port it to WebKitGtk+. You can find the code here, but please keep in mind that this is very preliminary work.

As usual, thanks to my employer Collabora for sponsoring this work.

WebGL support in WebKit-Clutter: Part 2

Following up on my work from last december, I have a branch of webkit-clutter that runs most of the WebGL demos that float around the web. The main differences are that the required new API has been accepted into Cogl and that there's a lot more code shared with the other WebKit ports.

Right now bigger applications are not working because of this bug which also affects the Gtk+ port, but hopefully it will be fixed soon.

The code lives in this branch and you will need this Cogl branch with a recent Clutter 1.10 release.

As before, thanks to my employer Collabora for allowing to work on this.

Multi-touch support in Clutter

Have been doing some tests regarding multi-touch support on Clutter and everything seems to be working fine:



You will need either a very recent master or clutter-1.10 branch because there was a bit of fixing needed, but hopefully that will get into a release soon.

That example can be find in ./tests/interactive/test-touch-events.c and can be run as:

$ cd clutter
$ ./tests/interactive/test-touch-events

Needless to say, you will need a recent Xorg (1.12) and support in the kernel for your multi-touch hardware.

Hope that soon I will be able to work on gestures support and automated tests for this.

Thanks for my employer Collabora for sponsoring this work.

Progress on generating Python API docs from introspection

The Docs hackfest in Brno is nearing to an end and I think enough progress has happened on generating API reference from introspection to warrant a report.

To see the current state, check the Telepathy-GLib reference generated here for Python.

Tomorrow morning will work on fixing some issues in the GIR files such as enum docs not making to it and then I will head home.

I'm not likely to work further on this, so if you think it can be of use to you, please consider lending a hand and contribute bugfixes and features. And if you don't like the design or the layout, just change it :)

Hope you have as much fun with this as I have had!

Once more, thanks to the GNOME Foundation, Red Hat and Collabora for sponsoring this work.

Below you can get a glance of what has been progressed on during these days, here is the changelog:

• Add some more stuff to Enum pages
• Display the hierarchy tree for classes
• Install the Mako template files
• Add AM_CHECK_PYMOD macro and use it to check for Mako
• Remove stale DocBook files
• Properly link to property pages
• Fix names of xref links
• Move constructor example to follow the class doc body
• Properly link to signal pages
• Fix the id of signal pages for Python
• Qualify the class name in the constructor with the module name
• Add links to properties in the constructor
• Fix link
• Show how to construct instances in the Class page
• Add details for properties in Python
• Add param information to signal pages in Python
• Implement intra-namespace cross references
• Use instances of MallardFormatter* instead of class methods
• Remove some occurrences of empty Returns: lines
• use True and False instead of %TRUE and %FALSE
• Remove module and class from the function signature
• Use template caching
• Use None instead of %NULL
• Use the ctype for the return type in signatures
• Display Returns: even if there's no args
• Update expected, some whitespace was removed
• Add Since: tag to classes
• Add Since: tag to mallard docs
• Add function argument docs to the C template
• Add argument docs to the Python template
• Don't instantiate MallardFormatter for unknown languages
• Update expected mallard for the doc tests

Documentation hackfest in Brno 2012

Tomorrow I'm taking the train to Brno to participate in the documentation hackfest, with plans to continue the work on generating API documentation from the GIR files that are generated by GObject Introspection.

I will be continuing the work that a bunch of us started in Berlin during the GObject Introspection hackfest last year.

Will be giving as well a workshop on GNOME application development with Python and GObject Introspection as part of the Developer Conference 2012 that Red Hat is organizing there.

Thanks to the GNOME Foundation for sponsoring my accommodation, to my employer Collabora for sponsoring my time there and Red Hat for sponsoring the venue and the food.

PyGObject gets a splash of new blood

Things as they are, few hackers get to work on the same project their whole life. No matter how deep one has been involved in a project, and how much effort was put in, there are many reasons why at some point one may decide to distance from it.

For different reasons, the people who have been maintaining PyGObject for the last couple of years (since the move to introspection) aren't currently using it much any more, which isn't ideal because it means they can allocate less time to maintenance and also lack the guidance of their own motivation.

Fortunately for PyGObject, a bunch of heroic hackers have stepped forward to take the responsibility of maintaining it:

• Martin Pitt
• Paolo Borelli
• Ignacio Casal Quintero
• Sebastian Pölsterl

For now I remain listed in the .doap file, but as I'm not using PyGObject myself any more (even though Collabora sponsors some of my time working on PyGObject), my involvement will be limited to occasional patches and code reviews as I find time.

My thanks and admiration to those who have maintained PyGObject in the past:

• Johan Dahlin
• James Henstridge
• John (J5) Palmieri
• Simon van der Linden
• Zach Goldberg
• Gustavo J Carneiro
• Paul Pogonyshev
• Gian Mario Tagliaretti

To end, just note that Martin is using his canonical.com address, so I assume that Canonical is sponsoring his work as maintainer upstream, so kudos to them as well.

WebGL support in WebKit-Clutter

In a previous post I wrote about how we can use accelerated compositing in the Clutter port of WebKit in order to take advantage of the graphics hardware when merging media elements into the finalized web page, which can be very dynamic (read: several updates per second) due to JavaScript usage and also because of CSS3 animations.

Another advantage of having the composition happening in the GPU is that whatever elements we render there (hw-accelerated video decoding or WebGL) doesn't need to be brought back to the CPU for composition, which would be very expensive.

For the last few weeks I have been working on a proof of concept of how we could implement the WebGL specification in our port, using Clutter. We cannot take the WebGL implementation that is already in use by the Gtk+, Qt and OSX ports because direct GL access isn't possible with Clutter, so it has taken quite a bit more of work than that. Here you can see the code in action, it's a WebGL canvas being rotated with CSS3:



What I ended up doing was adding a mechanism to Cogl (Clutter's graphics backend that abstracts OpenGL) so that applications could use raw GL to draw into an offscreen framebuffer without disturbing the rest of the application's graphics state, that is managed by Cogl. The branch that implements this is here.

Note that this is a proof of concept, in order to upstream this work we would need to find a way to share more code with the existing WebGL implementation in WebKit and also fix all the hackiness on the Cogl branch.

Other further work includes reducing the frequency with which GL contexts are switched, which should help enormously in GPUs such as Imagination's PVR and ARM's Mali.

To finish, I would like to thank my employer Collabora for allowing me to work on this and share the results, and to the Clutter guys that patiently answered my questions and showed me the right way, specially Robert Bragg and Neil Roberts.

If your company is interested in this, we'll be pleased in working with you, just drop an email to sales@collabora.com.

More acceleration support for video in webkit-clutter

The last few days I have been working on reworking how video gets rendered inside a web page in the Clutter port of WebKit, using Clutter-GStreamer. This work is contributed by Collabora and sponsored by Bosch.

Previously, we were doing the same as the Gtk+ port, that is, rendering each frame to a Cairo texture (with whatever format conversion that was needed) which would be then composited along the rest of the page (with Cairo as well). On the Clutter port that big surface would be uploaded to a texture in the GPU side of things and composed along the rest of the stage.

Joone and Gustavo have been working lately on adding accelerated compositing support to the Clutter port (and both will be blogging in the next few days about this work), which means that the page gets split in layers that get sent individually to the GPU where they get composited. Layers can be cached so only the ones that changed need to be re-rendered and re-uploaded.

This reduces dramatically the amount of work that happens on the CPU, so that animations with complex pages are feasible, even those involving video elements during playback, such as this:



Because page composition happens now in the GPU instead of in Cairo, we don't need to copy each frame to a Cairo surface (saving a copy) and don't need either to change the pixel format from whatever GStreamer gave us to whatever Cairo wants.

Another benefit of using Clutter-GStreamer is that alpha pre-multiplication can happen in the GPU (implemented as a shader), saving more CPU.

Furthermore (and more importantly), we can use standard GStreamer elements that use the hardware for decoding. Given the proper hardware and driver support, frames could be placed directly where the GPU can find them when compositing the page.

Hopefully this will make the Clutter port of WebKit more appealing for mobile platforms, where low-power GPUs are commonly available but the CPU is often taxed by other tasks.

All the code is here: https://gitorious.org/webkit-clutter/webkit-clutter/commits/ac2

Thanks to Bosch and Collabora for allowing me working on this. To know more about Collabora's involvement in WebKit: http://www.collabora.com/projects/webkit/

WebKit, Clutter and companies giving back

Time passes by and another project nears its end. For the past several months I have worked on a project for Bosch involving Clutter, Mx and WebKit and have had the pleasure of interacting daily with the upstream communities.

I feel I have been lucky because many of the companies that are starting to base their products on FOSS fail to understand the dynamics that will allow them to make the most from the components they are reusing. In this particular case, Bosch has understood that a big part of the value of FOSS resides in the possibility of adapting the software to their specific needs, and also in sharing the burden of maintaining the software with other organizations.

For that to be practical, you have to resist the urge to settle at the beginning of the project on specific stable releases, because if you do, you not only will miss new features and many bug fixes that will land only in the unstable branch, but you will have a much harder time integrating upstream whatever modifications you make. This situation often leads to companies having to maintain forks of the components they had to modify, and that's when the costs start spiraling up. David Neary wrote recently about in The Cost of Going it Alone.

Additionally, by working within the upstream community you will be doing so side-by-side with the people that are most knowledgeable about that specific component, either because they have been working on it for a long time, or quite often, because they are its creators. This is going to massively increase the chances that whatever modifications you end up doing are done in the best way and also that they will be integrated upstream and become maintained by other parties.

Besides technological soundness and reduced maintenance costs, there was an additional factor that made even more important the collaboration with upstream: Bosch had devised a new user experience that pushed the limits of the UI toolkit because of being innovative and because of scalability reasons. And understandably, they didn't want to compromise there because that innovative UI brought a lot of value to their product.

Getting into a FOSS community can be tough sometimes (specially if it's your first time), but in this case I have had the luck of finding a very open community in Clutter, well resourced and with a clear idea of how to deal with those contributions that can bring the most to the project. I take this opportunity to congratulate and thank the colleagues at the Clutter team at Intel OTC: Emmanuele Bassi, Robert Bragg, Neil Roberts, Thomas Wood, Brian Peel and Chris Lord (now at Mozilla).

Here comes a summary of the contributions sponsored by Collabora on behalf of Bosch:

- 16 commits to Clutter, most notably adding the base for gesture-based actions and several improvements to ClutterEffect subclasses.

- 16 commits to Mx, mainly on MxKineticScrollView

- 5 commits to WebKit's Clutter port, the most notable change was allowing the user to explicitly load images

- Filed 30 bugs in total

Besides thanking the individuals mentioned above, I would like to send big kudos to Bosch, Collabora and Intel for the courage and insight shown when investing in this new way of working together that we call Open Source.

You can read more about the Clutter port of WebKit here and about Collabora's involvement in WebKit here.

Clutter WebKit port at the Desktop Summit

Collabora is sponsoring work on the Clutter port for WebKit and we are going to take the chance to discuss this with the Clutter community during the Desktop Summit in Berlin this month.

Add a comment if you are interested and if there's enough people we'll register for a BoF.

See you in Berlin!

Wrap-up: Python ⊕ GNOME Hackfest 2011

A bunch of GNOME hackers spent the last week hacking on the Python bindings and this post gives an account of the achievements and tries to give credit where is due.

First, all attendants made substantial contributions and they and their employers (if not volunteers) should be thanked by their commitment: John (J5) Palmieri (Red Hat), Sebastian Pölsterl (volunteer), Laszlo Pandy (volunteer), Simon Schampijer (OLPC), Ignacio Casal Quinteiro (volunteer), Steve Frécinaux (volunteer), Simon van der Linden (volunteer), Martin Pitt (Canonical), Pavel Holejsovsky (volunteer) and myself (Collabora).

Also, Johan Dahlin (litl), Colin Walters (Red Hat), Paolo Borelli (volunteer) and Mikkel Kamstrup (Canonical) joined us remotely to hack together.

In numbers, at the hackfest were pushed 72 commits to PyGObject, 13 to gobject-introspection and 21 to Gtk+. There was quite a bit of Bugzilla activity as well.

One of the highlights was the push of *lots* of annotations into Gtk+, work submitted by Pavel and reviewed by Johan and some hackfest attendants. This makes a big part of the Gtk+ API accessible through introspection but there will be bugs so testing and corrections are still needed.

Another important contribution has been Martin's work on improving the GVariant and GBus API as exposed by PyGObject. He has a nice blog post explaining his work.

Steve did an amazing job delving into the ugliest innards of PyGObject and fixing some really hard memory leaks. Also, he added several test cases for other reference counting conditions which will help with not breaking it in the future.

Among other things, Laszlo worked on preserving GError information when propagating exceptions to Python and also has been looking at improving considerably our startup speed.

John kept hacking on his rewrite of the invoke path, including all argument validation and marshalling, so we get closer in speed to the static bindings. He gave a presentation explaining his strategy and people in the room seemed to like it. I expect a blog post from him soon on this subject.

Simon (svdlinden) did a lot of bug triaging, code reviews and rationalized our build system, needed after years of regular maintenance hacks.

Sebastian worked mainly on adding overrides, annotations and improving the perl script we provide to make porting easier. This as part of his porting of DVB Daemon to Gtk+ 3.

Nacho kept porting GEdit plugins to introspection and submitted misc. patches as he went, both to core PyGObject and to overrides.

Simon (erikos) started porting some containers in Sugar to Gtk+ 3 with great enthusiasm, but he found that the recent API changes in Gtk+ weren't supported by PyGObject nor gobject-introspection. Several fixes went in and others are still travelling through Bugzilla.

Johan and Colin helped out with reviews and answering our questions about how best to fix things in gobject-introspection, Paolo was there spotting any problems with the code that was being checked in and Mikkel was porting an Ubuntu app and reporting issues.

To finish, I spent most of the time reviewing and discussing approaches with the others but also managed to find some time to do some actual coding myself. Hope to find time during this week to push the last patches through bugzilla and make a new release with all that good stuff.

I'm pretty sure I have forgotten to give credit about something to someone so I apologize in advance and will gladly update this post if I'm told to :)

Thanks to Collabora for sponsoring the hotel, coffee, everynight's beer and the nice duckfest on Thursday at the U Slovanské lípy, check out the photos below:



The GNOME Foundation sponsored the travel and lodging for the participants of the hackfest:

We are also very obliged to the people at the brmlab hackerspace who hosted us during the whole week, they were really great:

Python ♥ GNOME Hackfest 2011

Looks like we are almost ready for one more push to Python development in GNOME 3, as originally announced by J5.

From 17th to 21th of January we'll be gathering at the brmlab hackerspace in Prague to hack on the applications of the brave souls that are taking the lead and port them to the future: Gtk+ 3, Python 3 and introspection.

We are having several GNOME volunteers, and Canonical, Collabora, OLPC, openSUSE and Red Hat  are sending their people to lend a hand.

If you think you should be there, please act quickly and add your name to the wiki before J5 secures the budget. If you would like to contribute to the funds or in any other way that would help this hackfest become a success, please send email to the GNOME Foundation board.