The last few days I have been working on reworking how video gets rendered inside a web page in the Clutter port of WebKit, using Clutter-GStreamer. This work is contributed by Collabora and sponsored by Bosch.
Previously, we were doing the same as the Gtk+ port, that is, rendering each frame to a Cairo texture (with whatever format conversion that was needed) which would be then composited along the rest of the page (with Cairo as well). On the Clutter port that big surface would be uploaded to a texture in the GPU side of things and composed along the rest of the stage.
Joone and Gustavo have been working lately on adding accelerated compositing support to the Clutter port (and both will be blogging in the next few days about this work), which means that the page gets split in layers that get sent individually to the GPU where they get composited. Layers can be cached so only the ones that changed need to be re-rendered and re-uploaded.
This reduces dramatically the amount of work that happens on the CPU, so that animations with complex pages are feasible, even those involving video elements during playback, such as this:
Because page composition happens now in the GPU instead of in Cairo, we don't need to copy each frame to a Cairo surface (saving a copy) and don't need either to change the pixel format from whatever GStreamer gave us to whatever Cairo wants.
Another benefit of using Clutter-GStreamer is that alpha pre-multiplication can happen in the GPU (implemented as a shader), saving more CPU.
Furthermore (and more importantly), we can use standard GStreamer elements that use the hardware for decoding. Given the proper hardware and driver support, frames could be placed directly where the GPU can find them when compositing the page.
Hopefully this will make the Clutter port of WebKit more appealing for mobile platforms, where low-power GPUs are commonly available but the CPU is often taxed by other tasks.
All the code is here: https://gitorious.org/webkit-clutter/webkit-clutter/commits/ac2
Thanks to Bosch and Collabora for allowing me working on this. To know more about Collabora's involvement in WebKit: http://www.collabora.com/projects/webkit/
