WebGL support in WebKit-Clutter

In a previous post I wrote about how we can use accelerated compositing in the Clutter port of WebKit in order to take advantage of the graphics hardware when merging media elements into the finalized web page, which can be very dynamic (read: several updates per second) due to JavaScript usage and also because of CSS3 animations.

Another advantage of having the composition happening in the GPU is that whatever elements we render there (hw-accelerated video decoding or WebGL) doesn't need to be brought back to the CPU for composition, which would be very expensive.

For the last few weeks I have been working on a proof of concept of how we could implement the WebGL specification in our port, using Clutter. We cannot take the WebGL implementation that is already in use by the Gtk+, Qt and OSX ports because direct GL access isn't possible with Clutter, so it has taken quite a bit more of work than that. Here you can see the code in action, it's a WebGL canvas being rotated with CSS3:



What I ended up doing was adding a mechanism to Cogl (Clutter's graphics backend that abstracts OpenGL) so that applications could use raw GL to draw into an offscreen framebuffer without disturbing the rest of the application's graphics state, that is managed by Cogl. The branch that implements this is here.

Note that this is a proof of concept, in order to upstream this work we would need to find a way to share more code with the existing WebGL implementation in WebKit and also fix all the hackiness on the Cogl branch.

Other further work includes reducing the frequency with which GL contexts are switched, which should help enormously in GPUs such as Imagination's PVR and ARM's Mali.

To finish, I would like to thank my employer Collabora for allowing me to work on this and share the results, and to the Clutter guys that patiently answered my questions and showed me the right way, specially Robert Bragg and Neil Roberts.

If your company is interested in this, we'll be pleased in working with you, just drop an email to sales@collabora.com.

PyGObject 3.0.3 Released

The Sugar guys are doing great progress porting their code from PyGTK+ to GObject Introspection and they have requested a new stable release, which comes with these goodies:

- Convert all modifier constants to Gdk.ModifierType (Manuel Quiñones)

- Convert all strings to utf-8 encoding when retrieving from TreeModel (Sebastian Pölsterl)

- add test for bytearray variants (John (J5) Palmieri)

- handle NULL arrays correctly for each array type (John (J5) Palmieri)

- Revert "Revert "Fix array termination and size calculation"" (John (J5) Palmieri)

- pygmainloop: avoid lockups if multiple glib.MainLoop exist (Owen W. Taylor)

- Properly chain up to the class that implements a given vfunc. (Tomeu Vizoso)

- Revert "Fix array termination and size calculation" (Tomeu Vizoso)

- Fix array termination and size calculation (Holger Berndt)

- pygi-convert: fix for Pango.Alignment (Daniel Drake)

- pygi-convert: fix for Gtk.Orientation (Daniel Drake)

- Add tests for calling closures (Martin Pitt)

- fix marshaling of arrays of GVariants (Mikkel Kamstrup Erlandsen)

Thanks all for your hard work!

More acceleration support for video in webkit-clutter

The last few days I have been working on reworking how video gets rendered inside a web page in the Clutter port of WebKit, using Clutter-GStreamer. This work is contributed by Collabora and sponsored by Bosch.

Previously, we were doing the same as the Gtk+ port, that is, rendering each frame to a Cairo texture (with whatever format conversion that was needed) which would be then composited along the rest of the page (with Cairo as well). On the Clutter port that big surface would be uploaded to a texture in the GPU side of things and composed along the rest of the stage.

Joone and Gustavo have been working lately on adding accelerated compositing support to the Clutter port (and both will be blogging in the next few days about this work), which means that the page gets split in layers that get sent individually to the GPU where they get composited. Layers can be cached so only the ones that changed need to be re-rendered and re-uploaded.

This reduces dramatically the amount of work that happens on the CPU, so that animations with complex pages are feasible, even those involving video elements during playback, such as this:



Because page composition happens now in the GPU instead of in Cairo, we don't need to copy each frame to a Cairo surface (saving a copy) and don't need either to change the pixel format from whatever GStreamer gave us to whatever Cairo wants.

Another benefit of using Clutter-GStreamer is that alpha pre-multiplication can happen in the GPU (implemented as a shader), saving more CPU.

Furthermore (and more importantly), we can use standard GStreamer elements that use the hardware for decoding. Given the proper hardware and driver support, frames could be placed directly where the GPU can find them when compositing the page.

Hopefully this will make the Clutter port of WebKit more appealing for mobile platforms, where low-power GPUs are commonly available but the CPU is often taxed by other tasks.

All the code is here: https://gitorious.org/webkit-clutter/webkit-clutter/commits/ac2

Thanks to Bosch and Collabora for allowing me working on this. To know more about Collabora's involvement in WebKit: http://www.collabora.com/projects/webkit/

WebKit, Clutter and companies giving back

Time passes by and another project nears its end. For the past several months I have worked on a project for Bosch involving Clutter, Mx and WebKit and have had the pleasure of interacting daily with the upstream communities.

I feel I have been lucky because many of the companies that are starting to base their products on FOSS fail to understand the dynamics that will allow them to make the most from the components they are reusing. In this particular case, Bosch has understood that a big part of the value of FOSS resides in the possibility of adapting the software to their specific needs, and also in sharing the burden of maintaining the software with other organizations.

For that to be practical, you have to resist the urge to settle at the beginning of the project on specific stable releases, because if you do, you not only will miss new features and many bug fixes that will land only in the unstable branch, but you will have a much harder time integrating upstream whatever modifications you make. This situation often leads to companies having to maintain forks of the components they had to modify, and that's when the costs start spiraling up. David Neary wrote recently about in The Cost of Going it Alone.

Additionally, by working within the upstream community you will be doing so side-by-side with the people that are most knowledgeable about that specific component, either because they have been working on it for a long time, or quite often, because they are its creators. This is going to massively increase the chances that whatever modifications you end up doing are done in the best way and also that they will be integrated upstream and become maintained by other parties.

Besides technological soundness and reduced maintenance costs, there was an additional factor that made even more important the collaboration with upstream: Bosch had devised a new user experience that pushed the limits of the UI toolkit because of being innovative and because of scalability reasons. And understandably, they didn't want to compromise there because that innovative UI brought a lot of value to their product.

Getting into a FOSS community can be tough sometimes (specially if it's your first time), but in this case I have had the luck of finding a very open community in Clutter, well resourced and with a clear idea of how to deal with those contributions that can bring the most to the project. I take this opportunity to congratulate and thank the colleagues at the Clutter team at Intel OTC: Emmanuele Bassi, Robert Bragg, Neil Roberts, Thomas Wood, Brian Peel and Chris Lord (now at Mozilla).

Here comes a summary of the contributions sponsored by Collabora on behalf of Bosch:

- 16 commits to Clutter, most notably adding the base for gesture-based actions and several improvements to ClutterEffect subclasses.

- 16 commits to Mx, mainly on MxKineticScrollView

- 5 commits to WebKit's Clutter port, the most notable change was allowing the user to explicitly load images

- Filed 30 bugs in total

Besides thanking the individuals mentioned above, I would like to send big kudos to Bosch, Collabora and Intel for the courage and insight shown when investing in this new way of working together that we call Open Source.

You can read more about the Clutter port of WebKit here and about Collabora's involvement in WebKit here.

GObject Introspection hackfest in Berlin

A quick (and late!) recollection of what happened in Berlin during the GObject Introspection hackfest:

• Red Hat's John (J5) Palmieri worked on new releases of PyGObject and on the documentation generators.
• Prezi's Laszlo Pandy worked mainly in resurrecting Zach Goldberg's DocBook generator, merging it into master and adding important stuff to it such as description parsing and class hierarchy rendering.
• Red Hat's Colin Walters reviewed code, gave input on people's direction and answered thousands of questions.
• Syllogist's Shaun McCance added features to Mallard to match gtk-doc and worked on the DocBook and mallard generators.
• Volunteer Torsten Schönfeld worked on improvements in gobject-introspection that are needed by his Perl bindings but that will benefit other bindings as well.
• Canonical's Martin Pitt worked on server-side GDBus support in PyGObject and some improvements trickled to gobject-introspection.
• Igalia's Guillaume Emont worked on adding default values to typelibs, #558620.
• Codethink's Richard Dale worked on GObject-Consume, making it much easier to consume GObject-based APIs from Qt.
• KDE volunteer Arno Rehn worked along with Richard on GObject-Consume.
• Volunteer Johan Dahlin, as one of the maintainers, also supported all others, added missing information to the .gir files and hacked on the DocBook generator.
• OLPC's Simon Schampijer worked on porting Sugar to use introspection instead of the static bindings.
• Volunteer Pavel Holejsovsky couldn't be with us in Berlin but hacked along in #introspection adding stuff to gobject-introspection that his advanced Lua bindings were already needing.
• Collabora's Tomeu Vizoso worked on the documentation generators and on the train back home rebased them to use Mako templates.

Nemein's Piotr Pokora and Nokia's Ivan Frade couldn't join us at the end because of unexpected obligations but we look forward to their contributions in the future.

Travel and lodging has been sponsored by the GNOME Foundation:

Openismus hosted us at their office when the Summit was over:

Nemein invited us to a delicious fondue dinner:

And Collabora treated us to tasty Ethiopian food and flooded us in beer afterwards:

Big thanks to those companies that sent their hackers, to the volunteers that devoted their time and talent, and to the sponsors for their support.

To close, I think this was a hackfest filled with fun and that we made good progress on the stated goals, most importantly having expanded the set of people that hack on gobject-introspection.

There were constant signs of appreciation towards Berlin, its food, beer, people and bike infrastructure. Looking forward for the next one!

Clutter WebKit port at the Desktop Summit

Collabora is sponsoring work on the Clutter port for WebKit and we are going to take the chance to discuss this with the Clutter community during the Desktop Summit in Berlin this month.

Add a comment if you are interested and if there's enough people we'll register for a BoF.

See you in Berlin!

PyGObject BoF at the Desktop Summmit 2011

A BoF has just been proposed for the Desktop Summit in Berlin in August, this is the description:

We'll get together to port applications from PyGtk+ to Gtk+ 3 (using GObject introspection). Seasoned developers of PyGObject will meet applications developers and will help each other port applications and improve PyGObject itself.

The session is still just a proposal and we'll only know if it has been accepted in a few days, but if you will be at the Summit and this session interests you, please feel free to add yourself to the list of prospective participants.

See you in Berlin!