WebGL support in WebKit-Clutter

In a previous post I wrote about how we can use accelerated compositing in the Clutter port of WebKit in order to take advantage of the graphics hardware when merging media elements into the finalized web page, which can be very dynamic (read: several updates per second) due to JavaScript usage and also because of CSS3 animations.

Another advantage of having the composition happening in the GPU is that whatever elements we render there (hw-accelerated video decoding or WebGL) doesn't need to be brought back to the CPU for composition, which would be very expensive.

For the last few weeks I have been working on a proof of concept of how we could implement the WebGL specification in our port, using Clutter. We cannot take the WebGL implementation that is already in use by the Gtk+, Qt and OSX ports because direct GL access isn't possible with Clutter, so it has taken quite a bit more of work than that. Here you can see the code in action, it's a WebGL canvas being rotated with CSS3:



What I ended up doing was adding a mechanism to Cogl (Clutter's graphics backend that abstracts OpenGL) so that applications could use raw GL to draw into an offscreen framebuffer without disturbing the rest of the application's graphics state, that is managed by Cogl. The branch that implements this is here.

Note that this is a proof of concept, in order to upstream this work we would need to find a way to share more code with the existing WebGL implementation in WebKit and also fix all the hackiness on the Cogl branch.

Other further work includes reducing the frequency with which GL contexts are switched, which should help enormously in GPUs such as Imagination's PVR and ARM's Mali.

To finish, I would like to thank my employer Collabora for allowing me to work on this and share the results, and to the Clutter guys that patiently answered my questions and showed me the right way, specially Robert Bragg and Neil Roberts.

If your company is interested in this, we'll be pleased in working with you, just drop an email to sales@collabora.com.

PyGObject 3.0.3 Released

The Sugar guys are doing great progress porting their code from PyGTK+ to GObject Introspection and they have requested a new stable release, which comes with these goodies:

- Convert all modifier constants to Gdk.ModifierType (Manuel Quiñones)

- Convert all strings to utf-8 encoding when retrieving from TreeModel (Sebastian Pölsterl)

- add test for bytearray variants (John (J5) Palmieri)

- handle NULL arrays correctly for each array type (John (J5) Palmieri)

- Revert "Revert "Fix array termination and size calculation"" (John (J5) Palmieri)

- pygmainloop: avoid lockups if multiple glib.MainLoop exist (Owen W. Taylor)

- Properly chain up to the class that implements a given vfunc. (Tomeu Vizoso)

- Revert "Fix array termination and size calculation" (Tomeu Vizoso)

- Fix array termination and size calculation (Holger Berndt)

- pygi-convert: fix for Pango.Alignment (Daniel Drake)

- pygi-convert: fix for Gtk.Orientation (Daniel Drake)

- Add tests for calling closures (Martin Pitt)

- fix marshaling of arrays of GVariants (Mikkel Kamstrup Erlandsen)

Thanks all for your hard work!

More acceleration support for video in webkit-clutter

The last few days I have been working on reworking how video gets rendered inside a web page in the Clutter port of WebKit, using Clutter-GStreamer. This work is contributed by Collabora and sponsored by Bosch.

Previously, we were doing the same as the Gtk+ port, that is, rendering each frame to a Cairo texture (with whatever format conversion that was needed) which would be then composited along the rest of the page (with Cairo as well). On the Clutter port that big surface would be uploaded to a texture in the GPU side of things and composed along the rest of the stage.

Joone and Gustavo have been working lately on adding accelerated compositing support to the Clutter port (and both will be blogging in the next few days about this work), which means that the page gets split in layers that get sent individually to the GPU where they get composited. Layers can be cached so only the ones that changed need to be re-rendered and re-uploaded.

This reduces dramatically the amount of work that happens on the CPU, so that animations with complex pages are feasible, even those involving video elements during playback, such as this:



Because page composition happens now in the GPU instead of in Cairo, we don't need to copy each frame to a Cairo surface (saving a copy) and don't need either to change the pixel format from whatever GStreamer gave us to whatever Cairo wants.

Another benefit of using Clutter-GStreamer is that alpha pre-multiplication can happen in the GPU (implemented as a shader), saving more CPU.

Furthermore (and more importantly), we can use standard GStreamer elements that use the hardware for decoding. Given the proper hardware and driver support, frames could be placed directly where the GPU can find them when compositing the page.

Hopefully this will make the Clutter port of WebKit more appealing for mobile platforms, where low-power GPUs are commonly available but the CPU is often taxed by other tasks.

All the code is here: https://gitorious.org/webkit-clutter/webkit-clutter/commits/ac2

Thanks to Bosch and Collabora for allowing me working on this. To know more about Collabora's involvement in WebKit: http://www.collabora.com/projects/webkit/

WebKit, Clutter and companies giving back

Time passes by and another project nears its end. For the past several months I have worked on a project for Bosch involving Clutter, Mx and WebKit and have had the pleasure of interacting daily with the upstream communities.

I feel I have been lucky because many of the companies that are starting to base their products on FOSS fail to understand the dynamics that will allow them to make the most from the components they are reusing. In this particular case, Bosch has understood that a big part of the value of FOSS resides in the possibility of adapting the software to their specific needs, and also in sharing the burden of maintaining the software with other organizations.

For that to be practical, you have to resist the urge to settle at the beginning of the project on specific stable releases, because if you do, you not only will miss new features and many bug fixes that will land only in the unstable branch, but you will have a much harder time integrating upstream whatever modifications you make. This situation often leads to companies having to maintain forks of the components they had to modify, and that's when the costs start spiraling up. David Neary wrote recently about in The Cost of Going it Alone.

Additionally, by working within the upstream community you will be doing so side-by-side with the people that are most knowledgeable about that specific component, either because they have been working on it for a long time, or quite often, because they are its creators. This is going to massively increase the chances that whatever modifications you end up doing are done in the best way and also that they will be integrated upstream and become maintained by other parties.

Besides technological soundness and reduced maintenance costs, there was an additional factor that made even more important the collaboration with upstream: Bosch had devised a new user experience that pushed the limits of the UI toolkit because of being innovative and because of scalability reasons. And understandably, they didn't want to compromise there because that innovative UI brought a lot of value to their product.

Getting into a FOSS community can be tough sometimes (specially if it's your first time), but in this case I have had the luck of finding a very open community in Clutter, well resourced and with a clear idea of how to deal with those contributions that can bring the most to the project. I take this opportunity to congratulate and thank the colleagues at the Clutter team at Intel OTC: Emmanuele Bassi, Robert Bragg, Neil Roberts, Thomas Wood, Brian Peel and Chris Lord (now at Mozilla).

Here comes a summary of the contributions sponsored by Collabora on behalf of Bosch:

- 16 commits to Clutter, most notably adding the base for gesture-based actions and several improvements to ClutterEffect subclasses.

- 16 commits to Mx, mainly on MxKineticScrollView

- 5 commits to WebKit's Clutter port, the most notable change was allowing the user to explicitly load images

- Filed 30 bugs in total

Besides thanking the individuals mentioned above, I would like to send big kudos to Bosch, Collabora and Intel for the courage and insight shown when investing in this new way of working together that we call Open Source.

You can read more about the Clutter port of WebKit here and about Collabora's involvement in WebKit here.

GObject Introspection hackfest in Berlin

A quick (and late!) recollection of what happened in Berlin during the GObject Introspection hackfest:

• Red Hat's John (J5) Palmieri worked on new releases of PyGObject and on the documentation generators.
• Prezi's Laszlo Pandy worked mainly in resurrecting Zach Goldberg's DocBook generator, merging it into master and adding important stuff to it such as description parsing and class hierarchy rendering.
• Red Hat's Colin Walters reviewed code, gave input on people's direction and answered thousands of questions.
• Syllogist's Shaun McCance added features to Mallard to match gtk-doc and worked on the DocBook and mallard generators.
• Volunteer Torsten Schönfeld worked on improvements in gobject-introspection that are needed by his Perl bindings but that will benefit other bindings as well.
• Canonical's Martin Pitt worked on server-side GDBus support in PyGObject and some improvements trickled to gobject-introspection.
• Igalia's Guillaume Emont worked on adding default values to typelibs, #558620.
• Codethink's Richard Dale worked on GObject-Consume, making it much easier to consume GObject-based APIs from Qt.
• KDE volunteer Arno Rehn worked along with Richard on GObject-Consume.
• Volunteer Johan Dahlin, as one of the maintainers, also supported all others, added missing information to the .gir files and hacked on the DocBook generator.
• OLPC's Simon Schampijer worked on porting Sugar to use introspection instead of the static bindings.
• Volunteer Pavel Holejsovsky couldn't be with us in Berlin but hacked along in #introspection adding stuff to gobject-introspection that his advanced Lua bindings were already needing.
• Collabora's Tomeu Vizoso worked on the documentation generators and on the train back home rebased them to use Mako templates.

Nemein's Piotr Pokora and Nokia's Ivan Frade couldn't join us at the end because of unexpected obligations but we look forward to their contributions in the future.

Travel and lodging has been sponsored by the GNOME Foundation:

Openismus hosted us at their office when the Summit was over:

Nemein invited us to a delicious fondue dinner:

And Collabora treated us to tasty Ethiopian food and flooded us in beer afterwards:

Big thanks to those companies that sent their hackers, to the volunteers that devoted their time and talent, and to the sponsors for their support.

To close, I think this was a hackfest filled with fun and that we made good progress on the stated goals, most importantly having expanded the set of people that hack on gobject-introspection.

There were constant signs of appreciation towards Berlin, its food, beer, people and bike infrastructure. Looking forward for the next one!

Clutter WebKit port at the Desktop Summit

Collabora is sponsoring work on the Clutter port for WebKit and we are going to take the chance to discuss this with the Clutter community during the Desktop Summit in Berlin this month.

Add a comment if you are interested and if there's enough people we'll register for a BoF.

See you in Berlin!

PyGObject BoF at the Desktop Summmit 2011

A BoF has just been proposed for the Desktop Summit in Berlin in August, this is the description:

We'll get together to port applications from PyGtk+ to Gtk+ 3 (using GObject introspection). Seasoned developers of PyGObject will meet applications developers and will help each other port applications and improve PyGObject itself.

The session is still just a proposal and we'll only know if it has been accepted in a few days, but if you will be at the Summit and this session interests you, please feel free to add yourself to the list of prospective participants.

See you in Berlin!

Time to port your Python application to GObject introspection?

From time to time I realize that not all maintainers of GNOME applications written in Python have been following what has been happening upstream during the GNOME 3 development cycle.

Some of the noise we have been making about introspection and hackfests may have reached them, but for someone who has only time to review patches and make releases, this new stuff may have seemed as of secondary relevance.

For those that are asking themselves whether they should invest time now into moving away from PyGtk and the other static bindings, I would summarize the decision as to whether you want your application to run on Gtk+ 3 (and other recent APIs in GNOME 3) or not.

Besides that, there are other advantages such as reduced memory usage, fast startup and being able to use libraries that have no static bindings.

About distribution, most major distros have already packaged the whole of GNOME 3 and the rest will follow in the following months. As a rule of thumb, if you can develop for GNOME 3 in C in a particular distro version, you have already all you need to port your app to introspection.

As always, I want to remember that PyGObject (and gobject-introspection for that matter) has no dedicated developers. If it's progressing forward is because some application authors decided to jump and lend a hand, so if you are producing code that uses GNOME and Python, please consider dedicating some time to file bugs, write test cases and maybe writing a patch yourself.

Session on GObject Introspection at the Ubuntu App Developer Week

The talk will be oriented to developers that use or want to use GNOME technologies and still haven't fully grasped how GObject Introspection is changing the game. It won't contain any distro specificities, so join without fear even if you don't use Ubuntu, if it's of your interest.

It should be of special interest to those willing to contribute to GNOME Shell or that plan to attend Martin Pitt's talk about life after PyGTK this Tuesday at 16 UTC.

You can find instructions about how to join in the link below, plus other interesting talks:

https://wiki.ubuntu.com/UbuntuAppDeveloperWeek

See you later today at 17 UTC.

GNOME 3 at Prague

Tomorrow we'll hold a GNOME 3 Launch Party here in Prague and between interesting and even controversial talks, I will be talking a bit about the changes in the development story that happened during GNOME 3.

http://live.gnome.org/ThreePointZero/LaunchParty/Czech

See you there!

GObject-Introspection hackfest

Following the success of the two last PyGObject hackfests, the idea of having one focused on GObject-introspection has been floating for a while.

After enough people agreed it could be a good idea, I have created a page in the wiki to start filling in the details. We should have a good provisional list of goals and participants before we get to things such as the place and date, which will be better decided once we have the participant list.

http://live.gnome.org/Hackfests/Introspection2011

So if you are planning to do some work on GObject-introspection that would benefit from happening in the hackfest, please add it to the goals list in the wiki.

If you won't be able to work on it yourself but you think something should really happen in the near future, please add it to the comments in this blog post.

I know we can count on consultancies such as my employer Collabora to sponsor the hackfest, but I would like to encourage downstreams to send their hackers and show they care, are able and deserve more good will.

Generating API docs from .gir files

From time to time people ask which are the plans about generating API documentation from the introspected information in the .gir files that g-ir-scanner outputs.

More often, I hear someone swearing about how hard is to setup gtk-doc to correctly generate documentation for your code. And getting g-ir-scanner to generate .gir files as expected is not completely trivial either.

It should be apparent that generating the docs from the .gir would save quite a bit of grief altogether but for one reason or another this hasn't happened yet. Docstrings are in the .gir files already along with the rest of the metadata but gtk-doc is still scanning the C sources.

© Juan José Sánchez Penas

During last GUADEC, the gobject-introspection people could be seen hacking in the lobby instead of attending your talk or rioting against the release team, and one of the outcomes was Zach Goldberg's g-ir-docgen script. His new employer forbids doesn't sponsor him for this work so this has been in the freezer since then.

So we can eventually get out from this situation, I would like to encourage interested non-coding people to add to CC on this bug: https://bugzilla.gnome.org/show_bug.cgi?id=625494

And to those that can actually code, please consider using that ticket to coordinate the effort, it may take less work than you think and it can be a fun hack. Think of all the GNOME hackers that will thank you!

Wrap-up: Python ⊕ GNOME Hackfest 2011

A bunch of GNOME hackers spent the last week hacking on the Python bindings and this post gives an account of the achievements and tries to give credit where is due.

First, all attendants made substantial contributions and they and their employers (if not volunteers) should be thanked by their commitment: John (J5) Palmieri (Red Hat), Sebastian Pölsterl (volunteer), Laszlo Pandy (volunteer), Simon Schampijer (OLPC), Ignacio Casal Quinteiro (volunteer), Steve Frécinaux (volunteer), Simon van der Linden (volunteer), Martin Pitt (Canonical), Pavel Holejsovsky (volunteer) and myself (Collabora).

Also, Johan Dahlin (litl), Colin Walters (Red Hat), Paolo Borelli (volunteer) and Mikkel Kamstrup (Canonical) joined us remotely to hack together.

In numbers, at the hackfest were pushed 72 commits to PyGObject, 13 to gobject-introspection and 21 to Gtk+. There was quite a bit of Bugzilla activity as well.

One of the highlights was the push of *lots* of annotations into Gtk+, work submitted by Pavel and reviewed by Johan and some hackfest attendants. This makes a big part of the Gtk+ API accessible through introspection but there will be bugs so testing and corrections are still needed.

Another important contribution has been Martin's work on improving the GVariant and GBus API as exposed by PyGObject. He has a nice blog post explaining his work.

Steve did an amazing job delving into the ugliest innards of PyGObject and fixing some really hard memory leaks. Also, he added several test cases for other reference counting conditions which will help with not breaking it in the future.

Among other things, Laszlo worked on preserving GError information when propagating exceptions to Python and also has been looking at improving considerably our startup speed.

John kept hacking on his rewrite of the invoke path, including all argument validation and marshalling, so we get closer in speed to the static bindings. He gave a presentation explaining his strategy and people in the room seemed to like it. I expect a blog post from him soon on this subject.

Simon (svdlinden) did a lot of bug triaging, code reviews and rationalized our build system, needed after years of regular maintenance hacks.

Sebastian worked mainly on adding overrides, annotations and improving the perl script we provide to make porting easier. This as part of his porting of DVB Daemon to Gtk+ 3.

Nacho kept porting GEdit plugins to introspection and submitted misc. patches as he went, both to core PyGObject and to overrides.

Simon (erikos) started porting some containers in Sugar to Gtk+ 3 with great enthusiasm, but he found that the recent API changes in Gtk+ weren't supported by PyGObject nor gobject-introspection. Several fixes went in and others are still travelling through Bugzilla.

Johan and Colin helped out with reviews and answering our questions about how best to fix things in gobject-introspection, Paolo was there spotting any problems with the code that was being checked in and Mikkel was porting an Ubuntu app and reporting issues.

To finish, I spent most of the time reviewing and discussing approaches with the others but also managed to find some time to do some actual coding myself. Hope to find time during this week to push the last patches through bugzilla and make a new release with all that good stuff.

I'm pretty sure I have forgotten to give credit about something to someone so I apologize in advance and will gladly update this post if I'm told to :)

Thanks to Collabora for sponsoring the hotel, coffee, everynight's beer and the nice duckfest on Thursday at the U Slovanské lípy, check out the photos below:



The GNOME Foundation sponsored the travel and lodging for the participants of the hackfest:

We are also very obliged to the people at the brmlab hackerspace who hosted us during the whole week, they were really great: