Multi-touch gestures in Mutter-based compositors

A customer has asked for documentation on handling multi-touch gestures in their Mutter-based compositor (see my previous post) and I thought that it could be a good idea to have it in the GNOME wiki, in case it helps when they are added to GNOME Shell:

Multi-touch gestures in Mutter-based compositors

I'm not sure of what would be the best use for multi-touch gestures in the Shell, probably for resizing windows (with a 4-finger pinch gesture) or for switching desktops (with a 3 or 4 finger swipe). Probably some ideas can be taken from the multi-tasking gestures in recent versions of iOS, such as using pinch gestures to activate hidden panels or to switch to another views.

Something I feel strongly is about restricting system-wide gestures to more than 3 fingers, because the user experience and the implementation gets quite complicated if the compositor and the applications need to compete for touch sequences in similar gestures.

It's currently a bit convoluted due to zero support in Mutter for touch events, but once the Shell starts using touch events, I think it will make sense to move some of the setup and boilerplate into Mutter.

Once more, thanks to my employer Collabora for sponsoring this work:

Multi-touch gestures in Gnome Shell

I'm keeping this small branch in which touch event support is added to Mutter. Plug-ins can register to get touch events before any other client and then accept or reject touch sequences depending on whether the shell is interested on the gesture or not.

As can be seen in this example of a mutter plug-in, any subclass of ClutterGestureAction can be used, which includes gesture recognizers for pan, zoom and rotate actions, but creating new recognizers is pretty easy.

The mutter branch is up for reviewing in bugzilla and any comments on the approach will be very welcome. And if anybody wants to play with multi-touch gestures in Gnome Shell, please link to your work from the wiki so we can track it.

If anybody from the design team has already started thinking about this, I would be very glad to hear their thoughts on this.

As always, I'm grateful to my employer Collabora for sponsoring this work, and I hope GNOME benefits from it.

Touch events within Xephyr

As part of my ongoing work on multi-touch, I have been looking at handling touch gestures within Mutter plugins. Developing a X window manager/shell can be quite a hassle because you are likely to want to do that in a separate X display so your testing doesn't disturb the session where you do the actual coding.

In the past I have used Xephyr to run a nested X display so I can run the window manager as I would run any normal application, but this time I found that Xephyr doesn't forward any XInput2 events right now, which is needed for MT. Having a separate machine where to test is an alternative but it has been quite uncomfortable.

My colleague at Collabora Daniel Stone encouraged me to give it a try and it indeed didn't take much work to get something running, though I still haven't tested it much.

So for the code, here is the Xephyr repo: http://cgit.collabora.com/git/user/tomeu/xserver/log/?h=xephyr-touch

And here the Mutter repo: http://cgit.collabora.com/git/user/tomeu/mutter/log/?h=touch-clutter

Remember that these are early proof-of-concepts, but if you give it a try and want to give feedback, it will be welcome.

As usual, thanks to my employee Collabora for sponsoring this work and letting me share it.

Multi-touch in WebKit-Clutter

Following my past work on multi-touch support in Clutter, have been playing lately in implementing the W3C Touch Events API in the Clutter port of WebKit.

A lot of code can be reused from WebCore without problems, but we'll need to do some mildly complex event translation because the W3C API and the one in Clutter (and in XInput and in Gtk+) are very different.

But for now, a quick demo of a web page drawing the touch events that it receives, limited to 2 touch points because that's the maximum supported by the hardware I have here:


This is still early work, but once event translation is done, this should be very close to be feature-complete. And a nice side-effect is that given that the touch API in Clutter is so similar to Gtk+'s, it should be pretty straightforward to port it to WebKitGtk+. You can find the code here, but please keep in mind that this is very preliminary work.

As usual, thanks to my employer Collabora for sponsoring this work.

WebGL support in WebKit-Clutter: Part 2

Following up on my work from last december, I have a branch of webkit-clutter that runs most of the WebGL demos that float around the web. The main differences are that the required new API has been accepted into Cogl and that there's a lot more code shared with the other WebKit ports.

Right now bigger applications are not working because of this bug which also affects the Gtk+ port, but hopefully it will be fixed soon.

The code lives in this branch and you will need this Cogl branch with a recent Clutter 1.10 release.

As before, thanks to my employer Collabora for allowing to work on this.

Multi-touch support in Clutter

Have been doing some tests regarding multi-touch support on Clutter and everything seems to be working fine:



You will need either a very recent master or clutter-1.10 branch because there was a bit of fixing needed, but hopefully that will get into a release soon.

That example can be find in ./tests/interactive/test-touch-events.c and can be run as:

$ cd clutter
$ ./tests/interactive/test-touch-events

Needless to say, you will need a recent Xorg (1.12) and support in the kernel for your multi-touch hardware.

Hope that soon I will be able to work on gestures support and automated tests for this.

Thanks for my employer Collabora for sponsoring this work.

WebGL support in WebKit-Clutter

In a previous post I wrote about how we can use accelerated compositing in the Clutter port of WebKit in order to take advantage of the graphics hardware when merging media elements into the finalized web page, which can be very dynamic (read: several updates per second) due to JavaScript usage and also because of CSS3 animations.

Another advantage of having the composition happening in the GPU is that whatever elements we render there (hw-accelerated video decoding or WebGL) doesn't need to be brought back to the CPU for composition, which would be very expensive.

For the last few weeks I have been working on a proof of concept of how we could implement the WebGL specification in our port, using Clutter. We cannot take the WebGL implementation that is already in use by the Gtk+, Qt and OSX ports because direct GL access isn't possible with Clutter, so it has taken quite a bit more of work than that. Here you can see the code in action, it's a WebGL canvas being rotated with CSS3:



What I ended up doing was adding a mechanism to Cogl (Clutter's graphics backend that abstracts OpenGL) so that applications could use raw GL to draw into an offscreen framebuffer without disturbing the rest of the application's graphics state, that is managed by Cogl. The branch that implements this is here.

Note that this is a proof of concept, in order to upstream this work we would need to find a way to share more code with the existing WebGL implementation in WebKit and also fix all the hackiness on the Cogl branch.

Other further work includes reducing the frequency with which GL contexts are switched, which should help enormously in GPUs such as Imagination's PVR and ARM's Mali.

To finish, I would like to thank my employer Collabora for allowing me to work on this and share the results, and to the Clutter guys that patiently answered my questions and showed me the right way, specially Robert Bragg and Neil Roberts.

If your company is interested in this, we'll be pleased in working with you, just drop an email to sales@collabora.com.

More acceleration support for video in webkit-clutter

The last few days I have been working on reworking how video gets rendered inside a web page in the Clutter port of WebKit, using Clutter-GStreamer. This work is contributed by Collabora and sponsored by Bosch.

Previously, we were doing the same as the Gtk+ port, that is, rendering each frame to a Cairo texture (with whatever format conversion that was needed) which would be then composited along the rest of the page (with Cairo as well). On the Clutter port that big surface would be uploaded to a texture in the GPU side of things and composed along the rest of the stage.

Joone and Gustavo have been working lately on adding accelerated compositing support to the Clutter port (and both will be blogging in the next few days about this work), which means that the page gets split in layers that get sent individually to the GPU where they get composited. Layers can be cached so only the ones that changed need to be re-rendered and re-uploaded.

This reduces dramatically the amount of work that happens on the CPU, so that animations with complex pages are feasible, even those involving video elements during playback, such as this:



Because page composition happens now in the GPU instead of in Cairo, we don't need to copy each frame to a Cairo surface (saving a copy) and don't need either to change the pixel format from whatever GStreamer gave us to whatever Cairo wants.

Another benefit of using Clutter-GStreamer is that alpha pre-multiplication can happen in the GPU (implemented as a shader), saving more CPU.

Furthermore (and more importantly), we can use standard GStreamer elements that use the hardware for decoding. Given the proper hardware and driver support, frames could be placed directly where the GPU can find them when compositing the page.

Hopefully this will make the Clutter port of WebKit more appealing for mobile platforms, where low-power GPUs are commonly available but the CPU is often taxed by other tasks.

All the code is here: https://gitorious.org/webkit-clutter/webkit-clutter/commits/ac2

Thanks to Bosch and Collabora for allowing me working on this. To know more about Collabora's involvement in WebKit: http://www.collabora.com/projects/webkit/

WebKit, Clutter and companies giving back

Time passes by and another project nears its end. For the past several months I have worked on a project for Bosch involving Clutter, Mx and WebKit and have had the pleasure of interacting daily with the upstream communities.

I feel I have been lucky because many of the companies that are starting to base their products on FOSS fail to understand the dynamics that will allow them to make the most from the components they are reusing. In this particular case, Bosch has understood that a big part of the value of FOSS resides in the possibility of adapting the software to their specific needs, and also in sharing the burden of maintaining the software with other organizations.

For that to be practical, you have to resist the urge to settle at the beginning of the project on specific stable releases, because if you do, you not only will miss new features and many bug fixes that will land only in the unstable branch, but you will have a much harder time integrating upstream whatever modifications you make. This situation often leads to companies having to maintain forks of the components they had to modify, and that's when the costs start spiraling up. David Neary wrote recently about in The Cost of Going it Alone.

Additionally, by working within the upstream community you will be doing so side-by-side with the people that are most knowledgeable about that specific component, either because they have been working on it for a long time, or quite often, because they are its creators. This is going to massively increase the chances that whatever modifications you end up doing are done in the best way and also that they will be integrated upstream and become maintained by other parties.

Besides technological soundness and reduced maintenance costs, there was an additional factor that made even more important the collaboration with upstream: Bosch had devised a new user experience that pushed the limits of the UI toolkit because of being innovative and because of scalability reasons. And understandably, they didn't want to compromise there because that innovative UI brought a lot of value to their product.

Getting into a FOSS community can be tough sometimes (specially if it's your first time), but in this case I have had the luck of finding a very open community in Clutter, well resourced and with a clear idea of how to deal with those contributions that can bring the most to the project. I take this opportunity to congratulate and thank the colleagues at the Clutter team at Intel OTC: Emmanuele Bassi, Robert Bragg, Neil Roberts, Thomas Wood, Brian Peel and Chris Lord (now at Mozilla).

Here comes a summary of the contributions sponsored by Collabora on behalf of Bosch:

- 16 commits to Clutter, most notably adding the base for gesture-based actions and several improvements to ClutterEffect subclasses.

- 16 commits to Mx, mainly on MxKineticScrollView

- 5 commits to WebKit's Clutter port, the most notable change was allowing the user to explicitly load images

- Filed 30 bugs in total

Besides thanking the individuals mentioned above, I would like to send big kudos to Bosch, Collabora and Intel for the courage and insight shown when investing in this new way of working together that we call Open Source.

You can read more about the Clutter port of WebKit here and about Collabora's involvement in WebKit here.

Clutter WebKit port at the Desktop Summit

Collabora is sponsoring work on the Clutter port for WebKit and we are going to take the chance to discuss this with the Clutter community during the Desktop Summit in Berlin this month.

Add a comment if you are interested and if there's enough people we'll register for a BoF.

See you in Berlin!

Now moving for real

In January I tried to move from Sugar to other projects, but Collabora prevented it by hiring me to work on Sugar and Telepathy integration.

Nine months and a release cycle later, I'm doing it for real. During the next months I will be switching to work on totally different stuff, involving Clutter and WebKit hacking.

Also plan to keep hacking a bit on Telepathy, would be a pity stop using what I learned during most of this year.