Friday, May 1, 2015

Lucid sleep in the free desktop

For the past year I have been working on the kernel side to bring some ChromeOS features to upstream.

One of the areas I'm currently working on is what Google calls Lucid Sleep, which is basically the ability of performing work while the machine is in a low power state such as suspend. I'm writing this blog post because there has been interest on this in different communities and the discussion is currently a bit dispersed.

Small mobile devices have been able to do that since basically always and this feature brings it to bigger devices that traditionally have been either on or off. It's similar to what Microsoft calls InstantGo (previously Connected Standby).

A few examples of tasks that the system could perform while apparently sleeping are:
  • Checking if the battery level is so low that it would be better to completely power down the machine
  • Starting a network backup if the present connectivity allows it (a known access point may have become accessible)
  • Downloading email
  • Checking for new instant messages

With regards to functionality and leaving performance considerations aside, userspace could implement this without requiring any new support in the kernel as illustrated in this scenario:
  • We assume that a video is currently playing in YouTube
  • User closes the lid
  • PM daemon notifies userspace of an impending sleep
  • Browser pauses playback
  • Compositor switches off the screen
  • Kernel freezes userspace, suspends devices and puts the CPUs to idle
  • Time passes...
  • RTC alarm fires off
  • Kernel resumes devices and unfreezes userspace
  • Userspace realizes there hasn't been any user activity since it went to sleep last, so stays in "dark resume" mode
  • Userspace does any lucid tasks it wants, then goes back to sleep again
  • Kernel freezes userspace, suspends devices and puts the CPUs to idle
  • Time passes...
  • User opens lid
  • Kernel resumes devices and unfreezes userspace
  • PM daemon notices the SW_LID event, so notifies userspace that this is a full-on resume
  • Compositor switches screen on
  • Browser resumes playback

No changes needed in the kernel is always good news, but there's two issues.

Lost input events

Sometimes the event from the input device that woke the system up gets lost before it reaches userspace, so we don't know if we can stay dark and do our lucid stuff, or if the user expects the machine to power completely on.

This is in any case a bug, but if it needs to be fixed in the firmware, we may not be able to do much about it. At most we could get the kernel to synthesize an input event, but sometimes it may not have enough information to do so.


When the system wakes up, there tends to be a lot to do in the kernel and userspace, so it could take several seconds for the screen to come up from the moment the user opened the lid in the scenario presented above.

For ChromeOS this isn't acceptable so they are carrying some patches in their kernel that make some shortcuts possible (the screen is left on at suspend time, and the kernel knows at resume time whether it has to power it on based on which was the wakeup source, thus not having to wait for userspace).

Fortunately, there have been some changes recently in the kernel PM subsystem that can speed up resumes quite a bit and we can make use of them to offset the penalty of dropping those shortcuts.

The first is idling the CPUs instead of suspending to firmware, which on modern SoCs should be quite efficient and much faster, by a few tenths of seconds.

The other is to leave idle devices that are already in a low power state alone when suspending, which means that we don't have to wait for them to resume when the system wakes up. In every system I have seen there's always a few devices that take a long time to resume, so this can shave several tenths of seconds from the total resume time.

Both need some amount of support in either the platform or in device drivers, and that's what I'm currently working on for the Tegra-based Chromebooks.

Wednesday, August 13, 2014

Dynamic scaling of the memory bus

The problem

These days there's quite good support for CPU scaling in the mainline kernel, and many ARM SoCs are making use of it already. But in modern hardware with lots of very fast external memory, running the memory bus at its maximum frequency drastically reduces the amount of time that the device can run when on battery.

A problem that many teams are finding when trying to upstream their power management code is that there's currently no way for several clock consumers to influence the frequency of the memory bus. There has been a few tries to upstream the solutions currently in vendor trees, but so far no acceptable solution has been found.

I'm helping to upstream some of the stuff in the ChromeOS tree, and this issue is currently blocking very interesting work from reaching mainline.

The past

In the vendor tree for Tegra this is addressed by creating virtual clocks that are child of the clock that wants to be influenced. Depending on the type of the virtual clock, setting its rate will influence the rate of its parent clock by setting a floor or ceiling value.

In Qualcomm's vendor tree for the Snapdragon family of SoCs, the concept of a voter clock is introduced. Drivers can vote on the rate of a given clock by "voting" through a child clock, so not that different to how Tegra does it.

Both approaches have the critical disadvantage of adding clk instances for things that aren't real clocks, thus making the API considerably more confusing for relatively little gain.

Both vendor trees have additional API for registering bandwidth needs: tegra_isomgr and msm_bus_scale. They bear quite some resemblance with each other and with pm_qos_interface, but both are tightly tied to specificities of their platforms.

The discussion was brought back to life a couple of months ago when a patch was posted for allowing the tegra-drm driver to set the frequency rate of the external memory controller based on the amount of bandwidth that was needed by the display controller for refreshing the display. Of course, that patch was rejected because there are other components that need to have a say in the frequency rate of the memory bus.

But in that discussion some kind of plan took form and I have been working on making something from it that can be merged upstream.

A possible future

There's so far two main additions to existing frameworks, with the rationale being explained further below:
  • Add per-user floor and ceiling constraints to the Common Clock Framework, so drivers can set maximum and minimum frequency rates that the clock should respect. Patchset here.
  • Add a PM_QOS_MEMORY_BANDWIDTH class to pm_qos, for drivers to register their expected bandwidth needs. Patchset here.
The idea is for the following agents to be able to influence the current frequency of the memory bus:
  • Thermal: a cooling device would call clk_set_ceiling_rate to cap the memory bus to a frequency based on the current temperature.
  • Power: a battery driver would set a ceiling in the same way, based on the remaining capacity.
  • Devfreq: a devfreq driver wrapping a power management unit such as the ACTMON on Tegra or the PPMU on Exynos would set a floor frequency based on the current load stats.
  • Cpufreq: a cpufreq driver would set a floor frequency based on the current CPU frequency.
  • Devices that can anticipate how much memory bandwidth will need (such as the display controller, the camera, multimedia codecs, an ISP, USB, etc) would register their requirements in the PM_QOS_MEMORY_BANDWIDTH class. The EMC driver would be listening for notifications and setting a floor frequency based on the aggregated bandwidth that is needed.
The impression so far is that this approach matches the needs of the Tegra and Exynos SoCs, and people working on Rockchip upstreaming are evaluating it. Others working on other SoCs are very welcome to look at it and comment, so the result is also useful to them and they can improve their power management in mainline without having to refactor things later.

Thursday, May 8, 2014

GNOME API reference at the DevX hackfest

Last week I spent a few days at the Developer Experience hackfest and got to have some fun again with the API reference generator in gobject-introspection.

Jon intended to hack on the XSLT stylesheets in yelp-tools/xsl, but after some trying he got discouraged by the amount of work that would take to get them to generate the HTML that we are interested in. We also discussed the benefits of using Mallard for generating the reference docs, and given that we want to generate a single output, we couldn't see much value in the level of indirection that Mallard adds.

Thus, we considered generating HTML directly from the GIR files, but shortly after Alberto Ruiz came by and offered to explore a client-side-only solution involving processing JSON files with JavaScript.

He very quickly got something relatively complete, which is very encouraging, but even more so is seeing how other projects are generating their API references that way, for example:!/api/Ext.Class

Aspects such as as-you-type search would bring the online documentation on par with the Devhelp experience. Plus they have some niceties such as a symbol browser and links to annotated source code.

I have hacked a (yet another) --write-json-files switch to g-ir-doc-tool that will output the content of the GIR file to JSON, but indexed and formatted as needed by the JS side of things. See this branch for the code.

That branch also adds support for Markdown rendering using python-markdown, but some more code needs to be written to implement the extensions to Markdown that the Gtk+ docs are using.

It was great to talk about all this and more with old and new friends in Berlin, so I'm very grateful to the GNOME Foundation for organizing it and sponsoring travel, and to Endocode for providing a venue. And special thanks to Chris Kühl for the great organization!

Thursday, March 14, 2013

Multi-touch gestures in Mutter-based compositors

A customer has asked for documentation on handling multi-touch gestures in their Mutter-based compositor (see my previous post) and I thought that it could be a good idea to have it in the GNOME wiki, in case it helps when they are added to GNOME Shell:

Multi-touch gestures in Mutter-based compositors

I'm not sure of what would be the best use for multi-touch gestures in the Shell, probably for resizing windows (with a 4-finger pinch gesture) or for switching desktops (with a 3 or 4 finger swipe). Probably some ideas can be taken from the multi-tasking gestures in recent versions of iOS, such as using pinch gestures to activate hidden panels or to switch to another views.

Something I feel strongly is about restricting system-wide gestures to more than 3 fingers, because the user experience and the implementation gets quite complicated if the compositor and the applications need to compete for touch sequences in similar gestures.

It's currently a bit convoluted due to zero support in Mutter for touch events, but once the Shell starts using touch events, I think it will make sense to move some of the setup and boilerplate into Mutter.

Once more, thanks to my employer Collabora for sponsoring this work:

Wednesday, September 5, 2012

Multi-touch gestures in Gnome Shell

I'm keeping this small branch in which touch event support is added to Mutter. Plug-ins can register to get touch events before any other client and then accept or reject touch sequences depending on whether the shell is interested on the gesture or not.

As can be seen in this example of a mutter plug-in, any subclass of ClutterGestureAction can be used, which includes gesture recognizers for pan, zoom and rotate actions, but creating new recognizers is pretty easy.

The mutter branch is up for reviewing in bugzilla and any comments on the approach will be very welcome. And if anybody wants to play with multi-touch gestures in Gnome Shell, please link to your work from the wiki so we can track it.

If anybody from the design team has already started thinking about this, I would be very glad to hear their thoughts on this.

As always, I'm grateful to my employer Collabora for sponsoring this work, and I hope GNOME benefits from it.

Thursday, August 2, 2012

Touch events within Xephyr

As part of my ongoing work on multi-touch, I have been looking at handling touch gestures within Mutter plugins. Developing a X window manager/shell can be quite a hassle because you are likely to want to do that in a separate X display so your testing doesn't disturb the session where you do the actual coding.

In the past I have used Xephyr to run a nested X display so I can run the window manager as I would run any normal application, but this time I found that Xephyr doesn't forward any XInput2 events right now, which is needed for MT. Having a separate machine where to test is an alternative but it has been quite uncomfortable.

My colleague at Collabora Daniel Stone encouraged me to give it a try and it indeed didn't take much work to get something running, though I still haven't tested it much.

So for the code, here is the Xephyr repo:

And here the Mutter repo:

Remember that these are early proof-of-concepts, but if you give it a try and want to give feedback, it will be welcome.

As usual, thanks to my employee Collabora for sponsoring this work and letting me share it.

Friday, July 13, 2012

Multi-touch in WebKit-Clutter

Following my past work on multi-touch support in Clutter, have been playing lately in implementing the W3C Touch Events API in the Clutter port of WebKit.

A lot of code can be reused from WebCore without problems, but we'll need to do some mildly complex event translation because the W3C API and the one in Clutter (and in XInput and in Gtk+) are very different.

But for now, a quick demo of a web page drawing the touch events that it receives, limited to 2 touch points because that's the maximum supported by the hardware I have here:

This is still early work, but once event translation is done, this should be very close to be feature-complete. And a nice side-effect is that given that the touch API in Clutter is so similar to Gtk+'s, it should be pretty straightforward to port it to WebKitGtk+. You can find the code here, but please keep in mind that this is very preliminary work.

As usual, thanks to my employer Collabora for sponsoring this work.