QtCompositor motivation

I have tried to give a brief overview over the QtCompositor architecture once before [1] but I want to revisit the subject.

One of the nice things about Qt 5 is that there it takes very little to port Qt to new graphics stacks. If there is a way to paint with Qt into a "graphics buffer" and then get this buffer to screen, then this is all it takes to get a basic Qt port up and running. The minimal and minimalegl QPA plugins are good examples for how to either get a "very" basic raster based stack or an OpenGL based stack up and running. Obviously the QtWidget stack corresponds well to the raster based graphic stack, while QtQuick2 uses OpenGL. Neither of these two plugins requires a windowing system. This is actually something which is crucial to stress. Qt 5 DOES NOT NEED A WINDOWING SYSTEM!

So why do you need a windowing system? Often there is actually not a need for a windowing system. The other day I made a xbmc box at home. For this box I don't need a windowing system. I just need to be able to get the screen resources and handle input in some way or the other, so that the xbmc application can show video. I'm not going to run any other GUI application on this box, so just giving all the screen resources to XBMC is ok.

However, sometimes you need to be able to have control the screen resources.
This doesn't need to be because you want to have two applications on the screen at the same time, but maybe you need a superior process that can override what is being sent to screen. Maybe in a full screen application setup you still need to be able to interrupt with some full screen notification.

But lets backtrack. Qt 5 works perfectly WITHOUT a windowing system! That's amazing. We have actually spent alot of time to figuring out how to make these abstractions so that Qt will work perfectly without a windowing system. We have separated out the input handling since this doesn't necessary need to be coupled with the graphics bit. We have made sure that it will work with as little functionality as possible. We added an interface for page-flipping buffers to screen.

All of these interfaces are also interfaces that are needed when you want to create a compositor that will target more than one graphics stack and more than one input system. The interfaces are there to make it possible and easy to port Qt to new windowing systems, but it just happens to be a perfect fit for creating new windowing systems.

There is also an added bonus that the backends will be developed/maintained by many developers. Not only developers that are making a Windowing system, but developers targeting single on screen applications.

Say you have used to QtCompositor api to develop your compositor. When you developed the compositor, you typically will use the xcb backend, since its very nice to have a window manager when you develop, so you can look up documentation, search the web etc. Then when you want to run your compositor as your main windowing system, you can run it with the kms backend, openwfd etc etc.

This takes care of the output, but what about the input? There are several available input plugins for QPA. So you can start it with evdevkeyboard and evdevmouse if its a more traditional desktop.

So what if the plugin for your HW does not exist? Then you have to do some work! But the upside is that the interfaces are very clearly defined*. It is EASY to target a new graphics stack, and it is EASY to target a new input system.

So lets backtrack again. Qt 5 doesn't require a window system. Wow, thats awsome. One of the side effects of this is that it is ideal to use the interfaceses developed to make Qt 5 window system agnostic to implement a windowing system.

QtCompositor sits on top of libwayland-server, implementing the boilerplate code that a compositor would need. By boilerplate code I mean code that every compositor needs. However, I don't mean trivial code. The buffer queuing and direct rendering schemes are not trivial. Making client surfaces to QML items is neither trivial, but using the items is. Handling all of the input in the correct way, is not difficult, but its code that is common for most compositors.

Also, often for some applications areas, you want to have additional protocol extensions (ie. not part of the Wayland core protocol). This is made very easy with the QtWayland module.

I strongly recommend people how need to create their own Wayland compositor to have a look at the QtWayland module. I believe its a "hidden" gem with loads of potential.

* Qt platform abstraction does not guarantee source or binary compatibility between Qt versions. However, it should be regarded as a stable api. Considering the amount of work it would require to change all the plugins that uses the apis.

[1]: http://labs.qt.nokia.com/2011/03/18/multi-process-lighthouse/

Wayland-scanner

Make no mistake about it. One of the coolest things about Qt 5 is going to be the Wayland module.
git://gitorious.org/qt/qtwayland.git

So in the start there was a wayland plugin. Then we made a QtCompositor module for the server side of things. Then we started making extensions. So we generated the extension headers and sources with wayland-scanner and checked them in.

At some point we came to our senses and merged the wayland QPA plugin and the QtCompositor project into one git repository. Then we made it a module, which uses the configure checks from QtBase to find the correct configuration. But we still pre-generated the wayland extensions, committing the pregenerated files.

The sollution was a bit messy, and the pattern was that we also checked in the xml extension files as part of the compositor api.

Last friday I sat down trying to implement a qmake compiler rule for wayland-scanner, and by monday afternoon (with some help from Marius SO) I got the compiler rule into QtBase.

That meant we could create this new directory in QtWayland called extensions, which will contain all the xml extension definitions. Now when you type make in the top level directory, it will compile the QPA plugin and QtCompositor with the generated extensions. If the xml changes, the extension headers and sources will be regenerated, and the files that uses them will be recompiled...

Anyway, these are babysteps, but as I said, one of coolest things with Qt 5 is the Wayland module.

Introducing QtPlatformSupport

So, in the beginning of june I wrote about how I wanted Lighthouse plugins to be built out of source. It turned into a big set of qmake prf files that had to be included as CONFIG += "someprffile".

Also, Qt would need to install alot of source files + some headers. It just didn't feel like a good solution. So, one afternoon Paul and I where sitting in the office discussing the solution. Then the discussion wandered onto static libraries, and how they are cool in such cases because the linker will only copy in the symbols it needs..... Then we discussed how syncqt would be cool to help out with "exporting" the headers. Specially now that we ship private headers....

So, the day after I made QtPlatformSupport. Its a static library which resides in Qt-Base. Because of the define magic needed by the platformdatabases I needed to make 2 prf files, but that was it!

When making a new platform plugin out of source now, all you need to do in the pro file is to:

QT += platformsupport-private

CONFIG += qpa/genericunixfontdatabase

This applies to the Qt 5 refactor branch as of now, but will hopfully move into Qt5 master in a month or so :)

Out of Qt source tree builds of Lighthouse plugins

Last week sometime, Paul blogged about that we where shutting down the Lighthouse repository. It had been a real fun project, but now it was moved into mainstream, so there was not need for that repository anymore.

But in one of the comments, someone noted that it was possible to build external lighthouse plugins. Then one of our qmake gurus looked at the pri files we have and said: "Using $$QTDIR is a big no no, you should use prf files instead."  Paul and I sat down and we thought it would be good to be able to build Lighthouse plugins as if they where just another Qt program or Library, and this was a good opportunity to have a stab at that.

The problem is that most Lighthouse plugins will depend on some source code from the Qt source tree. This is code that for some reason or the other is not compiled into one of the Qt libraries. The raster paint engine, and the opengl [es] 2 paintengine is compiled into their own modules. But the native font engines are for example not compiled in, as they pull in native "window system" libraries.(freetype2 isn't exactly a window system lib, but....)

 So for the xlib or the xcb backend to be able to be built out of source, it needs access to $QTSRCDIR/src/gui/text/qfontengine_ft_p.h and the cpp file.

So in $QTSRCDIR/src/plugins/platforms/install_rules I made a some pro files that installes some files that are required when building typical Lighthouse backends on Windows. It installes the correct files from harfbuzz, freetype2 and the correct fontengines. It installes them into: QT_INSTALL_DATA/platforms. mkspecs are installed into QT_INSTALL_DATA/mkspecs and often QT_INSTALL_DATA is the same as QT_INSTALL_PREFIX.

So when you do make install in the src/plugins/platforms dir, these files should be installed into your Qt installation.

Then when I had made the installation part, I added a few prfs into mkspecs/features/qpa. This is so they can be loaded with the load(qpa/....) syntax in qmake. So I added a prf that detects if your in a Qt src build or if your outside of the Qt src tree. Then it exports the qmake variable QT_PLATFORMS_DIR. This variable will point to QT_SRC/src/plugins/platforms if your building inside the Qt src tree, and QT_INSTALL_DATA/platforms if your building outside the source tree.

I also made a file called harfbuzz_dir.prf and freetype_dir.prf which exports the environment variable QT_HARFBUZZ_DIR and QT_FREETYPE_DIR.

So to be able to use these files, in the qmake project file do:

load(qpa/platform_dir)

load(qpa/harfbuzz_dir)

load(qpa/freetype_dir)

Now, since the essentials for building out of source where in place I removed the helper pri files in QT_SRC/plugins/platforms. ie. I removed basicunix_fontdatabase.pri eglconvenience.pri etc. And then I made prf files for them instead.

So to compile in the basic unix fontdatabase now into the plugin you should do:

load(qpa/fontdatabases/basicunix)

to compile in the egl convenience functions you should do:

load(qpa/egl/convenience)

This should setup the correct include path, and add the source and headers to the HEADERS and SOURCES variables in qmake.

I also made a file called plugin so that its possible to do:

load(qpa/plugin) which is suppose to replace the qt_plugin we used before. After all, the goal was to not make any difference between Qt Lighthouse plugins, and custom plugins :)

Wayland and GL integrations

I have spent the last few weeks trying to make Qt-Compositor as hardware and system friendly as possible. Running Qt-Compositor with OpenGL in the clients was not trivial. It required building libdrm and mesa and if you had a nvidia chip you would need to build the linux kernel.

The reason why it was so difficult to build Wayland was the dependencies to mesa for sharing hardware buffers. However, when you start out making a compositor, you just want to create a X window which symbolises your screen, and then paint Wayland clients inside this window.

In March, Benjamin Franzke abstracted the part of the Wayland protocol out and moved it into wayland. About the same time the Wayland repository was split into two. One demo and example repository, and one Wayland repository, only containing the protocol definition and the libwayland libraries. That resulted in that building Wayland only has one dependency, which is libffi. libffi is a pretty common library, so it should be part of any distribution.

So after Benjamin and Kristian had done that part of the job, building a full raster version of Qt-Compositor only required libffi + Wayland + Qt-Lighthouse. This was all fine and dandy, but it would be nice if it was possible to have actual GL rendering on the clients as well, even though its inside the test compositor running inside a GL window.

Then someone said: "If we just read back the pixels on the client side, and send that over as a shared memory buffer". I have to admit that I was against the idea in the beginning, but after some discussions, I thought: "ok, so we need an abstraction in the Wayland plugin that enables us to control GL drawing." "A GL pixels readback" back-end was a first nice step. It took a few days, but then it was possible to do GL rendering without sharing GPU surfaces. I was also quite surprised by how good the performance was. It didn't totally suck! It was very dependent of the size of the surfaces, but still OK.

Then I spoke to Matthias Ettrich. He had an example of how to use XComposite for making sub-compositors in a X Window. At that point I had an idea of just rendring into X Pixmaps and send the Xid's over, and use the texture for pixmap extension. This would resemble how other implementations would work, and I liked the simplicity of it.

So I started on making an EGL back-end where I sent X Pixmap ids over the "wire". I was so exited about the idea that I came in on a Saturday to work on it. And it worked nicely! So then I started working on a GLX backend for doing the same thing. And after tearing out my hair, complaining to Paul (the guy Im sharing an office with), and giving up on the idea a few times I managed to make it work. But this was done on a machine running the latest checkout of mesa from git, on Intel HW. Then I moved what I had done over onto my second machine with the Nvidia proprietary driver, and everything fell apart. I could not make it work... I chatted with Kristian online and he concurred that it probably wasn't possible with the proprietary driver.

So then I had another look at what Matthias had sent me and got working on making an XComposite back-end. This time I started the work on the machine with the Nvidia driver :) It didn't take too long before I had a GLX XComposite backend working. Then I made the EGL backend as well, sharing as much code as possible between the two implementations.

The back-ends are still not optimal, and performance isn't the same as you get in X when doing regular client side rendering, but its still work in progress. However, its better than all other options still when using Qt-Compositor inside X11, so yesterday I pushed a few commits making it the default back-ends for the Qt-Lighthouse Wayland plugin and Qt-Compositor.

It is possible to change what back-ends are compiled in to Qt-Compositor and the wayland plugin using the QT_WAYLAND_GL_CONFIG environment variable.

Here is a list of possible options for Qt-Compositor:
wayland_egl
dri2_xcb
nogl
custom

And here is the possible options for Qt-Lighthouse wayland plugin:
wayland_egl
readback

If the environment variable is undefined then the correct xcomposite back-end will be used.

Default plugin for Lighthouse

I didn't get time to write about this on Friday, but I made a small patch for qapplication_qpa.cpp on friday which let you compile in the default platform plugin which is going to be used by Lighthouse.
Sha1 for the patch: 4234ea4479ca858d058840eec01760a387e4afb1

This will typically be useful for environments where its not possible to set the environment variables, and where its unlikely that more than one platform plugin will be used.

So this is how platform plugins are picked up now:

First we define a variable called "platformName".
If qapplication_qpa.cpp is compiled with the define:
QT_QPA_DEFAULT_PLATFORM_NAME
then the value of platformName is set to the value of this define.

Then QApplication will see if the environment variable
QT_QPA_PLATFORM
exists. If the environment variable is set, platformName will be set to the value of this environment variable.

Then, if the command line argument -platform is set, the value of the application argument will be set to platformName.

Then QApplication will load all platformPlugins, evaluate their names, and pick the one which matches platformName. If QApplication doesn't find it, it will print all platform plugin names.

After I committed the patch, someone pinged me on irc and told me it would be cool if this could be done for where fontdatabases picked up the fonts as well. I agreed initially, and said I would look into it first thing on Monday. This morning I sat down to look at the problem. However, then I noticed that I had thought about this problem before:) QPlatformFontdatabase has the virtual function fontDir()  const.

So if you want to specify where your fontdatabase picks up your fonts, you sub-class either QPlatformFontdatabase or QBasicUnixFontdatabase and reimplement the fontDir function.

Qt-Compositor

Had to go the official way and blog about Qt-Compositor on labs:

http://labs.qt.nokia.com/2011/03/18/multi-process-lighthouse