New client languages for Qt WebChannel

At the company I’m working at, we’re employing Qt WebChannel for remote access to some of our software. Qt WebChannel was originally designed for interfacing with JavaScript clients, but it’s actually very well suited to interface with any kind of dynamic language.

We’ve created client libraries for a few important languages with as few dependencies as possible: pywebchannel (Python, no dependencies), (.NET/C#, depends on JSON.NET) and webchannel++ (header-only C++14, depends on Niels Lohmann’s JSON library).

Python and .NET are a pretty good match: Their dynamic language features make it possible to use remote methods and properties like they were native. Due to being completely statically typed, C++ makes the interface a little more clunky, although variadic templates help a lot to make it easier to use.

As with the original Qt WebChannel C++ classes, transports are completely user defined. When sensible, a default implementation of a transport is provided.

Long story short, here’s an example of how to use the Python client. It’s designed to talk with the chatserver example of the Qt WebChannel module over a WebSocket. It even supports the asyncio features of Python 3! Relevant excerpt without some of the boilerplate:

pywebchannel can also be used without the async/await syntax and should also be compatible with Python 2.

I would also really like to push the code upstream, but I don’t know when I’ll have the time to spare. Then there’s also the question of how to build and deploy the libraries. Would the qtwebchannel module install to $PYTHONPREFIX? Would it depend on a C# compiler (for which support would have to be added to qmake)?

In any case, I think the client libraries can come in handy and expand the spectrum of application of Qt WebChannel.

QRPC: A Qt remoting library

This project of mine has been around for quite a while already. I’ve never much publicised it, however, and for the past year the code hasn’t seen any changes (until a few days ago, anyway). But related to my current job, I’ve found a new need for remote procedure calls, so I came back to the code after all.

QRPC is a remoting library tightly integrated with Qt: in simplified terms, it’s an easy way to get signal-slot connections over the network. At least that was its original intention, and hence comes the name (QRPC as in “Qt Remote Procedure Call”).

But actually, it’s a little more than that. QRPC pipes the whole of Qt’s meta-object functionality over a given transport. That can be a network socket, unix domain socket or something more high-level like ZeroMQ. But it could, for example, also be a serial port (not sure why anybody would want that, though).
The actual remoting system works as follows: A server marks some objects for “export”, meaning they will be accessible by clients. Clients can then request any such object by its identifier. The server then serialises the whole QMetaObject hierarchy of the exported object and sends it to the client. There, it is reconstructed and used as the dynamic QMetaObject of a specialised QObject. Thus, the client not only has access to signals and slots, but also to properties and Q_CLASSINFOs (actually anything that a QMetaObject has to offer). The property support also includes dynamic properties, not only the statically defined QMetaProperties.
Method invocations, signal emissions and property changes are serialised and sent over the transport – after all, a QMetaObject without the dynamics isn’t that useful ;).

To give an impression of how QRPC is used, here’s an excerpt from the example included in the repository:



Argument or property types have to be registered metatypes (same as when using queued connections) and serialisable with QDataStream. The QDataStream operators also have to be registered with Qt’s meta type system (cf. QMetaType docs).
At the moment, one shortcoming is that references to QObjects (i.e. “QObject*”) can’t be transferred over the network, even if the relevant QObjects are “exported” by the server. Part of the problem is that Qt doesn’t allow us to register custom stream operators for “QObject*”. This could be solved by manually traversing the serialised values, but that will negatively impact the performance. Usually one can work around that limitation, though.

Furthermore, method return values are intentionally not supported. Supporting them would first require a more complex system to keep track of state and secondly impose blocking method invocations, both of which run contrary to my design goals.

In some final remarks, I’d like to point out two similar projects: For one, there’s QtRemoteObjects in Qt’s playground which works quite similarly to QRPC. I began working on QRPC at around the same time as QtRemoteObjects was started and I didn’t know it existed until quite some time later. Having glanced over its source code, I find it to be a little too complex and doing too much for my needs without offering the same flexibility (like custom transports). I must admit that I haven’t checked it out in more detail, though.
Then there’s also QtWebChannel which again offers much of the metaobject functionality over the web, but this time specifically geared towards JavaScript/HTML5 clients and Web apps. I thought about reusing part of its code, but ultimately its focus on JS was too strong to make this a viable path.

QuickPlot: A collection of native QtQuick plotting items

For a project at university I recently needed a plotting widget to display some data. Naturally, Qwt came to my mind. I’ve already been using it in a number of other projects and it works great.

The one drawback, however: The project was intended to be run on the Raspberry Pi. Now the X-Server on the R-Pi doesn’t have any 3D acceleration yet, so the performance of Qwt was subpar.

Luckily, Qt has the eglfs backend (using Broadcom’s EGL libraries), which works very well on the R-Pi. The problem: You have to use QML/QtQuick with that, so Qwt was out of the game (actually, I’ve played around with creating a QtQuick backend for Qwt, but with Qwt being QPainter-based, still, it was nowhere near performant enough). A more QtQuick-like solution was needed.

Since QtQuick supports the HTML5 Canvas API, I thought that maybe some HTML5 chart library like Chart.js could do the trick. The good news: It works pretty much out of the box and you get very pretty charts. The bad news: performance on the Raspberry Pi is more or less abysmal (on a laptop or desktop PC it’s fine, though). Even some hand-crafted plotting code based on the Canvas API ate the better part of the R-Pi’s CPU when displaying a continuous stream of data. This isn’t actually that much of a wonder, considering that it was all based on JavaScript and has redrawn the whole plot area in an FBO every time there was a data update.

So all of the “simple” solutions where dead ends. Eventually I ended up writing my own plot items, this time based on QtQuick’s Scene Graph to get the best performance out of the little computer. I was actually surprised that no-one had done something like this yet – or at least I couldn’t find anything.
Anyway, here’s a screenshot of the thing displaying a sine wave with axes:

The sine wave is actually scrolling through the view, always showing the most recent 300 data points of the data source. It’s even anti-aliased on most hardware, including the R-Pi, just not on my laptop’s Intel GPU on which I took the screenshot. The performance on the Raspberry Pi is now very good, using less than 5% of the CPU, while the other solutions where north of 70% for a continuously updated data source (values from top, not claiming any sort of accuracy).

The API is somewhat inspired by Qwt’s. There’s the PlotArea Item which is responsible for managing all items of the plot. It keeps a collection of PlotItems, displays the axes and makes the used ScaleEngines (used for the axis limits) known to every item and axis in the plot.
I haven’t implemented more than the most basic PlotItems and ScaleEngines at the moment (because those are currently the only things I need):

  • Curve: exactly what the name implies – a set of points connected by lines.
  • ScrollingCurve: A curve which only displays the last N data points. If the data is continuously updated, this looks as if the curve is scrolling from right to left.
  • TightScaleEngine: A ScaleEngine which sets the axis limits to the overall minima and maxima of all the plot items.
  • FixedScaleEngine: A ScaleEngine which sets the axis limits to constant values.

The axis ticks are placed based on the nicenum algorithm from the Graphics Gems book series from Academic Press. The API is designed with C++/Qt data sources in mind: Data is passed as QVector<float> or QVector<QPointF>, preferrably as an argument to a signal-slot connection. Note that the usage of QVector makes setting the data statically from QML a little difficult. I’m open for suggestions to improve this, but I would really like to avoid converting a batch of data to a QVariantList and back just to integrate better with QML/JS.

You can find the code on gitorious GitLab, licensed under the GPLv3:

Edit: Some example code would probably be nice, so here is the QML code used in the above screen shot:

Edit 2: Due to Gitorious being acquired by GitLab, the code has been moved to the latter:

Connecting C++11 lambdas to Qt4 Signals with libffi (now portable)

I know that this is probably not very useful, considering that it’s not portable and with Qt5 about to be released in June, but I found it interesting to experiment with the new language features and libffi.

The result is a class which lets you connect any function pointer or lambda, with or without bound variables, to Qt4 signals. Example of what you can do with it:

Quite nice, I think :) You can get the source for lambdaconnection.h here.

Its main drawback is the use of pointers to member functions as ordinary function pointers. It works with gcc and maybe clang, but it’s probably going to break with other compilers and/or operating systems. UPDATE: By converting everything to std::function, we can have one template function which is called by libffi and handles any further lambda calls. No need for member-function-pointers anymore. Portability shouldn’t be an issue anymore, too.

Additionally, to avoid code duplication, it simply converts function pointers to std::function objects. This adds some overhead when calling the function pointer.

Anyway, it’s only an afternoon’s work and it was fun to code. Maybe someone will find this interesting :)

Java bindings?

Since the C# bindings are apparently not really used/wanted by anyone (except for some Windows people – but that’s really not my main development platform) I thought that maybe some more people are interested in Java bindings.

There was a poll on two years ago that showed Java ahead of Ruby and C# – but it doesn’t seem to be really representative: there are at least some Ruby plasmoids on, but none in C# (which was still ahead of Ruby). Given that Trolltech/Nokia abandoned QtJambi I’m not too sure about Java bindings either.

So I’m asking the community directly before I start putting too much work into a project that noone really wants: So far we have (active) bindings projects for Python, Ruby and Perl – is there any demand for Qt/KDE Java bindings in the community? Or if not for Java, for any other language/platform?

Looking forward to your comments 🙂

And the bindings keep rocking: Writing Ruby KIO slaves

It seems like I got into a hacking frenzy after yesterday’s Ruby plugins for Kate. Today I sat down and took another look at the KRubyPluginFactory to implement KIO slave support.

So here it is, our very first Ruby Hello World KIO slave:

We’ll also need a rubytest.protocol file that describes our new protocol:

To test it, you just have to type rubytest:/ into konqi’s addressbar and you should see nice hello world greeting.

Some more info on the structure of a Ruby kio slave: the Ruby script always has to be named ‘main.rb’ and the SlaveBase derived class always ‘Main’. The module name is the camelized form of the protocol (so a protocol ‘foo_bar’ would map to a module name ‘FooBar’). The script has to reside in <kde prefix>/share/apps/kio_<protocol name>.
The .protocol file itself should be installed to <kde prefix>/share/kde4/services.

As with the Kate plugin, I packaged this example in a tarball. It ships a Makefile and can easily be installed to your home directory with ‘make install’.
I committed this feature to both trunk and the 4.5 branch – so you can soon start coding your own KIO slaves in Ruby! 🙂

Why KDE bindings simply rock (or: Creating our first Ruby plugin for Kate)

Akademy really speeds up development and communication between developers:
Yesterday I was poked by Milian Wolff and he told me how many people want to write Kate plugins in languages other than C++ and ECMAScript and how he always told them that’s not possible and probably a huge amount of work to get working. Well, that wasn’t entirely true:
If the target application has a SMOKE lib that wraps its API and an according bindings extensions, it works pretty much out of the box, thanks to the gorgeous KPluginFactory API and of course thanks to our gorgeous bindings.
Kate was a bit trickier to get working though, because it used old and deprecated API to load plugins. Well, I filed a merge request on gitorious about that, fixed a bug in KRubyPluginFactory and now I’m very proud to present

The absolutely gorgeous, simple, straight-forward, packed with all the Ruby goodness, very first Ruby Hello-World Kate plugin:

You’ll also need an ui.rc file that defines the actions to be merged into the Kate main window and a .desktop file which describes your plugin. Actually I’m too lazy to post them all here, so I made a nice package containing everything you need, together with a Makefile which can ‘make install’ the plugin into your home directory.

Since all SMOKE based bindings wrap the C++ API nearly 1:1 and only add syntactic language specific sugar on top, you can create nearly any plugin you like in any language, without modifying the host application, as long as the API is wrapped in a SMOKE lib and a bindings extension. In C# you can even create KIO slaves (a monodoc KIO slave example is shipped with the kdebindings tarball). That feature hasn’t been added to Ruby yet, but is on my ToDo list.

So praise the bindings, praise the KDE plugin infrastructure and start working on Ruby Kate plugins! 🙂

You can find the Ruby Kate binding and the SMOKE kate lib in current kdebindings trunk.

qyotoassemblygen progress

Some time back I have started a tool called qyotoassemblygen [0], which will hopefully ease the generation of .NET/Mono bindings based on SMOKE libraries. It basically introspects a SMOKE library and generates a System.CodeDom tree from that information. The CodeDom can then be used to generate C# code and/or compile an assembly.
Although some parts were rather difficult and had to be rewritten again and again (like checking whether a method has to be marked “override” or “virtual”), I can now announce that it”s quite stable and generates all of the Qt assemblies just fine :). I”m currently working on getting the KDE assemblies to build, which is really just a matter of sorting out *Private classes.

By using plugins the tool is not bound to Qt-based bindings, so we could as well generate assemblies for other toolkits, like Wt (for which there is already a smoke lib, thanks to Richard :)). Plugins will also make it easy to add syntactic sugar like event support for Qyoto (which I hope I can implement for KDE SC 4.5).

If everything works as expected, we can hopefully drop kalyptus completely in the next release and reduce our maintenance cost considerably 🙂


Current git branch in shell prompt

Since I”ve read much about git and how great it is, I recently thought that I had to try it myself. And indeed, it is quite amazing and is nice to work with. There was one thing that I disliked, though: I never quite knew in which branch of my repository I was, especially after I”ve been away from coding for a few hours.
Of course I could just look that up, but I tend to forget that and then end up with code that belongs to a different branch. Now the shell prompt looked like a good place to show the current branch, but simply executing “git branch | grep ^*” would only work in the top-level directory of the repository. So I wrote the following function which will traverse all containing directories and look for a .git directory. Having found one, it will nicely print the the branch in dark green color, with a @ sign in front of it:

Update: As many people have pointed out, bash-completion provides a function called __git_ps1 which will happily print the current git branch and can be used as well. I don”t quite like the rest of bash-completion, though, so I”ll stick with my own.
As Yuri pointed out, git branch works from any directory, so here”s the updated (and much shorter 🙂 ) version:

Having this in my .bashrc, I could integrate it quite nicely with my prompt:

git prompt

Qyoto on Windows!

Now that the new smokegenerator is in trunk and generating smoke on windows shouldn”t be a pain anymore, I thought that I could dig out my nearly unused Vista partition on my laptop and start porting Qyoto to Windows.
First thing I did was removing the preinstalled Vista and installing it again to get rid of all the preinstalled crap Dell thought I needed. Next was installing Visual C++ and C# (the Express editions are free as in beer and work ok – nothing in comparison to KDevelop4 though) and KDE 4.3. Thanks to all of the KDE Windows team – you make Windows much more bearable! I also installed msys – can”t live without a proper shell anymore.

Now came the actual porting: To support MS .NET and not only Mono, I needed to add support for csc.exe (the MS C# compiler) to the existing C# macros. As I recently discovered how to add support for a new language to cmake (which lacks documentation but is really easy once you understood it) I simply rewrote the C# macros as a new cmake language plugin. Compilation was pretty straight-forward, then.
But as was to be expected, the examples simply crashed upon start. Long story short: Some of them work now and this is how it looks:

Qt tutorial t5 with Qyoto

This is still really unstable and crashes every few minutes with an AccessViolationException. Have to figure out why it does that.. But first I have to do some other things, mainly porting the remaining kalyptus-generated smoke modules to the new smokegenerator. This shouldn”t take too long, so I think I can complete the Windows port soon 🙂 So stay tuned!

Update: The AccessViolationExceptions should be fixed now. If you want to try Qyoto on Windows and have problems getting it to compile or run, please send a mail to the kde-bindings mailinglist and include the complete error and backtrace. You can also search the mailing list archives since this topic came up quite often recently.