Benchmarking open source software: we can’t just focus on the numbers.
My last article was about the performance and memory consumption of the popular open source KDE and GNOME desktops. Well, it seems that that particular article was submitted to Digg. I looked at some of the comments that people posted, and I specifically wanted to address this one comment in particular.
The comment suggested that it is improper to compare desktop environments based on qualitative, rather than quantitative, measures. That attitude is, of course, incorrect. Open source software may not have been adopted as quickly as it could have been because many developers likely do ignore qualitative data, and thus did not get an adequate idea of what the users liked or disliked.
I understand why many developers would feel more comfortable dealing with numerical data. It is, in many cases, a lot easier to work with and analyze than qualitative data. We can easily calculate various statistics and measures from that data. We can plot graphs. But we have to remember that this is of little concern to many computer and software users, especially those who are relatively non-technical.
We can only play the numbers game up to a point. Suppose for a moment that it typically takes a KDE application running on a particular system 0.5 seconds to draw a menu. It takes a GNOME application on the same system 2.0 seconds to perform a similar operation. Windows Vista takes 1.0 seconds. We could go around playing number games all we want, saying things like KDE is four times faster than GNOME when it comes to drawing menus, and GNOME is twice as slow as Windows Vista. We could even say that Windows Vista’s menu drawing is twice as slow as KDE’s. Regardless, such data and claims are quite meaningless.
What matters is how users find the system to work. And this will likely involve qualitative, rather than quantitative, information. For the aforementioned example, they will say things like “KDE and Windows Vista feel quick”, or “GNOME’s menu redrawing seems a lot slower than KDE’s”. Non-technical users, who tend to be the majority of computer users these days, do not sit there with a stopwatch timing how long it takes for a menu to redraw, or for some other action to be performed.
This is something that I think a lot of developers have missed, both those working on open source projects and closed source commercial developments. For the users’ experience to be enjoyable, we as developers do need to pay a lot of attention to their qualitative assessments of our software. When they say it feels slow, we damn well better look into what they’re talking about, even if our benchmarking data may suggest we’re faster than our competitors’ systems.
Furthermore, we need to use such data when deciding what to improve. If our menu drawing speed is twice as slow as that of our competitors’ systems, but the users in general don’t seem to mind, then it’s likely not something we should bother improving. There are no doubt other features that we could better put our attention and efforts towards working on.
But I suspect that the person who posted that comment at Digg did not actually read my article, because he or she further requests some “charts and numbers”. Anyone who had read my article would have been directed to this page with desktop environment memory usage data. Instead of just providing the numbers, that site also describes the methodology used to obtain such data. So I did provide some numerical data to back up my claims regarding KDE. I invite you to check out that data, and the way it was collected, for yourself. Then you can see how it correlates to my qualitative analysis of the memory usage and responsiveness of GNOME and KDE.
When we as developers are working on software, we need to consider both quantitative benchmark data, and qualitative user analysis. Only when considering both of these data sources can we truly begin to make our applications better, not just in terms of raw performance, but also in terms of satisfying our users.