Thermostat goes to Red Hat Summit & JavaOne 2012

Last week, we were happy to receive notification that our (Mario & me) proposal for a presentation about Thermostat at JavaOne 2012 has been accepted! Thermostat is the project that both of us have been heavily involved with during the last 4-5 months (having joined Jon & Omair in February who have been involved much longer and deeper). It’s a next generation monitoring, diagnosing, profiling and controlling solution for OpenJDK. Just now we are adding heap analysis support and make Thermostat ready for its first milestone release at the end of this week, just in time for Deepak’s show at the Red Hat Summit in Boston. The second milestone release is planned to be ready roughly 3 months later, just before JavaOne. (Yes, we invented a new development cycle: Conference Driven Release Management :-) ) Hope to see lots of folks there!

Gnome3 usability

I have been, and still am, quite a fan of Gnome. I followed the development of Gnome3 and was relatively pleased when it came out, despite all the bashing. I even find some workflows sorely lacking when switching to other desktops, which is a good sign (for Gnome). It has its rough edges of course, and I hoped they would be ironed out, but I get to think that this might never happen, because some of the things seem to be intentional. Let me list a few of the things that I am most concerned about:
– The timezone-aware clock. I loved this thingy in Gnome2, I would add any location I want, and the clock would show me the time in those places in the popup view. Since I am working with heavily distributed team, this is an extremely useful feature. The clock in Gnome3 instead is very basic. I have seen a discussion somewhere to re-introduce the timezone aware clock back into Gnome3, but apparently it never happened. And doesn’t have anything like that either.
– The hopping notification icons. Unless you are very good with the mouse, it becomes a little bit of a chase to click one of those notification icons in the lower right corner. Why the hell do the need to show a name in the taskbar, and move around while you hover over them??? This is so usability-backwards! Moving things are always a bit of a double edged sword in any GUI. Yes, movements *can* be a very useful visual cue to something happening, but needs to be implemented with great care to not end up being totally confusing. In this particular case I simply don’t get it: why not simply show the icons and only show the ‘name’ (or whatever, sometimes it’s really just some crappy variable name or such) in a tooltip when hovering over it?? There is completely no good reason to shuffle those icons around!
– Epiphany’s new fullscreen mode. The idea is SO good!! But the implementation is so far off, I cannot believe it. First: once maximized there is *NO* way to unmaximize it. I mean, yeah, ALT+F5. After having looked it up in the *web*. Similar for *CLOSING* the frakking browser window. It requires to do 2 mouse moves and 2 clicks. Not to speak of the initial confusion of the close button being replaces by a menu, that simply should not be there. Yeah, CTRL+W. I know. It seems like one half of Gnome devs target idiots who need huge title bars in order to not miss them, and the other half targets superusers who never use the mouse and know gazillions of shortcuts. This is just wrong.
– Since Gnome 3.4 some applications seem to use a new theme with slick scrollbars and stuff, while the other half uses a different theme. I guess it’s because of GTK2 vs. GTK3 dichotomy, but why?? Why not make them look the same until every major app is switched to GTK3?? Now the whole experience is totally inconsistent. I hated this when it happened with Ubuntu’s new scrollbar (and the new Gnome3.4 scrollbar is only slightly better… requires mouse-superskills) now Gnome upstream repeats the same mess.
– For some unknown reason, Evolution becomes more and more broken with every release. In Gnome3.2, it would freeze every now and then when I enabled spam filtering. Now it freezes whenever the computer comes back from suspend, which is the new OFF. (Ok, it doesn’t strictly freeze, it just doesn’t seem to be able to connect to any network.) Fail. Ah, and it manages to always come up on the wrong screen.

I find the state of Gnome quite sad. On one side I really like many of the ideas. On the other side, it really feels a bit like 1998 in many respects. And this is not because of needing to re-learn new concepts, I actually like to try and learn new ways of work. It is really because some things are made unnecessarily hard, for no apparent reason. Or buggy. Or both. I really wish the that those issues get ironed out over time, because I do like Gnome and the underlying ideas.

How to use Cacio-tta with FEST for Java GUI testing

Now that Cacio is finally released and promoted to Maven central repositories, I want to provide you a quick HOWTO about testing user interfaces.

For writing GUI tests, I am using FEST. However, one problem with running GUI tests is that they need to create windows, grab keyboard focus, and do all sorts of interaction with the screen. This has several disadvantages:

  • When running on a developer machine, it requires to not touch anything while running the test suite, otherwise tests may fail or get blocked, e.g. because keyboard focus cannot be acquired or a test window suddenly goes into the background when it should not, etc. This can be quite annoying, especially with large test suites.
  • When running on a Unix-based continuous integration server, it’s possible to utilize xvfb driver to create a virtual framebuffer, but I still often noticed problems that seem to come from asynchronous behaviour of the X architecture. For example, a test would draw a border in RED, and when checking some pixels to be red, they are not. They would only turn red after some unknown time, when the drawing commands have been processed by the X server and graphics card.
  • When running on Windows based CI server, this sort of asynchronous problems don’t occur, but to make up for it, Windows has its own share of problems. First of all, on Windows you need a real screen/graphics card to be present (bad on servers). Even worse, on many Windows servers, you need a user to be logged in, and stay logged in, and the CI server running in that session to be able to access the screen. On other servers, multiple concurrent logons are possible, but not sharing a screen, e.g. when some guy logs into the CI server to do some admin work, it would grab the screen from the CI user, etc. All very complicated and time consuming to setup.

This is where Cacio-tta comes into play. It provides a graphics stack for the Java VM, that is completely independent from the environment. It renders everything into a virtual screen, which is simply a BufferedImage, and is driven solely by java.awt.Robot events. This makes it a perfect fit for GUI testing environments. And using it is very easy. There are two ways do it, and I usually combine them.

1. Include Cacio in your Maven dependencies

Simply add the following in your pom.xml:


2. Run your test with CacioTestRunner

Add the following annotation to your test class:


If you are using FEST and want to take advantage of its additional capabilities (create screenshots on failure, run tests in EDT, etc), use:


instead. Those annotations will make your test run in a Cacio-tta virtual screen environment

3. Optionally, run the whole test suite in Cacio

In some cases, it may be necessary to run the whole test suite in Cacio-tta. In order to do so, add the following to your pom:


This makes sure that Cacio is loaded instead of the default toolkit. This may be necessary, if any of your tests load any AWT, Java2d or Swing class, and are not annotated with the above annotation. This is because Java only allows to set the toolkit once, and it cannot be unloaded or unset. When you load any GUI class before loading the CacioTestRunner, the default toolkit will be loaded, and tests will not run in Cacio.

4. For non-Maven based builds

Of course, Cacio can also be used in non-Maven builds (ant, make, ..). Just download the Cacio sources ( build them, include the JARs in your project somehow, and add the annotations as described above, or set the system properties awt.toolkit, java.awt.graphicsenv and java.awt.headless as described in the Maven snippet above.

Last but not least, if you encounter any problems with Cacio or the above instructions, let us know!

Caciocavallo 1.1 released

I am very proud to announce that after 4 years of development, we publish the first release of Caciocavallo. Due to some funny problems with Maven and my network, it is numbered 1.1.

Cacio started out in 2008 as one of ten projects to take on the OpenJDK Innovators Challenge. This is how it all began, at FOSDEM 2008:

There, Rich Sand and Tom Marble told us about the OpenJDK Innovators Challenge, and Mario brought this fine cheese (on which Steph broke the above knife), and thus the idea to do *something* was born. The original idea was to improve OpenJDK’s GUI APIs implementation (AWT, Java2D and Swing) to make it easier to port the stack to new platforms and plug them in as separate modules. As a proof of concept, we provided a very simple implementation, it was based on Escher if I remember correctly. Back then, Cacio was nominated bronze for the challenge, which made us very proud.

Following that, the focus shifted from fixing OpenJDK to actually porting the GUI stack (although merging the patches into JDK7 would keep us busy for a while to come, I have some special memories of the FontManager in particular). At this time, Cacio evolved from a simple proof of concept implementation into a fully fledged framework for porting Java’s graphics stack. The idea was to make it as easy as possible to do it. And because a large part of that porting work would consist of implementing all those AWT widgets that almost nobody’s actually using (and which are not even available native on many interesting platforms), we thought out a little trick: why not let Swing do all the widget magic in AWT? I.e. from the standpoint of the programmer, you would code AWT, but under the hood, Swing would be used for rendering, event handling, etc. The only thing that a Java GUI port would need to provide is an implementation of Java2D (which is fairly easy to do if you don’t shoot for the fastest ever implementation), and some event handling. Using that, I was able to get a working prototype (working as in, can run Java2Demo) of a DirectFB backend working in 1 week!

After I left aicas and joined Sun in June 2009, things got a bit quiet around Cacio for a while. I was still merging back some of the FontManager fixes into OpenJDK7 mainline, but that was basically it. Until Clemens Eisserer (the guy who took the gold award in the OpenJDK Challenge in 2008 with his XRender work) hit the stage with his Google Summer of Code 2011 proposal to implement a Java GUI layer that would render a Swing application running on the server (in the cloud if you will) directly to a browser, using HTML5. What followed was a time of very intense development, and the outcome was (and still is) absolutely remarkable. We could get a number of Swing applications to run ‘in the cloud’ and talking to browsers running on all sorts of platforms without Java installed locally, including iPad/iPhone, Android phones, embedded devices and of course the usual desktop OSes. It was a huge success which sparked new life in Caciocavallo land.

By this time, I already left Sun and worked for JP Morgan, and there I got really hooked up with test driven development and all other sorts of agile practices. One of the problems that we hit there was running GUI tests on continuous integration servers, because they would either need access to a desktop (when running on a Windows server) and get disturbed whenever somebody logged on the machine, or hit problems with X servers (on Unix systems) due to asynchronicity and other funny stuff. At some point I realized that Cacio could help us here as well, and this gave birth to another side project of Cacio, called Cacio-TTA (pronounced Caciotta which is one special type of Caciocavallo in terms of cheese). This is basically a virtual GUI stack, made for running GUI tests in an isolated environment. It would render to a BufferedImage only, and process events solely through java.awt.Robot. Relatively simple to implement (took me 1 evening, thanks to the already existing Cacio framework), and enourmously useful: each GUI test would now run completely independent from any platform GUI stack, be it X server or Windows desktop or whatever. You can run tests in parallel if you like (just spawn enough JVMs), and every test would get its own GUI sandbox. This finally makes Java GUI testing reliable and predictable.

That was a long journey, and now we are here and proudly releasing Caciocavallo 1.1! Let me take this opportunity to say thanks to a couple of companies and individuals:

  • First of all, Sun Microsystems, which sparked the original idea(s) through the OpenJDK Innovators Challenge. And the folks in Sun’s graphics team, Phil Race, Jim Graham, Dmitri Trembovetski, Andrei Dmitriev and a bunch of invisible folks for following and supporting us.
  • Mario Torre, my friend and compagnon on this journey. It would not have been possible without you.
  • Of course, Clemens Eisserer for Cacio-Web and for revitalizing Cacio.
  • Andrei Dmitriev, he supported us a lot by initial testing, reporting issues, running the TCK and some then-Sun-internal test suites over it. And he presented at JavaOne 2009 about Cacio with us.
  • Aicas, who provided me and Mario with time and support on the initial OpenJDK Challenge project, and keep contributing to it, in particular:
  • Ralf Heini and Ingo Proetel, who keep submitting bug reports and fixes.
  • Glen Schrader who joined Cacio just some weeks ago to submit some important bugfixes.
  • A whole bunch of individuals who give Cacio a try, who write encouraging words in our blog comments, who lobby their development teams to use Cacio, etc etc.


Life doesn’t stop here of course. We are currently using and improving Cacio for our Thermostat at Red Hat. And we still have a long list of plans:

  • Finally get WebJDK project going. The idea is to build on top of Cacio-Web a distribution of OpenJDK that truly runs ‘in the cloud’ (i.e. on a stupid webserver farm in a boring server room). Swing applications running on WebJDK would talk directly to the client’s browser, would have (limited) access to the client filesystem and printer, will have their session managed properly, it will be scalable and easy to just throw more servers at it should resources become a problem, etc etc. I still have that filesystem support lying around on my harddisk, uncommitted.
  • Another working prototype that is uncommitted is the DirectFB port, to run Java on embedded systems, using DirectFB as graphics layer.
  • Finish a bunch of unfinished backends, in particular SDL.
  • Package Cacio for Fedora 17.
  • Plus all sorts of interesting things that we don’t know about yet, but which will surely happen.

In case you are interested in the code (*cough*), you can find source tarballs here. Or in the Cacio Mercurial repository. Or in Maven central (should appear soonish):


Please notice that Cacio only works on OpenJDK7 and most likely Oracle JDK 7¬† (no JDK6, no OpenJDK6, OpenJDK8 will be supported once it’s out, no other JVM runtimes like Classpath based ones are supported).

Thermostat CLI

The last weeks I’ve been working on a command line interface (CLI) for Thermostat. Now the first bunch of commands are available. It’s not much yet, but the groundwork for the really useful ones is laid:

Command line interface for Thermostat

3 Commands can be seen in the above screenshot:

  • thermostat service --start (to start the thermostat agent and storage)
  • thermostat list-vms (to list the currently monitored VMs)
  • thermostat help (to print out a list of available commands and get usage information

This command line interface will make Theromstat useful in settings where the already available Swing GUI is not feasible, like when running over SSH connections on a remote machine, or for automated/scripted monitoring.

Next step will be to add some really useful commands, like getting some actual statistics and analysis about the monitored VMs, find memory leaks, deadlocks, and so on.

How to test drive a main method

Test driven development is all nice and dandy, but there are some areas that most people find notoriously difficult to test, and often dismiss them as not worth it. The main() method, entry point for any application, is one of them. The problem with main methods is that they combine at least two of the patterns that make testing difficult: static method calls and many constructor calls. Let me illustrate it with a snippet of code from Thermostat:

public static void main(String[] args) {
    CommandContextFactory cmdCtxFactory = CommandContextFactory.getInstance();
    CommandRegistry registry = cmdCtxFactory.getCommandRegistry();
    ServiceLoader cmds = ServiceLoader.load(Command.class);
    if (hasNoArguments()) {
    } else {

This is a rather common way to implement a main method: setup a bunch of things by calling factories and constructors, then launch whatever needs to be launched.

The first thing that needs to be done to make this testable is to reduce the problematic static method and constructor calls by moving them into a separate class. How does this look?

public static void main(String[] args) {
    new Launcher().run(args);

This way, we can now test the bulk of the functionality using fairly normal testing techniques such as mocking, injection, etc. We still want to test the actual main method though (even though many will now cry out that this is so trival that it’s not worth even thinking about it… but wait, the solution is not so difficult either).

What would need to be tested here? Obviously, there is no state change, only interactions. Therefore, the test would need to be a verification of interaction of the main class with the Launcher class. More specifically, it would need to verify that the run() method is called on the newly constructed Launcher with the correct arguments (and not, say, null). However, we need to somehow get rid of the construction itself and open a way to inject the Launcher instance. That’s how I solved it:

public class Thermostat {

    private static Launcher launcher = new Launcher();

    public static void main(String[] args) {;

    static void setLauncher(Launcher launcher) {
        Thermostat.launcher = launcher;

This way, we can override the launcher to be tested with a mock launcher, and verify the correct interaction:

public class ThermostatTest {

    public void testThermostatMain() {
        Launcher launcher = Mockito.mock(Launcher.class);
        Thermostat.main(new String[] { "test1", "test2" });
        Mockito.verify(launcher).run(new String[] { "test1", "test2" });

The verifyNew() call verifies that we actually create a new instance of Launcher in the static initializer of Thermostat (the main class), and the last call verifies that run() is called on our mock launcher with the correct arguments.

And if you are interested in how the rest of the launching code now looks like, and how we test it (in fairly normal unit testing style), take a look at the Launcher and LauncherTest classes.

If you have other solutions to the problem of unit testing main methods, I would be very interested in hearing from them in the comments!

Test Driven Thermostat Development

I’ve recently become very excited about test driven development (TDD) and come to really integrate it into my daily work for Thermostat. The idea behind TDD is to only ever write production code to fix a failing test. Which implies that you write the test *first*. I found that the obvious advantage — the fact that you end up with a fully tested codebase — is really only a side effect, a very nice one though. What seems more important is the thought processes and change in work style that TDD provokes. It forces you to think first about what you actually want. It encourages a great design in the code you write. It encourages you to write clean code (because¬† messy code is difficult to test), it encourages you to write code with little dependencies and which separates concerns nicely (because otherwise, you guessed it, it’s difficult to test). It forces you to focus on one thing at a time. In the long run, this results in a continuously clean code base, low bug count and therefore high productivity.

One thing that must not be neglected in TDD is refactoring. TDD effectively turns around the classical work cycle of design-code-test. Instead of designing our software first, then writing the code, and then test it, we write the test first, then write the code to fix that test, and then refactor our code to yield the best possible design for that code. Test-code-design (or test-code-refactor if you want). And it works really well. But one must not forget that last step, refactoring. This is what keeps the code clean all the time. Some people tend to shy away from refactoring, because ‘it might break stuff’. But not here! We have tests in place as safety nets. Consider this: codebases need to change all the time. They need to adopt to new requirements, and the design of the code needs to be enhanced to support new ideas. We need to be able to change. And we prove that we’re able to change things, by changing them all the time! YAY. And it works surprisingly well.

Just recently we had a nice example how this way of working yields significant improvement in the design. Consider this MainWindowFacadeImpl class and its associate MainWindowFacadeImplTest. That’s basically a controller in an MVC triad. There was one thing that was bugging me a lot: the use of a spy (partial mock) for the class-under-test. I.e. we wanted the real object, but mock just a little bit with it. That was because we wanted to verify that stop() would be called under certain conditions (when a shutdown event got fired from the view). That stop() method would then stop the timer that sends updates to the view. (You don’t need to understand what exactly this stuff does, just that there’s a timer that fires periodically to update some data in the view.) Also, how should I go about testing the timer itself? I basically wanted to verify that, upon calling start(), the view would get updated every 10 seconds. Now, I could go on and write a test that calls start(), then wait 10 seconds (or a little more) and verify that the view got updated, then wait another 10 seconds, and verify again. This would have made the test rather long-running though. And one of the rules for TDD is: tests should be executed fast (because we want to run them all the time, and not wait minutes for them to complete). Short summary, that timer caused all sorts of troubles. In order to test it, I’d need to create a partial mock out of the class-under-test, and write brittle threading-aware tests. Not good.

So I took a step back and thought about what is actually going on? Why is this so difficult to test? And it occured to me that this controller class was mixing at least 2 concerns: the actual timing of stuff (including the associated threading, etc), and the actual timer *action*, that is updating the view. Clearly, the latter one is a valid thing that the controller should handle. But the timer? Maybe we can externalize it somehow. (And you need to know that we have those timers in other places as well, scatter throughout the code.)

What we actually needed here is a way to provide (and inject) timing capabilities to the controller and other APIs. So I created a TimerFactory that creates Timer objects. Those can be configured by the controller, and started and stopped. They are very similar to java.util.Timer or java.util.concurrent.ScheduledExecutorService (on purpose), but more object oriented as I find. Such a TimerFactory would be made available in the ApplicationContext (kindof a global application wide scope), so that each controller or whatever code needs a timer, can grab one from there. I made one implementation of that TimerFactory, which is called ThreadPoolTimerFactory, which is backed by a ScheduledExecutorService. This relatively simple change in the design yielded a number of big advantages:

  • The timing code is now separated nicely from the timer action. For the purpose of test, I can now inject a mock Timer to the application, which I can actually *control* from the test. I.e. I can fire the timer action at my sole descretion, instead of relying on some undeterministic threading behaviour. And I don’t need to wait 10 seconds to do that.
  • I can actually test the timer itself in one place, rather writing threading-aware tests for every component that uses such a timer.
  • I only get one (or very few) thread(s) to support all the timing needs of the various components of the application, rather than having one timer per component.
  • Finally, I am now able to get rid of that partial mock. Instead of testing that controller.stop() has been called, I can now more easily verify that stop() has been called on the timer (because I injected a mock timer). Bingo!

Now that was quite a little success. And all only because I was unnerved by that little seemingly innocent use of the partial mock. Some people would probably neglect such issue and say, what the hell? As long as it works! But it *is* a code smell, and code smells are not to be taken lightly. You understand this when you’ve worked with code bases which ended up to be only a godaweful amount of sh***. And it shows why the ‘refactoring’ step in test-code-refactor is so important. Paraphrasing some popular saying: Quality is not some magical thing that is sent by a great god (or flying spagetti monster if you will) in the sky, it is 100s of little acts of care.


Get every new post delivered to your Inbox.

Join 160 other followers