Thermostat CLI

The last weeks I’ve been working on a command line interface (CLI) for Thermostat. Now the first bunch of commands are available. It’s not much yet, but the groundwork for the really useful ones is laid:

Command line interface for Thermostat

3 Commands can be seen in the above screenshot:

  • thermostat service --start (to start the thermostat agent and storage)
  • thermostat list-vms (to list the currently monitored VMs)
  • thermostat help (to print out a list of available commands and get usage information

This command line interface will make Theromstat useful in settings where the already available Swing GUI is not feasible, like when running over SSH connections on a remote machine, or for automated/scripted monitoring.

Next step will be to add some really useful commands, like getting some actual statistics and analysis about the monitored VMs, find memory leaks, deadlocks, and so on.

How to test drive a main method

Test driven development is all nice and dandy, but there are some areas that most people find notoriously difficult to test, and often dismiss them as not worth it. The main() method, entry point for any application, is one of them. The problem with main methods is that they combine at least two of the patterns that make testing difficult: static method calls and many constructor calls. Let me illustrate it with a snippet of code from Thermostat:

public static void main(String[] args) {
    CommandContextFactory cmdCtxFactory = CommandContextFactory.getInstance();
    CommandRegistry registry = cmdCtxFactory.getCommandRegistry();
    ServiceLoader cmds = ServiceLoader.load(Command.class);
    registry.registerCommands(cmds);
    if (hasNoArguments()) {
        runHelpCommand();
    } else {
        runCommandFromArguments();
    }
}

This is a rather common way to implement a main method: setup a bunch of things by calling factories and constructors, then launch whatever needs to be launched.

The first thing that needs to be done to make this testable is to reduce the problematic static method and constructor calls by moving them into a separate class. How does this look?

public static void main(String[] args) {
    new Launcher().run(args);
}

This way, we can now test the bulk of the functionality using fairly normal testing techniques such as mocking, injection, etc. We still want to test the actual main method though (even though many will now cry out that this is so trival that it’s not worth even thinking about it… but wait, the solution is not so difficult either).

What would need to be tested here? Obviously, there is no state change, only interactions. Therefore, the test would need to be a verification of interaction of the main class with the Launcher class. More specifically, it would need to verify that the run() method is called on the newly constructed Launcher with the correct arguments (and not, say, null). However, we need to somehow get rid of the construction itself and open a way to inject the Launcher instance. That’s how I solved it:

public class Thermostat {

    private static Launcher launcher = new Launcher();

    public static void main(String[] args) {
        launcher.run(args);
    }

    static void setLauncher(Launcher launcher) {
        Thermostat.launcher = launcher;
    }
}

This way, we can override the launcher to be tested with a mock launcher, and verify the correct interaction:

public class ThermostatTest {

    @Test
    public void testThermostatMain() {
        Launcher launcher = Mockito.mock(Launcher.class);
        Thermostat.setLauncher(launcher);
        Thermostat.main(new String[] { "test1", "test2" });
        PowerMockito.verifyNew(Launcher.class).withNoArguments();
        Mockito.verify(launcher).run(new String[] { "test1", "test2" });
     }
}

The verifyNew() call verifies that we actually create a new instance of Launcher in the static initializer of Thermostat (the main class), and the last call verifies that run() is called on our mock launcher with the correct arguments.

And if you are interested in how the rest of the launching code now looks like, and how we test it (in fairly normal unit testing style), take a look at the Launcher and LauncherTest classes.

If you have other solutions to the problem of unit testing main methods, I would be very interested in hearing from them in the comments!

Test Driven Thermostat Development

I’ve recently become very excited about test driven development (TDD) and come to really integrate it into my daily work for Thermostat. The idea behind TDD is to only ever write production code to fix a failing test. Which implies that you write the test *first*. I found that the obvious advantage — the fact that you end up with a fully tested codebase — is really only a side effect, a very nice one though. What seems more important is the thought processes and change in work style that TDD provokes. It forces you to think first about what you actually want. It encourages a great design in the code you write. It encourages you to write clean code (becauseĀ  messy code is difficult to test), it encourages you to write code with little dependencies and which separates concerns nicely (because otherwise, you guessed it, it’s difficult to test). It forces you to focus on one thing at a time. In the long run, this results in a continuously clean code base, low bug count and therefore high productivity.

One thing that must not be neglected in TDD is refactoring. TDD effectively turns around the classical work cycle of design-code-test. Instead of designing our software first, then writing the code, and then test it, we write the test first, then write the code to fix that test, and then refactor our code to yield the best possible design for that code. Test-code-design (or test-code-refactor if you want). And it works really well. But one must not forget that last step, refactoring. This is what keeps the code clean all the time. Some people tend to shy away from refactoring, because ‘it might break stuff’. But not here! We have tests in place as safety nets. Consider this: codebases need to change all the time. They need to adopt to new requirements, and the design of the code needs to be enhanced to support new ideas. We need to be able to change. And we prove that we’re able to change things, by changing them all the time! YAY. And it works surprisingly well.

Just recently we had a nice example how this way of working yields significant improvement in the design. Consider this MainWindowFacadeImpl class and its associate MainWindowFacadeImplTest. That’s basically a controller in an MVC triad. There was one thing that was bugging me a lot: the use of a spy (partial mock) for the class-under-test. I.e. we wanted the real object, but mock just a little bit with it. That was because we wanted to verify that stop() would be called under certain conditions (when a shutdown event got fired from the view). That stop() method would then stop the timer that sends updates to the view. (You don’t need to understand what exactly this stuff does, just that there’s a timer that fires periodically to update some data in the view.) Also, how should I go about testing the timer itself? I basically wanted to verify that, upon calling start(), the view would get updated every 10 seconds. Now, I could go on and write a test that calls start(), then wait 10 seconds (or a little more) and verify that the view got updated, then wait another 10 seconds, and verify again. This would have made the test rather long-running though. And one of the rules for TDD is: tests should be executed fast (because we want to run them all the time, and not wait minutes for them to complete). Short summary, that timer caused all sorts of troubles. In order to test it, I’d need to create a partial mock out of the class-under-test, and write brittle threading-aware tests. Not good.

So I took a step back and thought about what is actually going on? Why is this so difficult to test? And it occured to me that this controller class was mixing at least 2 concerns: the actual timing of stuff (including the associated threading, etc), and the actual timer *action*, that is updating the view. Clearly, the latter one is a valid thing that the controller should handle. But the timer? Maybe we can externalize it somehow. (And you need to know that we have those timers in other places as well, scatter throughout the code.)

What we actually needed here is a way to provide (and inject) timing capabilities to the controller and other APIs. So I created a TimerFactory that creates Timer objects. Those can be configured by the controller, and started and stopped. They are very similar to java.util.Timer or java.util.concurrent.ScheduledExecutorService (on purpose), but more object oriented as I find. Such a TimerFactory would be made available in the ApplicationContext (kindof a global application wide scope), so that each controller or whatever code needs a timer, can grab one from there. I made one implementation of that TimerFactory, which is called ThreadPoolTimerFactory, which is backed by a ScheduledExecutorService. This relatively simple change in the design yielded a number of big advantages:

  • The timing code is now separated nicely from the timer action. For the purpose of test, I can now inject a mock Timer to the application, which I can actually *control* from the test. I.e. I can fire the timer action at my sole descretion, instead of relying on some undeterministic threading behaviour. And I don’t need to wait 10 seconds to do that.
  • I can actually test the timer itself in one place, rather writing threading-aware tests for every component that uses such a timer.
  • I only get one (or very few) thread(s) to support all the timing needs of the various components of the application, rather than having one timer per component.
  • Finally, I am now able to get rid of that partial mock. Instead of testing that controller.stop() has been called, I can now more easily verify that stop() has been called on the timer (because I injected a mock timer). Bingo!

Now that was quite a little success. And all only because I was unnerved by that little seemingly innocent use of the partial mock. Some people would probably neglect such issue and say, what the hell? As long as it works! But it *is* a code smell, and code smells are not to be taken lightly. You understand this when you’ve worked with code bases which ended up to be only a godaweful amount of sh***. And it shows why the ‘refactoring’ step in test-code-refactor is so important. Paraphrasing some popular saying: Quality is not some magical thing that is sent by a great god (or flying spagetti monster if you will) in the sky, it is 100s of little acts of care.

Red Hat, week 4

Oh my, how time flies, just a few days ago I joined Red Hat, and suddenly the 4th week is already almost over.

And what an amazing 4 weeks that have been.

  • I started out by helping with the IcedTea security release.
  • More or less at the same time I jumped into the cool project that is Thermostat. With Thermostat, we want to develop the most sophisticated and useful monitoring solution for OpenJDK. My collegues Omair and Jon presented a first prototype at FOSDEM. Too bad I couldn’t make it this year. :-( I hope I find the time to blog more about Thermostat soon.
  • Driven by some of Thermostat’s requirements, I became a Fedora packager. My first package in Fedora is mockito, and more will likely follow.

Overall, I am excited about my work like I haven’t been for a long time. The work athmosphere is really energizing.

I recently found those thoughs about my work place (I am lucky to be able to work from home) that hit it on the nail ;-) :

Red Hat

Just like my good friend Mario, I will join Red Hat, starting from February, 1st. It seems like we only come in pairs, like Laurel & Hardy. I just can’t decide who would be Laurel and who would be Hardy :-) The thing about us is that we don’t just add up. We amplify each other.

I am very excited about this, Red Hat is about the best company I can imagine working for. They stand for all the values that I stand for myself. Let me also document this important step in my life with a picture:

Roman joining Red Hat

Life after JP Morgan

By the end of November 2011 my contract with JP Morgan ended after 17 months. In many ways it was an interesting but difficult time. I could learn a lot of interesting (and not so interesting) things. Unfortunately, not so much technical stuff that I would have liked (I didn’t really expect that), but about how (really) big corporations work (I think they copy from dilbert.com), about software engineering processes and banking stuff. It was quite difficult for me and my family though, I needed to commute from south-west Germany to south-west Switzerland every second week, which was quite a struggle for all of us. I am happy that this is over now.

I took some free time to spend with my family for most of December, and now want to start through with cool new stuff. I am still not 100% sure which way to go. I am currently spending time to work on CacioWeb and want to turn it into a great product for LadyBug. We have some interested customers and could get some funding, so this is an obvious thing to do for the short term. We also have a bunch of other cool ideas in the pipeline. I am not sure if it can provide enough safe income for the long run to feed my family though, which is why I am also currently looking for an employment position in a nice company (if you happen to know a good one – here’s my CV – , let me know!).

To me that’s an exciting start into the new year. It will be interesting to see what will have happened one year from now :-)

 

How the SwingView works – painting

Now that the JavaFX source code starts to trickle out of Oracle, Mario proposed the SwingView of ThingsFX to the official code base of JavaFX. I thought I’d write a couple of technical storytelling posts about how the SwingView actually works and how we got there.

The first prototype

It all started when Mario, Robert and me were looking at JavaFX 2 for the first time a couple of weeks ago. We found that one of the big missing things is a Swing integration into JavaFX, to enable embedding/reusing work that has been done for Swing. Later that same day, Robert came to us and said, hey look what I’ve done, a SwingView to embed a Swing component into JavaFX. He showed us, and it worked indeed. It was a JButton in a JavaFX scene and when you pushed it, it changed it’s text. Very impressive! The way it worked was fairly simple but very smart at the same time: on the JavaFX view he registered a mouse clicked listener, and whenever that fires, he would call the button’s click() method and then paint the button to a BufferedImage, serialize it as PNG over ImageIO into a stream and read that stream from a JavaFX ImageView to show it on the JavaFX scene. Of course it was slow and inflexible (only handles specific events), but it was a good start and it got us working quickly.

Introducing the RepaintManager

One of the obvious shortcomings of the approach was that it only painted when the button got clicked. We needed to be more flexible to repaint more generic components whenever anything gets repainted (i.e. call to JComponent.repaint()). The only place in Swing that knows when to paint what is obviously the RepaintManager. So.. the first improvement was to install a custom RepaintManager, and keep track of dirty components by overriding addDirtyRegion() and then, when paintDirtyRegions() gets called, after all components have been painted by the superclass implementation, send the repainted components to JavaFX (using the same serialized BufferedImage approach as above). This is much more flexible and accurate than the original approach, it allowed us to implement full event handling (more on this in another post). It’s still not perfect though: whenever client code calls getGraphics() on the component to paint directly, it would blow up. On the way to implementing this RepaintManager solution, we needed to introduce a lot of infrastructure that is still around today (the RepaintManager is gone now as you will see in a bit), most importantly, we needed to provide the wrapped Swing component a Window and a WindowPeer, both of which we mocked up and have them delegate to JavaFX instead (stuff like getLocationOnScreen(), focus handling and other stuff) . In other words, the Swing component thinks it lives in its own window, while in reality the ProxyWindowPeer delegates most of its functionality to the JavaFX node that contains it (hence the name). We also got rid of the BufferedImage buffer that we introduced in the prototype, and now reuse the RepaintManager’s own backbuffer (otherwise we would get 3 layers of buffering).

Bye Bye RepaintManager

It soon became clear that the RepaintManager approach alone was not the ultimate solution. Most importantly, JFreeChart would eventually blow up since it calls getGraphics() on the Swing component directly to perform some sweet animation stuff. It was clear we need to support it. What we did was to implement a special Graphics2D subclass (ProxyGraphics2D) that wraps the BufferedImage buffer and delegates all drawing operations to the Graphics2D of that BufferedImage and copy that image to JavaFX from time to time. In that last bit lies the tricky part though. When to copy over to JavaFX? On Graphics2D.dispose() ? But what when code keeps the Graphics2D object and never disposes it? It still needs to paint something on screen. We could also trigger on certain events or on RepaintManager but none of this seems to be a fully working solution. What I ended up with was to paint whenever there was a couple of milliseconds (20-50 turned out to be good) of inactivity on the Graphics2D, or after it did not copy for too long, in case some code does a neverending drawing burst. This implementation fixes the NPE when code calls getGraphics(). And we soon realized that this obsoletes our own RepaintManager code above, since now all the painting can be done directly by the normal RepaintManager using the Graphics2D provided by the WindowPeer implementation (this also enables support for client side custom RepaintManagers). Also, after we have done that, we found that this new implementation failed on MacOS, and we tracked it down to the pointless use of SunGraphics2D in a couple of places in the Mac port where using plain Graphics2D is sufficient. Mario pushed a fix for this into the OpenJDK port.

Where we are now

The big remaining bottleneck now was the serialization of the BufferedImage over the stream. We contacted some folks at Oracle (Jonathan, Kevin, Richard: thanks a ton!) and they pointed us to some semi-internal API in JavaFX that can take a BufferedImage and turn it into a JavaFX Image. This solved the problem of serialization. We still have the problem that we always need to update the complete buffer, which takes too long for reasonable large Swing components. We hope to be able to resolve this as soon as the remaining parts of JavaFX get published. Also, Michael Paus pointed us to an interesting discussion that seems to indicate that the Oracle folks are working on a Canvas2D which provides Java2D access to a JavaFX node (you guys rock!), which is exactly what the SwingView needs for good performance.

In one of the next posts I will write something about the event handling, which is the other big/interesting part of the SwingView.

Follow

Get every new post delivered to your Inbox.

Join 105 other followers