How to use Cacio-tta with FEST for Java GUI testing

Now that Cacio is finally released and promoted to Maven central repositories, I want to provide you a quick HOWTO about testing user interfaces.

For writing GUI tests, I am using FEST. However, one problem with running GUI tests is that they need to create windows, grab keyboard focus, and do all sorts of interaction with the screen. This has several disadvantages:

• When running on a developer machine, it requires to not touch anything while running the test suite, otherwise tests may fail or get blocked, e.g. because keyboard focus cannot be acquired or a test window suddenly goes into the background when it should not, etc. This can be quite annoying, especially with large test suites.
• When running on a Unix-based continuous integration server, it’s possible to utilize xvfb driver to create a virtual framebuffer, but I still often noticed problems that seem to come from asynchronous behaviour of the X architecture. For example, a test would draw a border in RED, and when checking some pixels to be red, they are not. They would only turn red after some unknown time, when the drawing commands have been processed by the X server and graphics card.
• When running on Windows based CI server, this sort of asynchronous problems don’t occur, but to make up for it, Windows has its own share of problems. First of all, on Windows you need a real screen/graphics card to be present (bad on servers). Even worse, on many Windows servers, you need a user to be logged in, and stay logged in, and the CI server running in that session to be able to access the screen. On other servers, multiple concurrent logons are possible, but not sharing a screen, e.g. when some guy logs into the CI server to do some admin work, it would grab the screen from the CI user, etc. All very complicated and time consuming to setup.

This is where Cacio-tta comes into play. It provides a graphics stack for the Java VM, that is completely independent from the environment. It renders everything into a virtual screen, which is simply a BufferedImage, and is driven solely by java.awt.Robot events. This makes it a perfect fit for GUI testing environments. And using it is very easy. There are two ways do it, and I usually combine them.

1. Include Cacio in your Maven dependencies

Simply add the following in your pom.xml:

<dependency>
  <groupId>net.java.openjdk.cacio</groupId>
  <artifactId>cacio-tta</artifactId>
  <scope>test</scope>
</dependency>

2. Run your test with CacioTestRunner

Add the following annotation to your test class:

@RunWith(CacioTestRunner.class)

If you are using FEST and want to take advantage of its additional capabilities (create screenshots on failure, run tests in EDT, etc), use:

@RunWith(CacioFESTRunner.class)

instead. Those annotations will make your test run in a Cacio-tta virtual screen environment

3. Optionally, run the whole test suite in Cacio

In some cases, it may be necessary to run the whole test suite in Cacio-tta. In order to do so, add the following to your pom:

<build>
  <plugins>
    <plugin>
      <groupId>org.apache.maven.plugins</groupId>
      <artifactId>maven-surefire-plugin</artifactId>
      <version>2.12</version>
      <configuration>
        <systemPropertyVariables>
          <java.awt.headless>false</java.awt.headless>
          <awt.toolkit>net.java.openjdk.cacio.ctc.CTCToolkit</awt.toolkit>
          <java.awt.graphicsenv>net.java.openjdk.cacio.ctc.CTCGraphicsEnvironment</java.awt.graphicsenv>
        </systemPropertyVariables>
      </configuration>
    </plugin>

This makes sure that Cacio is loaded instead of the default toolkit. This may be necessary, if any of your tests load any AWT, Java2d or Swing class, and are not annotated with the above annotation. This is because Java only allows to set the toolkit once, and it cannot be unloaded or unset. When you load any GUI class before loading the CacioTestRunner, the default toolkit will be loaded, and tests will not run in Cacio.

4. For non-Maven based builds

Of course, Cacio can also be used in non-Maven builds (ant, make, ..). Just download the Cacio sources (http://ladybug-studio.com/downloads/cacio/) build them, include the JARs in your project somehow, and add the annotations as described above, or set the system properties awt.toolkit, java.awt.graphicsenv and java.awt.headless as described in the Maven snippet above.

Last but not least, if you encounter any problems with Cacio or the above instructions, let us know!

Caciocavallo 1.1 released

I am very proud to announce that after 4 years of development, we publish the first release of Caciocavallo. Due to some funny problems with Maven and my network, it is numbered 1.1.

Cacio started out in 2008 as one of ten projects to take on the OpenJDK Innovators Challenge. This is how it all began, at FOSDEM 2008:

There, Rich Sand and Tom Marble told us about the OpenJDK Innovators Challenge, and Mario brought this fine cheese (on which Steph broke the above knife), and thus the idea to do *something* was born. The original idea was to improve OpenJDK’s GUI APIs implementation (AWT, Java2D and Swing) to make it easier to port the stack to new platforms and plug them in as separate modules. As a proof of concept, we provided a very simple implementation, it was based on Escher if I remember correctly. Back then, Cacio was nominated bronze for the challenge, which made us very proud.

Following that, the focus shifted from fixing OpenJDK to actually porting the GUI stack (although merging the patches into JDK7 would keep us busy for a while to come, I have some special memories of the FontManager in particular). At this time, Cacio evolved from a simple proof of concept implementation into a fully fledged framework for porting Java’s graphics stack. The idea was to make it as easy as possible to do it. And because a large part of that porting work would consist of implementing all those AWT widgets that almost nobody’s actually using (and which are not even available native on many interesting platforms), we thought out a little trick: why not let Swing do all the widget magic in AWT? I.e. from the standpoint of the programmer, you would code AWT, but under the hood, Swing would be used for rendering, event handling, etc. The only thing that a Java GUI port would need to provide is an implementation of Java2D (which is fairly easy to do if you don’t shoot for the fastest ever implementation), and some event handling. Using that, I was able to get a working prototype (working as in, can run Java2Demo) of a DirectFB backend working in 1 week!

After I left aicas and joined Sun in June 2009, things got a bit quiet around Cacio for a while. I was still merging back some of the FontManager fixes into OpenJDK7 mainline, but that was basically it. Until Clemens Eisserer (the guy who took the gold award in the OpenJDK Challenge in 2008 with his XRender work) hit the stage with his Google Summer of Code 2011 proposal to implement a Java GUI layer that would render a Swing application running on the server (in the cloud if you will) directly to a browser, using HTML5. What followed was a time of very intense development, and the outcome was (and still is) absolutely remarkable. We could get a number of Swing applications to run ‘in the cloud’ and talking to browsers running on all sorts of platforms without Java installed locally, including iPad/iPhone, Android phones, embedded devices and of course the usual desktop OSes. It was a huge success which sparked new life in Caciocavallo land.

By this time, I already left Sun and worked for JP Morgan, and there I got really hooked up with test driven development and all other sorts of agile practices. One of the problems that we hit there was running GUI tests on continuous integration servers, because they would either need access to a desktop (when running on a Windows server) and get disturbed whenever somebody logged on the machine, or hit problems with X servers (on Unix systems) due to asynchronicity and other funny stuff. At some point I realized that Cacio could help us here as well, and this gave birth to another side project of Cacio, called Cacio-TTA (pronounced Caciotta which is one special type of Caciocavallo in terms of cheese). This is basically a virtual GUI stack, made for running GUI tests in an isolated environment. It would render to a BufferedImage only, and process events solely through java.awt.Robot. Relatively simple to implement (took me 1 evening, thanks to the already existing Cacio framework), and enourmously useful: each GUI test would now run completely independent from any platform GUI stack, be it X server or Windows desktop or whatever. You can run tests in parallel if you like (just spawn enough JVMs), and every test would get its own GUI sandbox. This finally makes Java GUI testing reliable and predictable.

That was a long journey, and now we are here and proudly releasing Caciocavallo 1.1! Let me take this opportunity to say thanks to a couple of companies and individuals:

• First of all, Sun Microsystems, which sparked the original idea(s) through the OpenJDK Innovators Challenge. And the folks in Sun’s graphics team, Phil Race, Jim Graham, Dmitri Trembovetski, Andrei Dmitriev and a bunch of invisible folks for following and supporting us.
• Mario Torre, my friend and compagnon on this journey. It would not have been possible without you.
• Of course, Clemens Eisserer for Cacio-Web and for revitalizing Cacio.
• Andrei Dmitriev, he supported us a lot by initial testing, reporting issues, running the TCK and some then-Sun-internal test suites over it. And he presented at JavaOne 2009 about Cacio with us.
• Aicas, who provided me and Mario with time and support on the initial OpenJDK Challenge project, and keep contributing to it, in particular:
• Ralf Heini and Ingo Proetel, who keep submitting bug reports and fixes.
• Glen Schrader who joined Cacio just some weeks ago to submit some important bugfixes.
• A whole bunch of individuals who give Cacio a try, who write encouraging words in our blog comments, who lobby their development teams to use Cacio, etc etc.

THANK YOU ALL!

Life doesn’t stop here of course. We are currently using and improving Cacio for our Thermostat at Red Hat. And we still have a long list of plans:

• Finally get WebJDK project going. The idea is to build on top of Cacio-Web a distribution of OpenJDK that truly runs ‘in the cloud’ (i.e. on a stupid webserver farm in a boring server room). Swing applications running on WebJDK would talk directly to the client’s browser, would have (limited) access to the client filesystem and printer, will have their session managed properly, it will be scalable and easy to just throw more servers at it should resources become a problem, etc etc. I still have that filesystem support lying around on my harddisk, uncommitted.
• Another working prototype that is uncommitted is the DirectFB port, to run Java on embedded systems, using DirectFB as graphics layer.
• Finish a bunch of unfinished backends, in particular SDL.
• Package Cacio for Fedora 17.
• Plus all sorts of interesting things that we don’t know about yet, but which will surely happen.

In case you are interested in the code (*cough*), you can find source tarballs here. Or in the Cacio Mercurial repository. Or in Maven central (should appear soonish):

<dependency>
<groupId>net.java.openjdk.cacio</groupId>
<artifactId>cacio-tta</artifactId>
<version>1.1</version>
</dependency>

Please notice that Cacio only works on OpenJDK7 and most likely Oracle JDK 7  (no JDK6, no OpenJDK6, OpenJDK8 will be supported once it’s out, no other JVM runtimes like Classpath based ones are supported).