How the SwingView works – painting

Now that the JavaFX source code starts to trickle out of Oracle, Mario proposed the SwingView of ThingsFX to the official code base of JavaFX. I thought I’d write a couple of technical storytelling posts about how the SwingView actually works and how we got there.

The first prototype

It all started when Mario, Robert and me were looking at JavaFX 2 for the first time a couple of weeks ago. We found that one of the big missing things is a Swing integration into JavaFX, to enable embedding/reusing work that has been done for Swing. Later that same day, Robert came to us and said, hey look what I’ve done, a SwingView to embed a Swing component into JavaFX. He showed us, and it worked indeed. It was a JButton in a JavaFX scene and when you pushed it, it changed it’s text. Very impressive! The way it worked was fairly simple but very smart at the same time: on the JavaFX view he registered a mouse clicked listener, and whenever that fires, he would call the button’s click() method and then paint the button to a BufferedImage, serialize it as PNG over ImageIO into a stream and read that stream from a JavaFX ImageView to show it on the JavaFX scene. Of course it was slow and inflexible (only handles specific events), but it was a good start and it got us working quickly.

Introducing the RepaintManager

One of the obvious shortcomings of the approach was that it only painted when the button got clicked. We needed to be more flexible to repaint more generic components whenever anything gets repainted (i.e. call to JComponent.repaint()). The only place in Swing that knows when to paint what is obviously the RepaintManager. So.. the first improvement was to install a custom RepaintManager, and keep track of dirty components by overriding addDirtyRegion() and then, when paintDirtyRegions() gets called, after all components have been painted by the superclass implementation, send the repainted components to JavaFX (using the same serialized BufferedImage approach as above). This is much more flexible and accurate than the original approach, it allowed us to implement full event handling (more on this in another post). It’s still not perfect though: whenever client code calls getGraphics() on the component to paint directly, it would blow up. On the way to implementing this RepaintManager solution, we needed to introduce a lot of infrastructure that is still around today (the RepaintManager is gone now as you will see in a bit), most importantly, we needed to provide the wrapped Swing component a Window and a WindowPeer, both of which we mocked up and have them delegate to JavaFX instead (stuff like getLocationOnScreen(), focus handling and other stuff) . In other words, the Swing component thinks it lives in its own window, while in reality the ProxyWindowPeer delegates most of its functionality to the JavaFX node that contains it (hence the name). We also got rid of the BufferedImage buffer that we introduced in the prototype, and now reuse the RepaintManager’s own backbuffer (otherwise we would get 3 layers of buffering).

Bye Bye RepaintManager

It soon became clear that the RepaintManager approach alone was not the ultimate solution. Most importantly, JFreeChart would eventually blow up since it calls getGraphics() on the Swing component directly to perform some sweet animation stuff. It was clear we need to support it. What we did was to implement a special Graphics2D subclass (ProxyGraphics2D) that wraps the BufferedImage buffer and delegates all drawing operations to the Graphics2D of that BufferedImage and copy that image to JavaFX from time to time. In that last bit lies the tricky part though. When to copy over to JavaFX? On Graphics2D.dispose() ? But what when code keeps the Graphics2D object and never disposes it? It still needs to paint something on screen. We could also trigger on certain events or on RepaintManager but none of this seems to be a fully working solution. What I ended up with was to paint whenever there was a couple of milliseconds (20-50 turned out to be good) of inactivity on the Graphics2D, or after it did not copy for too long, in case some code does a neverending drawing burst. This implementation fixes the NPE when code calls getGraphics(). And we soon realized that this obsoletes our own RepaintManager code above, since now all the painting can be done directly by the normal RepaintManager using the Graphics2D provided by the WindowPeer implementation (this also enables support for client side custom RepaintManagers). Also, after we have done that, we found that this new implementation failed on MacOS, and we tracked it down to the pointless use of SunGraphics2D in a couple of places in the Mac port where using plain Graphics2D is sufficient. Mario pushed a fix for this into the OpenJDK port.

Where we are now

The big remaining bottleneck now was the serialization of the BufferedImage over the stream. We contacted some folks at Oracle (Jonathan, Kevin, Richard: thanks a ton!) and they pointed us to some semi-internal API in JavaFX that can take a BufferedImage and turn it into a JavaFX Image. This solved the problem of serialization. We still have the problem that we always need to update the complete buffer, which takes too long for reasonable large Swing components. We hope to be able to resolve this as soon as the remaining parts of JavaFX get published. Also, Michael Paus pointed us to an interesting discussion that seems to indicate that the Oracle folks are working on a Canvas2D which provides Java2D access to a JavaFX node (you guys rock!), which is exactly what the SwingView needs for good performance.

In one of the next posts I will write something about the event handling, which is the other big/interesting part of the SwingView.


6 Responses to How the SwingView works – painting

  1. stardrive says:

    This is great progress! Immediate mode rendering is needed in JavaFX for more granular or complex rendering. Things like music notation scores for instance, where it would not be reasonable to made every little symbol aspect a retained node.

    Thanks for bringing Java2D back into the fold. Now if Oracle can produce the Canvas2D solution then we are rocking full tilt with the most versatile and powerful JavaFX possible.

  2. Pingback: Java desktop links of the week, December 4 | Jonathan Giles

  3. nuwanda says:

    Keep up the good work! A fully operational SwingView is sorely needed!

  4. Jay says:

    I am thankful that people with the know-how to do this integration are willing to work on it, and I am thankful that the JavaFX developers seem to be willing to integrate it.

  5. Josh says:

    What are the limitations on the SwingView? Will it support any Swing component? I ask because I’ve been trying to get vlcj running inside your SwingView because the JavaFX mediaplayer has such little codec support. It’s not really working though and I’m wondering if its a vlcj issue or the SwingView. Thanks.

    • Roman Kennke says:

      SwingView will support any Swing component that is implemented in plain Java. There are a few special components – and I am pretty sure vlcj is one of them – that access the native drawing surface for rendering, e.g. media players and such. Those will most likely not be supported.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: