Now that the JavaFX source code starts to trickle out of Oracle, Mario proposed the SwingView of ThingsFX to the official code base of JavaFX. I thought I’d write a couple of technical storytelling posts about how the SwingView actually works and how we got there.
The first prototype
It all started when Mario, Robert and me were looking at JavaFX 2 for the first time a couple of weeks ago. We found that one of the big missing things is a Swing integration into JavaFX, to enable embedding/reusing work that has been done for Swing. Later that same day, Robert came to us and said, hey look what I’ve done, a SwingView to embed a Swing component into JavaFX. He showed us, and it worked indeed. It was a JButton in a JavaFX scene and when you pushed it, it changed it’s text. Very impressive! The way it worked was fairly simple but very smart at the same time: on the JavaFX view he registered a mouse clicked listener, and whenever that fires, he would call the button’s click() method and then paint the button to a BufferedImage, serialize it as PNG over ImageIO into a stream and read that stream from a JavaFX ImageView to show it on the JavaFX scene. Of course it was slow and inflexible (only handles specific events), but it was a good start and it got us working quickly.
Introducing the RepaintManager
One of the obvious shortcomings of the approach was that it only painted when the button got clicked. We needed to be more flexible to repaint more generic components whenever anything gets repainted (i.e. call to JComponent.repaint()). The only place in Swing that knows when to paint what is obviously the RepaintManager. So.. the first improvement was to install a custom RepaintManager, and keep track of dirty components by overriding addDirtyRegion() and then, when paintDirtyRegions() gets called, after all components have been painted by the superclass implementation, send the repainted components to JavaFX (using the same serialized BufferedImage approach as above). This is much more flexible and accurate than the original approach, it allowed us to implement full event handling (more on this in another post). It’s still not perfect though: whenever client code calls getGraphics() on the component to paint directly, it would blow up. On the way to implementing this RepaintManager solution, we needed to introduce a lot of infrastructure that is still around today (the RepaintManager is gone now as you will see in a bit), most importantly, we needed to provide the wrapped Swing component a Window and a WindowPeer, both of which we mocked up and have them delegate to JavaFX instead (stuff like getLocationOnScreen(), focus handling and other stuff) . In other words, the Swing component thinks it lives in its own window, while in reality the ProxyWindowPeer delegates most of its functionality to the JavaFX node that contains it (hence the name). We also got rid of the BufferedImage buffer that we introduced in the prototype, and now reuse the RepaintManager’s own backbuffer (otherwise we would get 3 layers of buffering).
Bye Bye RepaintManager
It soon became clear that the RepaintManager approach alone was not the ultimate solution. Most importantly, JFreeChart would eventually blow up since it calls getGraphics() on the Swing component directly to perform some sweet animation stuff. It was clear we need to support it. What we did was to implement a special Graphics2D subclass (ProxyGraphics2D) that wraps the BufferedImage buffer and delegates all drawing operations to the Graphics2D of that BufferedImage and copy that image to JavaFX from time to time. In that last bit lies the tricky part though. When to copy over to JavaFX? On Graphics2D.dispose() ? But what when code keeps the Graphics2D object and never disposes it? It still needs to paint something on screen. We could also trigger on certain events or on RepaintManager but none of this seems to be a fully working solution. What I ended up with was to paint whenever there was a couple of milliseconds (20-50 turned out to be good) of inactivity on the Graphics2D, or after it did not copy for too long, in case some code does a neverending drawing burst. This implementation fixes the NPE when code calls getGraphics(). And we soon realized that this obsoletes our own RepaintManager code above, since now all the painting can be done directly by the normal RepaintManager using the Graphics2D provided by the WindowPeer implementation (this also enables support for client side custom RepaintManagers). Also, after we have done that, we found that this new implementation failed on MacOS, and we tracked it down to the pointless use of SunGraphics2D in a couple of places in the Mac port where using plain Graphics2D is sufficient. Mario pushed a fix for this into the OpenJDK port.
Where we are now
The big remaining bottleneck now was the serialization of the BufferedImage over the stream. We contacted some folks at Oracle (Jonathan, Kevin, Richard: thanks a ton!) and they pointed us to some semi-internal API in JavaFX that can take a BufferedImage and turn it into a JavaFX Image. This solved the problem of serialization. We still have the problem that we always need to update the complete buffer, which takes too long for reasonable large Swing components. We hope to be able to resolve this as soon as the remaining parts of JavaFX get published. Also, Michael Paus pointed us to an interesting discussion that seems to indicate that the Oracle folks are working on a Canvas2D which provides Java2D access to a JavaFX node (you guys rock!), which is exactly what the SwingView needs for good performance.
In one of the next posts I will write something about the event handling, which is the other big/interesting part of the SwingView.
The thing about us is that we don’t just add up. We amplify each other.
).
). The source code for the demo can be found 
