Hotspot Zero & OpenJDK8 & DaVinci

During the last couple of weeks, I brought the Zero (i.e. no-assembly) interpreter of OpenJDK up-to-date with OpenJDK8 and (most of) the DaVinci project. Zero is not usually built in standard and developer builds of OpenJDK, and thus often falls by the wayside when new features are developed in Hotspot. Various fixes needed to be made in order to be able to build and run Zero with the latest developments in OpenJDK8 and the DaVinci project again:

  • The Makefiles needed to be updated for copying the .debuginfo and/or .diz files into the right places.
  • Various small and fairly obvious changes needed to be made like renamed or extended method names. (Some of them are not only needed for Zero, even the X86 CPP interpreter was broken.)

With those changes in place (with help from Chris Phillips), it was possible to build Zero with OpenJDK8. The next big step for me was to make Zero build with the DaVinci patch set, in particular with the meth-lazy-7023639.patch. This turned out to be a fairly large refactoring. This patch introduced the so-called lambda-forms for method handles, which moves most of the invocation logic into Java code. The method handles now generate synthetic methods to implement the invocation logic on-the-fly and call that, instead of implementing all of it in the VM. The most important changes that I made was:

  • Rewrote the interpreter handler for the invokedynamic instruction. It now basically resolves the call site and the lambda form to call, and calls it.
  • Implement a handler for the new invokehandle instruction. This is a JVM-internal bytecode that is generated for calls to the various intrinsics (see below), which pushes an optional appendix parameter on top of the interpreter stack, which is then consumed by those intrinsics and is basically a pointer to the target method to call.
  • Implement 5 new intrinsics for MethodHandle: invokeBasic(), linkToStatic(), linkToSpecial(), linkToVirtual() and linkToInterface(). The first one is used to call into a lambda form using a polymorphic signature. The latter 4 call out to the target method and are basically signature-polymorphic versions of the various invoke* bytecodes.

With those changes in place I can now build a Zero version of OpenJDK8 with the DaVinci (MLVM) patches and all jtreg tests for java/lang/invoke are passing now. As a very welcome side effect, I was able to throw out a *lot* of convoluted code for invokedynamic support in the interpreter. The new code is fairly simple and straightforward (and thus, more maintainable) instead. The code is available from a personal repository for now (it’s a forest-clone of the upstream hotspot-comp forest, with MQ patch queues in hotspot and jdk modules). In order to check it out, do this (you need the mq extension):

 

hg clone http://icedtea.classpath.org/people/rkennke/hotspot-comp-zero/
cd hotspot-comp-zero
sh get-source.sh
cd hotspot
hg qpush -a
cd ../jdk
hg qpush -a
cd ..

Then build with Zero enabled, see my build script to get an overview of the build variables.

Update: If you’re only interested in the patches themselves, have a look at the patch repository.

Hacking Hotspot in Eclipse

I recently started working on Hotspot, with the goal of getting the Zero port up-to-date with respect to he recent developments in MLVM. In order to work most efficiently, I set up a work environment in Eclipse, and thought it’d be useful to share a little HOW-TO.

First of all, you need to install Eclipse/CDT if you haven’t already. This gives you a very powerful C/C++ development environment in Eclipse.

You can then open an OpenJDK source tree as C/C++ project by selecting: File -> New -> Other and then ‘C/C++’ -> ‘Makefile project with existing code’. Enter project name, e.g. OpenJDK8, and the path to the existing location. Select the appropriate ‘Toolchain for indexer Settings’ below. Click Finish.

Then open the project properties by right-clicking on the project, and selecting ‘Properties’. There we need to setup a couple of things.Under ‘C/C++ Build’ -> Environment, enter all environment variables that you would normally set on the command line for building OpenJDK. At the very least, you need ‘ALT_BOOTDIR’ and LANG=C. Under ‘C/C++ Build’, click the tab ‘Refresh Policy’ and remove the only path that is there (otherwise the whole workspace will be refreshed after a build, which takes looooong’). Optionally, add any paths under ‘build’ that you are interested in. Under ‘C/C++ General’ -> ‘Paths and Symbols’, select the tab ‘Source Location’ and remove the toplevel project path, and enter any source paths you are working with (e.g. hotspot/src). This limits what is visible to the indexer, etc. In order to take full advantage of Eclipse for debugging, I also changed ‘C/C++ Build’, ‘Behavior’ tab, replace ‘all’ with ‘debug_build’ This will normally do a debug build of OpenJDK, which means that you get all the symbols and no compiler optimizations in the binaries. In order to be able to load the symbols in gdb, add ‘ZIP_DEBUGINFO_FILES=0’ into the environment variables.Then click ‘Apply’ and ‘OK’ to close the settings dialog. Select ‘Project -> Build Project’ to launch the first build of OpenJDK in Eclipse.

Debugging with Eclipse is similarily straightforward, open Debug Configurations, add a new C/C++ application, set up its properties for the binary, arguments and environment variables (make sure you use a debug-build binary) and run the thing! Being able to fully debug in Eclipse, navigating stack, inspecting variables, setting breakpoints and stepping through the code is so much more useful than doing the same in plain GDB:

Debug Hotspot Zero with Eclipse

 

 

Please help get my son’s school class get a van

Today I have a somewhat unusual request to you: I need your vote to help get my son’s school class get a van. Let me explain. My son attends a local Montessori school in Ettenheim (Freiburg area in Germany). A local bank is doing a contest for all sorts of local associations to win a van (actually, 8 of them). All of the participating associations are asked to make a short video clip that shows why they need one. And since my son’s school class could really use such a van as sort of school bus or general purpose transportation tool, they made a short video clip. The video clip was made by the students themselves, and I think it turned out really great. The selection of the winners goes in 3 phases: there were 80 participating associations, in the first round 50 were selected by voting (we already passed that), in the next round the top 25 of those 50 are selected by voting again (that’s what I’m gonna ask you about) and finally, 8 of those 25 are selected by a jury. We need to get into that last round now. What can you do? Simply follow this link and click on the star. No registration required. You only need to fill in a captcha and copy+paste some crap in order to prove that you’re human. If you do this: THANK YOU! It’s a great help!

Update: If you already voted yesterday, you can vote again (everybody can vote once per day, up to July 8th)!

Thermostat 0.3 released

The Thermostat team (Jon, Omair, Mario and myself) are proud to announce Thermostat 0.3, codenamed ‘Not Yet’ (yeah yeah, I know). (For those who don’t know yet, Thermostat is a next generation and free/open source monitoring and diagnosing tool for OpenJDK.)

The major new feature is the support for creating and analysing memory by creating a heapdump and browsing a histogram of objects in memory. Additional features include a kill switch, using which you can terminate a running process (for example if it grossly misbehaves), a new Look and Feel (named Dolphin, made to look similar to Gnome 3’s Adwaita theme), new command line commands to support heap analysis, numerous little UI improvements and bugfixes.

The below screenshot shows the new heap analysis view. It monitors the heap usage of a selected JVM. When you click the big ‘Heap Dump’ button, a heap dump image is created and saved in the MongoDB, it shows up in the list below (you can take and analyze several heapdumps per VM.. they will be listed with the timestamp of their creation). Clicking on the heap dump in that list brings up a histogram view that lists all allocated objects, the number of their instances and the total size of their allocation. This list can be sorted by clicking on the respective table headers. (We are planning several more heap analysis features like searching specific objects and following their paths to GC roots, showing incoming and outgoing references, etc etc, but those will show up in the next release, in about 4 weeks from now.)

Thermostat heap analysis view

This was a fairly important release for us, it was our first milestone release, ready just in time for the Red Hat Summit (Deepak will present Thermostat there). Infact, it is the second milestone, milestone 0 was earlier this year when Jon and Omair showed Thermostat at FOSDEM. We took at lot of care and time to make Thermostat solid and useful, and while it’s not overflowing with features yet, we prepared the ground for a lot of greatness. Eventually, Thermostat will support monitoring remote VMs and whole clusters of VMs, and allow drilling down into problem areas and perform deep and fine grained analysis, utilizing information from the JVM itself as well as from the kernel, the operating system and even from the application server.

Please read the release announcement for more details about this new shiny release of Thermostat!

Journey Through The Past

During the last couple of days, Madeleine and I dedicated for a little adventure, which was to clean up our attic. And what a great journey it’s been. I met old friends that I haven’t seen in ages. I have met dreams that never came true. And things that came true, that I would never have dared to dream of. I met some ghosts of the past. I remembered smells and sounds that I thought long forgotten. Lots and lots of memories of nice people that I have met on my way. I have seen love that’s long gone, and love that somehow never happened. And we cleaned up lots of garbage and dirt. Oh boy, loads of garbage and dirt.

The above pictures were made in April 1999. A friend of mine made this picture during a short trip to the Baltic Sea. This was just days before my life would change forever. It was a very happy time. A few days later I would meet Madeleine for the first time. Had I known what sort of adventure I was heading into… I would do all of it again 🙂 She turned my life upside down, a few months later I would move in with her in Freiburg, say goodbye to many of my old friends for the last time (many of them I never met again), and it’s been my best decision ever.

Thermostat goes to Red Hat Summit & JavaOne 2012

Last week, we were happy to receive notification that our (Mario & me) proposal for a presentation about Thermostat at JavaOne 2012 has been accepted! Thermostat is the project that both of us have been heavily involved with during the last 4-5 months (having joined Jon & Omair in February who have been involved much longer and deeper). It’s a next generation monitoring, diagnosing, profiling and controlling solution for OpenJDK. Just now we are adding heap analysis support and make Thermostat ready for its first milestone release at the end of this week, just in time for Deepak’s show at the Red Hat Summit in Boston. The second milestone release is planned to be ready roughly 3 months later, just before JavaOne. (Yes, we invented a new development cycle: Conference Driven Release Management 🙂 ) Hope to see lots of folks there!

Gnome3 usability

I have been, and still am, quite a fan of Gnome. I followed the development of Gnome3 and was relatively pleased when it came out, despite all the bashing. I even find some workflows sorely lacking when switching to other desktops, which is a good sign (for Gnome). It has its rough edges of course, and I hoped they would be ironed out, but I get to think that this might never happen, because some of the things seem to be intentional. Let me list a few of the things that I am most concerned about:
– The timezone-aware clock. I loved this thingy in Gnome2, I would add any location I want, and the clock would show me the time in those places in the popup view. Since I am working with heavily distributed team, this is an extremely useful feature. The clock in Gnome3 instead is very basic. I have seen a discussion somewhere to re-introduce the timezone aware clock back into Gnome3, but apparently it never happened. And extensions.gnome.org doesn’t have anything like that either.
– The hopping notification icons. Unless you are very good with the mouse, it becomes a little bit of a chase to click one of those notification icons in the lower right corner. Why the hell do the need to show a name in the taskbar, and move around while you hover over them??? This is so usability-backwards! Moving things are always a bit of a double edged sword in any GUI. Yes, movements *can* be a very useful visual cue to something happening, but needs to be implemented with great care to not end up being totally confusing. In this particular case I simply don’t get it: why not simply show the icons and only show the ‘name’ (or whatever, sometimes it’s really just some crappy variable name or such) in a tooltip when hovering over it?? There is completely no good reason to shuffle those icons around!
– Epiphany’s new fullscreen mode. The idea is SO good!! But the implementation is so far off, I cannot believe it. First: once maximized there is *NO* way to unmaximize it. I mean, yeah, ALT+F5. After having looked it up in the *web*. Similar for *CLOSING* the frakking browser window. It requires to do 2 mouse moves and 2 clicks. Not to speak of the initial confusion of the close button being replaces by a menu, that simply should not be there. Yeah, CTRL+W. I know. It seems like one half of Gnome devs target idiots who need huge title bars in order to not miss them, and the other half targets superusers who never use the mouse and know gazillions of shortcuts. This is just wrong.
– Since Gnome 3.4 some applications seem to use a new theme with slick scrollbars and stuff, while the other half uses a different theme. I guess it’s because of GTK2 vs. GTK3 dichotomy, but why?? Why not make them look the same until every major app is switched to GTK3?? Now the whole experience is totally inconsistent. I hated this when it happened with Ubuntu’s new scrollbar (and the new Gnome3.4 scrollbar is only slightly better… requires mouse-superskills) now Gnome upstream repeats the same mess.
– For some unknown reason, Evolution becomes more and more broken with every release. In Gnome3.2, it would freeze every now and then when I enabled spam filtering. Now it freezes whenever the computer comes back from suspend, which is the new OFF. (Ok, it doesn’t strictly freeze, it just doesn’t seem to be able to connect to any network.) Fail. Ah, and it manages to always come up on the wrong screen.

I find the state of Gnome quite sad. On one side I really like many of the ideas. On the other side, it really feels a bit like 1998 in many respects. And this is not because of needing to re-learn new concepts, I actually like to try and learn new ways of work. It is really because some things are made unnecessarily hard, for no apparent reason. Or buggy. Or both. I really wish the that those issues get ironed out over time, because I do like Gnome and the underlying ideas.