Drastically reducing GC pause times for YQL

Update 2: JRockit Real Time 3.1.2 using -XgcPrio:deterministic performed even better than this configuration in testing though it is not yet an approved VM at Yahoo! we will continue to test with it.

Update: One issue we still have is that after many hours of deployment with this configuration the heap fragments and we start to get concurrent mode failures.  We only saw this though during periods of peak activity.

We were struggling with some long pause times due to GC for YQL that we couldn’t stomach for our internal property SLAs. The secret turned out to be a collection of parameters for the Java 6 garbage collector:

-XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode \
-XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+CMSIncrementalPacing

I have so far been blown away by the results of this combination of parameters. Not only does it work very efficiently under moderate load, under crushing load it rises the occasion and aggressively keeps down the heap in such a way as to never cause the:

Concurrent Mode Failure
The concurrent collector uses one or more garbage collector threads that run simultaneously with the
application threads with the goal of completing the collection of the tenured and permanent generations
before either becomes full. As described above, in normal operation, the concurrent collector does most
of its tracing and sweeping work with the application threads still running, so only brief pauses are seen
by the application threads. However, if the concurrent collector is unable to finish reclaiming the
unreachable objects before the tenured generation fills up, or if an allocation cannot be satisfied with
the available free space blocks in the tenured generation, then the application is paused and the
collection is completed with all the application threads stopped. The inability to complete a collection
concurrently is referred to as concurrent mode failure and indicates the need to adjust the concurrent
collector parameters.

If you have any other GC secrets for the JVM, leave them here. Just as a heads up I compared it with all the other GC’s available for Java 6 including the experimental G1 and none of them were as effective.