SOA Server OutOfMemoryError Issue in JRockit JVM

The Challenge

This is a real scenario from one of my clients. I built a killer Oracle BPM / ADF app and deployed to a newly configured SOA server running in JRockit JVM in Linux. I see that the server starting to behave rather slowly after couple of deployments. I check the memory settings, the default settings, recycle the server. After few deploys server consistently crashed with OutOfMemoryError error. I certainly did not expect that in an enterprise level infrastructure. I monitored the memory utilization using JRockit Mission Control to see what was happening.

Heap after SOA server startup with default settings of 1.5 GB memory allocation.
SOA Server Start Up

Out of Memory Error after couple of deployments.
SOA Server Start Up

The Fix

After digging into JRockit document and trying out few options, I found an optimum setting for my application. All I needed to do was allocate right amount of nursery size so that the young collection heap does not fill up too quickly or take longer time to garbage collect it. I have a decent size application with lots of things happening and 1.5GB of nursery size seemed just right. I also gave pause time as 300 millisecond, so that I get maximum throughput. At the end, my optimal garbage collection tuning for JRockit was simple and perfect.

Here my tuning is straight forward, I have allocated my initial and reserved heap to be 6GB at JVM start up. I have allocated nursery size of 1.5GB for young collection and have optimized GC for short and even pause time of 300 milliseconds.

Please refer to http://docs.oracle.com/cd/E13150_01/jrockit_jvm/jrockit/geninfo/pdf/diagnos.pdf for further info on Oracle JRockit JVM.

The Deductions

In the enterprise infrastructure, tuning of JVM garbage collection is crucial and must be done based on your application, whether it be throughput or responsiveness. After monitoring heap through JRockit Mission Control, I can definitely deduce few key factors. When I used the default settings and started the SOA server, it potentially had 2 issues. First, it hardly had enough memory to run SOA suite and other applications together. Secondly, it did not have heap explicitly define to process and garbage collect large numbers of de-referenced young objects. Therefore, SOA server would come to a grinding halt after few deployments.

When I increased the min and max heap to be 6GB, the SOA server did hold up but the memory utilization was always high. That means the de-referenced objects were not being collected efficiently and the heap was slowly getting full.

Immedialy after SOA startup.
SOA Server Start Up

Heap after 3 days, with 30+ deployments and 200+ BPM test instances.
SOA Server Start Up

Finally when I explicitly introduces the nursery size of 1.5GB and pause time of 300ms, the server heap always stayed under 3GB of utilization. Which evidently confirms that the de-referenced objects had enough heap to process and recollect the memory efficiently.

Heap after a week with proper tuning. It hardly ever touched the 3GB memory utilization point.

SOA Server Start Up

Sanjib

Director at Fusion Applied
Sanjib is an Oracle Middleware Architectwith over 15 years' experience developing software and managing projects in challenging, fast-paced consulting environments. An experienced OO developer with expertise in J2EE and Oracle Fusion Middleware technologies. His current focus is on helping customers build solutions using the Oracle Fusion Middleware stack including Oracle SOA Suite 11g, BPM, ADF & WebCenter Suite.
Sanjib

Leave a Reply

avatar
  Subscribe  
Notify of