This is a real scenario from one of my clients. I built a killer Oracle BPM / ADF app and deployed to a newly configured SOA server running in JRockit JVM in Linux. I see that the server starting to behave rather slowly after couple of deployments. I check the memory settings, the default settings, recycle the server. After few deploys server consistently crashed with OutOfMemoryError error. I certainly did not expect that in an enterprise level infrastructure. I monitored the memory utilization using JRockit Mission Control to see what was happening.
Heap after SOA server startup with default settings of 1.5 GB memory allocation.
Out of Memory Error after couple of deployments.
After digging into JRockit document and trying out few options, I found an optimum setting for my application. All I needed to do was allocate right amount of nursery size so that the young collection heap does not fill up too quickly or take longer time to garbage collect it. I have a decent size application with lots of things happening and 1.5GB of nursery size seemed just right. I also gave pause time as 300 millisecond, so that I get maximum throughput. At the end, my optimal garbage collection tuning for JRockit was simple and perfect.
-Xms6G -Xmx6G -XgcPrio:pausetime -XpauseTarget:300ms -Xgc:gencon -Xns:1536m
Here my tuning is straight forward, I have allocated my initial and reserved heap to be 6GB at JVM start up. I have allocated nursery size of 1.5GB for young collection and have optimized GC for short and even pause time of 300 milliseconds.
Please refer to http://docs.oracle.com/cd/E13150_01/jrockit_jvm/jrockit/geninfo/pdf/diagnos.pdf for further info on Oracle JRockit JVM.
In the enterprise infrastructure, tuning of JVM garbage collection is crucial and must be done based on your application, whether it be throughput or responsiveness. After monitoring heap through JRockit Mission Control, I can definitely deduce few key factors. When I used the default settings and started the SOA server, it potentially had 2 issues. First, it hardly had enough memory to run SOA suite and other applications together. Secondly, it did not have heap explicitly define to process and garbage collect large numbers of de-referenced young objects. Therefore, SOA server would come to a grinding halt after few deployments.
When I increased the min and max heap to be 6GB, the SOA server did hold up but the memory utilization was always high. That means the de-referenced objects were not being collected efficiently and the heap was slowly getting full.
Tuning params: -Xms6G -Xmx6G
Immedialy after SOA startup.
Heap after 3 days, with 30+ deployments and 200+ BPM test instances.
Finally when I explicitly introduces the nursery size of 1.5GB and pause time of 300ms, the server heap always stayed under 3GB of utilization. Which evidently confirms that the de-referenced objects had enough heap to process and recollect the memory efficiently.
Heap after a week with proper tuning. It hardly ever touched the 3GB memory utilization point.
Tuning params: -Xms6G -Xmx6G -XgcPrio:pausetime -XpauseTarget:300ms -Xgc:gencon -Xns:1536m