Oracle Application Server - Java OutOfMemory , Analyzing the javacore dump

1. Download the IBM Thread & Monitor Dump Analyzer for Java at http://www.alphaworks.ibm.com/tech/jca



2. Follow the guide in How to Diagnose Java Resource Starvation http://java.sys-con.com/node/921279?page=0,1



3. Use the JDK 1.5 to run the jca.jar and open the javacore*.txt file with all the same thread file name. e.g. javacore147902.xxxxx.txt



4. In the Compare Monitors of the tool, you will noticed the blocked by 1. Clicked the blocked for more information. In the state, “Monitor”, the action is “Owns Monitor Lock on Heap Lock” , that means it is currently owning the lock when the dump is initiated.



5. Notice the last thread information on the detail. If the information is something like below, then it is possible that your application is logging to many data in the logs file and caused the locking issue.








at
java.util.AbstractCollection.toString(AbstractCollection.java(Compiled Code))at
java.lang.String.valueOf(String.java(Compiled Code))at
java.lang.StringBuffer.append(StringBuffer.java(Compiled Code))at
org.apache.commons.lang.builder.ToStringStyle.appendDetail(ToStringStyle.java(Compiled
Code))at
org.apache.commons.lang.builder.ToStringStyle.appendInternal(ToStringStyle.java(Compiled
Code))at
org.apache.commons.lang.builder.ToStringStyle.append(ToStringStyle.java(Compiled
Code))at
org.apache.commons.lang.builder.ToStringBuilder.append(ToStringBuilder.java(Compiled
Code))at
org.apache.commons.lang.builder.ReflectionToStringBuilder.appendFieldsIn(ReflectionToStringBuilder.java(Compiled
Code))at
org.apache.commons.lang.builder.ReflectionToStringBuilder.toString(ReflectionToStringBuilder.java(Compiled
Code))at
org.apache.commons.lang.builder.ReflectionToStringBuilder.toString(ReflectionToStringBuilder.java(Compiled
Code))at
org.apache.commons.lang.builder.ReflectionToStringBuilder.toString(ReflectionToStringBuilder.java(Compiled
Code))at
org.apache.commons.lang.builder.ToStringBuilder.reflectionToString(ToStringBuilder.java(Compiled
Code))




6. In our case, the logging is set by the Log4j , which we found out that the settings is set to full debug mode, we changed it to WARN mode. "log4j.rootLogger=DEBUG, app" to "log4j.rootLogger=WARN, app" , more information at here http://www.laliluna.de/log4j-tutorial.html



7. After this, our application server java core dump is greatly reduce.