Oracle Wallet Manager(OWM) :Exception in thread "main" java.lang.UnsatisfiedLinkError: owm2

Recently, we encounter the error as below in the Oracle Wallet Manager,

Exception in thread "main" java.lang.UnsatisfiedLinkError: owm2 (No such file or directory)
at java.lang.ClassLoader.loadLibraryWithPath(ClassLoader.java:993)
at java.lang.ClassLoader.loadLibraryWithClassLoader(ClassLoader.java:962)
at java.lang.System.loadLibrary(System.java:465)
at oracle.security.wallet.NZNative.(NZNative)
at java.lang.J9VMInternals.initializeImpl(Native Method)
at java.lang.J9VMInternals.initialize(J9VMInternals.java:200)
at oracle.security.admin.wltmgr.owma.OwmaApp.init(OwmaApp)
at oracle.sysman.emSDK.client.appContainer.WebApplication.launch(WebApplication.java:3446)
at oracle.sysman.emSDK.client.appContainer.WebApplication.main(WebApplication.java:3327)

To try to fix this error, we try to set the path for LD_LIBRARY_PATH to $ORACLE_HOME/lib but still the error persists. In the end , we found out that the error is due to the JDK , which we have upgrade from the existing 1.5 JDK to 1.6 JDK 64bit.

To fix it, we change the path in the file $ORACLE_HOME/bin/owm
JRE_HOME=/ApplTop/ias10gr2/app/product/10.1.3/jdk/jre
to JRE_HOME=/ApplTop/ias10gr2/app/product/10.1.3/jdk.orig/jre/ (Points back to original 1.5 JDK)

Oracle Application Server - Java OutOfMemory , Analyzing the javacore dump

1. Download the IBM Thread & Monitor Dump Analyzer for Java at http://www.alphaworks.ibm.com/tech/jca



2. Follow the guide in How to Diagnose Java Resource Starvation http://java.sys-con.com/node/921279?page=0,1



3. Use the JDK 1.5 to run the jca.jar and open the javacore*.txt file with all the same thread file name. e.g. javacore147902.xxxxx.txt



4. In the Compare Monitors of the tool, you will noticed the blocked by 1. Clicked the blocked for more information. In the state, “Monitor”, the action is “Owns Monitor Lock on Heap Lock” , that means it is currently owning the lock when the dump is initiated.



5. Notice the last thread information on the detail. If the information is something like below, then it is possible that your application is logging to many data in the logs file and caused the locking issue.








at
java.util.AbstractCollection.toString(AbstractCollection.java(Compiled Code))at
java.lang.String.valueOf(String.java(Compiled Code))at
java.lang.StringBuffer.append(StringBuffer.java(Compiled Code))at
org.apache.commons.lang.builder.ToStringStyle.appendDetail(ToStringStyle.java(Compiled
Code))at
org.apache.commons.lang.builder.ToStringStyle.appendInternal(ToStringStyle.java(Compiled
Code))at
org.apache.commons.lang.builder.ToStringStyle.append(ToStringStyle.java(Compiled
Code))at
org.apache.commons.lang.builder.ToStringBuilder.append(ToStringBuilder.java(Compiled
Code))at
org.apache.commons.lang.builder.ReflectionToStringBuilder.appendFieldsIn(ReflectionToStringBuilder.java(Compiled
Code))at
org.apache.commons.lang.builder.ReflectionToStringBuilder.toString(ReflectionToStringBuilder.java(Compiled
Code))at
org.apache.commons.lang.builder.ReflectionToStringBuilder.toString(ReflectionToStringBuilder.java(Compiled
Code))at
org.apache.commons.lang.builder.ReflectionToStringBuilder.toString(ReflectionToStringBuilder.java(Compiled
Code))at
org.apache.commons.lang.builder.ToStringBuilder.reflectionToString(ToStringBuilder.java(Compiled
Code))




6. In our case, the logging is set by the Log4j , which we found out that the settings is set to full debug mode, we changed it to WARN mode. "log4j.rootLogger=DEBUG, app" to "log4j.rootLogger=WARN, app" , more information at here http://www.laliluna.de/log4j-tutorial.html



7. After this, our application server java core dump is greatly reduce.
   

Oracle Application Server - Java OutOfMemory Issue

If your Oracle application server 10.1.2 or Oracle Forms encounter the Java OutOfmemory issue, then you may need to changed your OC4J container java option in the opmn.xml to increase the heap memory. Add the following below , the start parameters and in the java-options.


data id="java-options" value="-Xms1024m -Xmx2048m -Xss256k "


The Xms specified the start of the heap memory , Xmx , specified the maximum memory settings for the java container. the Xss specified to set the maximum native stack size for any thread.





Starts from the 512m for the memory settings and the xss128k until the issue can be resolved.

After that, dcmctl updateconfig to update the configuration.





For more information, on the using of this Java parameters, please refer to



Fine-Tuning Memory Usage of 32-Bit Java on AIX

http://www.ibmsystemsmag.com/print/print.aspx?print_page=%2Faix%2Foctobernovember04%2Ftechnicalcorner%2F6643printp1.aspx&string_referer=/aix/octobernovember04/technicalcorner/6643p2.aspx



Forum on Java Outofmemory issue,

http://www.ibm.com/developerworks/forums/message.jspa?messageID=13853385



Metalink Note : 298551.1

OPMN and Process Monitoring



Metalink Note : 823504.1

What are the minimum and maximum sizes recommended for Java heap configuration for OC4J?

Migrating from IAS 10.1.2 to 10.1.3.4

Other than the different in Application Server control GUI, if your application ear is compiled using JDK 1.42 , it might has error if it used the JDK 1.5.

http://download-west.oracle.com/docs/cd/B25221_04/web.1013/b14432/diagnosis.htm#sthref517

http://forums.oracle.com/forums/thread.jspa?threadID=361021

Other issue, is that it could be a bug as in the note “java.lang.NoClassDefFoundError: org/apache/log4j/Category After Upgrading from 10.1.3.1.0” ,Note ID : 460448.1
Which mentioned

http://forums.oracle.com/forums/thread.jspa?threadID=616870

To resolved this follow the guide here

http://www.pascalalma.net/2008/07/14/using-commons-logging-with-oc4j/

Opmn logs: -

Caused by: com.evermind.server.ejb.deployment.InvalidEJBAssemblyException: Unable to load ejb-class com.bidm.misc.ejb.session.ejbeans.MiGenBean, see section 23.2 of the EJB 2.1 specificationjava.lang.NoClassDefFoundError: org.apache.log4j.Logger

at com.evermind.server.ejb.exception.ValidationExceptions.unableToLoadEJBClass(ValidationExceptions.java:36)

Caused by: java.lang.NoClassDefFoundError: org/apache/log4j/Categoryat java.lang.Class.getDeclaredConstructors0(Native Method)at java.lang.Class.privateGetDeclaredConstructors(Class.java:2328)at java.lang.Class.getConstructor0(Class.java:2640)

Different between IAS 10.1.2 and 10.1.3

1. In 10.1.3 the dcmctl tool is no longer available, to performed the task that normally using the dcmctl you need to use the oc4j_admin at $ORACLE_HOME/j2ee/home or download the admin_client.jar in http://download.oracle.com/otn/java/oc4j/10131/oc4j_admin_client_101310.zip.

2. For the Oracle® Containers for J2EE Configuration and Administration Guide, Using the admin_client.jar Utility please refered the link here.

http://download.oracle.com/docs/cd/B31017_01/web.1013/b28950/adminclient.htm#i675530

E.g. normally to deploy application in 10.1.2 , you use

dmctl deployapplication -file xxx.ear -a TestInstance -co TestName

In 10.1.3 ,

java -jar admin_client.jar deployer:oc4j:opmn://test-cycle.oracle.com/TestName oc4jadmin welcome1 -deploy -file d:\temp\xxx.ear -deploymentName TestInstance –bindAllWebApps


3. Another important consideration in when migrating is that the default behaviour of the 10.1.3. 10.1.10 Invocation by Servlet Name without Mapping Is Disabled by Default

The 10.1.3.x Oracle Containers for J2EE Servlet Developer's Guide notes that by default, servlet invocation by class name is now disabled, but can be enabled through the property setting -Dhttp.webdir.enable=true. (Servlet invocation by class name was enabled by default in versions 10.1.2.x and prior.)

What is similar, but undocumented, is that this also applies to invoking a servlet through its value when there is no corresponding entry. Invoking a servlet through its servlet name in this way was enabled by default in 10.1.2.x and prior, but requires the setting -Dhttp.webdir.enable=true in 10.1.3.x versions.

For example, consider the following web.xml entry, with no corresponding entry:

    
 TestIt
 mypackage.TestIt

In 10.1.2.x, you could access this servlet using /servlet/TestIt without any property setting. In 10.1.3.x, you must set -Dhttp.webdir.enable=true to access the servlet in this way with the given configuration.

Apache Slow performance with NFS mount

Recently we have a slow performance with our web site after we redirect our OC4J Application logs to the NFS mount, which is using servlet to logs the application log.

Metalink :

Log Directories In NFS Mounted Device Environment Causes Performance Problem With Apache , Note: 180522.1

Bad performance of Apache when filesystem is NFS mounted, Note : 467315.1