Oracle EBS 12.1 处理ISG 发布的wsdl 被请求时遇到500错误

Oracle 12.1.3 通过ISG 发布了一个服务，该服务在被频繁调用的时候，出现500 错误，临时解决方案是可以通过重启oafm组件解决，但是需要定位原因。

排查日志路径

$INST_TOP/logs/ora/10.1.3/opmn/default_group~oafm~default_group~1.log
$INST_TOP/logs/ora/10.1.3/j2ee/oafm/oafm_default_group_1/

查看第一个日志发现频繁出现full GC

328459.617: [Full GC 847029K->847029K(860416K), 1.3999040 secs]
328461.017: [Full GC 847029K->845177K(860416K), 1.4276150 secs]
328462.468: [Full GC 847026K->847026K(860416K), 1.3887370 secs]
328463.856: [Full GC 847026K->845177K(860416K), 1.3851960 secs]
328465.266: [Full GC 847026K->847026K(860416K), 1.3830720 secs]
328466.649: [Full GC 847026K->845188K(860416K), 1.3847090 secs]
328468.046: [Full GC 847031K->847031K(860416K), 1.3924150 secs]
328469.438: [Full GC 847031K->845181K(860416K), 1.3835700 secs]
328470.865: [Full GC 847030K->847030K(860416K), 1.3961220 secs]
328472.262: [Full GC 847030K->845179K(860416K), 1.4245020 secs]
328473.712: [Full GC 847030K->847030K(860416K), 1.3863100 secs]
328475.098: [Full GC 847030K->845178K(860416K), 1.3831550 secs]
328476.504: [Full GC 847027K->847027K(860416K), 1.3911750 secs]
328477.895: [Full GC[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor23603]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor23605]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor23606]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor23607]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor23604]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor23602]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor23608]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor23609]847027K->845181K(860416K), 1.3902780 secs]
328479.299: [Full GC 847027K->847027K(860416K), 1.3852630 secs]
328480.684: [Full GC 847027K->845176K(860416K), 1.3825570 secs]
328482.077: [Full GC 847027K->847027K(860416K), 1.3850380 secs]
328483.462: [Full GC 847027K->845180K(860416K), 1.3842770 secs]
328484.869: [Full GC 847026K->847026K(860416K), 1.3835840 secs]
328486.253: [Full GC 847026K->845177K(860416K), 1.3835210 secs]
328487.702: [Full GC

猜想可能遇到java OutOfMemoryError (OOM),在第二个日志中出现

25/05/26 14:37:35.986 Internal server error 
java.lang.OutOfMemoryError: Java heap space

证实了这一点

处理方式，调整jvm参数

调整前

只截取关键部分process-type id=“oafm”

<data id="java-options" value="-server -verbose:gc -Xmx1024M -Xms512M -XX:MaxPermSize=256M -XX:NewRatio=2 -XX:+PrintGCTimeStamps -XX:+UseTLAB -XX:+UseParallelGC -XX:ParallelGCThreads=2  -Djava.security.policy=$ORACLE_HOME/j2ee/oacore/config/java2.policy  -Djava.awt.headless=true  -Dhttp.webdir.enable=false -Doracle.security.jazn.config=/data/ebs/prod/inst/apps/PROD_ifeng-ebs01/ora/10.1.3/j2ee/oafm/config/jazn.xml"/>

调整后

调整如下选项

<jvm-options>-Xms1024m</jvm-options>         <!-- 初始堆内存 1GB -->
<jvm-options>-Xmx2048m</jvm-options>         <!-- 最大堆内存 2GB -->
<jvm-options>-XX:MaxPermSize=512m</jvm-options>  <!-- 永久代（类元数据） -->
<jvm-options>-XX:+HeapDumpOnOutOfMemoryError</jvm-options> <!-- OOM 自动生成 dump 文件 -->
<jvm-options>-XX:HeapDumpPath=/tmp/oafm_heapdump.hprof</jvm-options> <!-- dump 路径 -->

调整后

<data id="java-options" value="-server -verbose:gc -Xmx2048M -Xms1024M -XX:MaxPermSize=512M -XX:NewRatio=2 -XX:+PrintGCTimeStamps -XX:+UseTLAB -XX:+UseParallelGC -XX:ParallelGCThreads=2 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/oafm_heapdump.hprof -Djava.security.policy=$ORACLE_HOME/j2ee/oacore/config/java2.policy  -Djava.awt.headless=true  -Dhttp.webdir.enable=false -Doracle.security.jazn.config=/data/ebs/prod/inst/apps/PROD_ifeng-ebs01/ora/10.1.3/j2ee/oafm/config/jazn.xml"/>

然后重启oafm

opmnctl stopproc ias-component=oafm
opmnctl startproc ias-component=oafm