FRM-92101: there was a failure in the forms server during startup

[baig]

Hi Erman,

we updated jdk on applications node. after the udpate when we launch the forms it is throwing the below error.

using the doc For 12.2: Using the Latest JDK 7.0 Update with Oracle E-Business Suite Release 12.2 Note 1530033.1

followed all steps from the document and completed the update.

after restartin the services unable to launch forms with below error.

FRM-92101: there was a failure in the forms server during startup.this could happen due to invalid configuration .

from the form log received the below error.
       
error details from forms log

oracle.forms.engine.RunformException: Forms session <3> failed during startup: no response from runtime process
at oracle.forms.servlet.RunformProcess.connect(Unknown Source)
at oracle.forms.servlet.RunformProcess.dataToRunform(Unknown Source)
at oracle.forms.servlet.RunformSession.dataToRunform(Unknown Source)
at oracle.forms.servlet.ListenerServlet.doPost(Unknown Source)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)
at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)
at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:301)
at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:60)
at oracle.security.jps.ee.http.JpsAbsFilter$1.run(JpsAbsFilter.java:138)
at java.security.AccessController.doPrivileged(Native Method)
at oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:324)
at oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:464)
at oracle.security.jps.ee.http.JpsAbsFilter.runJaasMode(JpsAbsFilter.java:121)
at oracle.security.jps.ee.http.JpsAbsFilter.doFilter(JpsAbsFilter.java:211)
at oracle.security.jps.ee.http.JpsFilter.doFilter(JpsFilter.java:71)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:60)
at oracle.dms.servlet.DMSServletFilter.doFilter(DMSServletFilter.java:163)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:60)
at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3748)
at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3714)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120)
at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2283)
at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2182)
at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1493)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:263)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)

kindly advise.

[baig]

Hi,

tried regenerating jar files and forms but still same issue exist.

Thanks

[ErmanArslansOracleBlog]

relinking the forms may help..

We have 1 document for this, but it is for 12.1 systems.. Still, it gets you to a point ->

R12: "FRM-92101:There was a failure in the Forms Server during startup" Error When Attempting to Launch Forms (Doc ID 454427.1)

--that is source the 10.1.2 env and ;

$ cd $ORACLE_HOME/forms/lib32
/* If this directory does not exist, do the following: */
$ cd $ORACLE_HOME/forms/lib
$ make -f ins_forms.mk sharedlib install

Also ensure you applied the following patches;

OracleAS 10.1.2 Patch 16271876
OracleAS 10.1.2 Patch 17907988
OracleAS 10.1.2 Forms Patch 17653437
OracleAS 10.1.2 Patch 16241466
OracleAS 10.1.2 Patch 17645157

Also check this MOS note -> Forms Session Failed During Startup No Response From Runtime Process (Doc ID 2809465.1)

[baig]

Hi Erman,

when i ran the command  i got the below:
cat: /p02/applpnf/PNF/fs1/EBSapps/10.1.2/lib/ldflags: Too many levels of symbolic links
cat: /p02/applpnf/PNF/fs1/EBSapps/10.1.2/lib/ldflags: Too many levels of symbolic links
cat: /p02/applpnf/PNF/fs1/EBSapps/10.1.2/lib/ldflags: Too many levels of symbolic links
cat: /p02/applpnf/PNF/fs1/EBSapps/10.1.2/lib/ldflags: Too many levels of symbolic links
mv frmcmp /p02/applpnf/PNF/fs1/EBSapps/10.1.2/bin/
chmod 755 /p02/applpnf/PNF/fs1/EBSapps/10.1.2/bin//frmcmp
rm -f frmweb
gcc -m32 -o frmweb -L/p02/applpnf/PNF/fs1/EBSapps/10.1.2/forms/lib/ -L/p02/applpnf/PNF/fs1/EBSapps/10.1.2/lib/ -L/p02/applpnf/PNF/fs1/EBSapps/10.1.2/lib//stubs  -L/p02/applpnf/PNF/fs1/EBSapps/10.1.2/jdk/jre/lib/i386 -L/p02/applpnf/PNF/fs1/EBSapps/10.1.2/jdk/jre/lib/i386/server -L/p02/applpnf/PNF/fs1/EBSapps/10.1.2/jdk/jre/lib/i386/native_threads -Bstatic -L/p02/applpnf/PNF/fs1/EBSapps/10.1.2/jdk/jre/lib/i386 -L/p02/applpnf/PNF/fs1/EBSapps/10.1.2/jdk/jre/lib/i386/server -L/p02/applpnf/PNF/fs1/EBSapps/10.1.2/jdk/jre/lib/i386/native_threads \
/p02/applpnf/PNF/fs1/EBSapps/10.1.2/lib/s0nnmain.o \
/p02/applpnf/PNF/fs1/EBSapps/10.1.2/forms/lib/ssliftabw.o \
/p02/applpnf/PNF/fs1/EBSapps/10.1.2/forms/lib/ifzxtb.o \
/p02/applpnf/PNF/fs1/EBSapps/10.1.2/forms/lib/sixn.o \
/p02/applpnf/PNF/fs1/EBSapps/10.1.2/forms/lib/sixp.o \
/p02/applpnf/PNF/fs1/EBSapps/10.1.2/forms/lib/ui10.o /p02/applpnf/PNF/fs1/EBSapps/10.1.2/forms/lib/uiicxd.o \
-liplsn \
/p02/applpnf/PNF/fs1/EBSapps/10.1.2/forms/lib/istuic.o \
        /p02/applpnf/PNF/fs1/EBSapps/10.1.2/forms/lib/istiif.o \
/p02/applpnf/PNF/fs1/EBSapps/10.1.2/forms/lib/istplu.o \
/p02/applpnf/PNF/fs1/EBSapps/10.1.2/forms/lib/istrwe.o \
-u iiflmp -u iifget -u iiffcb -u iiflog -u iiflov -u iifgdl -u iifatr -u iifeue -u iifwcnew -u iifwru  \
/p02/applpnf/PNF/fs1/EBSapps/10.1.2/forms/lib/iwvgbm.o \
-libfrmw -liffw -lifcw -lijcw -liifw  -lipc -lipfw -lipc -limfw -limc -liwfw -liwcw -litw -licw -lirm -lsosdw -lihm -libfrmw -lixw -liffw -lifcw -lijcw -lipfw -lipc -liicw -liiiw -liwfw -liwcw -liqw -litw -licw -lipc -lsosdw -limfw -limc -liplsn -lioc \
 \
 \
-lrws \
-lnn -lobx -lzrc  -lvgs -ldeb -lucol  -lca  -luicc -lmma -lmmiw -lmmov -lmma -lmmos -lmmoi  -lmmia -lmmft -lmmcm  -luihx -luc -luipr -luimotif -lot -lrem -lree -lrec -luiimg -luimotif -luipr -luiimg -luc -lrem -luimotif -luia -ltknqap -luipr -luimotif -lutt -lix -lixd -lix -lixd -lrod -lror -lros -lrod -lror -lros -lrod -luat  -lutc -lutj -lutl -lutsl -lpls10  -lplp10 -lplc10 -lpls10  -lplp10 -lslax10 -lsql10   -Bdynamic -z lazyload /p02/applpnf/PNF/fs1/EBSapps/10.1.2/lib/nautab.o /p02/applpnf/PNF/fs1/EBSapps/10.1.2/lib/naeet.o /p02/applpnf/PNF/fs1/EBSapps/10.1.2/lib/naect.o /p02/applpnf/PNF/fs1/EBSapps/10.1.2/lib/naedhs.o /p02/applpnf/PNF/fs1/EBSapps/10.1.2/rdbms/lib/kpudfo.o -lncrypt10 -lpthread -lm -lc    `cat /p02/applpnf/PNF/fs1/EBSapps/10.1.2/lib/ldflags`    -lnsslb10 -lncrypt10 -lnsgr10 -lnzjs10 -ln10 -lnnz10 -lnl10 -lnro10 `cat /p02/applpnf/PNF/fs1/EBSapps/10.1.2/lib/ldflags`    -lnsslb10 -lncrypt10 -lnsgr10 -lnzjs10 -ln10 -lnnz10 -lnl10 -lclient10 -lnnetd10  -lvsn10 -lcommon10 -lgeneric10 -lmm  -lcore10 -lxml10 -lunls10 -lsnls10 -lnls10 -lcore10 -lnls10 `cat /p02/applpnf/PNF/fs1/EBSapps/10.1.2/lib/ldflags`    -lnsslb10 -lncrypt10 -lnsgr10 -lnzjs10 -ln10 -lnnz10 -lnl10 -lnro10 `cat /p02/applpnf/PNF/fs1/EBSapps/10.1.2/lib/ldflags`    -lnsslb10 -lncrypt10 -lnsgr10 -lnzjs10 -ln10 -lnnz10 -lnl10 -lclient10 -lnnetd10  -lvsn10 -lcommon10 -lgeneric10    -lcore10 -lxml10 -lunls10 -lsnls10 -lnls10 -lcore10 -lnls10 -lclient10 -lnnetd10  -lvsn10 -lcommon10 -lgeneric10  -lcore10 -lxml10 -lunls10 -lsnls10 -lnls10 -lcore10 -lnls10   `cat /p02/applpnf/PNF/fs1/EBSapps/10.1.2/lib/sysliblist` -Wl,-rpath,/p02/applpnf/PNF/fs1/EBSapps/10.1.2/lib,-rpath,/p02/applpnf/PNF/fs1/EBSapps/10.1.2/jdk/jre/lib/i386:/p02/applpnf/PNF/fs1/EBSapps/10.1.2/jdk/jre/lib/i386/xawt:/lib:/usr/lib  -lm    `cat /p02/applpnf/PNF/fs1/EBSapps/10.1.2/lib/sysliblist` -ldl -lpthread -lm   -L/p02/applpnf/PNF/fs1/EBSapps/10.1.2/lib -L/p02/applpnf/PNF/fs1/EBSapps/10.1.2/lib/stubs/ -lsnls10 -ljvm  -Wl,-rpath,/usr/X11R6/lib  -Wl,--as-needed -lXi -lXtst -Wl,--no-as-needed -L/usr/X11R6/lib -lXm -lXt -lX11 -lm -lXp -lXext
cat: /p02/applpnf/PNF/fs1/EBSapps/10.1.2/lib/ldflags: Too many levels of symbolic links
cat: /p02/applpnf/PNF/fs1/EBSapps/10.1.2/lib/ldflags: Too many levels of symbolic links
cat: /p02/applpnf/PNF/fs1/EBSapps/10.1.2/lib/ldflags: Too many levels of symbolic links
cat: /p02/applpnf/PNF/fs1/EBSapps/10.1.2/lib/ldflags: Too many levels of symbolic links
/usr/bin/ld: warning: -z lazyload ignored.
/usr/bin/ld: /p02/applpnf/PNF/fs1/EBSapps/10.1.2/lib//libnnz10.so: undefined reference to symbol 'ldap_memfree'
/p02/applpnf/PNF/fs1/EBSapps/10.1.2/lib/libclntsh.so.10.1: error adding symbols: DSO missing from command line
collect2: error: ld returned 1 exit status
make: *** [frmweb] Error 1

Thanks

[baig]

Hi,

i have verified there are no ldflags file or directpry or link under the specific location.

Thanks

[ErmanArslansOracleBlog]

What is your OS ? (Vendor/Distro and version)

[baig]

Hi,

Red Hat Enterprise Linux Server release 7.9 (Maipo)
Linux hostname 3.10.0-1160.53.1.el7.x86_64 #1 SMP Thu Dec 16 10:19:28 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

thanks

[ErmanArslansOracleBlog]

Those errors are not expected..

Please check your apps tier and ensure that you have all the rpms in place. -> Oracle E-Business Suite Installation and Upgrade Notes Release 12 (12.2) for Linux x86-64 (Doc ID 1330701.1)

EBS R12: After App Tier Restore Forms Cannot Be Launched / Started up (Doc ID 2457777.1)

I need the full story of this env..
I need a list of action that you've taken .. List of thing that you do and end up with this issue.

I may need a remote session for this.. Because the following errors are really unexpected;

/usr/bin/ld: /p02/applpnf/PNF/fs1/EBSapps/10.1.2/lib//libnnz10.so: undefined reference to symbol 'ldap_memfree'
/p02/applpnf/PNF/fs1/EBSapps/10.1.2/lib/libclntsh.so.10.1: error adding symbols: DSO missing from command line

Probably, there is missing patch or conf in the forms oracle home.. There nay be some OS patch missing as well..

[baig]

Hi,
i have attached the rda report for rpm verification. i have another instance in the same server which is working fine in which i have not updated jdk.


I need the full story of this env..===Apps and DB on same server.

I need a list of action that you've taken .. List of thing that you do and end up with this issue.

-- applied oct-2021 db security patches, ad txk 12 and atg8 patch. untill here everything was working fine.

==to update tls1.2 for cloud email office365 applied patches  (Doc ID 2816784.1)

For 12.2: Using the Latest JDK 7.0 Update with Oracle E-Business Suite Release 12.2 Note 1530033.1

the below patches were already present.
For 12.2
Patch 22323006 : SU Patch [DI8E]: UPDATE JAVAMAIL API LIBRARY VERSION TO 1.5.4 WITHIN WLS 10.3.6
Patch 29607457 : NEED HIGHER VERSION OF MAIL.JAR IN WEB TIER FOR IEM PRODUCT IN 12.2.8 & 12.2.9


before proceeding with the udpate i have checked for the pre req patches for oracle forms home, all were present.

== attached the procedure used to updated jdk.

now when i am trying to query the inventory to check patches facing a new and unique issue.

===attached the output.

Thanks

[baig]

opatch_error.txtjdk_update_steps.txtAPPS_HCVE_A_EBS122_lin_res.htm

[baig]

opatch_lsinverrorp02.txt

please ignore the previous opatch_lsinv error file.
that is from another instance which is facing the same issue.

Thanks

[baig]

Also want to update you regarding the java versions used to update jdk.

in $COMMON_TOP/util

jdk64 folder was replaced with 64 bit of jdk version for linux86_64 i.e.jdk-7u321-linux-x64.tar.gz

patch id : p33416555_170321_Linux-x86-64.zip

jdk32 folder was replaced with 32 bit of jdk version for linux x86 i.e. jdk-7u321-linux-i586.tar.gz

patch id : p33416555_170321_LINUX.zip

for forms home jdk32 was used and for fmw/webtier 64 bit jdk was used.

[ErmanArslansOracleBlog]

Your RPMs seem ok..
This seems to be related with the libraries present in the directory  /p02/applpnf/PNF/fs1/EBSapps/10.1.2/lib..

It doesn't look related with java.

We are failing while compiling the frmweb and it is a long command line used there.. There are java lib folder there as well, but gcc complains about the following ->

cat: /p02/applpnf/PNF/fs1/EBSapps/10.1.2/lib/ldflags: Too many levels of symbolic links
cat: /p02/applpnf/PNF/fs1/EBSapps/10.1.2/lib/ldflags: Too many levels of symbolic links
cat: /p02/applpnf/PNF/fs1/EBSapps/10.1.2/lib/ldflags: Too many levels of symbolic links
cat: /p02/applpnf/PNF/fs1/EBSapps/10.1.2/lib/ldflags: Too many levels of symbolic links
/usr/bin/ld: warning: -z lazyload ignored.
/usr/bin/ld: /p02/applpnf/PNF/fs1/EBSapps/10.1.2/lib//libnnz10.so: undefined reference to symbol 'ldap_memfree'
/p02/applpnf/PNF/fs1/EBSapps/10.1.2/lib/libclntsh.so.10.1: error adding symbols: DSO missing from command line


**So we may try backup (move) /p02/applpnf/PNF/fs1/EBSapps/10.1.2/lib to a backup folder and then copy that 10.1.2/lib directory from other instance which doesn't have the similar problems and then try recompiling the forms..

--you didn't install any patches to 10.1.2 home .. any patches that may change the files in 10.1.2/lib right? If you didn't, then you can try the action plan I suggested above.

A similar action plan given in note -> EBS R12: After App Tier Restore Forms Cannot Be Launched / Started up (Doc ID 2457777.1).. It is due to a corruption, but here we may be dealing with a similar thing.. These errors are unexpected, so this is worth a try...

[baig]

Hi,

I replaced  the jdk folder in the forms home with the downloaded file and untar and replaced and tried again to relink and launch forms. it worked.

now when i query the opatch lsinventory it gives me proper output.

Thanks for your support.

Question : how can i update jdk on multinode environment when we are on a older java version "1.7.0_25" and wanted a higher version ===step 3.5.2 in ( Doc ID 1530033.1 ).

single node db + apps primary node + dmz node.

Please advise. thanks for your valuable suggestions.
Thanks

[ErmanArslansOracleBlog]

So your issue is solved then.

Okay. You are asking something different now. Please create a seperate issue for this.

[baig]

Hi Erman,

Thanks a lot for your continuous support.