fs_clone in upgrade to 12.2.5

[Laurel]

Hi Erman!
as we apply as oracle supports all patches 12.2.6+postpatches in downtime mode in one shot  in upgrade process 12.1.3->12.2.5.
Can we start and release the system  to users  and fs_clone we will be postponed f.e for 12h till next night ?
I mean that its not needed to do straight way, only for the next adop prepare  cycle .
This way we distribute the load on the system. As as soon system starts up wil be big load, so we want to wait till users and jobs calm down :)
Do you agree ?

Thanks,Laurel

[ErmanArslansOracleBlog]

Hi Laurel,

Yes you can run fs_clone later, i.e 12h after the start. (ofcourse, if you are sure that, you are not going to apply any more patches during that period.)

Note : that fs_clone is required for the following purpose:
to copy the new run edition code and configuration to the other file system, to ensure that both file systems are in sync before applying patches using the regular adop cycle on the other file system.

So, if you are not going to apply any patches right away till the next night, then you can postpone fs_clone.

[Laurel]

Hi Erman!
Thanks alot of helping! You rule! :)

Another quick question on nice Sunday morning :)
OAM context  file update in OAm manager  gives
"Failed when writing Context Configuration files back to file system. Possible causes: General SSLEngine problem. "
FNDFS entries are fine /pinged fine and can open logs/outputs. So, its something with TLS enabling.
Reviewed how we implemented that -> all looks fine. All steps done.
If I run check on db node ,
select utl_http.request('https://testapps.tyre.ad:4567/robots.txt',null,'file:/testdb/oracle/TEST/12.1.0/appsutil/wallet', null) from dual;
It works...

Any good hints where I could look at ? No matching on Metalink.

Thanks again!
Laurel

[Laurel]

I checked all logfiles - nothing:
$IAS_ORACLE_HOME/instances/*/diagnostics/logs/OHS/EBS_web_*/*log
OPMN Log
$IAS_ORACLE_HOME/instances/*/diagnostics/logs/OPMN/opmn/*
Weblogic Logs
$IAS_ORACLE_HOME/../wlserver_10.3/common/nodemanager $EBS_DOMAIN_HOME/servers/oa*/logs/*
$EBS_DOMAIN_HOME/servers/forms*/logs/*
$EBS_DOMAIN_HOME/servers/AdminServer/logs/*
$EBS_DOMAIN_HOME/sysman/log/*

In more ssl_request_log.* I can see the following entries only:

[12/Mar/2017:00:24:02 +0200] 172.19.12.138 TLSv1.2 SSL_RSA_WITH_AES_256_GCM_SHA384 "GET /OA_HTML/cabo/images/skyros/focus.gif HTTP/1.1" 561
[12/Mar/2017:00:24:02 +0200] 172.19.12.138 TLSv1.2 SSL_RSA_WITH_AES_256_GCM_SHA384 "GET /OA_HTML/ HTTP/1.1" 4328
[12/Mar/2017:00:24:10 +0200] 172.19.12.138 TLSv1.2 SSL_RSA_WITH_AES_256_GCM_SHA384 "GET /OA_HTML/weboam/oam/functionMap$target=TEST HTTP/1.1" 357
[12/Mar/2017:00:24:10 +0200] 172.19.12.138 TLSv1.2 SSL_RSA_WITH_AES_256_GCM_SHA384 "GET /OA_HTML/weboam/oam/functionMap$target=TEST$index=0 HTTP/1.1" 20753
[12/Mar/2017:00:24:15 +0200] 172.19.12.138 TLSv1.2 SSL_RSA_WITH_AES_256_GCM_SHA384 "GET /OA_HTML/weboam/oam/functionMap$target=TEST$index=0?event=link&target=TEST&page=AD_CONFIG_FILES_TABLE HTTP/1.1" 377
[12/Mar/2017:00:24:15 +0200] 172.19.12.138 TLSv1.2 SSL_RSA_WITH_AES_256_GCM_SHA384 "GET /OA_HTML/weboam/oam/adconfig/adAppsCtxtFilesTable$target=TEST HTTP/1.1" 24442


But no errors..
Have you configured TLS in 12.2 and is it working for you ? :)

Thanks,Laurel

[ErmanArslansOracleBlog]

Hi Laurel,

Yes. Here is my article about it -> http://ermanarslan.blogspot.com.tr/2014/07/enabling-ssl-in-oracle-e-business-suite.html

1)Can you wiew Log/Output Files of the concurrent programs using Application screens?
(trying to generalize the cause, suspecting from FNDFS and Oracle HTTP Server and if it is so, you should not be able tosee the conc view out and log files as well)
2)What are the version of javas that are used in apps tier?
3)What does OAM log files say? (they are located under $APPLRGF/oam) ** this is important.

[Laurel]

Hi!
Yes.  We did excatly as in your article.
All logs and outputs can be viewed in Apps and Copy file works
In the log file you mention is this error, but no hints on metalink


[0:24:56:884, 3/12/17] OAM:*Imm_a0uONSubVyI9UOi1hw_nZAXRQxu6HQlaNib1ndr0G_daqGP3!395140316!1489270164366*oracle.apps.fnd.oam.sdk.srvc.functSecurity.AOLFunctionSecurity.canAccessFunction bc, fs, fn:Warning !   Function : OAM_AD_HISTORY.RESTOREis null
[0:25:5:172, 3/12/17] getDefaultConfigurationName = BaseUIPBCfg1
[0:25:5:172, 3/12/17] getDefaultConfigurationName = BaseUIPBCfg1
[0:25:5:280, 3/12/17] OAM:*Imm_a0uONSubVyI9UOi1hw_nZAXRQxu6HQlaNib1ndr0G_daqGP3!395140316!1489270164366*oracle.apps.fnd.oam.sdk.srvc.functSecurity.AOLFunctionSecurity.canAccessFunction bc, fs, fn:Warning !   Function : OAM_AD_HISTORY.RESTOREis null
[0:25:28:30, 3/12/17] getDefaultConfigurationName = BaseUIPBCfg1
[0:25:28:30, 3/12/17] getDefaultConfigurationName = BaseUIPBCfg1
[0:25:28:51, 3/12/17] OAM:WarningBread Crumb is empty
[0:25:28:153, 3/12/17] OAM:*Imm_a0uONSubVyI9UOi1hw_nZAXRQxu6HQlaNib1ndr0G_daqGP3!395140316!1489270164366*oracle.apps.fnd.oam.sdk.srvc.functSecurity.AOLFunctionSecurity.canAccessFunction bc, fs, fn:Warning !   Function : OAM_AD_HISTORY.RESTOREis null
[0:25:32:115, 3/12/17] oracle.apps.fnd.oam.bobj.adconfig.WriteBackFailureException: General SSLEngine problem
        at oracle.apps.fnd.oam.bobj.adconfig.AppsCtxtFiles._updateAppsCtxtFile(AppsCtxtFiles.java:1602)
        at oracle.apps.fnd.oam.bobj.adconfig.AppsCtxtFiles.updateAppsCtxtFile(AppsCtxtFiles.java:1199)
        at oracle.apps.fnd.oam.bobj.adconfig.AppsCtxtFiles.updateAppsCtxtFile(AppsCtxtFiles.java:1283)
        at oracle.apps.fnd.oam.servlet.ui.handlers.adconfig.OAMAppsCtxtConfHandler._saveTree(Unknown Source)
        at oracle.apps.fnd.oam.servlet.ui.handlers.adconfig.OAMAppsCtxtConfHandler.doSaveTree(Unknown Source)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:95)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:56)
        at java.lang.reflect.Method.invoke(Method.java:620)
        at oracle.cabo.servlet.event.MethodEventHandler.handleEvent(Unknown Source)
        at oracle.cabo.servlet.event.TableEventHandler.handleEvent(Unknown Source)
        at oracle.cabo.servlet.event.TableEventHandler.handleEvent(Unknown Source)
        at oracle.cabo.servlet.event.BasePageFlowEngine.handleRequest(Unknown Source)
        at oracle.apps.fnd.oam.servlet.ui.oamPageFlowEngine.handleRequest(oamPageFlowEngine.java:846)
        at oracle.cabo.servlet.AbstractPageBroker.handleRequest(Unknown Source)
        at oracle.cabo.servlet.ui.BaseUIPageBroker.handleRequest(Unknown Source)
        at oracle.cabo.servlet.PageBrokerHandler.handleRequest(Unknown Source)
        at oracle.apps.fnd.oam.servlet.ui.OAMServlet.doGet(OAMServlet.java:293)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
        at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)
        at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)
        at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:301)
        at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:60)
        at oracle.apps.jtf.cabo.interceptor.JTFWrapperFilter.doFilter(JTFWrapperFilter.java:149)
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:60)
        at oracle.apps.jtf.base.session.ReleaseResFilter.doFilter(Unknown Source)
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:60)
        at oracle.apps.fnd.security.AppsServletFilter.doFilter(AppsServletFilter.java:487)
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:60)
        at oracle.security.jps.ee.http.JpsAbsFilter$1.run(JpsAbsFilter.java:138)
        at java.security.AccessController.doPrivileged(AccessController.java:456)
        at oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:324)
        at oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:464)
        at oracle.security.jps.ee.http.JpsAbsFilter.runJaasMode(JpsAbsFilter.java:121)
        at oracle.security.jps.ee.http.JpsAbsFilter.doFilter(JpsAbsFilter.java:211)
        at oracle.security.jps.ee.http.JpsFilter.doFilter(JpsFilter.java:71)
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:60)
        at oracle.dms.servlet.DMSServletFilter.doFilter(DMSServletFilter.java:163)
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:60)
        at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3748)
        at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3714)
        at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
        at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120)
        at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2283)
        at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2182)
        at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1499)
        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:263)
        at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)

Thanks,Laurel

[Laurel]

And we dont use Loadbalancer in front.
1 apps node and 1 db node.
The apps/db  tier is not on shared file system.


So, can it be if Oracle using plsql to write the context file, it cannot be accessed as the Context file is on apps tier and  db user has no access to it ... ?

ANy hints ?
Thanks,Laurel

[ErmanArslansOracleBlog]

I see..
Please lets create a debug OAM log and review..

For gathering debug OAM log , please see the following note:
Gathering Debug OAM Log In Oracle Applications R12 (Doc ID 740767.1)

[Laurel]

Hi !

The update of the Context file that belong to the DB context file works fine.
At the same to the Apps tier not working fine - SSL error
How is this possible...

Thanks for the hint with  debug  In the log file I see that :
   Caused by: com.ibm.jsse2.util.h: PKIX path building failed: java.security.cert.CertPathBuilderException: PKIXCertPathBuilderImpl could not build a valid CertPath.; internal cause is:
        java.security.cert.CertPathValidatorException: The certificate issued by CN=CERV-01 is not trusted; internal cause is:
        java.security.cert.CertPathValidatorException: Certificate chaining error
        at com.ibm.jsse2.util.f.a(f.java:94)
        at com.ibm.jsse2.util.f.b(f.java:36)
        at com.ibm.jsse2.util.e.a(e.java:7)
        at com.ibm.jsse2.yc.a(yc.java:108)
        at com.ibm.jsse2.yc.a(yc.java:146)
        at com.ibm.jsse2.yc.checkServerTrusted(yc.java:51)
        at weblogic.security.SSL.jsseadapter.JaTrustManager.checkServerTrusted(JaTrustManager.java:125)
        at com.ibm.jsse2.wc.checkServerTrusted(wc.java:30)
        at com.ibm.jsse2.bb.a(bb.java:129)
        ... 72 more
Caused by: java.security.cert.CertPathBuilderException: PKIXCertPathBuilderImpl could not build a valid CertPath.; internal cause is:
        java.security.cert.CertPathValidatorException: The certificate issued by CN=CERV-01 is not trusted; internal cause is:
        java.security.cert.CertPathValidatorException: Certificate chaining error
        at com.ibm.security.cert.PKIXCertPathBuilderImpl.engineBuild(PKIXCertPathBuilderImpl.java:410)
        at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:256)
        at com.ibm.jsse2.util.f.a(f.java:135)
        ... 80 more
Caused by: java.security.cert.CertPathValidatorException: The certificate issued by CN=CERV-01 is not trusted; internal cause is:
        java.security.cert.CertPathValidatorException: Certificate chaining error


 I did the keytool step of importing to  cacerts . We have only 1 certificate. All our java framework/forms work without the security warning, as we have the same certificate in the browser and java. Why then DB context file edit works ?

Interesting. Can it be that error message is wrong and actually it wants on shared file system..
Or some extra steps in the weblogic should be done that not documented :)
Thanks for hints! Let me know  if you think of something :)

Br,Laurel

[ErmanArslansOracleBlog]

This is almost documented.
See this note , it is for EBS 12.1 and 12.0, but it is very similar : Receive: PKIX path building failed: java.security.cert.CertPathBuilderException (Doc ID 1481177.1)
-- issue "when writing Context Configuration files back to file system, you get the following error-stack:"

PKIX path building failed: java.security.cert.CertPathBuilderException: PKIXCertPathBuilderImpl could not build a valid CertPath
internal cause is: java.security.cert.CertPathValidatorException: the certificate issued by CN=xxxx is not trusted
internal cause is: java.security.cert.CertPathValidatorException: Certificate chain error


The document says: "one needs to ensure that the CA certificates are recognized in order to verify the certificate in the same way as the browser does."

So, you need to check the document : "Enabling SSL or TLS in Oracle E-Business Suite Release 12.2 (Doc ID 2143101.1)" - 3.9 Update the JDK Cacerts File .. root intermediate and server certificate should be in your cacerts file..

[ErmanArslansOracleBlog]

Any updates?