SHA-2 certificates query

[Raja]

Hi Erman

We have a load balancer infront of our EBS 12.1.3 and ssl certificates are also kept in load balancer. As per industry changes, our network administrator has updated the ssl certificates with SHA-2. Now, after the changes, the outbound notification mails were not going out. The errors were :
======
HTML content -> oracle.apps.fnd.wf.common.HTTPClientException: Unable to invoke method HTTPClient.HTTPConnection.Get caused by: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target"
===

So we had to revert back the certificates to again SHA-1. So my question to you is, if the certificate changes are done in Load balancer, do we need to again import the root and intermediate certificates to our Apache server using wallet manager?

[ErmanArslansOracleBlog]

1)What is the value of "WF_MAIL_WEB_AGENT" profile option?
2)What is your load balancer url of EBS?
3)What is your actual urls of EBS?

[Raja]

Hi Erman

Answers are:

1.I cannot see the profile at site level "WF_MAIL_WEB_AGENT" but i can see an another  profile option-
WF: Workflow Mailer Framework Web Agent" which is set to null.

2.https://ebsprd.familydollar.com

3.http://fdlxebspdapp1.familydollar.com:8012/

[Raja]

To add, it is a multi node setup with 3 forms/web tiers, 2 Concurrent tiers and 2 DMZ tiers.

[ErmanArslansOracleBlog]

What is the value of the profile named Application Framework Agent (APPS_FRAMEWORK_AGENT)?

[ErmanArslansOracleBlog]

Anyways, here is an answer and workaround for you ->

For certain types of email, workflow mailer calls the agent that is pointed by the WF_MAIL_WEB_AGENT profile.
If WF_MAIL_WEB_AGENT is null, then your Workflow mailer will use the url to reach that agent using the APPS_FRAMEWORK_AGENT profile.

so, if you enabled SSL properly, then your Apps Framework Agent should be set to "https/blabla...."

Well.. in order to make worklow mailer to reach the framework agent , you need to import the Application's certificates to the WF mailers truststore (cacerts file).

Workaround:
----------
Set WF_MAIL_WEB_AGENT profile to the http url "http://blabla", i.e: "http://fdlxebspdapp1.familydollar.com:8012 "

This action makes the NTF mailer to reach the web agent through http, but this was a workaround not the fix..

FIX :
-----
If you are missing any certificates in one of the cacerts files, import it, restart the mailer and retest the issue.
If your certificates are changed then reimport them, restart the mailer retest the issue.

READ THIS ONE
------------------------------------
Read this one to get the idea.. I m explaining it with every details here -> http://ermanarslan.blogspot.com.tr/2014/08/ebs-122-notification-mailer.html

[Raja]

Thanks Erman for your time. I have one more question for you-Is it required everytime to import the root and intermediate certificates to $JDK_TOP when we change the certs deployed in Load balancer?

The reason i am asking because last year when our network administrator renewed the SSL certs which are deployed in F5, then we didn't have to do anything from EBS perspective. But this time when the network administrators change the certificate type from SHA-1 to SHA-2, our outbound workflow notifications got impacted. Your thoughts on this, please.

[ErmanArslansOracleBlog]

Everytime, if you change a certificate, then you need to reimport it to your wallet or cacerts file.
If the intermediate and root certificates that are associated with your new server certificate, are the same as the old, then you don't need to reimport them.

Also, if you offloaded the SSL work to the load balancer, then you don't need to do anything about SSL in EBS.(except reimporting the SSL cert to the mailer truststore and other autoconfig related SSL things)

Note that, if you don't import the SSL certificate in to the Mailer Truststore and if you have an https url set in the related profile options, then WF mailer will fail if it tries to generate the content for Applications Framework regions that are embedded in notifications.
Note that, WF mailer will fail only delivering the noficiations which include Applications Framework Regions.

Maybe that time, you didn't recognize this problem, because you didn't have any Applications Framework regions that are embedded in notifications during that period.

[Raja]

Thanks Erman for the explanation. I will try as per your suggestion.

[Raja]

Hi Erman

I imported the root, intermediate and Server certificate to cacerts and restarted the workflow mailer. But it seems the issue is not resolved. I am getting the error :

 Caused by: oracle.apps.fnd.wf.mailer.NotificationFormatter$FormatterSAXException: Problem obtaining the HTML content -> oracle.apps.fnd.wf.common.HTTPClientException: Unable to invoke method HTTPClient.HTTPConnection.Get caused by: java.net.SocketException: Connection reset
[GC 37682K->27437K(47292K), 0.0052460 secs]


The profile Application Framework Agent is pointing to the load balancer url-https://ebssit.familydollar.com:443

[ErmanArslansOracleBlog]

Aha.. It goes to the load balancer to retrieve the framework region or body.
Can it be caused by something blocking when you go to load balancer?
Set the (WF_MAIL_WEB_AGENT profile) URL to the web server node (not the load balancer url, but local application node url and port) , restart workflow mailer and retry.
Lets see if it is caused by the load balancer or by something blocking our way while reaching it.

[Raja]

Hi Erman

i have set the profile pointing to the local node and now i am getting the below error:

 Caused by: oracle.apps.fnd.wf.mailer.NotificationFormatter$FormatterSAXException: Problem obtaining the HTML content -> oracle.apps.fnd.wf.common.HTTPClientException: Unable to invoke method HTTPClient.HTTPConnection.Get caused by: java.lang.NoClassDefFoundError: HTTPClient/RespInputStream
[Mar 14, 2017 3:41:46 AM EDT]:1489477306495:-1:-1:fdlxebsstcm1.familydollar.com:10.120.6.73:-1:-1:1:20420:SYSADMIN(0):-1:Thread[outboundThreadGroup1,5,outboundThreadGroup]:2138303037:50504:1489477236359:95:ERROR:[SVC-GSM-WFMLRSVC-191581-10006 : oracle.apps.fnd.wf.mailer.SMTPMessageHandler.prepareMessages(String)]:Problem getting HTML content -> oracle.apps.fnd.wf.mailer.FormatterException: Problem getting the HTML content -> oracle.apps.fnd.wf.mailer.NotificationFormatter$FormatterSAXException: Problem obtaining the HTML content -> oracle.apps.fnd.wf.common.HTTPClientException: Unable to invoke method HTTPClient.HTTPConnection.Get caused by: java.lang.NoClassDefFoundError: HTTPClient/RespInputStream
        at oracle.apps.fnd.wf.mailer.NotificationFormatter.handleResEndTag(NotificationFormatter.java:3461)
        at oracle.apps.fnd.wf.mailer.NotificationFormatter.endElement(NotificationFormatter.java:578)
        at oracle.xml.parser.v2.XMLContentHandler.endElement(XMLContentHandler.java:210)
        at oracle.xml.parser.v2.NonValidatingParser.parseElement(NonValidatingParser.java:1345)
        at oracle.xml.parser.v2.NonValidatingParser.parseRootElement(NonValidatingParser.java:362)
        at oracle.xml.parser.v2.NonValidatingParser.parseDocument(NonValidatingParser.java:308)
        at oracle.xml.parser.v2.XMLParser.parse(XMLParser.java:337)
        at oracle.apps.fnd.wf.mailer.NotificationFormatter.getFormattedMessages(NotificationFormatter.java:354)
        at oracle.apps.fnd.wf.mailer.SMTPMessageHandler.prepareMessages(SMTPMessageHandler.java:96)
        at oracle.apps.fnd.wf.mailer.SMTPOutboundProcessor.read(SMTPOutboundProcessor.java:796)
        at oracle.apps.fnd.cp.gsc.SvcComponentProcessor.process(SvcComponentProcessor.java:604)
        at oracle.apps.fnd.cp.gsc.Processor.run(Processor.java:283)
        at java.lang.Thread.run(Thread.java:619)
 Caused by: oracle.apps.fnd.wf.mailer.NotificationFormatter$FormatterSAXException: Problem obtaining the HTML content -> oracle.apps.fnd.wf.common.HTTPClientException: Unable to invoke method HTTPClient.HTTPConnection.Get caused by: java.lang.NoClassDefFoundError: HTTPClient/RespInputStream

[ErmanArslansOracleBlog]

This is not normal. Mailer can not see the definition of a core java class. (Unable to invoke method HTTPClient.HTTPConnection.Get caused by: java.lang.NoClassDefFoundError: HTTPClient/RespInputStream)
Looks like a configuration issue. Something with the classpath configuration can cause this. (AF_CLASSPATH)

1)shutdown apps tier
2)Run AutoConfig on apps tier
3)Then run ADADMIN to recompile the Applications JAR files with the FORCE option set to Y when prompted whether or not to force compilation of all the JAR files.
4) start the apps tier.
5) Re-test

If the issue persists, send me the detailed info for this environment. (info such as: recent changes)

[Raja]

Hi Erman

I have performed the steps and now i am not seeing any error in the workflow logs. The WF: Workflow Mailer Framework Web Agent profile is pointing to the local application node and not the load balancer URL.

My question to you is, what could be the reason that when we use load balancer URL, we are getting errors. Are they related to missing certs for any files?


As we test it for few days, i will keep you posted on the updates.

Thanks a ton for assistance.

[ErmanArslansOracleBlog]

there is a connection reset there.. So I suspect something is blocking you. Maybe, something disallows the call back request from the web server node.

Ask your Network admin about it..

[Raja]

Sorry to bother you again and again. Could you please explain, what could be the implications of using a non ssl url in the profile option:WF_MAIL_WEB_AGENT? And do you think that to make SSL load balancer url work, what could the possible steps to be followed?

[Raja]

And also i was going through your blog http://ermanarslan.blogspot.in/2014/08/ebs-122-notification-mailer.html and i have a doubt on one of the fix you have provided. Can you please explain how to do the below steps.

Alternative 2)
Change workflow mailer's keystore parameter to " the cacerts file located in the jdk top"
But then, Inbound will not able to work properly, because this time cacerts file does not include IMAP server's ssl certificate .. That's why import IMAP server's ssl certificate in to the cacerts file , as well.

[ErmanArslansOracleBlog]

If you provide non-ssl url, then your traffic will not be encrypted.
The traffic that I m talking about it between WF mailer node and the application node(where Oracle HTTP Server) that WF mailer will try to reach using http.
If your apps node and wf mailer node is the same, there is no risk of doing that.

For SSL Load Balancer problem, I already said to you.. There is something blocking you in the way.
Ask Network and Security admin about it. (ask to Load Balancer admin too)

[ErmanArslansOracleBlog]

That alternative 2, is all about pointing the right cacerts file to the wf mailer.
If you read that blog post carefully,  you will see that there is specific scenario and that specific scenario caused the wrong cacerts file to be used by wf mailer.. So that 's it.