adapcctl.sh: exiting with status 204 in Ebs r12.2.4

[baig]

Hi

hope you are doing good.


we rebooted our server, when starting the services the following services is failing to start.
adapcctl.sh: exiting with status 204


we see the following messages in /p05/appljvj/fs2/FMW_Home/webtier/instances/EBS_web_OHS1/diagnostics/logs/OHS/EBS_web/console~OHS~1.log

/p05/appljvj/fs2/FMW_Home/webtier/ohs/bin/apachectl startssl: execing httpd
ModSecurity: WARNING Using transformations in SecDefaultAction is deprecated (/p05/appljvj/fs2/FMW_Home/webtier/instances/EBS_web_OHS1/config/OHS/EBS_web/security2.conf:75).
ModSecurity: WARNING Using transformations in SecDefaultAction is deprecated (/p05/appljvj/fs2/FMW_Home/webtier/instances/EBS_web_OHS1/config/OHS/EBS_web/security2.conf:90).
[Mon Dec 21 15:55:17 2020] [warn] Errors will be logged into /p05/appljvj/fs2/FMW_Home/webtier/instances/EBS_web_OHS1/diagnostics/logs/OHS/EBS_web/EBS_web.log
ModSecurity: WARNING Using transformations in SecDefaultAction is deprecated (/p05/appljvj/fs2/FMW_Home/webtier/instances/EBS_web_OHS1/config/OHS/EBS_web/security2.conf:75).
ModSecurity: WARNING Using transformations in SecDefaultAction is deprecated (/p05/appljvj/fs2/FMW_Home/webtier/instances/EBS_web_OHS1/config/OHS/EBS_web/security2.conf:90).
[Mon Dec 21 15:55:18 2020] [warn] Errors will be logged into /p05/appljvj/fs2/FMW_Home/webtier/instances/EBS_web_OHS1/diagnostics/logs/OHS/EBS_web/EBS_web.log

we see the following messages in
/p05/appljvj/fs2/FMW_Home/webtier/instances/EBS_web_OHS1/diagnostics/logs/OHS/EBS_web/EBS_web.log

[2020-12-21T16:00:28.1259+03:00] [OHS] [ERROR:32] [] [core.c] [host_id: hydev.xyzmycomp.com.sa] [host_addr: 10.1.3.13] [pid: 19966] [tid: 139917253543808] [user: applmgr4] [VirtualHost: hydev.xyzmycomp.com.sa:0] NZ Library Error: Unknown error


ebs r12.2.4
db 12.1.0.2
ssl 1.2 enabled

please advise

[ErmanArslansOracleBlog]

What did you change in this environment?
Did you apply any patches? Did you update any SSL certificates? What is the story of this environment?

[baig]

Hi,
we applied OS patches and rebooted the server. i noticed that the trial ssl certificates got expired and i renewed the ssl certificates. the apacctl is refusing to start along with the managed servers.

i am able to start the admin servere.

[ErmanArslansOracleBlog]

Hmm..
It seems this is a certificate issue..
Are you sure that you replaced those expired certificates?

Also you sent me the following error message only ->

NZ Library Error: Unknown error

What else do you have there?

But again, this is probably related with the certificates.. Either you didn't replace them, or you couldn't replace them properly. Check the wallet and see you have only the new certificates inside it.

Please review and follow ->

Enabling TLS in Oracle E-Business Suite Release 12.2 (Doc ID 1367293.1)
*****Section 8: Renewing Expired Certificates

Note that ;
If you are using SSL or TLS 1.0 and need to review your current configuration or renew your certificate, follow Enabling SSL or TLS in Oracle E-Business Suite Release 12.2, Document 2143101.

However; it seems you are using TLS 1.2 and the document to follow is 1367293.1 in this case.

[baig]

Hi,

I have replaced the certificates with the new ones and also i see  NZ Library Error only in EBS_log.
yes only the new ones inside..

i followed exactly as you have suggested for replacing certificates. should i do it from the begining and verify it again?

thanks

[ErmanArslansOracleBlog]

Yes, it is better to do those steps from the beginning.. Just to be sure.. in an error-free and complete way..

Also what else do you have in OHS related logs?

I mean do you see additional error messages like the following ones? ->

nzos handshake error, nzos_Handshake returned 29014
or
nzos call nzosSetCredential returned 28791

etc..

[baig]

[2020-12-26T12:34:05.0145+03:00] [OHS] [NOTIFICATION:16] [OHS-9999] [core.c] [host_id: hodev.xyzmycomp.com.sa] [host_addr: 10.1.3.13] [pid: 25039] [tid: 140348947367808] [user: applmgr4] [VirtualHost: main]  ModSecurity for Apache/2.7.4 (http://www.modsecurity.org/) configured.

[2020-12-26T12:34:05.0145+03:00] [OHS] [NOTIFICATION:16] [OHS-9999] [core.c] [host_id: hodev.xyzmycomp.com.sa] [host_addr: 10.1.3.13] [pid: 25039] [tid: 140348947367808] [user: applmgr4] [VirtualHost: main]  ModSecurity: APR compiled version="1.4.5"; loaded version="1.4.5"

[2020-12-26T12:34:05.0145+03:00] [OHS] [NOTIFICATION:16] [OHS-9999] [core.c] [host_id: hodev.xyzmycomp.com.sa] [host_addr: 10.1.3.13] [pid: 25039] [tid: 140348947367808] [user: applmgr4] [VirtualHost: main]  ModSecurity: PCRE compiled version="5.0 "; loaded version="5.0 13-Sep-2004"

[2020-12-26T12:34:05.0145+03:00] [OHS] [NOTIFICATION:16] [OHS-9999] [core.c] [host_id: hodev.xyzmycomp.com.sa] [host_addr: 10.1.3.13] [pid: 25039] [tid: 140348947367808] [user: applmgr4] [VirtualHost: main]  ModSecurity: LIBXML compiled version="2.7.2"

[2020-12-26T12:34:06.1284+03:00] [OHS] [ERROR:32] [] [core.c] [host_id: hodev.xyzmycomp.com.sa] [host_addr: 10.1.3.13] [pid: 25039] [tid: 140348947367808] [user: applmgr4] [VirtualHost: hodev.xyzmycomp.com.sa:0]  Init: (hodev.xyzmycomp.com.sa:443) Unable to initialize SSL environment, nzos call nzosSetCredential returned 28791

[2020-12-26T12:34:06.1285+03:00] [OHS] [ERROR:32] [] [core.c] [host_id: hodev.xyzmycomp.com.sa] [host_addr: 10.1.3.13] [pid: 25039] [tid: 140348947367808] [user: applmgr4] [VirtualHost: hodev.xyzmycomp.com.sa:0]  NZ Library Error: Unknown error

[2020-12-26T12:37:42.0170+03:00] [OHS] [NOTIFICATION:16] [OHS-9999] [core.c] [host_id: hodev.xyzmycomp.com.sa] [host_addr: 10.1.3.13] [pid: 27079] [tid: 139876957730688] [user: applmgr4] [VirtualHost: main]  ModSecurity for Apache/2.7.4 (http://www.modsecurity.org/) configured.

[2020-12-26T12:37:42.0170+03:00] [OHS] [NOTIFICATION:16] [OHS-9999] [core.c] [host_id: hodev.xyzmycomp.com.sa] [host_addr: 10.1.3.13] [pid: 27079] [tid: 139876957730688] [user: applmgr4] [VirtualHost: main]  ModSecurity: APR compiled version="1.4.5"; loaded version="1.4.5"

[2020-12-26T12:37:42.0171+03:00] [OHS] [NOTIFICATION:16] [OHS-9999] [core.c] [host_id: hodev.xyzmycomp.com.sa] [host_addr: 10.1.3.13] [pid: 27079] [tid: 139876957730688] [user: applmgr4] [VirtualHost: main]  ModSecurity: PCRE compiled version="5.0 "; loaded version="5.0 13-Sep-2004"

[2020-12-26T12:37:42.0171+03:00] [OHS] [NOTIFICATION:16] [OHS-9999] [core.c] [host_id: hodev.xyzmycomp.com.sa] [host_addr: 10.1.3.13] [pid: 27079] [tid: 139876957730688] [user: applmgr4] [VirtualHost: main]  ModSecurity: LIBXML compiled version="2.7.2"

[2020-12-26T12:37:43.1315+03:00] [OHS] [ERROR:32] [] [core.c] [host_id: hodev.xyzmycomp.com.sa] [host_addr: 10.1.3.13] [pid: 27079] [tid: 139876957730688] [user: applmgr4] [VirtualHost: hodev.xyzmycomp.com.sa:0]  Init: (hodev.xyzmycomp.com.sa:443) Unable to initialize SSL environment, nzos call nzosSetCredential returned 28791

[2020-12-26T12:37:43.1315+03:00] [OHS] [ERROR:32] [] [core.c] [host_id: hodev.xyzmycomp.com.sa] [host_addr: 10.1.3.13] [pid: 27079] [tid: 139876957730688] [user: applmgr4] [VirtualHost: hodev.xyzmycomp.com.sa:0]  NZ Library Error: Unknown error

[2020-12-26T12:37:48.0147+03:00] [OHS] [NOTIFICATION:16] [OHS-9999] [core.c] [host_id: hodev.xyzmycomp.com.sa] [host_addr: 10.1.3.13] [pid: 27113] [tid: 139984445011840] [user: applmgr4] [VirtualHost: main]  ModSecurity for Apache/2.7.4 (http://www.modsecurity.org/) configured.

[2020-12-26T12:37:48.0147+03:00] [OHS] [NOTIFICATION:16] [OHS-9999] [core.c] [host_id: hodev.xyzmycomp.com.sa] [host_addr: 10.1.3.13] [pid: 27113] [tid: 139984445011840] [user: applmgr4] [VirtualHost: main]  ModSecurity: APR compiled version="1.4.5"; loaded version="1.4.5"

[2020-12-26T12:37:48.0147+03:00] [OHS] [NOTIFICATION:16] [OHS-9999] [core.c] [host_id: hodev.xyzmycomp.com.sa] [host_addr: 10.1.3.13] [pid: 27113] [tid: 139984445011840] [user: applmgr4] [VirtualHost: main]  ModSecurity: PCRE compiled version="5.0 "; loaded version="5.0 13-Sep-2004"

[2020-12-26T12:37:48.0147+03:00] [OHS] [NOTIFICATION:16] [OHS-9999] [core.c] [host_id: hodev.xyzmycomp.com.sa] [host_addr: 10.1.3.13] [pid: 27113] [tid: 139984445011840] [user: applmgr4] [VirtualHost: main]  ModSecurity: LIBXML compiled version="2.7.2"

[2020-12-26T12:37:49.1230+03:00] [OHS] [ERROR:32] [] [core.c] [host_id: hodev.xyzmycomp.com.sa] [host_addr: 10.1.3.13] [pid: 27113] [tid: 139984445011840] [user: applmgr4] [VirtualHost: hodev.xyzmycomp.com.sa:0]  Init: (hodev.xyzmycomp.com.sa:443) Unable to initialize SSL environment, nzos call nzosSetCredential returned 28791

[2020-12-26T12:37:49.1230+03:00] [OHS] [ERROR:32] [] [core.c] [host_id: hodev.xyzmycomp.com.sa] [host_addr: 10.1.3.13] [pid: 27113] [tid: 139984445011840] [user: applmgr4] [VirtualHost: hodev.xyzmycomp.com.sa:0]  NZ Library Error: Unknown error

[baig]

Hi  

i see the below error

(hodev.xyzmycomp.com.sa:443) Unable to initialize SSL environment, nzos call nzosSetCredential returned 28791


Thanks

[ErmanArslansOracleBlog]

Error 28791..

Either your wallet is in wrong place or your wallet is still have expired certificates inside..
I think you OHS is still using the old certificates..

List the contents of the wallet using orapki, and see if there are the old certificate still displayed there.. If they are still there remove them using orapki. Sometimes, owm may confuse these things.. This might be the cause.. Use orapki to list the contents of the wallet.. (check the expiry date and old certificates).. If they are still there, use orapki remove them. lastly use orapki add to add the new ones..

[baig]

Hi,

I was able to resolve the issue by following the below procedure.

with all apps services down.
remove any old files apart from the existing certificates from "<s_web_ssl_directory>/Apache"

Take backup of all existing wallets from all locations.

Launch the wallet

right click and remove the user certificate and trusted certificates from the wallet,
save the wallet and add the user certificate and trusted certificates back to the wallet.

copy the wallet to respective locations as mentioned in Enabling TLS in Oracle E-Business Suite Release 12.2 ( Doc ID 1367293.1 )
(step 6, step 7 & step 8 without re-configuring the ciphers)

started the services it came up gracefully.

Ran fs_clone and a patch cycle including prepare, finalize, cutover and cleanup successfully.

I still couldnt figure out what caused the issue but i was able to resolve it this way.

Thanks for your usual support.
thank you.

[baig]

Hi,

how can i distinguish if TLS 1.0 or tls 1.2 is in use in any ssl enabled environment.

please advise.

Thanks

[ErmanArslansOracleBlog]

Easy way:

Launch FireFox.
Enter the URL you wish to check in the browser.
In the address bar, click the icon to the left of the URL.
Click on More Information.
Select the Security tab.
Look for the Technical details. This will describe the version of TLS or SSL used.

[baig]

Hi,

thanks so much brother. Have a nice day.

Thanks for resolving the issue.

[baig]

Hi,

I wanted to configure TLS1.0 and update to TLS 1.2  as i have the TLS1.0 in my prod server.

so i made configurations as per TLS1.0 and applied ATG 6.0 patch for moving to jws. but when i checked the tls as per the suggestion above i see its TLS 1.2.

I didnt configure cipher suites neither i applied any fmw patches or prerequisite patches for TLS 1.2.

How it is possible without the prerequisites for TLS1.2 being applied it became compatible with TLS1.2

for your expert comments please.

Thanks

[ErmanArslansOracleBlog]

Probably, it is enabled by default. Check your opmn.xml..
If you don't specify the chiphers, probably you end up using the default ones, which are old and weak.
If using an ECC certificate, no need to specify any ciphers.

Follow ->  Enabling TLS in Oracle E-Business Suite Release 12.2 (Doc ID 1367293.1).