fs_clone fails on secondary nodes with unable to start OHS component error

[ky_17]

We are in the process of apply AD/TXK 16 patches along with security patches to fix EBS vulnerabilities. We applied 34386593 in hotpatch mode as instructed in Applying the Latest AD and TXK Release Update Packs to Oracle E-Business Suite Release 12.2 ( Doc ID 1617461.1 ).

After this we are running fs_clone to sync the patch fs. It finished successfully on primary node but fails on the other nodes with Unable to start OHS component error.


/uprd/fs2/FMW_Home/webtier/instances/EBS_web_OHS2/diagnostics/logs/OHS/EBS_web/console~OHS~1.log for root cause.
SEVERE : Nov 8, 2025 01:22:51 - ERROR - CLONE-20218 Cloning is not successful.
SEVERE : Nov 8, 2025 01:22:51 - CAUSE - CLONE-20218 An internal operation failed.
SEVERE : Nov 8, 2025 01:22:51 - ACTION - CLONE-20218 Provide the clone log and error file for investigation.
oracle.as.t2p.exceptions.FMWT2PPasteConfigException: java.lang.Exception: Unable to start OHS Component.
at oracle.as.clone.ohs.impl.OHSComponentApplyClonerImpl.doPostPasteConfig(OHSComponentApplyClonerImpl.java:269)
at oracle.as.clone.cloner.component.ComponentApplyCloner.doClone(ComponentApplyCloner.java:193)
at oracle.as.clone.cloner.Cloner.doFinalClone(Cloner.java:63)
at oracle.as.clone.request.ApplyCloneRequest.applyArchive(ApplyCloneRequest.java:198)
at oracle.as.clone.request.ApplyCloneRequest._clone(ApplyCloneRequest.java:77)
at oracle.as.clone.process.CloningExecutionProcess.execute(CloningExecutionProcess.java:131)
at oracle.as.clone.process.CloningExecutionProcess.execute(CloningExecutionProcess.java:114)
at oracle.as.clone.client.CloningClient.executeT2PCommand(CloningClient.java:236)
at oracle.as.clone.client.CloningClient.main(CloningClient.java:124)
Caused by: java.lang.Exception: Unable to start OHS Component.
at oracle.as.clone.ohs.impl.OHSComponentApplyClonerImpl.doPostPasteConfig(OHSComponentApplyClonerImpl.java:265)
... 8 more

our cwallet.so files are updated properly in the wallet folders under $FMW_HOME.
Only difference I notice is that for nodes 2,3 and 4. Then opmn.xml file is getting created with below:

<ssl enabled="true" wallet-file="/uprd/fs1/FMW_Home/webtier/instances/EBS_web_OHS1/config/OPMN/opmn/wallet"/>

It should be created like below (This is how it is for node 1 and fs_clone completed fine for node 1):

<ssl enabled="true" wallet-file="/uprd/fs1/FMW_Home/webtier/instances/EBS_web_OHS1/config/OPMN/opmn/wallet" ssl-versions="TLSv1.2" ssl-ciphers="SSL_RSA_WITH_AES_256_CBC_SHA,SSL_RSA_WITH_AES_128_CBC_SHA"/>

I suppose opmn is not coming up because of the missing ssl-versions and ssl-ciphers settings.

[ky_17]

We raised a Sev 1 SR with Oracle too but they are unable to help since yesterday. Everything is good on the RUN FS. OPMN is coming up fine with all other services.

EBS_web.log shows below:
[2025-11-08T09:32:05.0021-05:00] [OHS] [WARNING:32] [OHS-214] [core.c] [host_id: spfinapp3v.fastcard.local] [host_addr: 10.41.32.17] [pid: 864304] [tid: 139674149113152] [user: appprd] [VirtualHost: main]  Init: Session Cache is not configured [hint: SSLSessionCache]

[2025-11-08T09:32:05.0190-05:00] [OHS] [NOTIFICATION:16] [OHS-9999] [core.c] [host_id: spfinapp3v.fastcard.local] [host_addr: 10.41.32.17] [pid: 864304] [tid: 139674149113152] [user: appprd] [VirtualHost: main]  ModSecurity for Apache/2.7.4 (http://www.modsecurity.org/) configured.

[2025-11-08T09:32:05.0190-05:00] [OHS] [NOTIFICATION:16] [OHS-9999] [core.c] [host_id: spfinapp3v.fastcard.local] [host_addr: 10.41.32.17] [pid: 864304] [tid: 139674149113152] [user: appprd] [VirtualHost: main]  ModSecurity: APR compiled version="1.4.5"; loaded version="1.4.5"

[2025-11-08T09:32:05.0190-05:00] [OHS] [NOTIFICATION:16] [OHS-9999] [core.c] [host_id: spfinapp3v.fastcard.local] [host_addr: 10.41.32.17] [pid: 864304] [tid: 139674149113152] [user: appprd] [VirtualHost: main]  ModSecurity: PCRE compiled version="5.0 "; loaded version="5.0 13-Sep-2004"

[2025-11-08T09:32:05.0190-05:00] [OHS] [NOTIFICATION:16] [OHS-9999] [core.c] [host_id: spfinapp3v.fastcard.local] [host_addr: 10.41.32.17] [pid: 864304] [tid: 139674149113152] [user: appprd] [VirtualHost: main]  ModSecurity: LIBXML compiled version="2.7.2"

[2025-11-08T09:32:06.2022-05:00] [OHS] [NOTIFICATION:16] [OHS-9999] [core.c] [host_id: spfinapp3v.fastcard.local] [host_addr: 10.41.32.17] [pid: 864304] [tid: 139674149113152] [user: appprd] [VirtualHost: main]  WebLogic Server Plugin version 1.1 <WLSPLUGINS_11.1.1.9.0_LINUX.X64_150206.1116>

[2025-11-08T09:32:06.2273-05:00] [OHS] [NOTIFICATION:16] [OHS-9999] [core.c] [host_id: spfinapp3v.fastcard.local] [host_addr: 10.41.32.17] [pid: 864304] [tid: 139674149113152] [user: appprd] [VirtualHost: main]  Oracle-HTTP-Server/11.1.1.9.0 (Unix) mod_ssl/11.1.1.9.0 OtherSSL/0.0.0 mod_onsint/2.0 configured -- resuming normal operations

[2025-11-08T09:32:08.0071-05:00] [OHS] [ERROR:32] [] [core.c] [host_id: spfinapp3v.fastcard.local] [host_addr: 10.41.32.17] [pid: 864338] [tid: 139673929770752] [user: appprd] [VirtualHost: fiserp.incomm.com:10045]  nzos handshake error, nzos_Handshake returned 28860(server fiserp.incomm.com:10045, client 127.0.0.1)

[2025-11-08T09:32:08.0071-05:00] [OHS] [ERROR:32] [] [core.c] [host_id: spfinapp3v.fastcard.local] [host_addr: 10.41.32.17] [pid: 864338] [tid: 139673929770752] [user: appprd] [VirtualHost: fiserp.incomm.com:10045]  NZ Library Error: SSL fatal alert

[ErmanArslansOracleBlog]

This seems like an SSL/TLS config issue. It may be due to the incomplete config on patch fs, that is SSL TLS related configuration maybe done properly in run fs , but not properly done in patch fs..

Please compare opmn.xml files of patch and run fs. Check the SSL related parts, any clue there? any missing chipper values for the related tags?

I mean check and compare the value of xml tags like <ssl-ciphers>, <ssl enabled>, <ssl-versions>...
(Ref: New SSL Protocol and Cipher Options for Oracle Fusion Middleware 10.1.3.5/11.1.1.7 OPMN/ONS Component (Doc ID 1905314.1))

Also please send me the remaining part of that error stack. I need those  , those 8 more lines! Herre ->

Caused by: java.lang.Exception: Unable to start OHS Component.
at oracle.as.clone.ohs.impl.OHSComponentApplyClonerImpl.doPostPasteConfig(OHSComponentApplyClonerImpl.java:265)
... 8 more  --> What is there??

[ky_17]

Thanks for the update! Patch fs was dropped and recreated by fs_clone. It finished fine for primary node and failed on secondary nodes. Those 8 lines are the whole stack in the error log for fs_clone but more details are in the EBS_web.log and opmn.log which I have uploaded earlier  like nzos handshake error, nzos_Handshake returned 28860 (in the EBS_web.log) and [ERROR:1] [] [ons-secure] Connection OHS SSL handshake failed (29024).

29024 seems to be Certificate validation failure. So I was thinking this may be related to cwallet.sso under the FMW HOME.
The opmn.xml on RUN FS is same for all nodes. But I did notice that when creating opmn.xml on secondary nodes its not including the ssl-verion and ssl-ciphers after the wallet path ( ssl-versions="TLSv1.2" ssl-ciphers="SSL_RSA_WITH_AES_256_CBC_SHA,SSL_RSA_WITH_AES_128_CBC_SHA"/>)

But I tried removing that on the working and node and brought up opmn and it came up without those lines. So I suspect it may be related to the cwallet.sso file. But RUN FS is all good with certs.

[ErmanArslansOracleBlog]

Okay I see.


Is the wallet files (cwallet.sso is the auto-login part) in that problematic patch fs exactly the same as the one in run fs? Did you check that? Did you try copying that?

Some refs on wallet locations and other ssl-related stuff:

Note 2143101.1 - Enabling SSL or TLS in Oracle E-Business Suite Release 12.2
Note 1367293.1 - Enabling TLS in Oracle E-Business Suite Release 12.2
12.2 E-Business Suite HTTP Server / OPMN Startup Fails After Activating TLS With Error '[ERROR:1] [] [ons-secure] Failed to open wallet (file:/01/fs1/FMW_Home...) [default password]' And '[ERROR:1] [222] [ons-secure] SSL initialization failed' (Doc ID 2250713.1)

[ky_17]

I did compare it with RUN FS. We have been copying the autologin cwallet.sso file to the below locations:

$FMW_HOME/webtier/instances/EBS_web_OHS1/config/OHS/EBS_web/proxy-wallet/
$FMW_HOME/webtier/instances/EBS_web_OHS1/config/OHS/EBS_web/keystore/default/
$FMW_HOME/webtier/instances/EBS_web_OHS1/config/OPMN/opmn/wallet/
$FMW_HOME/user_projects/domains/EBS_domain/opmn/EBS_web_OHS1/wallet/
$FMW_HOME/user_projects/domains/EBS_domain/opmn/EBS_web_OHS1/EBS_web/wallet/

RUN FS has this file updated with CA Cert (not self sign).
FS_CLONE ran successfully for node 1. It copied all cwallet files from FUN FS. Only proxy-wallet location got created with self signed cwallet file. Other 4 locations got updated with CA signed cwallet as RUN FS.

On secondary nodes, where FS_CLONE is failing, all 5 location including proxy-wallet location has CA signed cwallet. So that is a difference we noticed.

Since fs_clone failed on secondary nodes, we cant source the patch fs on those nodes and try to start opmn forcefully.

[ErmanArslansOracleBlog]

So the difference is the presence of the CA-signed certificate in the proxy-wallet/ location on the secondary node.

Can't you just try equalizing the contents of proxy-wallet locations, correcting the opmn.xml (as suggested and as you identified the difference in chipers and versions) and then re-running the fsclone operation ?

[ky_17]

We did. But the problem is when we re run fs_clone (force=no) which should start fs_clone from where it failed. It removes the EBS_web_OHS2, EBS_web_OHS3 and EBS_web_OHS4 folders and recreates them. So our changes dont work. It again copies whatever it has staged and fails again.
I guess, something needs to be changed on RUN FS and fs_clone needs to run from beginning so it takes a new FMW archive.
But not sure, what to change on RUN FS as opmn is working there on all nodes. Would running autoconfig have any significance? We renewed our TLS certificates in August and have not run autoconfig since then. This is the first time we are running fs_clone post cert renewal.

[ErmanArslansOracleBlog]

Okay.. Your recent TLS certificate renewal is the likely cause of the issue  and the adop behavior of deleting and recreating the OHS folders confirms the fix must be applied to the run fs.

Since you confirmed that AutoConfig was run in August after the renewal, the configuration files on the $\text{Run FS}$ should be correct. This focuses the issue squarely back on the proxy-wallet's certificate content.

It is like; now in run fs you have the CA-signed certificate in all five wallet locations, while the FMW cloning process on the primary node correctly used a self-signed certificate for the internal proxy-wallet on the Patch FS during its successful clone. The secondary nodes are failing because they are forced to use the CA-signed certificate for an internal FMW handshake. Correct right?

What I 'm thinking is; getting the things from patch to run. (for proxy-wallet). From successfully created patch fs to the failed node's run fs and then retrying..

But!Did you revisit the SSL Enablement and Certificate Renewing process?

I mean -> Enabling TLS in Oracle E-Business Suite Release 12.2 (Doc ID 1367293.1)

Following helps?
---
As of June 2022, this document was changed to stop replacing the FMW auto-generated, self-signed certificate for the OPMN Remote Port and OHS Admin Port in the following locations:

$FMW_HOME/webtier/instances/<s_ohs_instance>/config/OHS/<s_ohs_componet>/proxy-wallet
$FMW_HOME/webtier/instances/<s_ohs_instance>/config/OPMN/opmn/wallet/cwallet.sso
$EBS_DOMAIN_HOME/opmn/<s_ohs_instance>/<s_ohs_component>/wallet
$EBS_DOMAIN_HOME/opmn/<s_ohs_instance>/wallet
If you have followed the procedures in this document in the past, you will most likely have short-lived CA signed certificates being used for FMW internal communication. You should create a self-sign certificate and copy them to the locations previously listed. Keep the CA signed certificate in the following location as-is:

$FMW_HOME/webtier/instances/<s_ohs_instance>/config/OHS/<s_ohs_component>/keystores/default/
-------------

Does it help?

[ky_17]

Thank!
We did not run autoconfig after the TLS cert renewals in August. I was wondering if running autoconfig on RUN FS helps?

Also, we have been on TLS for years and never faced this issue. We did check the process and the only difference is the note you copied.

We can try to remove the CA-signed cert from the given 4 locations and keep it only under the keystore/default location. And retry fs_clone again.

But RUN FS is bringing up opmn with it being on the 5 locations, as of now.
So, my understanding is that the FMW stage that fs_clone takes is not picking the correct certs or settings. That is why I was wondering if running autoconfig would help? We have not run it after cert renewals in August.

[ErmanArslansOracleBlog]

Please review "Enabling TLS in Oracle E-Business Suite Release 12.2 (Doc ID 1367293.1)" carefully.

Especially the section named -> 8.1 Renew Revoked or Expired Certificates.

Running Autoconfig is a good idea. Your EBS should be configured should be properly, and with autoconfig you may ensure that.. Then, you should check the wallet locations and copy the FMW generated correct one to the necessary locations if necessary.. Then when you are good, you can run fs clone again.

Note that: you can also review potential changes before running autoconfig by running check config.. Check config is used to review the configuration changes that would take effect on an E-Business Suite instance during the next AutoConfig run. It identifies the potential changes to both the File system as well as the Database. It can be run on both the applications tier and the database tier.

adchkcfg.sh contextfile=<CONTEXT_FILE>

If you didn't run autoconfig for a long time, be careful with that.