adding nodes failing

[latifa]

Hi Erman, it's Always me :)

So adding an other node to the master node recently cloned is getting error as Following:

Modifying /opt/application/V12/fs1/FMW_Home/webtier/instances/EBS_web_OHS2/config/OHS/EBS_web/ssl.conf to set SSLProtocol
opmnctl startall: starting opmn and all managed processes...
opmnctl startall: failed.

EBS_web is failed to start !!

.end std out.
[2020-10-20T13:35:15+02:00] [opmn] [ERROR:1] [] [internal] lxlinit - NLS boot file not found or invalid
 -- default linked-in boot block used
[2020-10-20T13:35:15+02:00] [opmn] [ERROR:1] [105] [internal] XML parser init: character set initialization failed.

.end err out.

ERROR: Unable to create OHS instance.
ERROR: OHS instance creation and configuration failed
NMProcess: <Oct 20, 2020 1:30:49 PM> <INFO> <Loading domains file: /opt/application/V12/fs1/FMW_Home/wlserver_10.3/common/nodemanager/nmHome2/nodemanager.domains>
NMProcess: Oct 20, 2020 1:30:49 PM weblogic.nodemanager.server.NMServerConfig initDomainsMap
NMProcess: INFO: Loading domains file: /opt/application/V12/fs1/FMW_Home/wlserver_10.3/common/nodemanager/nmHome2/nodemanager.domains
NMProcess: <Oct 20, 2020 1:30:49 PM> <WARNING> <Node manager configuration properties file '/opt/application/V12/fs1/FMW_Home/wlserver_10.3/common/nodemanager/nmHome2/nodemanager.properties' not found. Using default settings.>
NMProcess: Oct 20, 2020 1:30:49 PM weblogic.nodemanager.server.NMServer <init>
NMProcess: WARNING: Node manager configuration properties file '/opt/application/V12/fs1/FMW_Home/wlserver_10.3/common/nodemanager/nmHome2/nodemanager.properties' not found. Using default settings.
NMProcess: <Oct 20, 2020 1:30:49 PM> <INFO> <Saving node manager configuration properties to '/opt/application/V12/fs1/FMW_Home/wlserver_10.3/common/nodemanager/nmHome2/nodemanager.properties'>
NMProcess: Oct 20, 2020 1:30:49 PM weblogic.nodemanager.server.NMServer <init>
NMProcess: INFO: Saving node manager configuration properties to '/opt/application/V12/fs1/FMW_Home/wlserver_10.3/common/nodemanager/nmHome2/nodemanager.properties'
NMProcess: Node manager v10.3
NMProcess:
NMProcess: Configuration settings:
NMProcess:
NMProcess: NodeManagerHome=/opt/application/V12/fs1/FMW_Home/wlserver_10.3/common/nodemanager/nmHome2
NMProcess: ListenAddress=udkfini15.dalkia.net
NMProcess: ListenPort=5557
NMProcess: ListenBacklog=50
NMProcess: SecureListener=false
NMProcess: AuthenticationEnabled=true
NMProcess: NativeVersionEnabled=true
NMProcess: CrashRecoveryEnabled=false
NMProcess: JavaHome=/opt/application/V12/fs1/EBSapps/comn/util/jdk/jre
NMProcess: StartScriptEnabled=false
NMProcess: StopScriptEnabled=false
NMProcess: StartScriptName=startWebLogic.sh
NMProcess: StopScriptName=
NMProcess: LogFile=/opt/application/V12/fs1/FMW_Home/wlserver_10.3/common/nodemanager/nmHome2/nodemanager.log
NMProcess: LogLevel=INFO
NMProcess: LogLimit=0
NMProcess: LogCount=1
NMProcess: LogAppend=false
NMProcess: LogToStderr=true
NMProcess: LogFormatter=weblogic.nodemanager.server.LogFormatter
NMProcess: DomainsFile=/opt/application/V12/fs1/FMW_Home/wlserver_10.3/common/nodemanager/nmHome2/nodemanager.domains
NMProcess: DomainsFileEnabled=true
NMProcess: StateCheckInterval=500
NMProcess: QuitEnabled=true
NMProcess: UseMACBroadcast=false
NMProcess: DomainRegistrationEnabled=false
NMProcess: DomainsDirRemoteSharingEnabled=false
NMProcess:
NMProcess: Domain name mappings:
NMProcess:
NMProcess: EBS_domain -> /opt/application/V12/fs1/FMW_Home/user_projects/domains/EBS_domain
NMProcess:
NMProcess: <Oct 20, 2020 1:30:50 PM> <INFO> <Plain socket listener started on port 5557, host udkfini15.dalkia.net>
NMProcess: Oct 20, 2020 1:30:50 PM weblogic.nodemanager.server.Listener run
NMProcess: INFO: Plain socket listener started on port 5557, host udkfini15.dalkia.net
OHSUtil::generateOHSInstanceName::sSid: DBOFINI2

OHSUtil::generateOHSInstanceName::file_edition: run

OHSUtil::generateOHSInstanceName::newInstanceName: EBS_web_DBOFINI2_OHS0

OHSUtil::generateOHSInstanceName::objectNames.size(): 1

OHSUtil::generateOHSInstanceName::mbeanObjectName.length(): 71

OHSUtil::generateOHSInstanceName::ohsInstanceNames.length: 1

OHSUtil::generateOHSInstanceName::tempOHSInstance: EBS_web_OHS1

OHSUtil::generateOHSInstanceName::tempOHSInstance.indexOf(sSid): -1

OHSUtil::generateOHSInstanceName::ohsInstancePrefix: EBS_web_OHS

OHSUtil::getNextOHSInstanceIndex::ohsInstancePrefix: EBS_web_OHS

OHSUtil::getNextOHSInstanceIndex::element: EBS_web_OHS1

OHSUtil::getNextOHSInstanceIndex::nextOHSInstance: 2

OHSUtil::generateOHSInstanceName::nextInstanceNumber: 2

OHSUtil::generateOHSInstanceName::newInstanceName: EBS_web_OHS2

ERROR: An error has occurred while adding the node. Refer to the log file /opt/application/V12/fs1/inst/apps/DBOFINI2_udkfini15/admin/log/clone/run/RCloneApplyAppstier_10201313.log for details.
Please rectify the error and add the node back following My Oracle Support Knowledge Document 1383621.1


.end std out.

.end err out.

Script Executed in 300116 milliseconds, returning status 1

---> So first the error message is this one I think:
[opmn] [ERROR:1] [] [internal] lxlinit - NLS boot file not found or invalid
 -- default linked-in boot block used
[2020-10-20T13:35:15+02:00] [opmn] [ERROR:1] [105] [internal] XML parser init: character set initialization failed.


What do you think ?

Regards,
Latifa
 

[ErmanArslansOracleBlog]

Yes.. Those errors can be decisive but also deceptive.

Check the following MOS note ->

Same errors reported there, but look at the cause and the solution;

Adcfgclone.pl Application Tier Failing With "Permission denied" While Instantiating The OHS Config (Doc ID 2151181.1)

[latifa]

Erman,

You know, I checked the system where I was cloning the secondary node and I found that I didn't comment Calling to EBSapps.env in the .profile file for the session of clonning. Could it be the Reason ? :-(

Regards
Latifa

[latifa]

I deleted the node and restored all EBSapps folder from the master.
I re-ran the adcfgclone to add the node and it working fine :)

Thank you Erman for your patience
Latifa

[latifa]

Erman,

cloning the node finished but I checked in the database and I found that the server has an "INACTIVE" status ! What does it mean ?

Thank you for your reply.

Regards
Latifa

[ErmanArslansOracleBlog]

Latifa;

Where exactly does -Inactive- write?
What kind of check did you do in the database and saw this INACTIVE status?

[latifa]

with this:

select NODE_NAME
, to_char(CREATION_DATE, 'DD-MON-RR HH24:MI') creation_date
, PLATFORM_CODE
, decode(STATUS,'Y','ACTIVE','INACTIVE') Status
, decode(SUPPORT_CP,'Y', 'ConcMgr','No') ConcMgr
, decode(SUPPORT_FORMS,'Y','Forms', 'No') Forms
, decode(SUPPORT_WEB,'Y','Web', 'No') WebServer
, decode(SUPPORT_ADMIN, 'Y','Admin', 'No') Admin
, decode(SUPPORT_DB, 'Y','Rdbms', 'No') Database
/*
, to_char(LAST_MONITORED_TIME, 'DD-MON-RR HH24:MI:SS') last_monitored
, NODE_MODE
, SERVER_ADDRESS
, HOST
, DOMAIN
, WEBHOST
, VIRTUAL_IP
, SERVER_ID
*/
from apps.fnd_nodes
where node_name != 'AUTHENTICATION'
order by node_name;

[latifa]

May be because it is not started :) !!

[ErmanArslansOracleBlog]

Maybe :)
So what is the current situation?

[latifa]

Hi Erman,

Today I continue cloning the test environment (yesterday day off :) sorry for delay replying).

I will let you know if every thing is ok

Regards,
Latifa

[latifa]

Hi Erman,

Now I finished cloning and I vhave a new error:

ERROR
Error : APP-FND-01516 : Invalid Application user name ,password or database
Username <Applsyspub>
Database <Database SID>

I found this DocID: APP-FND-01516 While Launching Forms from Framework Page for some users (Doc ID 2175267.1)

However i can' Apply the solution because I should open a forms session !

Other thing I saw, when connecting to the database server I mean doing sqlplus / as sysdba, I have sometimes the Following error message: -bash: fork: Not enough space

I think we have a system problem also or may be it's the only cause !!

What is your opinion ?

Thank you in advance for your help.

Latifa

[ErmanArslansOracleBlog]

Yes, it is better to solve that  " -bash: fork: Not enough space" error and retry..
This error is related with the swap space.

Increase swap size(OS work) or find the process that consumes all the swap space and terminate it(if it is the case, and if the termination is applicable)

[latifa]

Hi Erman,

Thank you very much. Increasing the swap solved the problem.

The cloning has finished with this step :)

Thank you again -:)

Latifa

[latifa]

Hi Erman,

The situation is unstable, the application is inaccessible sometimes intermitently.

Shull we rebbot the global zone  ?

Regards,
Latifa

[ErmanArslansOracleBlog]

Global zone.. Hmm this is Solaris right..
Do you still get the same "bash: fork: Not enough space) error?
Did you see the new space added to swap? -- I don't know what your Solaris version is and what steps did you take to add more space to the swap, so I can't tell if you should reboot or not.. (Don't you have your Solaris Admin around?)
Did you monitor swap usage after resizing it? Maybe you occupy the swap space again..(even after adding some more space to it)

[latifa]

Yes it's a solaris server:
SunOS etic1pdb0a 5.11 11.4.16.4.0 sun4v sparc sun4v

I Don't have any more the error message. It takes time to respond when I do" ls -l for example".
Also it generate a core file : core:           ELF 64-bit MSB core file SPARCV9 Version 1, from 'oracle' !

I couldn't see that because I Don't have access. However, an DB admin told me that he didn't changes !!

Now it's Ok, I can again access to the application.

Let's see if on monday they can work.

Regards,
Latifa

[latifa]

Hi Erman,

just to give you status of the situation :)

System administrator rebooted thz zone. It looks like stable. However database administrator decided to adjust databases parameters. Actually it's ok hope it will continue like this.

Regards,
Latifa

[ErmanArslansOracleBlog]

Okay Latifa . Good :)