Issue while starting managed server

[satish]

Dear Erman,

We are unable to start the oacore_server3 on secondary node.we are facing the below error.could you please help us

<21 Mar, 2018 4:45:37 PM IST> <Error> <Configuration Management> <BEA-150021> <The admin server failed to authenticate the identity of the user  starting the managed server. The reason for the error is .>
<21 Mar, 2018 4:45:37 PM IST> <Emergency> <Management> <BEA-141151> <The admin server could not be reached at <a href="http://erpuatapp01.ttd.com:7003.&gt;">http://erpuatapp01.ttd.com:7003.>
<21 Mar, 2018 4:45:37 PM IST> <Info> <Configuration Management> <BEA-150018> <This server is being started in managed server independence mode in the absence of the admin server.>
<21 Mar, 2018 4:45:37 PM IST> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STARTING>
<21 Mar, 2018 4:45:37 PM IST> <Info> <WorkManager> <BEA-002900> <Initializing self-tuning thread pool>
<21 Mar, 2018 4:45:37 PM IST> <Notice> <Log Management> <BEA-170019> <The server log file /u01/ROOKAPPS/fs2/FMW_Home/user_projects/domains/EBS_domain_UATDB/servers/oacore_server3/logs/oacore_server3.log is opened. All server side log events will be written to this file.>
Mar 21, 2018 4:45:42 PM oracle.security.jps.internal.idstore.util.LibOvdUtil pushLdapNamesToLibOvd
INFO: Pushed ldap name and types info to libOvd. Ldaps : DefaultAuthenticator:idstore.ldap.provideridstore.ldap.
<21 Mar, 2018 4:45:42 PM IST> <Notice> <Security> <BEA-090082> <Security initializing using security realm myrealm.>
<21 Mar, 2018 4:45:42 PM IST> <Critical> <Security> <BEA-090403> <Authentication for user  denied>
<21 Mar, 2018 4:45:42 PM IST> <Critical> <WebLogicServer> <BEA-000386> <Server subsystem failed. Reason: weblogic.security.SecurityInitializationException: Authentication for user  denied
weblogic.security.SecurityInitializationException: Authentication for user  denied
        at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.doBootAuthorization(CommonSecurityServiceManagerDelegateImpl.java:966)
        at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.initialize(CommonSecurityServiceManagerDelegateImpl.java:1054)
        at weblogic.security.service.SecurityServiceManager.initialize(SecurityServiceManager.java:888)
        at weblogic.security.SecurityService.start(SecurityService.java:141)
        at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
        Truncated. see log file for complete stacktrace
Caused By: javax.security.auth.login.FailedLoginException: [Security:090304]Authentication Failed: User  javax.security.auth.login.LoginException: [Security:090301]Password Not Supplied
        at weblogic.security.providers.authentication.LDAPAtnLoginModuleImpl.login(LDAPAtnLoginModuleImpl.java:261)
        at com.bea.common.security.internal.service.LoginModuleWrapper$1.run(LoginModuleWrapper.java:110)
        at java.security.AccessController.doPrivileged(Native Method)
        at com.bea.common.security.internal.service.LoginModuleWrapper.login(LoginModuleWrapper.java:106)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        Truncated. see log file for complete stacktrace
>
<21 Mar, 2018 4:45:43 PM IST> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to FAILED>

Please note oacore_server1 and oacore_server2 are running and we see the issue with this oacore_server3

Thanks for your support
                                                                                                         

[ErmanArslansOracleBlog]

Read this -> 12.2 E-Business Suite Technology Stack Weblogic Server Does Not Start Due To Error 'weblogic.security.SecurityInitializationException: Authentication denied: Boot identity not valid' And 'LoginException: [Security:090301]Password Not Supplied' (Doc ID 2272165.1)

[ErmanArslansOracleBlog]

Note that,

if you can start all other oacore servers but oacore_server3, then I recommend you to skip the boot.properties related actions and do -> "Take backup and Remove $EBS_DOMAIN_HOME/servers/oacore_server3/data/ldap" and retry.

[satish]

CONTENTS DELETED

The author has deleted this message.

[satish]

Dear erman,

We are able to start all the managed servers on node1,we are facing the issue only with 2nd node.please find the log attached

Thanks for you ontime support
oacore_server3.txt

[ErmanArslansOracleBlog]

your main issue is "The admin server could not be reached"

This may happen, when your nodes have different subnet masks.
Are these app nodes in the same subnet?

This may also happen if your start script is trying to reach the wrong admin server port.
Is 7003 correct? ( I m not asking about telnet.. Is it the admin server port? Is it the port where the weblogic console is running? Is it up & running in a good health state?)

Also send me the start log of a proper managed server? (for ex: the log of "admanagedsrvctl.sh start oacore_server2")

Note that, t3://erpuatapp02.blablabla.com:7203 is your managed server url , oacore_server3 url.. So it is not important.. In the log, it can not be reached, which is normal, as it cannot be started..

[satish]

Dear Erman,

Thanks for the update.

7003 is the admin port and we are able to access the console.please find the logs for successfully started managed server

[appluat@erpuatapp01 ~]$ admanagedsrvctl.sh start oacore_server2

You are running admanagedsrvctl.sh version 120.14.12020000.11

Enter the WebLogic Admin password:

Calling txkChkEBSDependecies.pl to perform dependency checks for oacore_server2
*** ALL THE FOLLOWING FILES ARE REQUIRED FOR RESOLVING RUNTIME ERRORS
*** Log File = /u02/UATINST/fs2/inst/apps/UATDB_erpuatapp01/logs/appl/rgf/TXK/txkChkEBSDependecies_Wed_Mar_21_21_30_46_2018/txkChkEBSDependecies_Wed_Mar_21_21_30_46_2018.log
Perl script txkChkEBSDependecies.pl got executed successfully


Starting oacore_server2...

Server specific logs are located at /u01/ROOKAPPS/fs2/FMW_Home/user_projects/domains/EBS_domain_UATDB/servers/oacore_server2/logs

admanagedsrvctl.sh: exiting with status 0

admanagedsrvctl.sh: check the logfile /u02/UATINST/fs2/inst/apps/UATDB_erpuatapp01/logs/appl/admin/log/adoacorectl.txt for more information ...

[appluat@erpuatapp01 ~]$



Logs
===
03/21/18-21:30:42 :: admanagedsrvctl.sh version 120.14.12020000.11


Calling txkChkEBSDependecies.pl to perform dependency checks for oacore_server2
Perl script txkChkEBSDependecies.pl got executed successfully


Validated the passed arguments for the option ebs-get-serverstatus
oacore_server2 is in SHUTDOWN mode.

Validated the passed arguments for the option ebs-nmstart-managedsrv
The Server oacore_server2 is not up. Proceeding to start the server.
FMW Version is 11.1.1.9
Checking if the Node Manager is already up..
Connecting to Node Manager ...
Successfully Connected to Node Manager.

The Node Manager is already up.
FMW Version is 11.1.1.9
JSSE is already enabled on oacore_server2
Starting server oacore_server2 ...
Successfully started server oacore_server2 ...

Started up the managed server oacore_server2

Server specific logs are located at /u01/ROOKAPPS/fs2/FMW_Home/user_projects/domains/EBS_domain_UATDB/servers/oacore_server2/logs

03/21/18-21:31:37 :: admanagedsrvctl.sh: exiting with status 0

================================================================================


Issue is with oacore_server3 which is running on node2 but opmn,apache and fndlibr processes are up and running on node2.how can we check if these nodes are in same subnet?

Thanks for your support

[satish]

Even i face same issue with oafm_server2 on node2

logs
===
<21 Mar, 2018 9:18:20 PM IST> <Error> <Configuration Management> <BEA-150021> <The admin server failed to authenticate the identity of the user  starting the managed server. The reason for the error is .>
<21 Mar, 2018 9:18:20 PM IST> <Emergency> <Management> <BEA-141151> <The admin server could not be reached at <a href="http://erpuatapp01.ttd.com:7003.&gt;">http://erpuatapp01.ttd.com:7003.>
<21 Mar, 2018 9:18:20 PM IST> <Info> <Configuration Management> <BEA-150018> <This server is being started in managed server independence mode in the absence of the admin server.>
<21 Mar, 2018 9:18:20 PM IST> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STARTING>
<21 Mar, 2018 9:18:20 PM IST> <Info> <WorkManager> <BEA-002900> <Initializing self-tuning thread pool>
<21 Mar, 2018 9:18:21 PM IST> <Notice> <Log Management> <BEA-170019> <The server log file /u01/UATAPPS/fs2/FMW_Home/user_projects/domains/EBS_domain_UATDB/servers/oafm_server2/logs/oafm_server2.log is opened. All server side log events will be written to this file.>
Mar 21, 2018 9:18:29 PM oracle.security.jps.internal.idstore.util.LibOvdUtil pushLdapNamesToLibOvd
INFO: Pushed ldap name and types info to libOvd. Ldaps : DefaultAuthenticator:idstore.ldap.provideridstore.ldap.
<21 Mar, 2018 9:18:30 PM IST> <Notice> <Security> <BEA-090082> <Security initializing using security realm myrealm.>
<21 Mar, 2018 9:18:30 PM IST> <Critical> <Security> <BEA-090403> <Authentication for user  denied>
<21 Mar, 2018 9:18:30 PM IST> <Critical> <WebLogicServer> <BEA-000386> <Server subsystem failed. Reason: weblogic.security.SecurityInitializationException: Authentication for user  denied
weblogic.security.SecurityInitializationException: Authentication for user  denied
        at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.doBootAuthorization(CommonSecurityServiceManagerDelegateImpl.java:966)
        at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.initialize(CommonSecurityServiceManagerDelegateImpl.java:1054)
        at weblogic.security.service.SecurityServiceManager.initialize(SecurityServiceManager.java:888)
        at weblogic.security.SecurityService.start(SecurityService.java:141)
        at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
                                                                                  40,1-8    

[ErmanArslansOracleBlog]

1)Reproduce the error and check the admin server log..
Does admin server log tell you anything? (it may tell you the reason, why it is unable to establish the connection..)

2)Can you see these managed servers (managed servers on node2) in weblogic admin console?  Can you start them by using weblogic admin console?

3)Please check the server start arguments;

Connect to weblogic admin console

Domain Configurations> Sub Category: Environment > click the link "Servers"
Click > oacore_server**
Click Server Start ("Sub Tab")

Check arguments there..
Compare it with the working oacore_servers, do you see any differences?

[satish]

Thanks for the update erman

we are able to see services on secondary node from weblogic console
Below is the error from admin server logs

<22 Mar, 2018 3:27:39 PM IST> <Warning> <DeploymentService> <BEA-290014> <Invalid user name or password.>
<22 Mar, 2018 3:27:40 PM IST> <Error> <Configuration Management> <BEA-150035> <An attempt was made to download the configuration for the server oafm_server2 by the user  with an invalid password.>
 
same error for oacore_server3 and oafm_server2 which are running on second node

<22 Mar, 2018 3:34:59 PM IST> <Warning> <DeploymentService> <BEA-290014> <Invalid user name or password.>
<22 Mar, 2018 3:35:00 PM IST> <Error> <Configuration Management> <BEA-150035> <An attempt was made to download the configuration for the server oacore_server3 by the user  with an invalid password.>


But i could see forms_server3 running on secondary node without any issues

Thanks for your support

[ErmanArslansOracleBlog]

Please implement the following actions and update me with the outcome;

Take backup and Remove $EBS_DOMAIN_HOME/servers/<problematic_managed_server>/data/ldap" and retry.

[Satish]

Thanks for the updates erman, I have done this earlier but didn't work.i have also done the below

Created boot.properties file and added weblogic username and password to it.Then I have started the server but no luck.
I think we also need to encrypt it in EBS.but didn't do that.
Can you plz update the command with which we have to encrypt the boot.properties file after adding weblogic username and password to the problematic server

[satish]

Erman Arslan's Oracle Forum › EBS 12.2
satish

Issue while starting managed server
classic Classic list List threaded Threaded
      13 messages  Options Options
Mar 21 satish Issue while starting managed server – Dear Erman, We are unable to start the oacore_server3 on secondary node.we are facing the below error.could you please help us <21 Mar, 2...
Mar 21 ErmanArslansOracleBlog Read this -> 12.2 E-Business Suite Technology Stack Weblogic Server Does Not Start Due To Error 'weblogic.security.SecurityInitializationExcept...
Mar 21 ErmanArslansOracleBlog Note that, if you can start all other oacore servers but oacore_server3, then I recommend you to skip the boot.properties related actions and d...
Mar 21 satish - deleted -
Mar 21 satish Dear erman, We are able to start all the managed servers on node1,we are facing the issue only with 2nd node.please find the log attached Th...
Mar 21 ErmanArslansOracleBlog your main issue is "The admin server could not be reached" This may happen, when your nodes have different subnet masks. Are these a...
Mar 21 satish Dear Erman, Thanks for the update. 7003 is the admin port and we are able to access the console.please find the logs for successfully starte...
Mar 21 satish Even i face same issue with oafm_server2 on node2 logs === <21 Mar, 2018 9:18:20 PM IST> <The admin server failed to authenti...
Mar 22 ErmanArslansOracleBlog 1)Reproduce the error and check the admin server log.. Does admin server log tell you anything? (it may tell you the reason, why it is unable to ...
Mar 22 satish Thanks for the update erman we are able to see services on secondary node from weblogic console Below is the error from admin server logs &...
Mar 22 ErmanArslansOracleBlog Please implement the following actions and update me with the outcome; Take backup and Remove $EBS_DOMAIN_HOME/servers//data/ldap" and retry.
9:58am Satish Thanks for the updates erman, I have done this earlier but didn't work.i have also done the below Created boot.properties file and added weblog...
Classic  Threaded   Reply   More     Close Mar 23, 2018; 10:47am  satishonline satish
Dear Erman,

I have taken backup and move the oacore_server3 dir.Then started the server but didnt help.I could see the dir oacore_server3 created but no boot.properties file created as below

[appluat@erpuatapp01 logs]$ pwd
/u01/ROOKAPPS/fs2/FMW_Home/user_projects/domains/EBS_domain_UATDB/servers/AdminServer/logs
[appluat@erpuatapp01 logs]$ cd ../..
[appluat@erpuatapp01 servers]$ ls
AdminServer         forms-c4ws_server2  oacore_server1      oaea_server1
AdminServerTag      forms_server1       oacore_server2      oafm_server1
domain_bak          forms_server2       oacore_server3      oafm_server2
forms-c4ws_server1  forms_server3       oacore_server3_bkp  oafm_server2_bkp
[appluat@erpuatapp01 servers]$ cd oacore_server3
[appluat@erpuatapp01 oacore_server3]$ pwd
/u01/ROOKAPPS/fs2/FMW_Home/user_projects/domains/EBS_domain_UATDB/servers/oacore_server3
[appluat@erpuatapp01 oacore_server3]$ ls
cache  data  logs  security  tmp
[appluat@erpuatapp01 oacore_server3]$ cd security/
[appluat@erpuatapp01 security]$ ls
[appluat@erpuatapp01 security]$



Below error is recorded in admin server logs

[appluat@erpuatapp01 servers]$ admanagedsrvctl.sh start oacore_server3

You are running admanagedsrvctl.sh version 120.14.12020000.11

Enter the WebLogic Admin password:

Calling txkChkEBSDependecies.pl to perform dependency checks for oacore_server3
*** ALL THE FOLLOWING FILES ARE REQUIRED FOR RESOLVING RUNTIME ERRORS
*** Log File = /u02/UATINST/fs2/inst/apps/UATDB_erpuatapp01/logs/appl/rgf/TXK/txkChkEBSDependecies_Fri_Mar_23_10_07_51_2018/txkChkEBSDependecies_Fri_Mar_23_10_07_51_2018.log
Perl script txkChkEBSDependecies.pl got executed successfully


Starting oacore_server3...

Server specific logs are located at /u01/ROOKAPPS/fs2/FMW_Home/user_projects/domains/EBS_domain_UATDB/servers/oacore_server3/logs

admanagedsrvctl.sh: exiting with status 1

admanagedsrvctl.sh: check the logfile /u02/UATINST/fs2/inst/apps/UATDB_erpuatapp01/logs/appl/admin/log/adoacorectl.txt for more information ...

[appluat@erpuatapp01 servers]$

Error

<23 Mar, 2018 10:08:02 AM IST> <Warning> <DeploymentService> <BEA-290014> <Invalid user name or password.>
<23 Mar, 2018 10:08:03 AM IST> <Error> <Configuration Management> <BEA-150035> <An attempt was made to download the configuration for the server oacore_server3 by the user  with an invalid password.>
             

[ErmanArslansOracleBlog]

1)You may try starting your problematic managed servers using weblogic console.. Did you try that?

2)Did you see this thread? -> "BEA-150035 An attempt was made to download the configuration for the server wls_forms by the user with an invalid password" -> Community Discussion ID 3489175 (Oracle Support)

[Satish]

Thanks for the update erman, can you plz provide the complete link.

[ErmanArslansOracleBlog]

Read this for starting/stopping managed servers from weblogic console->

https://docs.oracle.com/cd/E92951_01/wls/WLACH/taskhelp/startstop/StartManagedServersFromTheAdministrationConsole.html

Those community links can not be shared..

Just login to oracle support
search for the key : "BEA-150035 An attempt was made to download the configuration for the server wls_forms by the user with an invalid password"

Then, check the links under the Community Search Results section.. You will see it there..

 

[satish]

Dear Erman,

I started the server from admin console but same error repeats

<26 Mar, 2018 3:42:41 PM IST> <Error> <Configuration Management> <BEA-150035> <An attempt was made to download the configuration for the server oacore_server3 by the user  with an invalid password.>
<26 Mar, 2018 3:55:03 PM IST> <Error> <NodeManager> <BEA-300048> <Unable to start the server oacore_server3 : Exception while starting server 'oacore_server3'>

I observed boot.properties was not created,i have created the file manually and place the username and password but the entries are not being encrypted

[appluat@erpuatapp01 security]$ cd /u01/UATAPPS/fs2/FMW_Home/user_projects/domains/EBS_domain_UATDB/servers/oacore_server3/security
[appluat@erpuatapp01 security]$ ls -lrt
total 0
[appluat@erpuatapp01 security]$

could you please suggest any ideas

[ErmanArslansOracleBlog]

which boot.properties did you modify?

Also you can try to create one under that "security" folder and retry.

[satish]

I have created one under security folder but didnt work.wheni tried to start the managed server from admin console,i could see boot.properties created under nodemanager folder as below

[appluat@erpuatapp01 nodemanager]$ pwd
/u01/UATAPPS/fs2/FMW_Home/user_projects/domains/EBS_domain_UATDB/servers/oacore_server3/data/nodemanager
[appluat@erpuatapp01 nodemanager]$

[appluat@erpuatapp01 nodemanager]$ cat boot.properties
#Mon Mar 26 16:47:00 IST 2018
TrustKeyStore=DemoTrust
password={AES}i8G2JjZB2gk1IQExZP8WZKPVTajh+vhIhRGRvwRPai0\=
username={AES}n6lthM4wOMxx30/q8GDkwVceZaH/FL78vr4DDvvx5ww\=

But server ended up with error

[ErmanArslansOracleBlog]

Strange..
change the Weblogic admin password and retry.

In order to change the weblogic admin password in EBS 12.2 follow->

How To Change or Reset The WebLogic Administration Password In 12.2 E-Business Suite (Doc ID 1385751.1)