EBS 12.2.10 Multi-node Installation - Admin Server Configuration on 2-nodes

[burneyj]

I have installed EBS 12.2.10 in a Multi-node setup.

1 DB node and 2 Application Nodes.

I followed the Documentation:Cloning Oracle E-Business Suite Release 12.2 with Rapid Clone (Doc ID 1383621.1)

I followed steps 5.3 Adding a New Application Tier Node to an Existing System
Running adcfgclone.pl in the interactive mode

Run the following command:

$ cd <COMMON_TOP>/clone/bin

$ perl adcfgclone.pl appsTier dualfs

When asked the questions:

"Do you want to add a node (yes/no)" you should answer 'yes'

==

My Application is running fine BUT

The Admin Server is running on the first node only, the Primary Node. If that node goes down the application will go down because the Admin Server starts and stops from the Primary Node.

WebLogic Admin Server Control can only be accessed using the URL from the Primary node.



My question is:

Is there a way to Install/Configure Admin Server on BOTH the Application nodes so that if one node goes down the application keeps running?

I have many EBS 12.1.3 environments running on 2-Application nodes and if one node goes down the concurrent managers start on the other node and vice-versa and application is keeps running.

[ErmanArslansOracleBlog]

That's normal. Adminserver runs on one node only.
Normally, a admin server failure doesn't effect the applications..
However; if there is NFS-like configuration and if that node ( for instance node1), that is hosting NFS shared fails, all the nodes will fail.. Because the NFS will be down.. In that case, we do some extra stuff..

Read -> https://ermanarslan.blogspot.com/2017/09/ebs-122-shared-application-filesystem.html

[burneyj]

Thanks a lot for you reply.

I did take NODE 1 offline and the application continued to work and concurrent managers failed over to NODE 2 as well.

However, I noticed something after taking NODE 1 offline and have no explanation for that. I logged in the application on NODE 2 but it took really long for the forms to launch almost 2 minutes. Once the forms were launched there was no problem. I could launch and close forms. When I brought NODE 1 back online everything was back to normal. I could launch forms in normal time.

When I took  NODE 2 offline and connected to NODE 1 there was no problem with launching the forms.

Any idea why the forms take so long to launch from NODE 2 when the NODE 1 Admin server is offline?

[ErmanArslansOracleBlog]

1)What about launching the forms on NODE2, while everyting is working normally, I mean while NODE1 and NODE 2 both are running? That may be general issue.. We first need to identify it clearly..
2)Do you have a load balancer in front of the application? What happens when you directly launch the forms on node 2 by setting ICX Forms Launcher to the url that includes node 2's direct hostname? Do you see the same performance problem?
3)Did you check the client side? I mean, client may be the reason.. Did you check the java console and see what is taking too much time during the launch of the forms applet?

[burneyj]

1)What about launching the forms on NODE2, while everyting is working normally, I mean while NODE1 and NODE 2 both are running? That may be general issue.. We first need to identify it clearly..

When BOTH Application NODES are available and working normally, the forms come up in usual time like in few seconds. There is no problem there.

2)Do you have a load balancer in front of the application? What happens when you directly launch the forms on node 2 by setting ICX Forms Launcher to the url that includes node 2's direct hostname? Do you see the same performance problem?

Yes I am using a Load Balancer in front my application BUT if it was the Load Balancer issue it should also happen when NODE 1 is up and NODE 2 is down. When NODE 2 is down and application is running on NODE 1, the forms come up
in normal time just like when BOTH the nodes are up.

3)Did you check the client side? I mean, client may be the reason.. Did you check the java console and see what is taking too much time during the launch of the forms applet?

The java console comes up and hangs for about 2 mins and then the forms are launched.

This delay is happening only when NODE 1 is down. I think it has something to do with Admin Server being not available.

It is only the initial connection, when that connection is made things run fine. I think it is trying to make the connection to the forms server "forms_server1" on NODE 1 and when it does not get connection or times out it connects to the forms server "forms_server2" on NODE2.

Do you think it can be because forms has a cluster "forms_cluster1" at Weblogic level. The two managed forms servers are in that cluster "forms_server1" and "forms_server2". The two managed forms servers go round-robin.
I tried to change that to round-robin affinity but that didnt work either. I tried random as well but nothing different.

[ErmanArslansOracleBlog]

Yes, that weblogic level configuration may be the cause..
According to your inputs, probably, we wait for some timeout and then proceed with node 2 / forms 2..

Please check mod_wls.conf . What is the DynamicServerList set? Is it OFF or ON? ( for forms..)

Dynamic update list provide a mechanism to add any new servers in the cluster and delete any that have been shut down, or are being suspended, or are no longer part of the cluster or that have failed to respond to requests....

That parameter is also a little buggy, but I think it is related with the context.

Ref Threads:

http://erman-arslan-s-oracle-forum.2340467.n4.nabble.com/DynamicServerList-on-in-R12-2-td8048.html
http://erman-arslan-s-oracle-forum.2340467.n4.nabble.com/question-on-mod-wl-ohs-conf-and-apps-conf-td4475.html

[ErmanArslansOracleBlog]

That timeout may be caused the value of WLIOTimeoutSecs ... Check that as well..  

REF: MOS - if the server doesn’t respond within the WLIOTimeoutSecs, the plug-in looks for fail over and makes further attempts based on the ConnectRetrySecs and ConnectTimeoutSecs parameters set in seconds to other member in the cluster list as defined ..

[ErmanArslansOracleBlog]

So this seems to be related with the weblogic plugin.. You can debug the wls plugin and get more details about the problem. But! first consider the things given in my previous updates.

[burneyj]

I have figured out the reason why the forms take longer to launch when NODE 1 is down.


1. On the Primary Server, NODE 1,  (where Admin Server is running)  I have the following directory:

acacstap03:IMR03S:applmgr /app/applmgr/IMR03S/fs2/FMW_Home/webtier/instances $ls -al
total 12
drwxr-xr-x.  3 applmgr finusers 4096 Mar 25 23:04 .
drwxr-xr-x. 53 applmgr finusers 4096 Mar 25 23:06 ..
drwx------.  8 applmgr finusers 4096 Mar 25 23:04 EBS_web_OHS1
acacstap03:IMR03S:applmgr /app/applmgr/IMR03S/fs2/FMW_Home/webtier/instances $


acacstap03:IMR03S:applmgr /app/applmgr/IMR03S/fs2/FMW_Home/webtier/instances/EBS_web_OHS1/config/OHS/EBS_web $more mod_wl_ohs.conf

####################################################################
LoadModule weblogic_module   "${ORACLE_HOME}/ohs/modules/mod_wl_ohs.so"
<IfModule mod_weblogic.c>
        DynamicServerList OFF    <<<<<<< IT IS ALREADY OFF
        <Location /OA_HTML>
                SetHandler weblogic-handler
                WebLogicCluster acacstap03.es.ad.adp.com:7202
                WLTempDir ${ORACLE_INSTANCE}/tmp
                #SecureProxy ON
                #WLProxySSL ON
                #WLSSLWallet "${ORACLE_INSTANCE}/config/${COMPONENT_TYPE}/${COMPONENT_NAME}/keystores/default"
        </Location>

========================

acacstap03::applmgr /app/applmgr/IMR03S/fs2/FMW_Home/webtier/instances/EBS_web_OHS1/config/OHS/EBS_web/apps.conf

ONLY HAS entries for NODE 1.

######################
 # for forms
 ######################
 <Location /OA_MEDIA>
  ProxyPass balancer://formscluster_oamedia
  ProxyPassReverse balancer://formscluster_oamedia
 </Location>
 <Proxy balancer://formscluster_oamedia>
   BalancerMember http://acacstap03.es.ad.adp.com:7402/OA_HTML/media
 </Proxy>


2. On the Secondary Node (NODE 2) I have 2 directories:

acacstap04:IMR03S:applmgr /app/applmgr/IMR03S/fs2/FMW_Home/webtier/instances $ls -al
total 20
drwxr-xr-x.  5 applmgr finusers 4096 Mar 25 23:38 .
drwxr-xr-x. 53 applmgr finusers 4096 Mar 25 23:40 ..
drwx------.  8 applmgr finusers 4096 Mar 25 23:17 EBS_web_OHS1
drwx------.  8 applmgr finusers 4096 Mar 25 23:38 EBS_web_OHS2


acacstap04:IMR03S:applmgr /app/applmgr/IMR03S/fs2/FMW_Home/webtier/instances/EBS_web_OHS1/config/OHS/EBS_web $more mod_wl_ohs.conf

####################################################################
LoadModule weblogic_module   "${ORACLE_HOME}/ohs/modules/mod_wl_ohs.so"
<IfModule mod_weblogic.c>
DynamicServerList OFF   <<<<<<<<<<
<Location /OA_HTML>
SetHandler weblogic-handler
WebLogicCluster acacstap03.es.ad.adp.com:7201
WLTempDir ${ORACLE_INSTANCE}/tmp
#SecureProxy ON
#WLProxySSL ON
#WLSSLWallet "${ORACLE_INSTANCE}/config/${COMPONENT_TYPE}/${COMPONENT_NAME}/keystores/default"

====

acacstap04:IMR03S:applmgr /app/applmgr/IMR03S/fs2/FMW_Home/webtier/instances/EBS_web_OHS1/config/OHS/EBS_web $more apps.conf

######################
 # for forms
 ######################
 <Location /OA_MEDIA>
  ProxyPass balancer://formscluster_oamedia
  ProxyPassReverse balancer://formscluster_oamedia
 </Location>
 <Proxy balancer://formscluster_oamedia>
   BalancerMember http://acacstap03.es.ad.adp.com:7401/OA_HTML/media
 </Proxy>



UNDER the EBS_web_OHS2 directory (/app/applmgr/IMR03S/fs2/FMW_Home/webtier/instances/EBS_web_OHS2/config/OHS/EBS_web) on NODE 2 the file apps.conf

HAS ENTRIES FOR BOTH NODES.

=================
LoadModule proxy_balancer_module   "${ORACLE_HOME}/ohs/modules/mod_proxy_balancer.so"
ProxyRequests off

<IfModule proxy_balancer_module>

 ProxyRequests Off

 ######################
 # for oacore
 ######################
 <Location /OA_MEDIA>
   ProxyPass balancer://oacorecluster_oamedia
   ProxyPassReverse balancer://oacorecluster_oamedia
 </Location>
 <Proxy balancer://oacorecluster_oamedia>
   BalancerMember http://acacstap04.es.ad.adp.com:7202/OA_HTML/media
   BalancerMember http://acacstap03.es.ad.adp.com:7202/OA_HTML/media
 </Proxy>


 <Location /OA_JAVA>
   ProxyPass balancer://oacorecluster_oajava
   ProxyPassReverse balancer://oacorecluster_oajava
 </Location>
 <Proxy balancer://oacorecluster_oajava>
   BalancerMember http://acacstap04.es.ad.adp.com:7202/OA_HTML/classes
   BalancerMember http://acacstap03.es.ad.adp.com:7202/OA_HTML/classes
 </Proxy>


 <Location /OA_CGI/FNDWRR.exe>
  ProxyPass balancer://oacorecluster_oacgi/OA_HTML/txkFNDWRR.pl
  ProxyPassReverse balancer://oacorecluster_oacgi/OA_HTML/txkFNDWRR.pl
 </Location>
 <Proxy balancer://oacorecluster_oacgi/OA_HTML/txkFNDWRR.pl>
   BalancerMember http://acacstap04.es.ad.adp.com:7202
   BalancerMember http://acacstap03.es.ad.adp.com:7202
 </Proxy>

 ######################
 # for forms
 ######################
 <Location /OA_MEDIA>
  ProxyPass balancer://formscluster_oamedia
  ProxyPassReverse balancer://formscluster_oamedia
 </Location>
 <Proxy balancer://formscluster_oamedia>
   BalancerMember http://acacstap03.es.ad.adp.com:7402/OA_HTML/media
   BalancerMember http://acacstap04.es.ad.adp.com:7402/OA_HTML/media
 </Proxy>


 <Location /OA_JAVA>
  ProxyPass balancer://formscluster_oajava
  ProxyPassReverse balancer://formscluster_oajava
 </Location>
 <Proxy balancer://formscluster_oajava>
   BalancerMember http://acacstap03.es.ad.adp.com:7402/OA_HTML/classes
   BalancerMember http://acacstap04.es.ad.adp.com:7402/OA_HTML/classes
 </Proxy>


</IfModule>

#
# Environment settings to be passed to CGI programs from external env
#
PassEnv NLS_LANG


#
# Set Limit of ENV Variables
#
# The LimitRequestFieldSize directive allows the server administrator to
# reduce or increase the limit on the allowed size of an HTTP request header
# field. A server needs this value to be large enough to hold any one header
# field from a normal client request

LimitRequestFieldSize 4096

# The LimitRequestFields directive allows the server administrator to
# modify the limit on the number of request header fields allowed in an HTTP
# request. A server needs this value to be larger than the number of fields
# that a normal client request might include. The number of request header
# fields used by a client rarely exceeds 20, but this may vary among different
# client implementations, often depending upon the extent to which a user has
# configured their browser to support detailed content negotiation

LimitRequestFields 32

# The LimitRequestBody directive allows the user to set a limit on
# the allowed size of an HTTP request message body within the context in which
# the directive is given (server, per-directory, per-file or per-location). If
# the client request exceeds that limit, the server will return an error
# response instead of servicing the request. The size of a normal request
# message body will vary greatly depending on the nature of the resource and
# the methods allowed on that resource

LimitRequestBody 0

# The LimitRequestLine directive allows the server administrator to
# reduce the limit on the allowed size of a client's HTTP request-line
# below the normal input buffer size compiled with the
# server. Since the request-line consists of the HTTP method, URI, and protocol
# version, the LimitRequestLine directive places a restriction on the length of
# a request-URI allowed for a request on the server. A server needs this value
# to be large enough to hold any of its resource names, including any
# information that might be passed in the query part of a GET request

LimitRequestLine 4096


#
# Caching static contents
#
<IfModule expires_module>
 ExpiresActive On

 <Location /OA_MEDIA>
   ExpiresByType image/gif  "access plus 1 month"
   ExpiresByType image/jpeg  "access plus 1 month"
   ExpiresByType text/txt "access plus 1 month"
 </Location>

 <Location /OA_HTML>
   ExpiresByType image/gif  "access plus 1 month"
   ExpiresByType image/jpg  "access plus 1 month"
#   ExpiresByType text/html "access plus 1 month"
   ExpiresByType text/css "access plus 1 month"
#   ExpiresByType text/js "access plus 1 month"
 </Location>

</IfModule>


#
# To facilitate reloading of the html on change
#

<Location ~ "/oam/nonUix/launchMode/restricted">
Header always add Cache-Control no-cache
Header always merge Cache-Control must-revalidate
Header always merge Cache-Control no-store
</Location>
================

I removed the entries for NODE 1 (ap03) from the apps.conf on NODE 2 file under /app/applmgr/IMR03S/fs2/FMW_Home/webtier/instances/EBS_web_OHS2/config/OHS/EBS_web and now I can launch the forms without delay.

I am not sure if this is the right thing to do but it works.

Let me know please this was the right thing to do.

acacstap04:IMR03S:applmgr /app/applmgr/IMR03S/fs2/FMW_Home/webtier/instances/EBS_web_OHS2/config/OHS/EBS_web $more mod_wl_ohs.conf

LoadModule weblogic_module   "${ORACLE_HOME}/ohs/modules/mod_wl_ohs.so"
<IfModule mod_weblogic.c>
        DynamicServerList OFF  <<<<<<<<<<<<<<<<<  
        <Location /OA_HTML>
                SetHandler weblogic-handler
                WebLogicCluster acacstap04.es.ad.adp.com:7202,acacstap03.es.ad.adp.com:7202
                WLTempDir ${ORACLE_INSTANCE}/tmp
                #SecureProxy ON
                #WLProxySSL ON
                #WLSSLWallet "${ORACLE_INSTANCE}/config/${COMPONENT_TYPE}/${COMPONENT_NAME}/keystores/default"
        </Location>
        <Location /forms>
                SetHandler weblogic-handler
                WLCookieName JsessionIDForms
                WebLogicCluster acacstap03.es.ad.adp.com:7402,acacstap04.es.ad.adp.com:7402
                WLTempDir ${ORACLE_INSTANCE}/tmp
                #SecureProxy ON
                #WLProxySSL ON
                #WLSSLWallet "${ORACLE_INSTANCE}/config/${COMPONENT_TYPE}/${COMPONENT_NAME}/keystores/default"
        </Location>


I could not find WLIOTimeoutSecs settings in any conf files and I do not know where to set this value in WLS 10.3.6

[ErmanArslansOracleBlog]

Well, apps.conf is actually read by Oracle HTTP Server.. So it is actually a configuration in HTTP Server..( we see it included in httpd.conf)
That directive you removed, actually make the requests load balance between nodes..
For instance; when you reach the web server (load balancer redirects you), the web server sees the request as a form request and then check that apps.conf-configuration to see the available nodes for processing that requests.. But! you alread go through the load balancer, so when you reach a web tier/server , it is okay for you to be processed in that node.. So you can leave that BalancerMember as removed. One disadvantage for this configuration is this question -> "If your HTTP server 1 is up , but oacore servers on Node 1 are down, what will happen??"

We have also an option to specify timeout there, but this is a pure Apache config, and as we have EBS there, we don't really need to get into it..
However; according to the documentation, we see that Node2's configuration is correct.. Both of node urls should be there as Load Balancer members..

Bytheway , you have different in configuration of mod_wl_ohs.conf files.. Node 1 has single node for the WebLogicCluster parameter, but Node 2 has both nodes set for that parameters..

So at the bottom line, I think you have a missing config on Node 1.. I don't know the story of this environment,  but an autoconfig may correct his config on Node1..

You actual issue is however; will be solved by setting a timeout in the line you give BalancerMember directive..

Example config: BalancerMember "<<a href="http://your_node:your_port">http://your_node:your_port>" connectiontimeout=5

So please try this and update me..

Noe that, Connect timeout in seconds. The number of seconds Apache httpd waits for the creation of a connection to the backend to complete. By adding a postfix of ms, the timeout can be also set in milliseconds.

+

this is an undocumented solution offering.. It is not a supported move, but this is worth a try..

[burneyj]

Great Stuff !!! 

I really appreciate you helping me on this. You certainly know your stuff.

That connectiontimeout works. I put back everything and just added the connectiontimeout=5. Removed NODE 1 from the network
and I can still connect to NODE 2 within reasonable time.

acacstap04:IMR03S:applmgr /app/applmgr/IMR03S/fs1/FMW_Home/webtier/instances/EBS_web_OHS2/config/OHS/EBS_web $more apps.conf
#
###################################################################################


LoadModule proxy_balancer_module   "${ORACLE_HOME}/ohs/modules/mod_proxy_balancer.so"
ProxyRequests off

<IfModule proxy_balancer_module>

 ProxyRequests Off

 ######################
 # for oacore
 ######################
 <Location /OA_MEDIA>
   ProxyPass balancer://oacorecluster_oamedia
   ProxyPassReverse balancer://oacorecluster_oamedia
 </Location>
 <Proxy balancer://oacorecluster_oamedia>
   BalancerMember http://acacstap04.es.ad.adp.com:7201/OA_HTML/media connectiontimeout=5
   BalancerMember http://acacstap03.es.ad.adp.com:7201/OA_HTML/media connectiontimeout=5
 </Proxy>


 <Location /OA_JAVA>
   ProxyPass balancer://oacorecluster_oajava
   ProxyPassReverse balancer://oacorecluster_oajava
 </Location>
 <Proxy balancer://oacorecluster_oajava>
   BalancerMember http://acacstap04.es.ad.adp.com:7201/OA_HTML/classes connectiontimeout=5
   BalancerMember http://acacstap03.es.ad.adp.com:7201/OA_HTML/classes connectiontimeout=5
 </Proxy>


 <Location /OA_CGI/FNDWRR.exe>
  ProxyPass balancer://oacorecluster_oacgi/OA_HTML/txkFNDWRR.pl
  ProxyPassReverse balancer://oacorecluster_oacgi/OA_HTML/txkFNDWRR.pl
 </Location>
 <Proxy balancer://oacorecluster_oacgi/OA_HTML/txkFNDWRR.pl>
   BalancerMember http://acacstap04.es.ad.adp.com:7201 connectiontimeout=5
   BalancerMember http://acacstap03.es.ad.adp.com:7201 connectiontimeout=5
 </Proxy>

 ######################
 # for forms
 ######################
 <Location /OA_MEDIA>
  ProxyPass balancer://formscluster_oamedia
  ProxyPassReverse balancer://formscluster_oamedia
 </Location>
 <Proxy balancer://formscluster_oamedia>
   BalancerMember http://acacstap03.es.ad.adp.com:7401/OA_HTML/media connectiontimeout=5
   BalancerMember http://acacstap04.es.ad.adp.com:7401/OA_HTML/media connectiontimeout=5
 </Proxy>


 <Location /OA_JAVA>
  ProxyPass balancer://formscluster_oajava
  ProxyPassReverse balancer://formscluster_oajava
 </Location>
 <Proxy balancer://formscluster_oajava>
   BalancerMember http://acacstap03.es.ad.adp.com:7401/OA_HTML/classes connectiontimeout=5
   BalancerMember http://acacstap04.es.ad.adp.com:7401/OA_HTML/classes connectiontimeout=5
 </Proxy>

================

Bytheway , you have different in configuration of mod_wl_ohs.conf files.. Node 1 has single node for the WebLogicCluster parameter, but Node 2 has both nodes set for that parameters..

So at the bottom line, I think you have a missing config on Node 1.. I don't know the story of this environment,  but an autoconfig may correct his config on Node1..


ANSWER:

I am surprised myself as to why NODE 1 only has NODE 1 entries and NOT NODE 2 entries. All this setup was done by adclone utility
I did not do anything manually.
I will run autoconfig on BOTH NODES and see NODE 2 entries are inserted into NODE 1 mod_wl_ohs.conf file.

[ErmanArslansOracleBlog]

Good :) + thanks for the feedback Burneyj..