RAC failover
Locked
 
|
Hi Erman,
result of telnet and application error. First node is throwing (10.210.228.6) connection refused. Error: 2017-05-07 14:20:26,311 [10.210.227.13-RH-CBS-WSO2-01.mtg.local] [HttpClientWorker-7] INFO LogMediator To: http://www.w3.org/2005/08/addressing/anonymous,WSAction: ,SOAPAction: ,MessageID: urn:uuid:5B696DD59A8E1F2ABD9415563493003498-793749240,Direction: response,Envelope: <?xml version='1.0' encoding='utf-8'?><soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"><soap:Body><ns1:processRequestResponse xmlns:ns1="http://schema.concierge.com/Masters"><Masters xmlns="http://schema.concierge.com/Masters"><Response><ResultStatusResponse><resultCode>-1</resultCode><resultMessage>Could not create connection; - nested throwable: (java.sql.SQLException: Io exception: The Network Adapter could not establish the connection); - nested throwable: (org.jboss.resource.JBossResourceException: Could not create connection; - nested throwable: (java.sql.SQLException: Io exception: The Network Adapter could not establish the connection))</resultMessage></ResultStatusResponse></Response></Masters></ns1:processRequestResponse></soap:Body></soap:Envelope> Telenet Log: [cbsmiddleware@RH-CBS-WSO2-01 bin]$ telnet cbs-scan.telecom.mu 1521 Trying 10.210.228.6... telnet: connect to address 10.210.228.6: Connection refused Trying 10.210.228.7... Connected to cbs-scan.telecom.mu. Escape character is '^]'. ^C Connection closed by foreign host. [cbsmiddleware@RH-CBS-WSO2-01 bin]$ [cbsmiddleware@RH-CBS-WSO2-01 bin]$ telnet 10.210.228.6 1521 Trying 10.210.228.6... telnet: connect to address 10.210.228.6: Connection refused [cbsmiddleware@RH-CBS-WSO2-01 bin]$ telnet 10.210.228.7 1521 Trying 10.210.228.7... Connected to 10.210.228.7. Escape character is '^]'. ^C ^CConnection closed by foreign host. Do I need to check with network team? Regards, Roshan |
Administrator
|
First, check the rac nodes.. Ensure one scan listener is running on that ip. ensure it is listening the port for that ip. 7 May 2017 13:33 tarihinde "Roshan [via Erman Arslan's Oracle Forum]" <[hidden email]> yazdı: Hi Erman, |
|
|
Thanks Erman. Listener on node 1 was down. I have checked and started it
listenerscan1.png But still we are unable to telnet node1(10.***.6) 1521 |
|
Administrator
|
is it listening for the port? Use lsnrctl and netstat commands to ensure this. 7 May 2017 16:43 tarihinde "Roshan [via Erman Arslan's Oracle Forum]" <[hidden email]> yazdı: Thanks Erman. Listener on node 1 was down. I have checked and started it |
|
|
In reply to this post by Roshan
I have checked firewall on node1. It was enabled.
|
|
Administrator
|
so?
|
|
|
In reply to this post by ErmanArslansOracleBlog
Hi Erman,
I have disabled firewall, still we cannot connect. netstat shows it is not listening to port 1521 netstat.PNG |
|
|
The listener on node 1 with IP 10.210.228.6 is not registered with the listener
listener.PNG |
|
Administrator
|
In reply to this post by Roshan
Then you have a listener configuration problem.. Send me the listener status command output ( specifically for that scan listener) Also send me ifconfig( from both of the nodes) and "srvctl status scan_listener" outputs.. 7 May 2017 18:11 tarihinde "Roshan [via Erman Arslan's Oracle Forum]" <[hidden email]> yazdı: Hi Erman, |
|
Administrator
|
10.210.228.6 is one of the scan ips right?
So, your scan listener associated with this ip is probably not running. That's why I m requesting the following from you. 1)srvctl status scan 2)srvctl status scan_listener 3)ifconfig (from both of the nodes) 4)lsnrctl status LISTENER_SCAN* (for ex: LISTENER_SCAN1, LISTENER_SCAN2 etc..) --both of the nodes, whereever the related scan listener is running. 5)ps -ef|grep inh (from both of the nodes) |
|
Administrator
|
Also
6)netstat -anp|grep your_scan_llistener_port (from both of the nodeS) |
|
|
Grid12c>srvctl status scan
SCAN VIP scan1 is enabled SCAN VIP scan1 is running on node rh-cbs-db01 SCAN VIP scan2 is enabled SCAN VIP scan2 is running on node rh-cbs-db02 SCAN VIP scan3 is enabled SCAN VIP scan3 is running on node rh-cbs-db02 Grid12c>srvctl status scan_listener SCAN Listener LISTENER_SCAN1 is enabled SCAN listener LISTENER_SCAN1 is running on node rh-cbs-db01 SCAN Listener LISTENER_SCAN2 is enabled SCAN listener LISTENER_SCAN2 is running on node rh-cbs-db02 SCAN Listener LISTENER_SCAN3 is enabled SCAN listener LISTENER_SCAN3 is running on node rh-cbs-db02 node1 Grid12c>ifconfig bond0 Link encap:Ethernet HWaddr 00:17:A4:77:00:8C inet addr:10.210.228.6 Bcast:10.210.228.15 Mask:255.255.255.240 UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1 RX packets:1085053 errors:0 dropped:0 overruns:0 frame:0 TX packets:922628 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:224854065 (214.4 MiB) TX bytes:223073135 (212.7 MiB) bond0:1 Link encap:Ethernet HWaddr 00:17:A4:77:00:8C inet addr:10.210.228.9 Bcast:10.210.228.15 Mask:255.255.255.240 UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1 bond0:2 Link encap:Ethernet HWaddr 00:17:A4:77:00:8C inet addr:10.210.228.12 Bcast:10.210.228.15 Mask:255.255.255.240 UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1 bond1 Link encap:Ethernet HWaddr 00:17:A4:77:00:AC inet addr:192.168.124.80 Bcast:192.168.124.255 Mask:255.255.255.0 UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1 RX packets:8103551 errors:0 dropped:0 overruns:0 frame:0 TX packets:6927258 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:5908491151 (5.5 GiB) TX bytes:4939927995 (4.6 GiB) bond1:1 Link encap:Ethernet HWaddr 00:17:A4:77:00:AC inet addr:169.254.172.135 Bcast:169.254.255.255 Mask:255.255.0.0 UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1 eth0 Link encap:Ethernet HWaddr 00:17:A4:77:00:8C UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1 RX packets:999735 errors:0 dropped:0 overruns:0 frame:0 TX packets:461525 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:179361467 (171.0 MiB) TX bytes:111831023 (106.6 MiB) Interrupt:32 Memory:fa000000-fa7fffff eth1 Link encap:Ethernet HWaddr 00:17:A4:77:00:8C UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1 RX packets:85318 errors:0 dropped:0 overruns:0 frame:0 TX packets:461103 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:45492598 (43.3 MiB) TX bytes:111242112 (106.0 MiB) Interrupt:36 Memory:f8800000-f8ffffff eth2 Link encap:Ethernet HWaddr 00:17:A4:77:00:AC UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1 RX packets:4051795 errors:0 dropped:0 overruns:0 frame:0 TX packets:3464221 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:2954348962 (2.7 GiB) TX bytes:2471315144 (2.3 GiB) Interrupt:36 Memory:f7000000-f77fffff eth3 Link encap:Ethernet HWaddr 00:17:A4:77:00:AC UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1 RX packets:4051756 errors:0 dropped:0 overruns:0 frame:0 TX packets:3463037 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:2954142189 (2.7 GiB) TX bytes:2468612851 (2.2 GiB) Interrupt:37 Memory:f5800000-f5ffffff lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 UP LOOPBACK RUNNING MTU:65536 Metric:1 RX packets:1831875 errors:0 dropped:0 overruns:0 frame:0 TX packets:1831875 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:556654070 (530.8 MiB) TX bytes:556654070 (530.8 MiB) node2 Grid12c>ifconfig bond0 Link encap:Ethernet HWaddr 00:17:A4:77:00:84 inet addr:10.210.228.7 Bcast:10.210.228.15 Mask:255.255.255.240 UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1 RX packets:88083628 errors:0 dropped:0 overruns:0 frame:0 TX packets:78967799 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:19752876704 (18.3 GiB) TX bytes:47950716655 (44.6 GiB) bond0:1 Link encap:Ethernet HWaddr 00:17:A4:77:00:84 inet addr:10.210.228.10 Bcast:10.210.228.15 Mask:255.255.255.240 UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1 bond0:3 Link encap:Ethernet HWaddr 00:17:A4:77:00:84 inet addr:10.210.228.13 Bcast:10.210.228.15 Mask:255.255.255.240 UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1 bond0:4 Link encap:Ethernet HWaddr 00:17:A4:77:00:84 inet addr:10.210.228.11 Bcast:10.210.228.15 Mask:255.255.255.240 UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1 bond1 Link encap:Ethernet HWaddr 00:17:A4:77:00:88 inet addr:192.168.124.81 Bcast:192.168.124.255 Mask:255.255.255.0 UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1 RX packets:262025625 errors:0 dropped:0 overruns:0 frame:0 TX packets:240071206 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:121538599349 (113.1 GiB) TX bytes:99705538621 (92.8 GiB) bond1:1 Link encap:Ethernet HWaddr 00:17:A4:77:00:88 inet addr:169.254.122.141 Bcast:169.254.255.255 Mask:255.255.0.0 UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1 eth0 Link encap:Ethernet HWaddr 00:17:A4:77:00:84 UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1 RX packets:87075985 errors:0 dropped:0 overruns:0 frame:0 TX packets:39625244 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:19451263259 (18.1 GiB) TX bytes:24209816512 (22.5 GiB) Interrupt:32 Memory:fa000000-fa7fffff eth1 Link encap:Ethernet HWaddr 00:17:A4:77:00:84 UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1 RX packets:1007643 errors:0 dropped:0 overruns:0 frame:0 TX packets:39342555 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:301613445 (287.6 MiB) TX bytes:23740900143 (22.1 GiB) Interrupt:36 Memory:f8800000-f8ffffff eth2 Link encap:Ethernet HWaddr 00:17:A4:77:00:88 UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1 RX packets:131025530 errors:0 dropped:0 overruns:0 frame:0 TX packets:120035654 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:60785548804 (56.6 GiB) TX bytes:49851147383 (46.4 GiB) Interrupt:36 Memory:f7000000-f77fffff eth3 Link encap:Ethernet HWaddr 00:17:A4:77:00:88 UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1 RX packets:131000213 errors:0 dropped:0 overruns:0 frame:0 TX packets:120035635 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:60753165741 (56.5 GiB) TX bytes:49854468277 (46.4 GiB) Interrupt:37 Memory:f5800000-f5ffffff lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 UP LOOPBACK RUNNING MTU:65536 Metric:1 RX packets:12249888 errors:0 dropped:0 overruns:0 frame:0 TX packets:12249888 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:60218524919 (56.0 GiB) TX bytes:60218524919 (56.0 GiB) lsnrctl status LISTENER_SCAN* Grid12c>lsnrctl status LISTENER_SCAN1 LSNRCTL for Linux: Version 12.1.0.2.0 - Production on 08-MAY-2017 11:11:30 Copyright (c) 1991, 2014, Oracle. All rights reserved. Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN1))) STATUS of the LISTENER ------------------------ Alias LISTENER_SCAN1 Version TNSLSNR for Linux: Version 12.1.0.2.0 - Production Start Date 07-MAY-2017 23:35:41 Uptime 0 days 11 hr. 35 min. 48 sec Trace Level off Security ON: Local OS Authentication SNMP OFF Listener Parameter File /u01/grid12c/12.1.2/network/admin/listener.ora Listener Log File /u01/oracle/diag/tnslsnr/RH-CBS-DB01/listener_scan1/alert/log.xml Listening Endpoints Summary... (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER_SCAN1))) (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=10.210.228.12)(PORT=1521))) Services Summary... Service "cbs" has 2 instance(s). Instance "cbs1", status READY, has 1 handler(s) for this service... Instance "cbs2", status READY, has 1 handler(s) for this service... Service "cbsXDB" has 2 instance(s). Instance "cbs1", status READY, has 1 handler(s) for this service... Instance "cbs2", status READY, has 1 handler(s) for this service... The command completed successfully Grid12c> Grid12c>lsnrctl status LISTENER_SCAN2 LSNRCTL for Linux: Version 12.1.0.2.0 - Production on 08-MAY-2017 11:12:29 Copyright (c) 1991, 2014, Oracle. All rights reserved. Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN2))) TNS-12541: TNS:no listener TNS-12560: TNS:protocol adapter error TNS-00511: No listener Linux Error: 2: No such file or directory Grid12c>lsnrctl status LISTENER_SCAN3 LSNRCTL for Linux: Version 12.1.0.2.0 - Production on 08-MAY-2017 11:12:58 Copyright (c) 1991, 2014, Oracle. All rights reserved. Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN3))) TNS-12541: TNS:no listener TNS-12560: TNS:protocol adapter error TNS-00511: No listener Linux Error: 2: No such file or directory Grid12c> node2 Grid12c>srvctl status scan_listener SCAN Listener LISTENER_SCAN1 is enabled SCAN listener LISTENER_SCAN1 is running on node rh-cbs-db01 SCAN Listener LISTENER_SCAN2 is enabled SCAN listener LISTENER_SCAN2 is running on node rh-cbs-db02 SCAN Listener LISTENER_SCAN3 is enabled SCAN listener LISTENER_SCAN3 is running on node rh-cbs-db02 node2 Grid12c>lsnrctl status LISTENER_SCAN1 LSNRCTL for Linux: Version 12.1.0.2.0 - Production on 08-MAY-2017 11:14:35 Copyright (c) 1991, 2014, Oracle. All rights reserved. Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN1))) TNS-12541: TNS:no listener TNS-12560: TNS:protocol adapter error TNS-00511: No listener Linux Error: 2: No such file or directory Grid12c>lsnrctl status LISTENER_SCAN2 LSNRCTL for Linux: Version 12.1.0.2.0 - Production on 08-MAY-2017 11:18:02 Copyright (c) 1991, 2014, Oracle. All rights reserved. Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN2))) STATUS of the LISTENER ------------------------ Alias LISTENER_SCAN2 Version TNSLSNR for Linux: Version 12.1.0.2.0 - Production Start Date 07-MAY-2017 23:33:34 Uptime 0 days 11 hr. 44 min. 28 sec Trace Level off Security ON: Local OS Authentication SNMP OFF Listener Parameter File /u01/grid12c/12.1.2/network/admin/listener.ora Listener Log File /u01/oracle/diag/tnslsnr/RH-CBS-DB02/listener_scan2/alert/log.xml Listening Endpoints Summary... (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER_SCAN2))) (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=10.210.228.13)(PORT=1521))) Services Summary... Service "cbs" has 2 instance(s). Instance "cbs1", status READY, has 1 handler(s) for this service... Instance "cbs2", status READY, has 1 handler(s) for this service... Service "cbsXDB" has 2 instance(s). Instance "cbs1", status READY, has 1 handler(s) for this service... Instance "cbs2", status READY, has 1 handler(s) for this service... The command completed successfully Grid12c> Grid12c>lsnrctl status LISTENER_SCAN3 LSNRCTL for Linux: Version 12.1.0.2.0 - Production on 08-MAY-2017 11:18:15 Copyright (c) 1991, 2014, Oracle. All rights reserved. Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN3))) STATUS of the LISTENER ------------------------ Alias LISTENER_SCAN3 Version TNSLSNR for Linux: Version 12.1.0.2.0 - Production Start Date 07-MAY-2017 23:33:35 Uptime 0 days 11 hr. 44 min. 40 sec Trace Level off Security ON: Local OS Authentication SNMP OFF Listener Parameter File /u01/grid12c/12.1.2/network/admin/listener.ora Listener Log File /u01/oracle/diag/tnslsnr/RH-CBS-DB02/listener_scan3/alert/log.xml Listening Endpoints Summary... (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER_SCAN3))) (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=10.210.228.11)(PORT=1521))) Services Summary... Service "cbs" has 2 instance(s). Instance "cbs1", status READY, has 1 handler(s) for this service... Instance "cbs2", status READY, has 1 handler(s) for this service... Service "cbsXDB" has 2 instance(s). Instance "cbs1", status READY, has 1 handler(s) for this service... Instance "cbs2", status READY, has 1 handler(s) for this service... The command completed successfully Grid12c> node1 Grid12c>ps -ef|grep inh grid12c 23048 1 0 May07 ? 00:00:02 /u01/grid12c/12.1.2/bin/tnslsnr LISTENER -no_crs_notify -inherit grid12c 23066 1 0 May07 ? 00:00:01 /u01/grid12c/12.1.2/bin/tnslsnr LISTENER_SCAN1 -no_crs_notify -inherit grid12c 55918 55102 0 11:20 pts/1 00:00:00 grep inh node2 Grid12c>ps -ef|grep inh grid12c 24863 21284 0 11:21 pts/0 00:00:00 grep inh grid12c 56353 1 0 May07 ? 00:00:02 /u01/grid12c/12.1.2/bin/tnslsnr LISTENER -no_crs_notify -inherit grid12c 57085 1 0 May07 ? 00:00:01 /u01/grid12c/12.1.2/bin/tnslsnr LISTENER_SCAN2 -no_crs_notify -inherit grid12c 57160 1 0 May07 ? 00:00:01 /u01/grid12c/12.1.2/bin/tnslsnr LISTENER_SCAN3 -no_crs_notify -inherit |
|
Administrator
|
Your scan listeners are listening on 10.210.228.11, 10.210.228.12, 10.210.228.13
So this issue is not related with scan. Your first node's bond0 interface is on 10.210.228.6. So this is due to normal listener. (not scan) Your previous update: [cbsmiddleware@RH-CBS-WSO2-01 bin]$ telnet cbs-scan.telecom.mu 1521 Trying 10.210.228.6... If so, why "cbs-scan.telecom.mu" is trying to reach 10.210.228.6? Is your DNS or hosts configuration wrong? Send me the "lsnrctl status" command output from both of the nodes (execute it using grid user) -- I want to see the local listeners .. |
«
Return to RAC
|
1 view|%1 views
| Free forum by Nabble | Edit this page |