RAC restore on single node

[Roshan]

Hi Erman,

I am trying to restore a RAC(ASM) backup 12c on a test instance(single node). When I do startup nomount, I am getting
SQL> startup nomount
ORA-27504: IPC error creating OSD context
ORA-27300: OS system dependent operation:if_not_found failed with status: 0
ORA-27301: OS failure message: Error 0
ORA-27302: failure occurred at: skgxpvaddr9
ORA-27303: additional information: requested interface 10.28.1.58 not found. Check output from ifconfig command
SQL>

I performed the steps in doc
http://ermanarslan.blogspot.com/2016/01/ora-27303-and-ora-27302-on-instance.html

Still same issue.
Please advise.

Thanks,
Roshan

[ErmanArslansOracleBlog]

Send me your db parameters and their values. (the init.ora that you use for starting this single instance)
I think you didn't convert RAC to single. (you still have some RAC related parameters there..)
Check cluster_interconnects parameter, unset it if it is set.

[Roshan]

initcbs2.ora

Bash profile

export ORACLE_HOME=/archicom/ora12c/product/12.1.0
export ORACLE_SID=cbs2
export PATH=$ORACLE_HOME/bin:$PATH
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$ORACLE_HOME/lib:$ORACLE_HOME
# User specific environment and startup programs

#PATH=$PATH:$HOME/.local/bin:$HOME/bin

export PATH

Thanks,

[Roshan]

Extract from alert log when I issue startup nomount

[ora12c@rh-archicom-01 trace]$ tail -f alert_cbs2.log
  undo_retention           = 9000
  remote_login_passwordfile= "EXCLUSIVE"
  dispatchers              = "(PROTOCOL=TCP) (SERVICE=cbsXDB)"
  audit_file_dest          = "/archicom/ora12c/admin/cbs/adump"
  audit_trail              = "DB"
  db_name                  = "cbs"
  open_cursors             = 5000
  pga_aggregate_target     = 0
  diagnostic_dest          = "/archicom/ora12c"
NOTE: remote asm mode is local (mode 0x1; from cluster type)
Wed May 31 16:08:16 2017
WARNING: failed to retrieve DB spfile location (unable to communicate with CRSD/OHASD)
Starting ORACLE instance (normal) (OS id: 11324)
Wed May 31 16:08:16 2017
CLI notifier numLatches:29 maxDescs:3414
Wed May 31 16:08:16 2017
**********************************************************************
Wed May 31 16:08:16 2017
Dump of system resources acquired for SHARED GLOBAL AREA (SGA)

Wed May 31 16:08:16 2017
 Per process system memlock (soft) limit = 64K
Wed May 31 16:08:16 2017
 Expected per process system memlock (soft) limit to lock
 SHARED GLOBAL AREA (SGA) into memory: 13G
Wed May 31 16:08:16 2017
 Available system pagesizes:
  4K, 2048K
Wed May 31 16:08:16 2017
 Supported system pagesize(s):
Wed May 31 16:08:16 2017
  PAGESIZE  AVAILABLE_PAGES  EXPECTED_PAGES  ALLOCATED_PAGES  ERROR(s)
Wed May 31 16:08:16 2017
        4K       Configured         3276808         3276808        NONE
Wed May 31 16:08:16 2017
 Reason for not supporting certain system pagesizes:
Wed May 31 16:08:16 2017
  2048K - Dynamic allocate and free memory regions
Wed May 31 16:08:16 2017
RECOMMENDATION:
Wed May 31 16:08:16 2017
 1. Increase per process memlock (soft) limit to at least 13GB
 to lock 100% of SHARED GLOBAL AREA (SGA) pages into physical memory
Wed May 31 16:08:16 2017
**********************************************************************
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Initial number of CPU is 16
Number of processor cores in the system is 16
Number of processor sockets in the system is 2
2017-05-31 16:08:16.776
[unknown process(11324)]CRS-2317:Cannot get local GPnP security keys (wallet).
2017-05-31 16:08:16.776
[unknown process(11324)]CRS-2316:Cannot initialize GPnP, CLSGPNP_ERR (Generic GPnP error).
kggpnpInit: failed to init gpnp
  WARNING: No cluster interconnect has been specified. Depending on
           the communication driver configured Oracle cluster traffic
           may be directed to the public interface of this machine.
           Oracle recommends that RAC clustered databases be configured
           with a private interconnect for enhanced security and
           performance.
Picked latch-free SCN scheme 3
Using LOG_ARCHIVE_DEST_1 parameter default value as USE_DB_RECOVERY_FILE_DEST
Autotune of undo retention is turned on.
IMODE=BR
ILAT =341
LICENSE_MAX_USERS = 0
SYS auditing is enabled
NOTE: remote asm mode is local (mode 0x1; from cluster type)
NOTE: Using default ASM root directory ASM
NUMA system with 2 nodes detected
Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP, Advanced Analytics
and Real Application Testing options.
ORACLE_HOME = /archicom/ora12c/product/12.1.0
System name:    Linux
Node name:      rh-archicom-01.mtg.local
Release:        3.10.0-327.el7.x86_64
Version:        #1 SMP Thu Oct 29 17:29:29 EDT 2015
Machine:        x86_64
Using parameter settings in server-side pfile /archicom/ora12c/product/12.1.0/dbs/initcbs2.ora
System parameters with non-default values:
  processes                = 2048
  sessions                 = 3104
  event                    = "28401 TRACE NAME CONTEXT FOREVER, LEVEL 1"
  sga_max_size             = 12800M
  sga_target               = 0
  memory_target            = 12800M
  memory_max_target        = 12800M
  control_files            = "/archicom/ora12c/oradata/control01.ctl"
  control_files            = "/home/ora12c/control02.ctl"
  control_file_record_keep_time= 30
  db_block_size            = 16384
  compatible               = "12.1.0.2.0"
  log_archive_format       = "%t_%s_%r.arc"
  db_files                 = 1024
  cluster_database         = FALSE
  db_recovery_file_dest    = "/archicom/ora12c/oradata"
  db_recovery_file_dest_size= 200G
  undo_tablespace          = "UNDOTBS1"
  undo_retention           = 9000
  remote_login_passwordfile= "EXCLUSIVE"
  dispatchers              = "(PROTOCOL=TCP) (SERVICE=cbsXDB)"
  audit_file_dest          = "/archicom/ora12c/admin/cbs/adump"
  audit_trail              = "DB"
  db_name                  = "cbs"
  open_cursors             = 5000
  pga_aggregate_target     = 0
  diagnostic_dest          = "/archicom/ora12c"
NOTE: remote asm mode is local (mode 0x1; from cluster type)

[ErmanArslansOracleBlog]

Why your instance name is cbs2?

This is single instance right?
->
make it cbs (your db name)

Then,

relink the related oracle off (oracle that belongs only to the single instance database) with the following;

make -f ins_rdbms.mk rac_off
make -f ins_rdbms.mk ioracle

Then,

Remove all the parameters from init.ora which points to second instance and it is on other node.

Remove cluster_database parameter

Remove undo_tablespace parameter which points to second instance

Start-up the database in mount stage and execute..

alter database disable thread <thread of other instance> 

open the database;

[Roshan]

Thanks a lot :)
It is ok now after following the steps.

[ErmanArslansOracleBlog]

Good.

[I.Pastor]

Hello,
I have this error starting ASM instance.
We have 2 node Oracle RAC  GI 12.2.0.1 and we want to clone 1 node to single instance.
Are these steps (relink libraries from GI HOME) necessary to startup ASM instance?
Thanks!

[ErmanArslansOracleBlog]

Which error?

Tell me exactly what did you do for this cloning operation?

If you are going to clone from RAC to single. What I recommend you is to do the following;

Install grid on your single node
use rman to restore/duplicate/clone your RAC database into that single node.

Note that,  a fresh install of the Grid Install is recommended and is always less problematic  than cloning..

If you still want to clone your GRID/ASM -- please follow this note ->

How to Clone an 11.2 Grid Infrastructure Home and Clusterware (Doc ID 1413846.1)

If you do it appropriately, your Grid will be relinked automatically when you run clone.pl

However, as I said earlier; a fresh install of the Grid Install is recommended and is always less problematic  than cloning..

[I.pastor]

Hello,
We are cloning node disks (Operating System + ASM diskgroups) using Storage feature (FlashCopy).

So we have running 2 node Oracle RAC in Primary Site and 1 node (cloned by Storage) Oracle Rac in Secondary Site.

When we try to startup cloned node CRS doesn't start and ASM raise the following error:
Trying to startup the ASM (as sysasm) instance manually we get error:
SQL>startup
ORA-27504: IPC error creating OSD

I think the Oracle software must be relinked after Storage clone. What do you think?

Thanks in advance!

[ErmanArslansOracleBlog]

You have a interconnect problem..
I think your spfile is still configured according to your source RAC instance..

cluster_interconnects parameter of ASM instance causes this issue.

This is a single instance right?

İf so, try the following;

connect to ASM instance and issue ->

SQL> alter system reset cluster_interconnects SCOPE=SPFILE SID='*';

*restart asm.

[I.pastor]

Hello,

I've tried startup ASM instance using this pfile but the error is the same.

initASM.ora
*.asm_diskgroups='DG_GIMR','DATOS_MAPR12','FRA_MAPR12','REDO_MAPR12','DATOS_FR041','FRA_FR041','REDO_FR041','DATOS_FR941','FRA_FR941','REDO_FR941'#Manual Mount
*.asm_diskstring='/dev/mapper/ASM_*'
*.asm_power_limit=1
*.large_pool_size=12M
*.memory_max_target=0
*.memory_target=0
*.pga_aggregate_target=1073741824
*.remote_login_passwordfile='EXCLUSIVE'
*.sga_max_size=3221225472
*.sga_target=2147483648

And cluster_interconnects  parameter is empty in Primary Site.

Kind Regards!

[ErmanArslansOracleBlog]

show me..

send me the screenshot (I want to see your sqlplus command as well as the error in one screenshot)

[I.pastor]

Hello

I send you the screenshot.

Kind Regards!

[ErmanArslansOracleBlog]

Please upload the screenshot here.

[I.pastor]

Image

[ErmanArslansOracleBlog]

Send me the following:

1)the output of "ifconfig" command
2)the ASM alert log