rapidwiz failed during installation 12.2 on 2 node rac environment.

[raiq1]

Dear sir,

AS per your advice, create the new thread for solving the issue.

I have successfully installed a Grid infrastructure, now planning to install EBS12.2 on 2 node rac environement.

EBSRAC1
EBSRAC2

once run the rapidwizard , its start the database installaton , but it has failed with below message.

 Phase 3 Rename Database



Executing command: sh -c "ORACLE_SID=ORCLPROD;export ORACLE_SID;ORACLE_HOME=/u01/oracle/PROD/12.1.0;export ORACLE_HOME; /u01/oracle/PROD/12.1.0/bin/rman target / nocatalog @/u01/oracle/PROD/12.1.0/appsutil/out/PROD_ebsrac1/restore-rac3.rman > /u01/oracle/PROD/12.1.0/appsutil/out/PROD_ebsrac1/restore-rac3.log"

Unable to rename database

    RW-50010: Error: - script has returned an error:   1
RW-50004: Error code received when running external process.  Check log file for details.
Running Database Install Driver for PROD instance


Need to solve the above issue, and also need your advice to start the fresh installation .

before fresh installation , what directories i have to clean to start the new installation.

also in ASM, how to remove the data there , because it has occupied the space because if we start

the new installation again, the previous installation data are there in the asm DISK1, so need to clean it.

Please guide us to solve the above error.

ASMCMD> lsdg
State    Type    Rebal  Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name
MOUNTED  EXTERN  N         512   4096  4194304    127992    39700                0           39700              0             Y  DATA/
ASMCMD>

log file are attached.

05061616.logrestore-rac3.log

[ErmanArslansOracleBlog]

You got a ORA-01110, a corruption on system01.dbf

ORA-01578 in the stack is indicating that : The corruption was a software block corruption which was a former LOGICAL corrupt block marked as formally corrupt.

So you are using rapidwiz to install a fresh EBS 12.2 and getting this error, huh?
Interesting..
If it was a EBS 12.1 cloning operation or something like that , I would recommend resizing system01.dbf on the source as it is a must in EBS 12.1 installations -- for EBS 12.1 system01.dbf comes with a corruption, but your case is on EBS 12.2...

During the installation, rapidwiz uses rman to restore a precreated database, but in your case there is a corruption..
Are those installation files stable?  
So what I currently think is, you have a installation file problem.. Maybe some of the files are corrupted during the download.

But , what happens when you install this EBS database using rapidwiz on a cooked filesystem --like ext3,ext4 or others? (rather than ASM) Did you try it? Just to be sure that this corruption is not caused by Grid ..

What do you see in the alert log of this newly created database?
What do you see in the GRID -ASM 's alert log?
Are you sure that you have healthy EBS installation files? (please check them, and do the test that I mentioned above..)

[raiq1]

Dear Erman sir,

I have tried again to run rapidwizard and again it failed with below errors..

channel ORA_DISK_2: restored backup piece 1
channel ORA_DISK_2: restore complete, elapsed time: 00:00:55
channel ORA_DISK_2: starting datafile backup set restore
channel ORA_DISK_2: specifying datafile(s) to restore from backup set
channel ORA_DISK_2: restoring datafile 00003 to +DATA/dbfiles/system03.dbf
channel ORA_DISK_2: restoring datafile 00007 to +DATA/dbfiles/owad01.dbf
channel ORA_DISK_2: restoring datafile 00009 to +DATA/dbfiles/odm.dbf
channel ORA_DISK_2: restoring datafile 00011 to +DATA/dbfiles/sysaux01.dbf
channel ORA_DISK_2: restoring datafile 00020 to +DATA/dbfiles/system14.dbf
channel ORA_DISK_2: restoring datafile 00025 to +DATA/dbfiles/system19.dbf
channel ORA_DISK_2: restoring datafile 00030 to +DATA/dbfiles/system24.dbf
channel ORA_DISK_2: restoring datafile 00351 to +DATA/dbfiles/system07.dbf
channel ORA_DISK_2: restoring datafile 00379 to +DATA/dbfiles/undo01.dbf
channel ORA_DISK_2: restoring datafile 00393 to +DATA/dbfiles/a_txn_ind01.dbf
channel ORA_DISK_2: restoring datafile 00394 to +DATA/dbfiles/a_ref01.dbf
channel ORA_DISK_2: restoring datafile 00401 to +DATA/dbfiles/a_txn_data02.dbf
channel ORA_DISK_2: reading from backup piece /u01/oracle/software/ebs122/EBSInstallMedia/AppDB/PROD/Disk6/data/stage/0dqq4vh5_1_1
channel ORA_DISK_3: ORA-19870: error while restoring backup piece /u01/oracle/software/ebs122/EBSInstallMedia/AppDB/PROD/Disk2/data/stage/0fqq4vh5_1_1
ORA-19502: write error on file "+DATA/dbfiles/system10.dbf", block number 16448 (block size=8192)
ORA-15081: failed to submit an I/O operation to a disk
ORA-15081: failed to submit an I/O operation to a disk
ORA-15186: ASMLIB error function = [kfk_asm_ioerror],  error = [0],  mesg = [I/O Error
c. dbms_backup_restore.restoreCancel() failed
[root@ebsrac1 PROD_ebsrac1]#



Make sure i am trying to install Fresh EBS 12.2 database with 2 node rac environement, Grid are installed successfully but the issue is on EBS 12.2 rapidwizard.

REgards,

[ErmanArslansOracleBlog]

Hmm this time a different block number.. different than we see in your last update.. (the rman  log)
So this seems an ASM problem. asmlib problem actually.

Check the following MOS note and ensure you have the correct config.

Repeated Error: ORA-15186: ASMLIB Error Function = [kfk_asm_ioerror], Error = [0], Mesg = [I/O Error] (Doc ID 2066399.1)

[raiq1]

Dear Erman sir,

As per your advice, i have download again latest copy of ebs12.2 from edelevery oracle and try again to run the rapidwizard with Fresh files.

Previously i was using old ebs12.2 installation files.

Will update you again all the progress.

Regards,

[ErmanArslansOracleBlog]

Okay good. Lets see..

[raiq1]

Dear Erman,

I have downloaded all ebs 12.2 files and create new stage and installedagain but unfortunately again , it has failed with rman restore stage.

I dont know what is the issue as i have using latest ebs 12.2 installation files.

Kindly check the  log files and guide me , how to solve the issue, already i have attempted more than 5 time

but  always 05110116.logrestore-single2.logrestore-single-logfile.sqlrapid wizard are failed.

attached the 3 log files .

Regards,

[ErmanArslansOracleBlog]

RMAN 's DB session is failing suddenly..
You need to check all the thing under the hood. ASM , GRID , CRS-like processes... You also need to ensure there is no OS firewall blocking us..
This is a wizard-type install, it works everywhere and it is based on a rman restore.. While rman is restoring, we lost contact.. According to your earlier updates on this subject, I think you have some problems in the ASM or GRID config.. But again anything on OS Side can also cause this.
You need to check you env based on this info..

Common Causes RMAN ORA-03113 ORA-03114 Error From Recovery Catalog Database (Doc ID 1263456.1)

[raiq1]

Dear Erman sir,

Thanks for suggestion, as per my knowledge when i have check grid, Asm , status are showing stable & online. both nodes are online.

For more kindly suggest me in depth how to troubleshoot.

Is there any RDA to run & check grid, asm stability status.?

there is no firewall blocking.


Need your further advice.

Regards

[ErmanArslansOracleBlog]

Check the logs of the Grid processes --including & especially of the ASM instance.
Check the alert log of the database (while its is restoring.. If there is an alert log produced during that time..)

Analyze the logs and understand why the rman loses the db connection .