Restore RAC database to different client

Envirenment

• Master Server: nbumaster (NBU Appliance 5230, 7.6.0.1)
• RAC node1: mls-rac01 (OEL6.7, Oracle database 11.2.0.4)
• Destination Client: rac-restore (OEL6.7, Oracle database 11.2.0.4)
• DBID=920906249

requirements

1. Successful Netbackup backup of set.
2. Same version of database.
3. Record Source DBID/DATAFILE/TABLESPACE.
4. Install database software only on Destination client.
5. Install Netbackup client software on Destination client.

SQL> select name from v$datafile;
Press Return to Continue

NAME
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+EAS_DATA/easdb/datafile/system.256.943035169
+EAS_DATA/easdb/datafile/sysaux.257.943035169
+EAS_DATA/easdb/datafile/undotbs1.258.943035169
+EAS_DATA/easdb/datafile/users.259.943035169
+EAS_DATA/easdb/datafile/undotbs2.264.943035365
+EAS_DATA/easdb/datafile/eas_d_mls2015db_index.268.945306733
+EAS_DATA/easdb/datafile/eas_d_mls2015db_standard.270.945306749
+EAS_DATA/easdb/datafile/eas_d_mls2015db_temp2.279.945306769
+EAS_DATA/easdb/datafile/eas_d_mls2015db_standard.269.945306785
+EAS_DATA/easdb/datafile/eas_d_mls2015db_standard.272.945306829
+EAS_DATA/easdb/datafile/eas_d_mls2015db_standard.273.945307171
+EAS_DATA/easdb/datafile/eas_d_mls2015db_standard.274.945307197
+EAS_DATA/easdb/datafile/eas_d_mls2015db_standard.275.945307213
+EAS_DATA/easdb/datafile/eas_d_mls2015db_standard.276.945307231
+EAS_DATA/easdb/datafile/eas_d_mls2015db_standard.277.945307251
+EAS_DATA/easdb/datafile/eas_d_mls2015db_standard.281.945307259
+EAS_DATA/easdb/datafile/eas_d_mls2015db_index.280.945307301
+EAS_DATA/easdb/datafile/mls_d_mlsmid_standard.278.945360691
+EAS_DATA/easdb/datafile/eas_d_eas8_index.261.945083521
+EAS_DATA/easdb/datafile/eas_d_eas8_standard.266.945083533
+EAS_DATA/easdb/datafile/eas_d_eas8_temp2.265.945083541
+EAS_DATA/easdb/datafile/mls_d_dsp1_standard.283.945362643

22 rows selected.

SQL> select name from v$tempfile;
Press Return to Continue

NAME
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+EAS_DATA/easdb/tempfile/temp.263.943035277
+EAS_DATA/easdb/tempfile/eas_t_mls2015db_standard.271.945306775
+EAS_DATA/easdb/tempfile/eas_t_eas8_standard.262.945085313
+EAS_DATA/easdb/tempfile/mls_t_mlsmid_standard.282.945360745
+EAS_DATA/easdb/tempfile/mls_t_dsp1_standard.284.945362645

SQL> select name from v$controlfile;
Press Return to Continue

NAME
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
+EAS_DATA/easdb/controlfile/current.260.943035273
+CRS/easdb/controlfile/current.258.943637341

Master Server hosts file

nbumaster:/home/maintenance # tail -5 /etc/hosts

192.168.24.73   it003   it003
192.168.17.161  mls-rac01
192.168.17.162  mls-rac02
192.168.17.168  eas-new
192.168.17.169  rac-restore

Client/Media Server

• Master&Media

nbumaster:/home/maintenance # cat /usr/openv/netbackup/bp.conf 
SERVER = nbumaster
SERVER = TEST_SEVER
SERVER = opscenter
CLIENT_NAME = nbumaster
USE_VXSS = PROHIBITED
VXSS_SERVICE_TYPE = INTEGRITYANDCONFIDENTIALITY
EMMSERVER = nbumaster
VXDBMS_NB_DATA = /usr/openv/db/data
ENABLE_ROBUST_LOGGING = YES
VERBOSE = 5
NETBACKUP_APPLIANCE_MODE = MANUAL_MASTER_APPLIANCE
IP_ADDRESS_FAMILY = AF_UNSPEC
VM_PROXY_SERVER = nbumaster
OPS_CENTER_SERVER_NAME = TEST_SEVER
OPS_CENTER_SERVER_NAME = opscenter
TELEMETRY_UPLOAD = YES

• Source Client (mls-rac01)

[oracle@mls-rac01 ~]$ cat /usr/openv/netbackup/bp.conf
SERVER = nbumaster
CLIENT_NAME = mls-rac01
CONNECT_OPTIONS = localhost 1 0 2

• Destination Client (rac-restore)

[root@rac-restore ~]# cat /usr/openv/netbackup/bp.conf 
SERVER = nbumaster
CLIENT_NAME = rac-restore
CONNECT_OPTIONS = localhost 1 0 2

Excute restore

Create initEASDB.ora & orapwd file

copy pfile from Source client mls-rac02 OR create new one.

EASDB.__db_cache_size=5737807872
EASDB.__java_pool_size=117440512
EASDB.__large_pool_size=134217728
EASDB.__pga_aggregate_target=3221225472
EASDB.__sga_target=7516192768
EASDB.__shared_io_pool_size=0
EASDB.__shared_pool_size=1308622848
EASDB.__streams_pool_size=0
*.audit_file_dest='/u01/app/oracle/admin/EASDB/adump'
*.audit_trail='db'
*.compatible='11.2.0.4.0'
*.control_files='/rac-restore/EASDB/controlfile/ctrl_dEASDB.ctl'
*.db_block_size=8192
*.db_create_file_dest='/rac-restore'
*.db_domain=''
*.db_name='EASDB'
*.diagnostic_dest='/u01/app/oracle'
*.dispatchers='(PROTOCOL=TCP) (SERVICE=EASDBXDB)'
*.log_archive_dest_1='LOCATION=/rac-restore/EASDB/arch'
*.log_archive_format='arch_%t_%s_%r.arc'
*.open_cursors=300
*.pga_aggregate_target=3221225472
*.processes=300
*.remote_login_passwordfile='exclusive'
*.sessions=335
*.sga_target=7516192768
*.undo_tablespace='UNDOTBS1'

[oracle@rac-restore dbs]$ orapwd password=Systec36 file=orapwEASDB

[oracle@rac-restore dbs]$ ls
hc_EASDB.dat  initEASDB.ora  initEASDB.ora.old  init.ora  orapwEASDB

create directories

[root@rac-restore ]# mkdir -pv /rac-restore/EASDB/{controlfile,arch}
[root@rac-restore ]# chown -Rv oracle:oinstall /rac-restore/EASDB/


[oracle@rac-restore dbs]$ mkdir -pv /u01/app/oracle/admin/EASDB/{adump,bdump,cdump,ddump,udump,pfile}
mkdir: created directory `/u01/app/oracle/admin/EASDB'
mkdir: created directory `/u01/app/oracle/admin/EASDB/adump'
mkdir: created directory `/u01/app/oracle/admin/EASDB/bdump'
mkdir: created directory `/u01/app/oracle/admin/EASDB/cdump'
mkdir: created directory `/u01/app/oracle/admin/EASDB/ddump'
mkdir: created directory `/u01/app/oracle/admin/EASDB/udump'
mkdir: created directory `/u01/app/oracle/admin/EASDB/pfile'

restore controlfile

• find out correct controlfile in Netbackup

/usr/openv/netbackup/bin/bplist -S nbumaster -C mls-rac02 -t 4 -l -R / | tee nbu_backups20170704_mls-rac02.txt

  export ORACLE_SID=EASDB
  rman target / nocatalog

RMAN> startup nomount pfile='/u01/app/oracle/product/11.2.0/dbhome_1/dbs/initEASDB.ora';

RMAN> set DBID=920906249
RMAN> run {
ALLOCATE CHANNEL ch00 TYPE 'SBT_TAPE' PARMS 'SBT_LIBRARY=/usr/openv/netbackup/bin/libobk.so64';
SEND 'NB_ORA_SERV=nbumaster, NB_ORA_CLIENT=mls-rac02';
# restore controlfile from '/ctrl_dEASDB_um5s8a6ll_s1733_p1_t948247221';
restore controlfile from '/ctrl_dEASDB_ukps84cjs_s1689_p1_t948056700';
RELEASE CHANNEL ch00;
}

RMAN> startup mount

• output

run {
ALLOCATE CHANNEL ch00 TYPE 'SBT_TAPE' PARMS 'SBT_LIBRARY=/usr/openv/netbackup/bin/libobk.so64';
SEND 'NB_ORA_SERV=nbumaster, NB_ORA_CLIENT=mls-rac02';
restore controlfile from '/ctrl_dEASDB_u09s8ehds_s9_p1_t948389308';
RELEASE CHANNEL ch00;

6> }

allocated channel: ch00
channel ch00: SID=122 device type=SBT_TAPE
channel ch00: Veritas NetBackup for Oracle - Release 7.6 (2013111313)

sent command to channel: ch00

Starting restore at 04-7月 -2017 16:08:44

channel ch00: restoring control file
channel ch00: restore complete, elapsed time: 00:00:15
output file name=/rac-restore/EASDB/controlfile/ctrl_dEASDB.ctl
Finished restore at 04-7月 -2017 16:08:59

released channel: ch00

RMAN> startup mount;

database is already started
database mounted

restore database by UNTIL TIME

• Following is the very important script that one can use to check the recoverability of the DB.

spool log to rman_listbackup02.log
set echo on
show all;
report schema;
list backup summary;
list backup;
list copy;
run {
ALLOCATE CHANNEL ch00 TYPE 'SBT_TAPE' PARMS 'SBT_LIBRARY=/usr/openv/netbackup/bin/libobk.so64';
SEND 'NB_ORA_SERV=nbumaster, NB_ORA_CLIENT=mls-rac01';
ALLOCATE CHANNEL ch01 TYPE 'SBT_TAPE' PARMS 'SBT_LIBRARY=/usr/openv/netbackup/bin/libobk.so64';
SEND 'NB_ORA_SERV=nbumaster, NB_ORA_CLIENT=mls-rac02';

#allocate channel ch01 type 'sbt_tape' parms 'ENV=(NB_ORA_CLIENT=hosta.com)';
#allocate channel ch02 type 'sbt_tape' parms 'ENV=(NB_ORA_CLIENT=hostb.com)';
#set UNTIL TIME "to_date('2017-07-01 17:00:00','YYYY-MM-DD HH24:MI:SS')";

SET UNTIL TIME "to_date('2017-07-01:17:00:00','YYYY-MM-DD:HH24:MI:SS')";
 restore database preview;
 restore database validate;
RELEASE CHANNEL ch00;
RELEASE CHANNEL ch01;
}
spool log off

RMAN> run {
ALLOCATE CHANNEL ch00 TYPE 'SBT_TAPE' PARMS 'SBT_LIBRARY=/usr/openv/netbackup/bin/libobk.so64';
SEND 'NB_ORA_SERV=nbumaster, NB_ORA_CLIENT=mls-rac02';
set UNTIL TIME "to_date('2017-07-01 17:00:00','YYYY-MM-DD HH24:MI:SS')";
restore database;
recover database;
RELEASE CHANNEL ch00;
}

RMAN> run {
ALLOCATE CHANNEL ch00 TYPE 'SBT_TAPE' PARMS 'SBT_LIBRARY=/usr/openv/netbackup/bin/libobk.so64';
SEND 'NB_ORA_SERV=nbumaster, NB_ORA_CLIENT=mls-rac02';
set UNTIL TIME "to_date('2017-07-01 17:00:00','YYYY-MM-DD HH24:MI:SS')";
restore validate archivelog from scn 31231 until scn 33224;
RELEASE CHANNEL ch00;
}

RMAN> run {
ALLOCATE CHANNEL ch00 TYPE 'SBT_TAPE' PARMS 'SBT_LIBRARY=/usr/openv/netbackup/bin/libobk.so64';
SEND 'NB_ORA_SERV=nbumaster, NB_ORA_CLIENT=mls-rac02';
ALLOCATE CHANNEL ch01 TYPE 'SBT_TAPE' PARMS 'SBT_LIBRARY=/usr/openv/netbackup/bin/libobk.so64';
SEND 'NB_ORA_SERV=nbumaster, NB_ORA_CLIENT=mls-rac02';
#set UNTIL TIME "to_date('2017-07-01 17:00:00','YYYY-MM-DD HH24:MI:SS')";
restore validate archivelog all;
#restore archivelog from time "TRUNC(sysdate)" until time "sysdate";
#restore validate archivelog from scn 31231 until scn 33225;
#restore validate archivelog from scn 31178 until scn 33171;
RELEASE CHANNEL ch00;
RELEASE CHANNEL ch01;
}

archived log file name=/rac-restore/EASDB/arch/arch_1_45_948384202.arc thread=1 sequence=45
unable to find archived log
archived log thread=1 sequence=46
released channel: ch00
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of recover command at 07/04/2017 16:20:36
RMAN-06054: media recovery requesting unknown archived log for thread 1 with sequence 46 and starting SCN of 1025994

run {
ALLOCATE CHANNEL ch00 TYPE 'SBT_TAPE' PARMS 'SBT_LIBRARY=/usr/openv/netbackup/bin/libobk.so64';
3> SEND 'NB_ORA_SERV=nbumaster, NB_ORA_CLIENT=mls-rac02';
4> RESTORE ARCHIVELOG FROM SEQUENCE 46;
5> RELEASE CHANNEL ch00;
6> }

allocated channel: ch00
channel ch00: SID=122 device type=SBT_TAPE
channel ch00: Veritas NetBackup for Oracle - Release 7.6 (2013111313)

sent command to channel: ch00

Starting restore at 04-7月 -2017 16:24:53
released channel: ch00
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of restore command at 07/04/2017 16:24:53
RMAN-20242: specification does not match any archived log in the repository

https://www.veritas.com/support/en_US/article.000021590

RMAN> run {
ALLOCATE CHANNEL ch00 TYPE 'SBT_TAPE' PARMS 'SBT_LIBRARY=/usr/openv/netbackup/bin/libobk.so64';
SEND 'NB_ORA_SERV=nbumaster, NB_ORA_CLIENT=mls-rac02';
#restore controlfile from '/ctrl_dEASDB_u09s8ehds_s9_p1_t948389308';
restore controlfile from '/ctrl_dEASDB_u0is8epih_s18_p1_t948397649';

RELEASE CHANNEL ch00;

Open database

[oracle@rac-restore dbs]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.4.0 Production on 星期二 7月 4 16:43:54 2017

Copyright (c) 1982, 2013, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

SQL> select instance_name,status from v$instance;

INSTANCE_NAME    STATUS
---------------- ------------
EASDB            MOUNTED

SQL> alter database open;
alter database open
*
ERROR at line 1:
ORA-01589: 要打开数据库则必须使用 RESETLOGS 或 NORESETLOGS 选项

SQL> alter database open resetlogs;
Database altered.

SQL> select instance_name,status from v$instance;

INSTANCE_NAME    STATUS
---------------- ------------
EASDB            OPEN

SQL> create spfile from pfile;

NBU restore script

• recovery catalog (if used). At the rman command prompt, enter the following:

crosscheck backupset of database;

Use restore ... validate when you want RMAN to choose the backups to test.

NetBackup 7.7.1 AdministratorGuide => 131 to 138 pages

https://www.veritas.com/support/en_US/article.HOWTO64200

1. Make sure you have installed Oracle with the same user as used on the original source client machine

2. Install NetBackup Client and Oracle agent

3. Patch if needed

4. Touch the file on the master server: => /usr/openv/netbackup/db/altnames/No.Restrictions

5. Browse the Oracle backups by running /usr/openv/netbackup/bin/bplist -S <master server> - C <Oracle source client name> -t 4 -l -R /

   • => Make sure you can browse the backups

6. Restore the control file (in NOMOUNT mode) - See exact steps for this procedure in technote HOWTO63971

   •    => SQL> STARTUP NOMOUNT pfile='/u01/xxx.ora'
       rman nocatalog
        => RMAN> connect target 
        => RMAN> SET DBID=XXXXXX
           RMAN> restore controlfile from '/tmp/xxx';

7. Restore the database (in MOUNT mode)

    RMAN> mount database
    
   
    8. Restore the database using the script in /usr/openv/netbackup/ext/db_ext/oracle/samples/rman/database_restore.sh
   {{{
    SQL> ALTER DATABASE BACKUP CONTROLFILE TO TRACE RESETLOGS;
    ALTER DATABASE RENAME FILE '/old/location' TO '/new/location';
   
    SQL> select log_mode from v$database;
    SQL> select name from v$datafile;
    SQL> select member from v$logfile;
    SQL> select name from v$tempfile;
    SQL> select name from v$controlfile;

8. SQL> SELECT NAME,FILE#,STATUS FROM V$DATAFILE;

RUN {
  ALLOCATE CHANNEL ch00 TYPE 'SBT_TAPE';
  ALLOCATE CHANNEL ch01 TYPE 'SBT_TAPE';
  SEND 'NB_ORA_SERV=<master server>, NB_ORA_CLIENT=<Oracle source client name>’;
  set newname for datafile 1 to '/data/lsdb/datafile/system01.dbf'
  set newname for datafile 2 to '/data/lsdb/datafile/sysaux01.dbf'
  RESTORE DATABASE;
  RECOVER DATABASE;
  sql 'alter database open resetlogs';
  RELEASE CHANNEL ch00;
  RELEASE CHANNEL ch01;
} 

Troubleshooting

NetBackup 7.7.1 AdministratorGuide => 208 pages

/usr/openv/netbackup/logs/user_ops

UNIX: /usr/openv/netbackup/logs/dbclient/log.mmddyy

The debug logs are located in /usr/openv/netbackup/logs .

To set the debug level on a UNIX client
Enter the following line in the bp.conf file.

VERBOSE = X

Where X is the debug level you want.

Modifying the maxjobs for "NBU_POLICY"

• Info nbrb (pid=3037) Limit has been reached for the logical resource <Master server>.NBU_POLICY.MAXJOBS.<Policy name>

you can chage that form your Policy attributes

but do make sure that it will not effect the other jobs... if you increase value there may be a possiblity that other polices run out of resources for backups..

its you need to make sure that ,chaning this value will not impact the other jobs to get the resources..

Select the Policy in the GUI and adjust Limit Jobs per Policy