Friday, October 1, 2010

SOLARIS USEFUL COMMANDS FOR DBA'S

Not just for DBAs: this is to remind anyone of syntax and options.

1. Handy Solaris commands HW config, swap, patches etc.
2. Real-time performance monitoring vmstat, iostat, mpstat etc.
3. Unix Commands find, awk, sed, grep etc.
4. vi editor commands the 22 most useful vi commands
5. Unix file permissions explained sounds basic, but you may learn something new here!
6. The Solaris Name Service explained how hostname and username lookups work


1. Handy Solaris commands

show hardware config (memory, CPUs etc) :
/usr/platform/sun4u/sbin/prtdiag -v |more
/usr/sbin/psrinfo
show kernel settings:
/usr/sbin/sysdef |more
more /etc/system

show list of Solaris patches:
showrev -p |more

show software installed:
pkginfo |more

check swap space:
/usr/sbin/swap -s

check system load average: an uptime more than double the number of CPUs is getting a bit busy
uptime

top CPU users:
/usr/ucb/ps uax |head

top memory users:
/usr/ucb/ps vax |head

check disk space:
df -k (shows all mounted filesystems -including NFS etc)
df -lk (shows only local filesystems)
df -k -Fufs (shows only local Sun UFS filesystems (normal hard disks)
df -k . (show just the filesystem you are currently in)

2. Real-time performance monitoring:


memory usage :
vmstat 5 -look at memory/free field and page/sr field. Ignore the first line output as it’s historical
procs memory page disk faults cpu
r b w swap free re mf pi po fr de sr s0 s1 s6 s3 in sy cs us sy id
0 0 83 4456 456 1 431 266 70 167 0 35 6 6 0 2 523 567 31 14 9 76
0 0 62 3588464 46824 0 196 64 0 0 0 0 5 4 0 0 606 9743 882 86 7 7
0 0 62 3587960 42672 1 552 41 1 1 0 0 2 2 0 0 789 5488 1040 84 7 9
0 1 62 3584704 38848 0 471 3 38 38 0 0 5 5 0 1 1426 5270 968 64 9 27
0 0 62 3586464 38456 0 451 0 0 0 0 0 2 2 0 0 929 6039 1265 70 6 24
Also make sure that cpu/us is at least double cpu/sy


disk busy-ness
mpstat 5 (or iostat -c 5 )
CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl
0 221 3 544 227 75 582 61 31 28 7 267 18 12 2 68
2 209 2 446 395 178 328 37 31 32 6 299 11 7 2 80
-look at the wt field, this is wait-for-I/O which can be network or disk I/O, and should not be more than 30-40 (percent)

To see individual disk performance and find slow ('hot') disks :
iostat -d 5
sd0 sd1 sd6 sd37
kps tps serv kps tps serv kps tps serv kps tps serv
123 6 44 123 6 42 0 0 42 66 2 8
33 1 3 37 1 1 0 0 0 3 0 5
-check the serv column for each disk: this is the disk service time in milliseconds.
However iostat by default shows only the first four disks: if you have more use -x and/or -l options :
iostat -xdnl 7 5 (e.g. if you have seven hard disks)
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.4 1.6 3.2 70.4 0.0 0.0 0.0 2.7 0 1 c0t0d0
0.2 1.6 1.6 70.4 0.0 0.0 0.0 3.0 0 1 c0t1d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c0t6d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c1t8d0
88.8 19.8 1276.8 158.4 0.0 0.6 0.0 5.7 0 51 c1t9d0
0.0 0.4 0.0 100.8 0.0 0.0 0.0 17.4 0 1 c1t10d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c1t11d0


3. Unix Commands

at - run a command or script once in the future
bash$ at 0815 tomorrow
at> rm $HOME/zxc
at> (type Control-D)
commands will be executed using /usr/bin/bash
job 984032100.a at Thu Mar 8 06:15:00 2001



awk


cron
crontab -l to see your crontab
crontab -e to edit it (NB but first do EDITOR=vi (or EDITOR=emacs) ; export EDITOR )
Remember fields are : minute hour day month day_of_week (0=Sun)
0-59 0-23 1-31 1-12 0-6
e.g. run a script only on Tuesdays to Fridays, at 10:12 am, 3:12pm, 4:12pm and 5:12pm
12 10,15-17 * * 2-5/home/oracle/bin/script



du -show disk space usage
du -k -show disk usage in kB in all subdirectories below the current one
du -sk -show just the total of everything below the current dir
du -sk * -show individual totals for each file or directory in the current dir



find
find things you can’t remember the exact name of :
find /usr/lib -name "*socket*"
find big files (more than 20000 blocks, ie about 10MB )
find . -size +20000
find files or directories modified in last day
find . -mtime -1
find files not accessed (read) in 45 days
find /tmp -atime +45
Add -ls to the end of any of the above commands to see a full ls -l listing of the files found.

You can combine multiple options , e.g look for which big files have filled up a filesystem recently
find . -size +20000 -mtime -1 -ls
Or combine options using an "OR" syntax, e.g. find files which were modified either less than 1 week ago or more than 2 weeks ago
find . \( -mtime +14 -o -mtime -7 \) -ls
You can send the ouput of find to another command : the xargs command is ideal for this: e.g. interactively delete all core files under your $HOME
find ~ -type f -name core |xargs rm -i
or look for a particular word in all your .c files:
find . -name "*.c" |xargs grep -l libsocket



grep and its more flexible and faster cousins egrep and fgrep
Search for either of two strings in all log files:
egrep "error|PROBLEM" *log
case-insensitive search
fgrep -i error *log (will find Error, ERROR, etc.)
just see the filenames where the search text was found
grep -l error *log



sort
-n sorts by number instead of alphabetically
e.g. cd /export/home; du -sk * | sort -nr lists who is using most space in their home directory



tar
e.g. to copy a directory tree, preserving symlinks:
cd fromdir; tar cf - . | (cd todir; tar xpf -)



tr translate characters
convert lower-case to upper:
cat myfile | tr [a-z] [A-Z]
change colons to newlines:
tr : "\n" < myfile > newfile



who show who is currently logged on and where from



w show what people are doing


other useful Unix commands for you to check out:


cut, paste , split

diff, sdiff, cmp, dircmp

head, tail, tail -f

id, groups

4. vi commands
The basic vi commands:
i insert before current character a append after current char
I insert at beginning of line A append after end of line
x delete current character X backspace-delete (deletes char to left)
dd delete current line
22dd delete 22 lines u undo the last thing you did
p paste those 22 lines below the current line
12yy copy 12 lines into the ‘clipboard’ for later pasting
to get out of editing (insert/append) mode
:wq to save ('write') and quit (:q! to quit without writing)


vi moving-around Timesavers:


using h,j,k,l to move left,down,up and right is quicker than using the arrow keys (once you get used to it!). w move one word forward b move back a word e go to end of word
Ctl-F move Forward a whole page Ctl-B move Back a page
0 (zero) go to beginning of line $ go to end of line
:242 go to line 242 Ctl-G see what line you are on


vi modifying things:
cw change word 3cw change 3 words C change to end-of-line
dw delete word D delete to end-of-line

cc replace current line with what you type next
r replace one character R endless replace (like over-typing)
o open/add a new line below the current one O open new line above current one
xp swap two characters (e.g. when you make a typo)
/bob search forward for 'bob' n repeat previous search ? search backwards


Probably the handiest vi command:
. (dot) -repeat your last command


useful extra vi commands:
~ swap case of current character (capitalize or lower-case)
Ctl-L re-draw the screen (e.g when something from a background process writes to your screen and messes up your vi window)
:set nu show line numbers (:set nonu to remove line numbers)
:set list show hidden characters, line endings etc. (: set nolist)


Global replace:
:%s/old/new/g (% means all lines)
or :g/old/s//new/g
or :%s/old/new/gc (c is to confirm each replacement, type ‘y’ to accept)



5. Unix file permissions
You probably already know the basics:
Each user in Unix belongs to at least one group, each file or directory on the system belongs to one user and one group. When you do an ls -l on a file you see what permissions the file owner, group owner and everyone else (‘world’ or ‘others’) have on it
-rwxr-xr-- 1 robins devteam 180 Mar 8 13:50 instbb
so on the file instbb there is full access (rwx- read,write,execute) for the owner (robins), read and execute for the group devteam and read-only for everyone else.


What you may not know is that whether someone can delete that file is not determined by the file permissions, but by the permissions of the directory the file is in :
bash$ ls -al
total 598
drwxrwxr-x 3 robins devteam 512 Mar 8 12:02 .
drwxr-xr-x 24 root wheel 1024 Mar 8 12:02 ..
-rwxr-xr-- 1 robins devteam 180 Mar 8 13:50 instbb
In this case the directory (.) is group writeable, which means anyone in the group devteam can delete the file instbb. Although they can’t modify it, they could copy it to a new file in that directory, modify it, then delete the original and rename the new file as instbb. So the file isn’t as secure as it may appear..



6. Solaris Name services
Solaris provides a mechanism for getting hostnames and usernames etc from several sources (e.g DNS, NIS, or the traditional /etc/hosts file) : the file /etc/nsswitch.conf , which may contain
hosts: files dns
this means that when you try to access a remote host by name (e.g. ping neptune) it will look for neptune first in /etc/hosts, then do an lookup in DNS (nslookup using the server specified in /etc/resolv.conf)


Similarly for usernames, you may have
passwd: files nis
So when you run a command which refers to a username (e.g. cd ~oracle ) , it first looks in /etc/passwd then does a lookup in NIS (ypmatch oracle passwd).


However you can look up hosts and users without worrying about where they are stored, using the getent command :
bash$ getent passwd oracle
oracle:##oracle:3008:5001:Oracle DBA:/export/home/oracle:/usr/local/bin/bash
bash$ getent hosts www.inkq.com
192.168.245.12 www.inkq.com
bash$ getent hosts 10.0.0.91
10.0.0.91 plop.inkq.com plop mailhost

HAPPY LEARNING!

SRVCTL FAILS TO START RAC DATABASE

SRVCTL has problems while starting the database (SRVCTL STOP DATABASE works ok).
Also, the database can be started and stopped fine using SQL*Plus. No problems there.

$ srvctl start database -d HLPR
PRKP-1001 : Error starting instance HLPR1 on node db01-lisqa
CRS-0215: Could not start resource 'ora.HLPR.HLPR1.inst'.
PRKP-1001 : Error starting instance HLPR2 on node db02-lisqa
CRS-0215: Could not start resource 'ora.HLPR.HLPR2.inst'.
PRKP-1001 : Error starting instance HLPR3 on node db03-lisqa
CRS-0215: Could not start resource 'ora.HLPR.HLPR3.inst'.
PRKP-1001 : Error starting instance HLPR4 on node db04-lisqa
CRS-0215: Could not start resource 'ora.HLPR.HLPR4.inst'.
$
As seen above, it is a 4-node RAC database.

2 Issue is here
1) REQUIRED_RESOURCE missing for one of the nodes
===========================================
Doing a CRS_STAT on the above resources, shows that the REQUIRED_RESOURCES for node 4 instance (HLPR4) is missing. For the other 3 nodes the ASM dependency is defined. It is already verified that ASM is up and running on the 4th node (also there are other RAC databases on the same cluster that do not have any issues).
$ crs_stat -p ora.HLPR.HLPR1.inst|grep REQUIRED
REQUIRED_RESOURCES=ora.db01-lisqa.ASM1.asm
$ crs_stat -p ora.HLPR.HLPR2.inst|grep REQUIRED
REQUIRED_RESOURCES=ora.db02-lisqa.ASM2.asm
$ crs_stat -p ora.HLPR.HLPR3.inst|grep REQUIRED
REQUIRED_RESOURCES=ora.db03-lisqa.ASM3.asm
$ crs_stat -p ora.HLPR.HLPR4.inst|grep REQUIRED
REQUIRED_RESOURCES=
$

Fix
=====
Add the REQUIRED_RESOURCE using "srvctl modify". First lets look at the syntax (-h for help) -
$ srvctl modify instance -h
Usage: srvctl modify database -d [-n ] [-m ] [-p ] [-r {PRIMARY | PHYSICAL_STANDBY | LOGICAL_STANDBY}] [-s ] [-y {AUTOMATIC | MANUAL}]
-d Unique name for the database
-n Database name (DB_NAME), if different from the unique name given by the -d option
-o ORACLE_HOME path
-m Domain for cluster database
-p Server parameter file path
-r Role of the database (primary, physical_standby, logical_standby)
-s Startup options for the database
-y Management policy for the database (automatic, manual)
-h Print usage
$
$ srvctl modify instance -d HLPR -i HLPR4 -s +ASM4
$
$ crs_stat -p ora.HLPR.HLPR4.inst|grep REQUIRE
REQUIRED_RESOURCES=ora.db04-lisqa.ASM4.asm
$


2) SPFILE was incorrectly defined in the OCR
====================================
The imon log for the database under /log//racg shows this --
$ cd $ORACLE_HOME/log//racg
$ vi imon_HLPR.log
...
...
2009-03-19 20:36:02.729: [ RACG][1] [8400][1][ora.HLPR.HLPR1.inst]: racgimon exiting

2009-03-19 20:43:34.680: [ RACG][1] [12432][1][ora.HLPR.HLPR1.inst]: racgimon started

2009-03-19 20:43:37.137: [ RACG][6] [12432][6][ora.HLPR.HLPR1.inst]:
SQL*Plus: Release 10.2.0.4.0 - Production on Thu Mar 19 20:43:36 2009

Copyright (c) 1982, 2007, Oracle. All Rights Reserved.
Enter user-name: Connected to an idle instance.

SQL> ORA-01565: error in identifying file '+RAC_REDO02/HLPR/spfileHLPR.ora'

2009-03-19 20:43:37.137: [ RACG][6] [12432][6][ora.HLPR.HLPR1.inst]: ORA-17503: ksfdopn:2 Failed to open file
+RAC_REDO02/HLPR/spfileHLPR.ora
ORA-15056: additional error message
ORA-17503: ksfdopn:DGOpenFile05 Failed to open file +RAC_REDO02/hlpr/spfile
ORA-01078: failure in processing system parameters
SQL> Disconnected
This shows that the SPFILE was incorrectly defined in the OCR

Fix
====
Modified OCR for the correct location of the SPFILE -
$ cat $ORACLE_HOME/dbs/initHLPR1.ora
spfile='+HLPR_DATA/HLPR/PARAMETERFILE/spfile.286.640265465'
$
$ srvctl modify database -d HLPR -p +HLPR_DATA/HLPR/PARAMETERFILE/spfile.286.640265465
$


This fixed the OCR and database could then be started using SRVCTL.

There is also one more issue that I noticed today in the listener configuration w/ CRS. On node4, the listener was up under db_home (typically it is seen under ASM home). Doing a "ps -ef|grep tns" will tell what OH is being used for the listener home. Also, the crs_stat -p output shows that.
[oracle@ajithpathiyil /home/oracle] $ ps -ef|grep tns
oracle 1982 1 0 May 13 ? 2:06 /d01/app/oracle/product/10.2/oracledb/bin/tnslsnr LISTENER_DB04-LISQA -inherit
oracle 1965 1 0 May 13 ? 5:49 /d01/app/oracle/product/10.2/oracledb/bin/tnslsnr LISTENER_CGQA_DB04-LISQA -inherit
oracle 2007 1 0 May 13 ? 4:33 /d01/app/oracle/product/10.2/oracledb/bin/tnslsnr LISTENER_HLPR_DB04-LISQA -inherit
oracle 2041 1 0 May 13 ? 5:43 /d01/app/oracle/product/10.2/oracledb/bin/tnslsnr LISTENER_ORACLE_DB04-LISQA -inherit
oracle 2024 1 0 May 13 ? 6:33 /d01/app/oracle/product/10.2/oracledb/bin/tnslsnr LISTENER_PLQA_DB04-LISQA -inherit
oracle 22092 23479 0 11:18:42 pts/1 0:00 grep tns


As we can see from above, listener "LISTENER_DB04-LISQA" is running under the database home (/d01/app/oracle/product/10.2/oracledb). The crs_stat -p output for this resource shows this --
[oracle@ajithpathiyil /home/oracle] $ crs_stat -p ora.db04-lisqa.LISTENER_DB04-LISQA.lsnr
NAME=ora.db04-lisqa.LISTENER_DB04-LISQA.lsnr
TYPE=application
ACTION_SCRIPT=/d01/app/oracle/product/10.2/oracledb/bin/racgwrap
ACTIVE_PLACEMENT=0
AUTO_START=1
CHECK_INTERVAL=600
DESCRIPTION=CRS application for listener on node
FAILOVER_DELAY=0
FAILURE_INTERVAL=0
FAILURE_THRESHOLD=0
HOSTING_MEMBERS=db04-lisqa
OPTIONAL_RESOURCES=
PLACEMENT=restricted
REQUIRED_RESOURCES=ora.db04-lisqa.vip
RESTART_ATTEMPTS=5
SCRIPT_TIMEOUT=600
START_TIMEOUT=0
STOP_TIMEOUT=0
UPTIME_THRESHOLD=7d
USR_ORA_ALERT_NAME=
USR_ORA_CHECK_TIMEOUT=0
USR_ORA_CONNECT_STR=/ as sysdba
USR_ORA_DEBUG=0
USR_ORA_DISCONNECT=false
USR_ORA_FLAGS=
USR_ORA_IF=
USR_ORA_INST_NOT_SHUTDOWN=
USR_ORA_LANG=
USR_ORA_NETMASK=
USR_ORA_OPEN_MODE=
USR_ORA_OPI=false
USR_ORA_PFILE=
USR_ORA_PRECONNECT=none
USR_ORA_SRV=
USR_ORA_START_TIMEOUT=0
USR_ORA_STOP_MODE=immediate
USR_ORA_STOP_TIMEOUT=0
USR_ORA_VIP=

[oracle@ajithpathiyil /home/oracle] $

The action script clearly points to the database home.
To change this, we again use the "srvctl modify" command.

[oracle@ajithpathiyil /home/oracle] $ srvctl modify listener -h
Usage: srvctl modify listener -n [-l ] -o
-n Node name
-l "" Comma separated listener names
-o ORACLE_HOME path
-h Print usage
[oracle@ajithpathiyil /home/oracle] $

[oracle@ajithpathiyil /home/oracle] $ srvctl modify listener -n db04-lisqa -l LISTENER_DB04-LISQA -o /d01/app/oracle/product/10.2/ASM

[oracle@ajithpathiyil /home/oracle] $ crs_stat -p ora.db04-lisqa.LISTENER_DB04-LISQA.lsnr
NAME=ora.db04-lisqa.LISTENER_DB04-LISQA.lsnr
TYPE=application
ACTION_SCRIPT=/d01/app/oracle/product/10.2/ASM/bin/racgwrap
ACTIVE_PLACEMENT=0
AUTO_START=1
CHECK_INTERVAL=600
DESCRIPTION=CRS application for listener on node
FAILOVER_DELAY=0
FAILURE_INTERVAL=0
FAILURE_THRESHOLD=0
HOSTING_MEMBERS=db04-lisqa
OPTIONAL_RESOURCES=
PLACEMENT=restricted
REQUIRED_RESOURCES=ora.db04-lisqa.vip
RESTART_ATTEMPTS=5
SCRIPT_TIMEOUT=600
START_TIMEOUT=0
STOP_TIMEOUT=0
UPTIME_THRESHOLD=7d
USR_ORA_ALERT_NAME=
USR_ORA_CHECK_TIMEOUT=0
USR_ORA_CONNECT_STR=/ as sysdba
USR_ORA_DEBUG=0
USR_ORA_DISCONNECT=false
USR_ORA_FLAGS=
USR_ORA_IF=
USR_ORA_INST_NOT_SHUTDOWN=
USR_ORA_LANG=
USR_ORA_NETMASK=
USR_ORA_OPEN_MODE=
USR_ORA_OPI=false
USR_ORA_PFILE=
USR_ORA_PRECONNECT=none
USR_ORA_SRV=
USR_ORA_START_TIMEOUT=0
USR_ORA_STOP_MODE=immediate
USR_ORA_STOP_TIMEOUT=0
USR_ORA_VIP=

[oracle@ajithpathiyil /home/oracle] $

The CRS registration has been changed.
Now we only need to stop the listener and restart it under the ASM home.

HAPPY LEARNING!

Monday, March 1, 2010

Standby Database Recovery Using RMAN Incremental Backup

CASE 1- NON ASM SYSTEM...all datafiles and control file is on file system.
#####################################################################################


DGMGRL> show configuration;

Configuration
Name: DR
Enabled: YES
Protection Mode: MaxPerformance
Fast-Start Failover: DISABLED
Databases:
PRSP01_VE - Primary database
PRSP01_BS - Physical standby database

Current status for "DR":
Warning: ORA-16607: one or more databases have failed


DGMGRL> show database 'PRSP01_VE';

Database
Name: PRSP01_VE
Role: PRIMARY
Enabled: YES
Intended State: ONLINE
Instance(s):
PRSP01

Current status for "PRSP01_VE":
Error: ORA-16778: redo transport error for one or more databases


DGMGRL> show database 'PRSP01_BS';

Database
Name: PRSP01_BS
Role: PHYSICAL STANDBY
Enabled: YES
Intended State: ONLINE
Instance(s):
PRSP01

Current status for "PRSP01_BS":
Error: ORA-16766: Redo Apply unexpectedly offline


DGMGRL> SHOW DATABASE 'PRSP01_VE' 'StatusReport';
STATUS REPORT
INSTANCE_NAME SEVERITY ERROR_TEXT
PRSP01 ERROR ORA-16737: the redo transport service for standby database "PRSP01_BS" has an error

DGMGRL> SHOW DATABASE 'PRSP01_VE' 'LogXptStatus';
LOG TRANSPORT STATUS
PRIMARY_INSTANCE_NAME STANDBY_DATABASE_NAME STATUS
PRSP01 PRSP01_BS ORA-00254: error in archive control string ''



Thu Jan 14 04:07:05 2010
Redo Shipping Client Connected as PUBLIC
-- Connected User is Valid
RFS[9777]: Assigned to RFS process 25364
RFS[9777]: Identified database type as 'physical standby'
RFS[9777]: Error in standby_archive_dest '/local/ORAPRSP01/DBDUMPS/ARCHIVE/PRSP01'
RFS: Forced Shutdown due to RFS_ERROR state
Thu Jan 14 04:07:05 2010
Redo Shipping Client Connected as PUBLIC
-- Connected User is Valid
RFS[9778]: Assigned to RFS process 25366
RFS[9778]: Identified database type as 'physical standby'
RFS[9778]: Error in standby_archive_dest '/local/ORAPRSP01/DBDUMPS/ARCHIVE/PRSP01'
RFS: Forced Shutdown due to RFS_ERROR state
Thu Jan 14 04:07:07 2010
Redo Shipping Client Connected as PUBLIC
-- Connected User is Valid
RFS[9779]: Assigned to RFS process 25368
RFS[9779]: Identified database type as 'physical standby'
RFS[9779]: Error in standby_archive_dest '/local/ORAPRSP01/DBDUMPS/ARCHIVE/PRSP01'
RFS: Forced Shutdown due to RFS_ERROR state
Thu Jan 14 04:07:07 2010
Redo Shipping Client Connected as PUBLIC
-- Connected User is Valid
RFS[9780]: Assigned to RFS process 25370
RFS[9780]: Identified database type as 'physical standby'
RFS[9780]: Error in standby_archive_dest '/local/ORAPRSP01/DBDUMPS/ARCHIVE/PRSP01'
RFS: Forced Shutdown due to RFS_ERROR state



Logged on to standby successfully
Client logon and security negotiation successful!
*** 2010-01-13 19:39:16.857
Redo shipping client performing standby login
*** 2010-01-13 19:39:17.016 65194 kcrr.c
Logged on to standby successfully
Client logon and security negotiation successful!
Error 254 creating standby archive log file at host '(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=eqd-oradev02)(PORT=1575)))(CONNECT_DATA=(SERVICE_NAME=PRSP01_BS_XPT)(INSTANCE_NAME=PRSP01)(SERVER=dedicated)))'
*** 2010-01-13 19:39:17.078 61287 kcrr.c
ARC1: Attempting destination LOG_ARCHIVE_DEST_2 network reconnect (254)
*** 2010-01-13 19:39:17.078 61287 kcrr.c
ARC1: Destination LOG_ARCHIVE_DEST_2 network reconnect abandoned
ORA-00254: error in archive control string ''



(1)At standby site.
=================
SQL> column current_scn format 9999999999999
SQL> select current_scn from v$database;

CURRENT_SCN
--------------
8918544365814


(2) At primary site
================
BACKUP DEVICE TYPE DISK INCREMENTAL FROM SCN 8918544365814 DATABASE FORMAT '/local/ORAPRMP01/DBDUMPS/bkup_%U';

(3) At BCP site
============
3a) Stop MRP at BCP databae side

At BCP site
===========
3b) Note location of controlfiles

SQL> select name from v$controlfile;

NAME
--------------------------------------------------------------------------------
/local/ORAPRSP01/data01/PRSP01/control01.ctl
/local/ORAPRSP01/data02/PRSP01/control02.ctl
/local/ORAPRSP01/data03/PRSP01/control03.ctl


At BCP Site
============

4) Incremental backup
==================

RMAN> catalog start with '/local/ORAUASP01/DBDUMPS/incr_bkup';

Starting implicit crosscheck backup at 14-JAN-10
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: sid=38 devtype=DISK
Crosschecked 803 objects
Finished implicit crosscheck backup at 14-JAN-10

Starting implicit crosscheck copy at 14-JAN-10
using channel ORA_DISK_1
Finished implicit crosscheck copy at 14-JAN-10

searching for all files in the recovery area
cataloging files...
no files cataloged

searching for all files that match the pattern /local/ORAUASP01/DBDUMPS/incr_bkup

List of Files Unknown to the Database
=====================================
File Name: /local/ORAUASP01/DBDUMPS/incr_bkup/bkup_n4l3dti1_1_1
File Name: /local/ORAUASP01/DBDUMPS/incr_bkup/bkup_n5l3du33_1_1
File Name: /local/ORAUASP01/DBDUMPS/incr_bkup/bkup_n3l3dti1_1_1
File Name: /local/ORAUASP01/DBDUMPS/incr_bkup/bkup_n6l3dua5_1_1

Do you really want to catalog the above files (enter YES or NO)? YES
cataloging files...
cataloging done

List of Cataloged Files
=======================
File Name: /local/ORAUASP01/DBDUMPS/incr_bkup/bkup_n4l3dti1_1_1
File Name: /local/ORAUASP01/DBDUMPS/incr_bkup/bkup_n5l3du33_1_1
File Name: /local/ORAUASP01/DBDUMPS/incr_bkup/bkup_n3l3dti1_1_1
File Name: /local/ORAUASP01/DBDUMPS/incr_bkup/bkup_n6l3dua5_1_1


5)At BCP site


RMAN> recover database noredo;

Starting recover at 14-JAN-10
using channel ORA_DISK_1
channel ORA_DISK_1: starting incremental datafile backupset restore
channel ORA_DISK_1: specifying datafile(s) to restore from backup set
destination for restore of datafile 00008: /local/ORAPRSP01/data01/PRSP01/PRSP01_PRISM_DATA_03.dbf
destination for restore of datafile 00009: /local/ORAPRSP01/data01/PRSP01/PRSP01_PRISM_INDEX_02.dbf
destination for restore of datafile 00014: /local/ORAPRSP01/data03/PRSP01/PRSP01_PRISM_DATA_04.dbf
destination for restore of datafile 00018: /local/ORAPRSP01/data02/PRSP01/PRSP01_PRISM_DATA_06.dbf
destination for restore of datafile 00022: /local/ORAPRSP01/data03/PRSP01/capacity01.dbf
destination for restore of datafile 00023: /local/ORAPRSP01/data02/PRSP01/PRSP01_sysaux02.dbf
destination for restore of datafile 00024: /local/ORAPRSP01/data02/PRSP01/PRSP01_system02.dbf
destination for restore of datafile 00025: /local/ORAPRSP01/data02/PRSP01/dbus01.dbf
channel ORA_DISK_1: reading from backup piece /local/ORAUASP01/DBDUMPS/incr_bkup/bkup_n4l3dti1_1_1
channel ORA_DISK_1: restored backup piece 1
piece handle=/local/ORAUASP01/DBDUMPS/incr_bkup/bkup_n4l3dti1_1_1 tag=TAG20100114T064416
channel ORA_DISK_1: restore complete, elapsed time: 00:04:36
channel ORA_DISK_1: starting incremental datafile backupset restore
channel ORA_DISK_1: specifying datafile(s) to restore from backup set
destination for restore of datafile 00004: /local/ORAPRSP01/data01/PRSP01/PRSP01_users01.dbf
destination for restore of datafile 00012: /local/ORAPRSP01/data02/PRSP01/PRSP01_CERD_CORPORATE_DATA_02.dbf
destination for restore of datafile 00013: /local/ORAPRSP01/data01/PRSP01/PRSP01_CERD_CORPORATE_DATA_03.dbf
destination for restore of datafile 00015: /local/ORAPRSP01/data03/PRSP01/PRSP01_PRISM_INDEX_03.dbf
destination for restore of datafile 00021: /local/ORAPRSP01/data03/PRSP01/PRSP01_PRISM_INDEX_06.dbf
channel ORA_DISK_1: reading from backup piece /local/ORAUASP01/DBDUMPS/incr_bkup/bkup_n5l3du33_1_1
channel ORA_DISK_1: restored backup piece 1
piece handle=/local/ORAUASP01/DBDUMPS/incr_bkup/bkup_n5l3du33_1_1 tag=TAG20100114T064416
channel ORA_DISK_1: restore complete, elapsed time: 00:01:25
channel ORA_DISK_1: starting incremental datafile backupset restore
channel ORA_DISK_1: specifying datafile(s) to restore from backup set
destination for restore of datafile 00001: /local/ORAPRSP01/data01/PRSP01/PRSP01_system01.dbf
destination for restore of datafile 00003: /local/ORAPRSP01/data01/PRSP01/PRSP01_sysaux01.dbf
destination for restore of datafile 00005: /local/ORAPRSP01/data03/PRSP01/PRSP01_PRISM_DATA_01.dbf
destination for restore of datafile 00007: /local/ORAPRSP01/data02/PRSP01/PRSP01_PRISM_DATA_02.dbf
destination for restore of datafile 00010: /local/ORAPRSP01/data01/PRSP01/PRSP01_undotbs02.dbf
destination for restore of datafile 00011: /local/ORAPRSP01/data01/PRSP01/PRSP01_CERD_CORPORATE_DATA_01.dbf
destination for restore of datafile 00016: /local/ORAPRSP01/data01/PRSP01/PRSP01_PRISM_DATA_05.dbf
destination for restore of datafile 00017: /local/ORAPRSP01/data01/PRSP01/PRSP01_PRISM_INDEX_04.dbf
destination for restore of datafile 00019: /local/ORAPRSP01/data02/PRSP01/PRSP01_PRISM_INDEX_05.dbf
channel ORA_DISK_1: reading from backup piece /local/ORAUASP01/DBDUMPS/incr_bkup/bkup_n3l3dti1_1_1
channel ORA_DISK_1: restored backup piece 1
piece handle=/local/ORAUASP01/DBDUMPS/incr_bkup/bkup_n3l3dti1_1_1 tag=TAG20100114T064416
channel ORA_DISK_1: restore complete, elapsed time: 00:10:36
channel ORA_DISK_1: starting incremental datafile backupset restore
channel ORA_DISK_1: specifying datafile(s) to restore from backup set
destination for restore of datafile 00002: /local/ORAPRSP01/data03/PRSP01/PRSP01_undotbs01.dbf
destination for restore of datafile 00006: /local/ORAPRSP01/data02/PRSP01/PRSP01_PRISM_INDEX_01.dbf
destination for restore of datafile 00020: /local/ORAPRSP01/data03/PRSP01/PRSP01_CERD_CORPORATE_DATA_04.dbf
channel ORA_DISK_1: reading from backup piece /local/ORAUASP01/DBDUMPS/incr_bkup/bkup_n6l3dua5_1_1
channel ORA_DISK_1: restored backup piece 1
piece handle=/local/ORAUASP01/DBDUMPS/incr_bkup/bkup_n6l3dua5_1_1 tag=TAG20100114T064416
channel ORA_DISK_1: restore complete, elapsed time: 00:13:17
Finished recover at 14-JAN-10



6) At BCP site
starting MRP asks for same archive gap still as the control file is not updated.


MRP0: Background Managed Standby Recovery process started (PRSP01)
Managed Standby Recovery not using Real Time Apply
parallel recovery started with 16 processes
Media Recovery Waiting for thread 1 sequence 94040
Fetching gap sequence in thread 1, gap sequence 94040-94052
Thu Jan 14 08:32:03 2010
Completed: alter database recover managed standby database disconnect from session
Thu Jan 14 08:32:15 2010
alter database recover managed standby database cancel
Thu Jan 14 08:32:33 2010
MRP0: Background Media Recovery cancelled with status 16037
Thu Jan 14 08:32:33 2010
Errors in file /local/ORAPRSP01/sw/oracle/admin/PRSP01/bdump/prsp01_mrp0_27018.trc:
ORA-16037: user requested cancel of managed recovery operation
Recovery interrupted!
Thu Jan 14 08:32:35 2010
Errors in file /local/ORAPRSP01/sw/oracle/admin/PRSP01/bdump/prsp01_mrp0_27018.trc:
ORA-16037: user requested cancel of managed recovery operation
Thu Jan 14 08:32:35 2010
MRP0: Background Media Recovery process shutdown (PRSP01)

7)At priamry and ship to BCP site
create a new standby control file at primary and restart the standby using that new satndby controlfile .




CASE 2- ASM SYSTEM...all datafiles and control file is on ASM.
#####################################################################################



1) get latest SCN number from standby database.

SQL> column current_scn format 999999999999
SQL> SELECT CURRENT_SCN FROM V$DATABASE;

CURRENT_SCN
-------------
18934679323

SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE CANCEL;

Database altered.


2)Take incremantal SCN backup on primary side.


RMAN> BACKUP DEVICE TYPE DISK INCREMENTAL FROM SCN 18934679323 database format '/backup/oracle/bkup_%U';

Starting backup at 19-JAN-10
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: sid=705 instance=PWHGRD21 devtype=DISK
allocated channel: ORA_DISK_2
channel ORA_DISK_2: sid=712 instance=PWHGRD21 devtype=DISK
allocated channel: ORA_DISK_3
channel ORA_DISK_3: sid=792 instance=PWHGRD21 devtype=DISK
allocated channel: ORA_DISK_4
channel ORA_DISK_4: sid=677 instance=PWHGRD21 devtype=DISK
channel ORA_DISK_1: starting compressed full datafile backupset
channel ORA_DISK_1: specifying datafile(s) in backupset
input datafile fno=00014 name=+ARCHDG1/pwhgrd20/datafile/mgmt_tablespace.994.686140129
input datafile fno=00005 name=+DATADG1/pwhgrd20/datafile/mgmt_tablespace.279.629708905
input datafile fno=00007 name=+DATADG1/pwhgrd20/datafile/bearmon.268.619705149
channel ORA_DISK_1: starting piece 1 at 19-JAN-10
channel ORA_DISK_2: starting compressed full datafile backupset
channel ORA_DISK_2: specifying datafile(s) in backupset
channel ORA_DISK_3: starting compressed full datafile backupset
channel ORA_DISK_3: specifying datafile(s) in backupset
input datafile fno=00010 name=+DATADG1/pwhgrd20/datafile/mgmt_tablespace.276.641683063
input datafile fno=00017 name=+DATADG1/pwhgrd20/datafile/mgmt_tablespace.292.689859511
input datafile fno=00015 name=+ARCHDG1/pwhgrd20/datafile/sysaux.478.686918653
input datafile fno=00012 name=+DATADG1/pwhgrd20/datafile/bearaudit.282.645449195
input datafile fno=00004 name=+DATADG1/pwhgrd20/datafile/users.266.619034639
channel ORA_DISK_3: starting piece 1 at 19-JAN-10
channel ORA_DISK_4: starting compressed full datafile backupset
channel ORA_DISK_4: specifying datafile(s) in backupset
input datafile fno=00008 name=+DATADG1/pwhgrd20/datafile/mgmt_tablespace.277.629285163
input datafile fno=00013 name=+DATADG1/pwhgrd20/datafile/mgmt_tablespace.273.685143701
input datafile fno=00006 name=+DATADG1/pwhgrd20/datafile/mgmt_ecm_depot_ts.280.629709499
input datafile fno=00003 name=+DATADG1/pwhgrd20/datafile/sysaux.264.619034633
input datafile fno=00016 name=+ARCHDG1/pwhgrd20/datafile/mgmt_ecm_depot_ts.938.688263225
channel ORA_DISK_4: starting piece 1 at 19-JAN-10
including current SPFILE in backupset
including current control file in backupset
input datafile fno=00011 name=+DATADG1/pwhgrd20/datafile/mgmt_tablespace.287.657955169
input datafile fno=00009 name=+DATADG1/pwhgrd20/datafile/undotbs2.278.631236955
input datafile fno=00002 name=+DATADG1/pwhgrd20/datafile/undotbs1.263.619034627
input datafile fno=00001 name=+DATADG1/pwhgrd20/datafile/system.262.619034617
channel ORA_DISK_2: starting piece 1 at 19-JAN-10
channel ORA_DISK_3: finished piece 1 at 19-JAN-10
piece handle=/backup/oracle/bkup_tol3rslj_1_1 tag=TAG20100119T135458 comment=NONE
channel ORA_DISK_3: backup set complete, elapsed time: 00:21:48
channel ORA_DISK_4: finished piece 1 at 19-JAN-10
piece handle=/backup/oracle/bkup_tpl3rslj_1_1 tag=TAG20100119T135458 comment=NONE
channel ORA_DISK_4: backup set complete, elapsed time: 00:22:03
channel ORA_DISK_1: finished piece 1 at 19-JAN-10
piece handle=/backup/oracle/bkup_tml3rslj_1_1 tag=TAG20100119T135458 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:26:28
channel ORA_DISK_2: finished piece 1 at 19-JAN-10
piece handle=/backup/oracle/bkup_tnl3rslj_1_1 tag=TAG20100119T135458 comment=NONE
channel ORA_DISK_2: backup set complete, elapsed time: 00:28:43
Finished backup at 19-JAN-10


3) cancel MRP on standby side and ship the backup pieces to satndby site in a separate directory.

4) MAke a note of all control files,datafiles and satndby redologs if there. Als

5) connect to standby database using rman and catalog the backup pieces.

RMAN> catalog start with '/backup/oracle/incr_bkup';


6) recover standby database with noredo.

RMAN> recover database noredo.

7) make a new standby controlfile from primary and ship to satndby site. And now comes the different part as controlfiles and datafiles are in the ASM.

8) Shut down standby database.

sql> shut immediate; (make a backup of old standby control file now)

sql> startup nomount;

9) Now restore the new standby control file in to ASM DG from file system using rman.

rman> RESTORE STANDBY CONTROLFILE TO '' FROM '/tmp/ForStandbyCTRL.ctl';

ie rman > RESTORE STANDBY CONTROLFILE TO '+DATADG1/bnygrd20/controlfile/current.274.693183845' FROM '/tmp/ForStandbyCTRL.ctl';

10) If directory structure is same then the workis done just mount the database and start MRP.
But if primary and Standby datafile/redolog location is different then do following.

The new standby control file will be showing the datafiles locations of the primary now.

####
SQL> select name from v$datafile;

NAME
--------------------------------------------------------------------------------
+DATADG1/pwhgrd20/datafile/system.262.619034617
+DATADG1/pwhgrd20/datafile/undotbs1.263.619034627
+DATADG1/pwhgrd20/datafile/sysaux.264.619034633
+DATADG1/pwhgrd20/datafile/users.266.619034639
+DATADG1/pwhgrd20/datafile/mgmt_tablespace.279.629708905
+DATADG1/pwhgrd20/datafile/mgmt_ecm_depot_ts.280.629709499
+DATADG1/pwhgrd20/datafile/bearmon.268.619705149
+DATADG1/pwhgrd20/datafile/mgmt_tablespace.277.629285163
+DATADG1/pwhgrd20/datafile/undotbs2.278.631236955
+DATADG1/pwhgrd20/datafile/mgmt_tablespace.276.641683063
+DATADG1/pwhgrd20/datafile/mgmt_tablespace.287.657955169

NAME
--------------------------------------------------------------------------------
+DATADG1/pwhgrd20/datafile/bearaudit.282.645449195
+DATADG1/pwhgrd20/datafile/mgmt_tablespace.273.685143701
+ARCHDG1/pwhgrd20/datafile/mgmt_tablespace.994.686140129
+ARCHDG1/pwhgrd20/datafile/sysaux.478.686918653
+ARCHDG1/pwhgrd20/datafile/mgmt_ecm_depot_ts.938.688263225
+DATADG1/pwhgrd20/datafile/mgmt_tablespace.292.689859511

17 rows selected.
####

While the datafiles arre in ASM in differeent directory structure

###
ASMCMD [+] > ls
ARCHDG1/
DATADG1/
ASMCMD [+] > cd DATADG1/
ASMCMD [+DATADG1] > ls
BNYGRD20/
DB_UNKNOWN/
ASMCMD [+DATADG1] > cd BNYGRD20/
ASMCMD [+DATADG1/BNYGRD20] > ls
CONTROLFILE/
DATAFILE/
DATAGUARDCONFIG/
ONLINELOG/
TEMPFILE/
dr1.dat
dr2.dat
spfilebnygrd20.ora
ASMCMD [+DATADG1/BNYGRD20] > cd DATAFILE/
ASMCMD [+DATADG1/BNYGRD20/DATAFILE] > ls
BEARAUDIT.313.693183871
BEARMON.302.693183873
MGMT_ECM_DEPOT_TS.304.693183871
MGMT_TABLESPACE.269.693183869
MGMT_TABLESPACE.272.693183863
MGMT_TABLESPACE.273.693183865
MGMT_TABLESPACE.281.693183869
MGMT_TABLESPACE.307.693183867
MGMT_TABLESPACE.310.693183867
SYSAUX.315.693183871
SYSTEM.295.693183871
UNDOTBS1.293.693183871
UNDOTBS2.278.693183871
USERS.301.693183871

######


Starting MRP with this control file without modifying this file structure will give error like below when starting MRP.

###

Errors in file /opt/oracle/admin/PWHGRD20/bdump/pwhgrd21_mrp0_32314.trc:
ORA-01110: data file 1: '+DATADG1/pwhgrd20/datafile/system.262.619034617'
ORA-01157: cannot identify/lock data file 1 - see DBWR trace file
ORA-01110: data file 1: '+DATADG1/pwhgrd20/datafile/system.262.619034617'
Tue Jan 19 23:52:43 2010
MRP0: Background Media Recovery process shutdown (PWHGRD21)
###


11) So If the primary and standby database data file directories are different, then in RMAN, connect to the standby database, catalog the standby data files, and switch the standby database to use the just-cataloged data files. For example:

RMAN> CATALOG START WITH '+DATA_1/CHICAGO/DATAFILE/';
RMAN> SWITCH DATABASE TO COPY;



#########
$ rman target /

Recovery Manager: Release 10.2.0.4.0 - Production on Wed Jan 20 00:10:43 2010

Copyright (c) 1982, 2007, Oracle. All rights reserved.

connected to target database: PWHGRD20 (DBID=1812226798, not open)

RMAN> CATALOG START WITH '+DATADG1/BNYGRD20/DATAFILE/';

using target database control file instead of recovery catalog
searching for all files that match the pattern +DATADG1/BNYGRD20/DATAFILE/

List of Files Unknown to the Database
=====================================
File Name: +datadg1/BNYGRD20/DATAFILE/MGMT_TABLESPACE.272.693183863
File Name: +datadg1/BNYGRD20/DATAFILE/MGMT_TABLESPACE.273.693183865
File Name: +datadg1/BNYGRD20/DATAFILE/MGMT_TABLESPACE.310.693183867
File Name: +datadg1/BNYGRD20/DATAFILE/MGMT_TABLESPACE.307.693183867
File Name: +datadg1/BNYGRD20/DATAFILE/MGMT_TABLESPACE.269.693183869
File Name: +datadg1/BNYGRD20/DATAFILE/MGMT_TABLESPACE.281.693183869
File Name: +datadg1/BNYGRD20/DATAFILE/UNDOTBS2.278.693183871
File Name: +datadg1/BNYGRD20/DATAFILE/UNDOTBS1.293.693183871
File Name: +datadg1/BNYGRD20/DATAFILE/SYSTEM.295.693183871
File Name: +datadg1/BNYGRD20/DATAFILE/MGMT_ECM_DEPOT_TS.304.693183871
File Name: +datadg1/BNYGRD20/DATAFILE/BEARAUDIT.313.693183871
File Name: +datadg1/BNYGRD20/DATAFILE/SYSAUX.315.693183871
File Name: +datadg1/BNYGRD20/DATAFILE/USERS.301.693183871
File Name: +datadg1/BNYGRD20/DATAFILE/BEARMON.302.693183873

Do you really want to catalog the above files (enter YES or NO)? YES
cataloging files...
cataloging done

List of Cataloged Files
=======================
File Name: +datadg1/BNYGRD20/DATAFILE/MGMT_TABLESPACE.272.693183863
File Name: +datadg1/BNYGRD20/DATAFILE/MGMT_TABLESPACE.273.693183865
File Name: +datadg1/BNYGRD20/DATAFILE/MGMT_TABLESPACE.310.693183867
File Name: +datadg1/BNYGRD20/DATAFILE/MGMT_TABLESPACE.307.693183867
File Name: +datadg1/BNYGRD20/DATAFILE/MGMT_TABLESPACE.269.693183869
File Name: +datadg1/BNYGRD20/DATAFILE/MGMT_TABLESPACE.281.693183869
File Name: +datadg1/BNYGRD20/DATAFILE/UNDOTBS2.278.693183871
File Name: +datadg1/BNYGRD20/DATAFILE/UNDOTBS1.293.693183871
File Name: +datadg1/BNYGRD20/DATAFILE/SYSTEM.295.693183871
File Name: +datadg1/BNYGRD20/DATAFILE/MGMT_ECM_DEPOT_TS.304.693183871
File Name: +datadg1/BNYGRD20/DATAFILE/BEARAUDIT.313.693183871
File Name: +datadg1/BNYGRD20/DATAFILE/SYSAUX.315.693183871
File Name: +datadg1/BNYGRD20/DATAFILE/USERS.301.693183871
File Name: +datadg1/BNYGRD20/DATAFILE/BEARMON.302.693183873

RMAN> CATALOG START WITH '+ARCHDG1/BNYGRD20/DATAFILE/';

searching for all files that match the pattern +ARCHDG1/BNYGRD20/DATAFILE/

List of Files Unknown to the Database
=====================================
File Name: +archdg1/BNYGRD20/DATAFILE/MGMT_TABLESPACE.689.693183863
File Name: +archdg1/BNYGRD20/DATAFILE/MGMT_ECM_DEPOT_TS.631.693183871
File Name: +archdg1/BNYGRD20/DATAFILE/SYSAUX.393.693183871

Do you really want to catalog the above files (enter YES or NO)? YES
cataloging files...
cataloging done

List of Cataloged Files
=======================
File Name: +archdg1/BNYGRD20/DATAFILE/MGMT_TABLESPACE.689.693183863
File Name: +archdg1/BNYGRD20/DATAFILE/MGMT_ECM_DEPOT_TS.631.693183871
File Name: +archdg1/BNYGRD20/DATAFILE/SYSAUX.393.693183871

RMAN> switch database to copy;

datafile 1 switched to datafile copy "+DATADG1/bnygrd20/datafile/system.295.693183871"
datafile 2 switched to datafile copy "+DATADG1/bnygrd20/datafile/undotbs1.293.693183871"
datafile 3 switched to datafile copy "+DATADG1/bnygrd20/datafile/sysaux.315.693183871"
datafile 4 switched to datafile copy "+DATADG1/bnygrd20/datafile/users.301.693183871"
datafile 5 switched to datafile copy "+DATADG1/bnygrd20/datafile/mgmt_tablespace.307.693183867"
datafile 6 switched to datafile copy "+DATADG1/bnygrd20/datafile/mgmt_ecm_depot_ts.304.693183871"
datafile 7 switched to datafile copy "+DATADG1/bnygrd20/datafile/bearmon.302.693183873"
datafile 8 switched to datafile copy "+DATADG1/bnygrd20/datafile/mgmt_tablespace.272.693183863"
datafile 9 switched to datafile copy "+DATADG1/bnygrd20/datafile/undotbs2.278.693183871"
datafile 10 switched to datafile copy "+DATADG1/bnygrd20/datafile/mgmt_tablespace.310.693183867"
datafile 11 switched to datafile copy "+DATADG1/bnygrd20/datafile/mgmt_tablespace.273.693183865"
datafile 12 switched to datafile copy "+DATADG1/bnygrd20/datafile/bearaudit.313.693183871"
datafile 13 switched to datafile copy "+DATADG1/bnygrd20/datafile/mgmt_tablespace.281.693183869"
datafile 14 switched to datafile copy "+ARCHDG1/bnygrd20/datafile/mgmt_tablespace.689.693183863"
datafile 15 switched to datafile copy "+ARCHDG1/bnygrd20/datafile/sysaux.393.693183871"
datafile 16 switched to datafile copy "+ARCHDG1/bnygrd20/datafile/mgmt_ecm_depot_ts.631.693183871"
datafile 17 switched to datafile copy "+DATADG1/bnygrd20/datafile/mgmt_tablespace.269.693183869"

RMAN> exit


Recovery Manager complete.

#######

12) Now Start the MRP (database is already mounted)

It will pick correct file to apply th elogs.

SQL> alter database recover managed standby database disconnect from session;

Database altered.


RFS IDLE 0 0 0 0
RFS IDLE 0 0 0 0
MRP0 APPLYING_LOG 501841 237473 2 21956

Happy Learning!