We're here to help!

How do I eliminate a conflict between an RMAN backup and the Dbvisit Standby log transfer?

I have the latest Dbvisit Standby version installed. During last weekend, there was an RMAN failure saying as following, that it was due to some conflict with the Dbvisit Standby process. Please advise - can I run an RMAN backup during the Dbvisit Standby process?

Starting backup at 20-JAN-13
current log archived
released channel: CH1
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of backup command at 01-20-2013 17:08:13
RMAN-06059: expected archived log not found, lost of archived log compromises recoverability
ORA-19625: error identifying file H:\FLASH_RECOVERY_AREA\ORAPRD\ARCHIVELOG\2013_01_17\O1_MF_1_96309_8HKGPFP8_.ARC
ORA-27041: unable to open file
OSD-04002: unable to open file
O-S-Error: (OS 2) The system cannot find the file specified.

Recovery Manager complete.

And when I was doing an RMAN backup just now, I met this error:

channel CH1: starting piece 1 at 21-JAN-13
channel CH1: finished piece 1 at 21-JAN-13
piece handle=H:\BACKUP\RMAN\PDCO1NIS01_ARCH_ORAPRD_KCO001NK_4748_20130121 tag=TAG20130121T163739 comment=NONE
channel CH1: backup set complete, elapsed time: 00:00:33
channel CH1: starting compressed archive log backupset
released channel: CH1
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of backup command on CH1 channel at 01-21-2013 16:43:02
ORA-19588: archived log recid 96343 stamp 805066839 is no longer valid

Recovery Manager complete.

I believe that this is due to a Dbvisit Standby process conflicting with the RMAN backup.

In summary this is how it worked in versions before 6.0.50 when using compression (dbvisit, gzip):

When compression is set to gzip or dbvisit, and Dbvisit Standby is run on the primary nodes:

1. The archive logs are one by one, compressed.
2. The "original" archivelog is now a compressed log, meaning the original uncompressed file does not exist anymore, only the compressed version.
3. The compressed log is then transferred to the standby server.
4. Once the transfer is complete, if LEAVE_COMPRESS_SOURCE=N the archive log on the primary is uncompressed. If LEAVE_COMPRESS_SOURCE=Y it will just leave the archive log compressed and continue.

The issue now with RMAN is that if Dbvisit Standby runs at exactly the same time that RMAN is running and attempting to backup the archive logs, RMAN will only find the compressed archive log (only the one that is busy being transferred will be compressed) and you then get the RMAN failure. Effectively both Dbvisit Standby and RMAN are looking at the same log at the same time, but, as Dbvisit Standby has compressed it, RMAN cannot find it.

In version 6.0.50 we improved this mechanism and it is now recommended to set KEEP_ORIG_FILE_BEFORE_COMPRESS=Y. This is required if you are using COMPRESS=dbvisit or gzip and you have LEAVE_COMPRESS_SOURCE=N.

What will happen now, is when Dbvisit Standby runs, it will create a compressed version of the archive log, but will leave the original archive log in tact, so that if a backup is running it will pick it up. Once the transfer of the compressed file is finished, it will be removed if LEAVE_COMPRESS_SOURCE=N otherwise it will just remove the original file and keep the compressed file.

So if you are making use of RMAN backup in versions prior to 6.0.50, there is a possibility to run into this issue.

We strongly recommend that you upgrade to 6.0.50 and then make sure you have KEEP_ORIG_FILE_BEFORE_COMPRESS=Y.

It is possible that your Dbvisit Standby Archive Log Management Module (AMM) is configured to remove archive logs older than a specified time. This option is used in many sites where Dbvisit Standby is used to manage the archive log deletion. When RMAN is trying to backup the archive log, it picks up the archive log does not exist anymore and you could get the errors you experienced.

It is always good practice to ensure that all archive logs are backed up, and it is recommended you review your backup schedule as well as your Dbvisit Archivelog Management Module (AMM) settings to ensure you DO NOT delete logs before they are backed up.

Example: you can have Dbvisit Standby remove archive logs older than 2 days (or only remove them if at least one backup has been performed with RMAN) if you are running daily RMAN backups of the archive logs. For more details on the configuration of the AMM module please see the online documentation, HERE.

To validate the archive logs prior to executing the backup command you can run the following in RMAN:

rman>
rman> validate archivelog all;

You can also review the rman Crosscheck command.

Once you have run the validate archivelog all command you can retry the backup. But note if archive logs were deleted before being backed up, it could lead to not having sufficient backups to perform full recovery, if required.

Dbvisit Standby's AMM module and RMAN work well together, you just need to make sure you configure the AMM module to fit into your backup strategy to ensure logs are not removed before being backed up.

Mike Donovan July 07, 2013 23:18