We recently performed our first ever failover. I’ll cover the actual steps of the failover including those that bite us because the standbys were built only with the protecting the data but never really using those databases. After I successfully failed the database over to the physical standby I immediately started a level 0 backup. The backup ran with incident until the archivelogs. That’s when I received the dreaded archivelog not found message:
The interesting thing about the message was the archivelog that the backup balked on — it wasn’t a log from the current primary but the previous primary. Actually it was the most recent archivelog that was applied to the now current primary when it was the standby. I decided the first step would be a crosscheck:
rman target / nocatalog
crosscheck archivelog all;
I noticed right away it started with a directory 2009 and slowly scrolled through about 70k of archivelogs including the most recent on from the previous primary. The only archives that were found of course were those from the current primary. No worries it knows the files don’t exist and marked them as such. I started the archivelog backup again and it immediately failed same reason. So this time I decided to run a delete expired and delete obsolete.
delete archivelog expired;
delete archivelog obsolete;
The backup once again scrolled through 70k plus archive logs received the same error message that they were not found. Odd since these archivelogs contained the previous DBID and even showed they were the previous primary when reviewing the v$archived_log view.
I found the following note on My Oracle Support. I first set out to uncatalog one by one each archive. But as I quickly discovered that process would take forever even after I scripted the uncataloged. I was hoping I could perform the uncatalog at the directory level after all it is possible to catalog a directory and all the archivelogs are then registered. That would not be the case. I actually had to uncatalog all the archivelogs following the note. And then recataloged the archives. When I recataloged I did by the directory.
RMAN target / nocatalog — we choose the daring life using the controlfile instead of recovery catalog
catalog start with ‘/u02/app/oracle/archives’;
I restarted the backup and everyone is happy now.