一旦rhel5.8造成只读文件系统ORA-00354: corrupt redo log block header

Wed Jun 03 03:31:24 CST 2015Thread 1 advanced to log sequence 396 (LGWR switch)  Current log# 3 seq# 396 mem# 0: /u01/app/oracle/oradata/orcl/redo03.logWed Jun 03 19:05:48 CST 2015Thread 1 advanced to log sequence 397 (LGWR switch) ----------------->从这个地方能够看出来：sequence 397为current redo log  Current log# 1 seq# 397 mem# 0: /u01/app/oracle/oradata/orcl/redo01.logWed Jun 03 20:44:19 CST 2015Errors in file /u01/app/oracle/admin/orcl/bdump/orcl_j000_30071.trc:ORA-07445: exception encountered: core dump [kglats()+106] [SIGSEGV] [Address not mapped to object] [0x000000000] [] []ORA-01403: no data foundWed Jun 03 22:00:26 CST 2015Errors in file /u01/app/oracle/admin/orcl/bdump/orcl_j001_30285.trc:ORA-07445: exception encountered: core dump [kglhdiv_callback()+79] [SIGSEGV] [Address not mapped to object] [0x0000000D4] [] []Thu Jun 04 20:28:17 CST 2015Starting ORACLE instance (normal)

后来数据库实例重新启动（人为重新启动）：

Completed: ALTER DATABASE   MOUNTThu Jun 04 20:28:24 CST 2015ALTER DATABASE OPENThu Jun 04 20:28:24 CST 2015Beginning crash recovery of 1 threads parallel recovery started with 3 processesThu Jun 04 20:28:24 CST 2015Started redo scanThu Jun 04 20:28:24 CST 2015Completed redo scan 1210 redo blocks read, 287 data blocks need recoveryThu Jun 04 20:28:25 CST 2015Started redo application at Thread 1: logseq 397, block 660827Thu Jun 04 20:28:25 CST 2015Recovery of Online Redo Log: Thread 1 Group 1 Seq 397 Reading mem 0  Mem# 0: /u01/app/oracle/oradata/orcl/redo01.logThu Jun 04 20:28:25 CST 2015Completed redo applicationThu Jun 04 20:28:29 CST 2015Completed crash recovery at Thread 1: logseq 397, block 662037, scn 1359645 287 data blocks read, 287 data blocks written, 1210 redo blocks readThu Jun 04 20:28:30 CST 2015LGWR: STARTING ARCH PROCESSESARC0 started with pid=19, OS id=3609Thu Jun 04 20:28:30 CST 2015ARC0: Archival startedARC1: Archival startedLGWR: STARTING ARCH PROCESSES COMPLETEARC1 started with pid=20, OS id=3611Thu Jun 04 20:28:30 CST 2015Thread 1 advanced to log sequence 398 (thread open) --->已经使用sequence 398，此时，sequence 398为current redo logThread 1 opened at log sequence 398  Current log# 2 seq# 398 mem# 0: /u01/app/oracle/oradata/orcl/redo02.logSuccessful open of redo thread 1Thu Jun 04 20:28:30 CST 2015MTTR advisory is disabled because FAST_START_MTTR_TARGET is not setThu Jun 04 20:28:30 CST 2015ARC0: Becoming the 'no FAL' ARCHARC0: Becoming the 'no SRL' ARCHThu Jun 04 20:28:30 CST 2015SMON: enabling cache recoveryThu Jun 04 20:28:30 CST 2015ARC1: Becoming the heartbeat ARCHThu Jun 04 20:28:35 CST 2015ARC0: Log corruption near block 385521 change 1314765 time ?   --->发现redo log 有损坏Thu Jun 04 20:28:35 CST 2015Errors in file /u01/app/oracle/admin/orcl/bdump/orcl_arc0_3609.trc:ORA-00354: corrupt redo log block header --->发现redo log header有损坏ORA-00353: log corruption near block 385521 change 1314765 time 06/03/2015 19:05:47ORA-00312: online log 1 thread 1: '/u01/app/oracle/oradata/orcl/redo01.log'ARC0: All Archive destinations made inactive due to error 354Thu Jun 04 20:28:35 CST 2015ARC0: Closing local archive destination LOG_ARCHIVE_DEST_1: '/home/oracle/1_397_880950378.dbf' (error 354) (orcl)Committing creation of archivelog '/home/oracle/1_397_880950378.dbf' (error 354)Thu Jun 04 20:28:43 CST 2015Successfully onlined Undo Tablespace 1.Thu Jun 04 20:28:43 CST 2015SMON: enabling tx recoveryThu Jun 04 20:28:45 CST 2015ARCH: Archival stopped, error occurred. Will continue retryingThu Jun 04 20:28:45 CST 2015ORACLE Instance orcl - Archival ErrorThu Jun 04 20:28:45 CST 2015Database Characterset is ZHS16GBKThu Jun 04 20:28:45 CST 2015ORA-16038: log 1 sequence# 397 cannot be archivedORA-00354: corrupt redo log block headerORA-00312: online log 1 thread 1: '/u01/app/oracle/oradata/orcl/redo01.log'Thu Jun 04 20:28:45 CST 2015Errors in file /u01/app/oracle/admin/orcl/bdump/orcl_arc0_3609.trc:ORA-16038: log 1 sequence# 397 cannot be archivedORA-00354: corrupt redo log block headerORA-00312: online log 1 thread 1: '/u01/app/oracle/oradata/orcl/redo01.log'Opening with internal Resource Manager plan replication_dependency_tracking turned off (no async multimaster replication found)Starting background process QMNCQMNC started with pid=21, OS id=3637Thu Jun 04 20:28:49 CST 2015Completed: ALTER DATABASE OPEN ------->db open了。open归open，无法归档的话。业务早晚会hang。--->此时/u01/app/oracle/oradata/orcl/redo02.log，是sequence 398，是current redo log。

 

Thread 1 advanced to log sequence 399 (LGWR switch)  Current log# 3 seq# 399 mem# 0: /u01/app/oracle/oradata/orcl/redo03.log--->此时/u01/app/oracle/oradata/orcl/redo03.log，是sequence 399，是current redo log。

 

 

--->然后，sqlplus中发出了日志切换的语句：alter system switch logfile;Thread 1 cannot allocate new log, sequence 400---->已经不能进行日志切换，由于sequence 397未被归档，不同意被覆盖，此时db hang住。

All online logs needed archiving Current log# 3 seq# 399 mem# 0: /u01/app/oracle/oradata/orcl/redo03.log Thu Jun 04 21:28:31 CST 2015 ARC0: Log corruption near block 385521 change 1314765 time ? Thu Jun 04 21:28:31 CST 2015 Errors in file /u01/app/oracle/admin/orcl/bdump/orcl_arc0_3609.trc: ORA-00354: corrupt redo log block header ORA-00353: log corruption near block 385521 change 1314765 time 06/03/2015 19:05:47 ORA-00312: online log 1 thread 1: '/u01/app/oracle/oradata/orcl/redo01.log' ARC0: All Archive destinations made inactive due to error 354 Thu Jun 04 21:28:31 CST 2015 ARC0: Closing local archive destination LOG_ARCHIVE_DEST_1: '/home/oracle/1_397_880950378.dbf' (error 354) (orcl) ARCH: Archival stopped, error occurred. Will continue retrying Thu Jun 04 21:28:34 CST 2015 ORACLE Instance orcl - Archival Error Thu Jun 04 21:28:34 CST 2015 ORA-16038: log 1 sequence# 397 cannot be archived ORA-00354: corrupt redo log block header ORA-00312: online log 1 thread 1: '/u01/app/oracle/oradata/orcl/redo01.log' Thu Jun 04 21:28:34 CST 2015 Errors in file /u01/app/oracle/admin/orcl/bdump/orcl_arc0_3609.trc: ORA-16038: log 1 sequence# 397 cannot be archived ORA-00354: corrupt redo log block header ORA-00312: online log 1 thread 1: '/u01/app/oracle/oradata/orcl/redo01.log' ARCH: Archival stopped, error occurred. Will continue retrying Thu Jun 04 21:28:34 CST 2015 ORACLE Instance orcl - Archival Error Thu Jun 04 21:28:34 CST 2015 ORA-16014: log 1 sequence# 397 not archived, no available destinations ORA-00312: online log 1 thread 1: '/u01/app/oracle/oradata/orcl/redo01.log' Thu Jun 04 21:28:34 CST 2015 Errors in file /u01/app/oracle/admin/orcl/bdump/orcl_arc0_3609.trc: ORA-16014: log 1 sequence# 397 not archived, no available destinations ORA-00312: online log 1 thread 1: '/u01/app/oracle/oradata/orcl/redo01.log'

 

启动db到mount状态。进行处理

[oracle@lei1 ~]$ sqlplus / as sysdbaSQL*Plus: Release 10.2.0.5.0 - Production on Fri Jun 12 23:14:24 2015Copyright (c) 1982, 2010, Oracle.  All Rights Reserved.Connected to an idle instance.SQL> startup mount;ORACLE instance started.Total System Global Area 2147483648 bytesFixed Size                  2097696 bytesVariable Size             536874464 bytesDatabase Buffers         1593835520 bytesRedo Buffers               14675968 bytesDatabase mounted.SQL> set line 200SQL> select * from v$log;    GROUP#    THREAD#  SEQUENCE#      BYTES    MEMBERS ARC STATUS           FIRST_CHANGE# FIRST_TIM---------- ---------- ---------- ---------- ---------- --- ---------------- ------------- ---------         1          1        397  536870912          1 NO  INACTIVE               1294462 03-JUN-15         3          1        399  536870912          1 NO  CURRENT                1361682 04-JUN-15         2          1        398  536870912          1 YES INACTIVE               1359647 04-JUN-15SQL> alter database clear logfile group 1;alter database clear logfile group 1*ERROR at line 1:ORA-00350: log 1 of instance orcl (thread 1) needs to be archivedORA-00312: online log 1 thread 1: '/u01/app/oracle/oradata/orcl/redo01.log'SQL> alter database clear unarchived logfile group 1;Database altered.SQL> select * from v$log;    GROUP#    THREAD#  SEQUENCE#      BYTES    MEMBERS ARC STATUS           FIRST_CHANGE# FIRST_TIM---------- ---------- ---------- ---------- ---------- --- ---------------- ------------- ---------         1          1          0  536870912          1 YES UNUSED                 1294462 03-JUN-15         3          1        399  536870912          1 NO  CURRENT                1361682 04-JUN-15         2          1        398  536870912          1 YES INACTIVE               1359647 04-JUN-15SQL> archive log list;Database log mode              Archive ModeAutomatic archival             EnabledArchive destination            /home/oracleOldest online log sequence     0Next log sequence to archive   399Current log sequence           399SQL>

此时alert日志的输出例如以下：

Fri Jun 12 23:20:27 CST 2015alter database clear logfile group 1Fri Jun 12 23:20:27 CST 2015ORA-350 signalled during: alter database clear logfile group 1...Fri Jun 12 23:22:30 CST 2015alter database clear unarchived logfile group 1Fri Jun 12 23:22:31 CST 2015WARNING! CLEARING REDO LOG WHICH HAS NOT BEEN ARCHIVED. BACKUPS TAKEN    BEFORE 06/04/2015 20:28:30 (CHANGE 1359647) CANNOT BE USED FOR RECOVERY.Clearing online log 1 of thread 1 sequence number 397Fri Jun 12 23:22:43 CST 2015Completed: alter database clear unarchived logfile group 1