今天西北某电信客户的一个系统有点小问题，无法切换归档日志。通过临时将归档路径都放到本地后，切换ok。检查发现之前的arch目录有点问题，如下：
[root@dtdb1 arch]# ls -ltr ls: cannot access 1_3883_855680193.dbf: Invalid argument ls: cannot access 2_2408_855680193.dbf: No such file or directory ls: cannot access 2_2408_855680193.dbf: No such file or directory ls: cannot access 1_4776_855680193.dbf: Invalid argument total 7521476 -????????? ? ? ? ? ? 2_2408_855680193.dbf -????????? ? ? ? ? ? 2_2408_855680193.dbf -????????? ? ? ? ? ? 1_4776_855680193.dbf -????????? ? ? ? ? ? 1_3883_855680193.dbf drwx------ 2 oracle asmadmin 65536 Nov 5 2014 lost+found drwxr-xr-x 4 oracle asmadmin 4096 Nov 5 2014 backup1 -rw-r----- 1 oracle asmadmin 23993856 Jan 1 10:06 2_2461_855680193.dbf -rw-r----- 1 oracle asmadmin 460232704 Jan 1 10:06 1_4858_855680193.dbf -rw-r----- 1 oracle asmadmin 460146176 Jan 1 12:26 1_4859_855680193.dbf -rw-r----- 1 oracle asmadmin 460609536 Jan 1 22:05 1_4860_855680193.dbf ......
我们可以看到有几个归档文件的属主、权限等信息都是问号，而且都无法删除。通过VNC登录查看该目录下的这几个文件都不存在，这确实感觉很怪异。通过rm -rf 删除报错，如下所示：
[root@dtdb1 arch]# rm -rf 1_4776_* rm: cannot remove `1_4776_855680193.dbf': Invalid argument
经过查询发生该arch本质上是acfs集群文件系统，如下：
[grid@dtdb1 ~]$ asmcmd
ASMCMD> lsdg
State Type Rebal Sector Block AU Total_MB Free_MB Req_mir_free_MB Usable_file_MB Offline_disks Voting_files Name
MOUNTED EXTERN N 512 4096 1048576 307196 920 0 920 0 N ARCHVG/
MOUNTED NORMAL N 512 4096 1048576 3105 2179 1035 572 0 Y CRS/
MOUNTED EXTERN N 512 4096 4194304 375752 4572 0 4572 0 N DATA01/
MOUNTED EXTERN N 512 4096 1048576 204797 1902 0 1902 0 N DATA02/
MOUNTED EXTERN N 512 4096 4194304 389104 158476 0 158476 0 N DATA03/
MOUNTED EXTERN N 512 4096 4194304 30724 27380 0 27380 0 N FRA01/
ASMCMD> cd archvg
ASMCMD> ls
ASMCMD> volinfo -a
Diskgroup Name: ARCHVG
Volume Name: ARCHVOL
Volume Device: /dev/asm/archvol-9
State: ENABLED
Size (MB): 306176
Resize Unit (MB): 32
Redundancy: UNPROT
Stripe Columns: 4
Stripe Width (K): 128
Usage: ACFS
Mountpath: /arch
首先将arch下面的文件都先tar 备份一下，然后准备重建下这个acfs文件系统，如下是重建的步骤：
1、删除原diskgroup
[grid@dtdb1 ~]$ sqlplus "/as sysasm" SQL*Plus: Release 11.2.0.3.0 Production on Sun Jan 3 12:43:55 2016 Copyright (c) 1982, 2011, Oracle. All rights reserved. Connected to: Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production With the Real Application Clusters and Automatic Storage Management options SQL> drop diskgroup archvg including contents; drop diskgroup archvg including contents * ERROR at line 1: ORA-15039: diskgroup not dropped ORA-15027: active use of diskgroup "ARCHVG" precludes its dismount SQL> alter diskgroup archvg dismount force; Diskgroup altered. SQL> drop diskgroup archvg including contents ; drop diskgroup archvg including contents * ERROR at line 1: ORA-15039: diskgroup not dropped ORA-15027: active use of diskgroup "ARCHVG" precludes its dismount SQL> alter diskgroup archvg dismount force; Diskgroup altered. SQL> drop diskgroup archvg force including contents ; Diskgroup dropped.
2、重建diskgroup
SQL> CREATE DISKGROUP archvg EXTERNAL REDUNDANCY DISK 'ORCL:ARCHVOL01' SIZE 306176M ATTRIBUTE 'compatible.asm' = '11.2', 'compatible.advm' = '11.2'; Diskgroup created. SQL> exit Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production With the Real Application Clusters and Automatic Storage Management options
3、创建卷组
[grid@dtdb1 ~]$ asmcmd volcreate -G archvg -s 299g ARCHVOL
ORA-15032: not all alterations performed
ORA-15041: diskgroup "ARCHVG" space exhausted (DBD ERROR: OCIStmtExecute)
[grid@dtdb1 ~]$ asmcmd volcreate -G archvg -s 297g ARCHVOL
[grid@dtdb1 ~]$ asmcmd volinfo -a
Diskgroup Name: ARCHVG
Volume Name: ARCHVOL
Volume Device: /dev/asm/archvol-396
State: ENABLED
Size (MB): 304128
Resize Unit (MB): 32
Redundancy: UNPROT
Stripe Columns: 4
Stripe Width (K): 128
Usage:
Mountpath:
4、格式acfs文件系统
[grid@dtdb1 ~]$ /sbin/mkfs -t acfs /dev/asm/archvol-396 mkfs.acfs: version = 11.2.0.3.0 mkfs.acfs: on-disk version = 39.0 mkfs.acfs: volume = /dev/asm/archvol-396 mkfs.acfs: volume size = 318901321728 mkfs.acfs: Format complete. [root@dtdb1 ~]# ls -ltr /dev/asm total 0 brwxrwx--- 1 root asmadmin 252, 202753 Jan 3 13:17 archvol-396
5、在其他节点激活卷组
[root@dtdb2 u01]# su - grid [grid@dtdb2 ~]$ ls -ltr /dev/asm total 0 [grid@dtdb2 ~]$ sqlplus "/as sysasm" SQL*Plus: Release 11.2.0.3.0 Production on Sun Jan 3 13:28:51 2016 Copyright (c) 1982, 2011, Oracle. All rights reserved. Connected to: Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production With the Real Application Clusters and Automatic Storage Management options SQL> alter diskgroup archvg enable volume 'ARCHVOL'; Diskgroup altered. SQL> exit Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production With the Real Application Clusters and Automatic Storage Management options [grid@dtdb2 ~]$ ls -ltr /dev/asm total 0 brwxrwx--- 1 root asmadmin 252, 202753 Jan 3 13:28 archvol-396
说明：如果节点2不激活，那么无法看到磁盘，需要手工激活。如果是通过asmca图形化界面来创建acfs不会有这个问题。
6、所有节点mount acfs文件系统
[root@dtdb2 u01]# mount -t acfs /dev/asm/archvol-396 /arch
[root@dtdb2 u01]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/rootvg-lv02
12G 450M 11G 4% /
tmpfs 30G 891M 29G 3% /dev/shm
/dev/sda1 388M 38M 331M 11% /boot
/dev/mapper/rootvg-lv03
6.0G 426M 5.2G 8% /home
/dev/mapper/rootvg-lv07
5.0G 168M 4.6G 4% /opt
/dev/mapper/rootvg-lv04
6.0G 203M 5.5G 4% /tmp
/dev/mapper/datavg-datalv01
50G 28G 19G 60% /u01
/dev/mapper/rootvg-lv06
7.9G 4.0G 3.6G 53% /usr
/dev/mapper/rootvg-lv05
7.9G 452M 7.1G 6% /var
/dev/mapper/rootvg-netbackuplv
9.9G 8.2G 1.2G 88% /usr/openv
/dev/asm/archvol-396 297G 668M 297G 1% /arch
[root@dtdb2 arch]# chown -R oracle:asmadmin /arch
之前检查/var/log/messages有不少错误，怀疑可能是acfs的问题，通过重建后不再出现任何错误.
发表回复