当NFS server 宕机后，Oracle 数据库 冻结并且alert 文件里没有任何错误
翻译自mos文章：When NFS Server Is Down, Oracle Server Freezes With No Errors In Alert Log File (文档 ID 1316251.1)
适用于：Oracle Server – Enterprise Edition – Version: 10.2.0.4 and later [Release: 10.2 and later ]IBM AIX on POWER Systems (64-bit)症状：AIX上的Oracle instance 有一个NFS 挂载点，该挂载点基于backup 的目的。 该挂载点mount时的选项如下：
bg,hard,intr,rsize=32768,wsize=32768,sec=sys,noac,rw
当NFS Server 宕机时，Ooracle RDMBS 冻结，并且alert 日志中没有任何错误。当NFS Sserver 恢复后，database 也正常工作，没有任何问题。
改变：环境没有改变，仅仅是丢失了NAS connectivity（to NFS Server）,因此远程的目录不可访问。原因：
从上传的sqlplus 和df 的 tusss跟踪来看，我们可以看到statx 命令在 /backup处挂起。
462940: statx("./../../../../backup", 0x0FFFFFFFFFFF5980, 176, 021) (sleeping…)561338: kread(14, " ÿ ÿ J ø\0\0\0\0\0\0\010".., 64) Err#82 ERESTART561338: Received signal #2, SIGINT [caught]561338: sigprocmask(0, 0x0FFFFFFFFFFF3620, 0x0000000000000000) = 0561338: sigprocmask(1, 0x0FFFFFFFFFFF3620, 0x0000000000000000) = 0561338: ksetcontext_sigreturn(0x0FFFFFFFFFFF37A0, 0x0000000000000000, 0x00000001100F04F0,0x800000000000D032, 0x3000000000000000, 0x0000000000000360, 0x0000000000000000, 0x0000000000000000)561338: kread(14, " ÿ ÿ J ø\0\0\0\0\0\0\010".., 64) Err#82 ERESTART561338: Received signal #2, SIGINT [caught]561338: sigprocmask(0, 0x0FFFFFFFFFFF3620, 0x0000000000000000) = 0561338: sigprocmask(1, 0x0FFFFFFFFFFF3620, 0x0000000000000000) = 0561338: ksetcontext_sigreturn(0x0FFFFFFFFFFF37A0, 0x0000000000000000, 0x00000001100F04F0,0x800000000000D032, 0x3000000000000000, 0x0000000000000320, 0x0000000000000000, 0x0000000000000000)561338: kread(14, " ÿ ÿ J ø\0\0\0\0\0\0\010".., 64) Err#82 ERESTART561338: Received signal #2, SIGINT [caught]561338: sigprocmask(0, 0x0FFFFFFFFFFF3620, 0x0000000000000000) = 0561338: sigprocmask(1, 0x0FFFFFFFFFFF3620, 0x0000000000000000) = 0561338: ksetcontext_sigreturn(0x0FFFFFFFFFFF37A0, 0x0000000000000000, 0x00000001100F04F0,0x800000000000D032, 0x3000000000000000, 0x0000000000000310, 0x0000000000000000, 0x0000000000000000)561338: kread(14, " ÿ ÿ J ø\0\0\0\0\0\0\010".., 64) Err#82 ERESTART561338: Received signal #2, SIGINT [caught]561338: sigprocmask(0, 0x0FFFFFFFFFFF3620, 0x0000000000000000) = 0561338: sigprocmask(1, 0x0FFFFFFFFFFF3620, 0x0000000000000000) = 0561338: ksetcontext_sigreturn(0x0FFFFFFFFFFF37A0, 0x0000000000000000, 0x00000001100F04F0,0x800000000000D032, 0x3000000000000000, 0x0000000000000310, 0x0000000000000000, 0x0000000000000000)561338: kread(14, " ÿ ÿ J ø\0\0\0\0\0\0\010".., 64) Err#82 ERESTART561338: Received signal #2, SIGINT [caught]561338: sigprocmask(0, 0x0FFFFFFFFFFF3620, 0x0000000000000000) = 0561338: sigprocmask(1, 0x0FFFFFFFFFFF3620, 0x0000000000000000) = 0561338: ksetcontext_sigreturn(0x0FFFFFFFFFFF37A0, 0x0000000000000000, 0x00000001100F04F0,0x800000000000D032, 0x3000000000000000, 0x0000000000000320, 0x0000000000000000, 0x0000000000000000)561338: kread(14, " ÿ ÿ J ø\0\0\0\0\0\0\010".., 64) (sleeping…)462940: statx("./../../../../backup", 0x0FFFFFFFFFFF5980, 176, 021) = 0462940: statx("./../../../../usr", 0x0FFFFFFFFFFF5980, 176, 021) = 0462940: statx("./../../../../lib", 0x0FFFFFFFFFFF5980, 176, 021) = 0462940: statx("./../../../../audit", 0x0FFFFFFFFFFF5980, 176, 021) = 0462940: statx("./../../../../dev", 0x0FFFFFFFFFFF5980, 176, 021) = 0462940: statx("./../../../../etc", 0x0FFFFFFFFFFF5980, 176, 021) = 0462940: statx("./../../../../u", 0x0FFFFFFFFFFF5980, 176, 021) = 0462940: statx("./../../../../lpp", 0x0FFFFFFFFFFF5980, 176, 021) = 0462940: statx("./../../../../mnt", 0x0FFFFFFFFFFF5980, 176, 021) = 0462940: statx("./../../../../proc", 0x0FFFFFFFFFFF5980, 176, 021) = 0462940: statx("./../../../../sbin", 0x0FFFFFFFFFFF5980, 176, 021) = 0462940: statx("./../../../../bin", 0x0FFFFFFFFFFF5980, 176, 021) = 0462940: statx("./../../../../oracle", 0x0FFFFFFFFFFF5980, 176, 021) = 0
问题在下面的地方：
statx("./../../../../backup", 0x0FFFFFFFFFFF5980, 176, 021) (sleeping…)
Oracle 程序（代码）调用一个Unix 系统调用（system call）， ‘getcwd’来得到当前的工作目录。在这之后，所有的控制权归还给操作系统。从我们所看到的， 函数’getcwd’ 调用’getwd’，，而’getwd’会依次调用’statx’。一旦’statx’被执行，它就按照 下面的顺序 通过执行’statx’ 开始处理directory entries
././…/../…/../../.. (this goes on until the root directory is reached)
一旦 root 目录（/）被到达，那么对于目录中的每个entry，’lstat’会调用 ‘statx’。Oracle 完全不控制这个处理过程，因此为了防止此情况发生，我们做不了任何事情（这完全是os级别的事情）
解决方法：从一个类似问题中，IBM 已经建议下面的 action plan 以避免这个问题。来自IBM的回答是：
Here's a solution to avoid the problem described by Oracle: DO NOT have the NFS mounts directly under /, but put them one level lower. Then, we can use symbolic links to them.NFS mount point on node /nfs/backup (/nfs is a directory we'll create, it can have any name) and create a softlink /backup -> /nfs/backup. $ ln -s /nfs/backup /backupThis will avoid the statx problem without having to make changes in the setup (because /backup is still there).Additionally you can ask IBM about APAR # IZ85027, IZ85029, IZ85032, IZ86102, IZ87374, IZ90533.Check with IBM which one applies to your configuration.
天下没有不散的宴席，也许这人间真的只有朦朦胧胧才是真。
