nfs

During a recent hardware upgrade, i received the following error on a ASM (NO-RAC) startup. It was very confusing, because i was custom to seeing ASM on Storage LUNS and in this case it was using ASM (NO-RAC) and NFS storage. Unfortunately, as always with late night changes, you don’t notice things until you get fresh eyes on the issue. A simple “df -kh” would’ve told me that it was NFS, but made too many presumptions.

When i first told about the NFS settings, i thought i needed to restart nfs, but that didn’t work. Than after getting another fresh pair of eyes from the Linux team, we realized that the directory “/var/lib/nfs” was set to 750??? After some investigation, we saw that puppet was changing it to 750 every 5 minutes, so even when we changed it to 755, it was 750. so the next question, why ?

Well it appears that FTI requirements are the culprit. Apparently the auditors don’t understand nfs and O/S’s, probably don’t care either. Apparently, FTI companies are required to set this to 750…. Now i understand security, but this directory needs to be readable by Oracle to start up the nfs drives and ASM.

Until we get a permanent solution, the linux team offered the following solution:

the setting of 755 is only needed for startup, so the temporary solution is (as root):

root> chmod 755 /var/lib/nfs
root> chatter +i /var/lib/nfs <== protects
root> crsctl start has
== after successful startup ====
root> chatter -i /var/lib/nfs <== takes off the protection

The following is the error description:

grid> srvctl start asm

PRCR-1079 : Failed to start resource ora.asm
CRS-5017: The resource action "ora.asm start" encountered the following error:
ORA-10997: another startup/shutdown operation of this instance in progress
ORA-09968: unable to lock file
Linux-x86_64 Error: 37: No locks available
Additional information: 64
. For details refer to "(:CLSN00107:)" in "/u01/app/grid/product/11.2.0.4/grid_home/log/<machine_name>/agent/ohasd/oraagent_grid/oraagent_grid.log".
CRS-2674: Start of 'ora.asm' on '<machine name>' failed