ORA-10997 and ORA-09968 on ASM startup (With NFS)

During a recent hardware upgrade, i received the following error on a ASM (NO-RAC) startup.   It was very confusing, because i was custom to seeing ASM on Storage LUNS and in this case it was using ASM (NO-RAC) and NFS storage.   Unfortunately, as always with late night changes, you don’t notice things until you get fresh eyes on the issue.   A simple “df -kh” would’ve told me that it was NFS, but made too many presumptions.

When i first told about the NFS settings, i thought i needed to restart nfs, but that didn’t work.   Than after getting another fresh pair of eyes from the Linux team, we realized that the directory “/var/lib/nfs” was set to 750???   After some investigation, we saw that puppet was changing it to 750 every 5 minutes, so even when we changed it to 755, it was 750.  so the next question, why ?

Well it appears that FTI requirements are the culprit.  Apparently the auditors don’t understand nfs and O/S’s, probably don’t care either.   Apparently, FTI companies are required to set this to 750…. Now i understand security, but this directory needs to be readable by Oracle to start up the nfs drives and ASM.

Until we get a permanent solution, the linux team offered the following solution:

the setting of 755 is only needed for startup, so the temporary solution is (as root):

root> chmod 755 /var/lib/nfs
root> chatter +i /var/lib/nfs <== protects
root> crsctl start has
== after successful startup ====
root> chatter -i /var/lib/nfs <== takes off the protection

The following is the error description:

grid> srvctl start asm

PRCR-1079 : Failed to start resource ora.asm
CRS-5017: The resource action "ora.asm start" encountered the following error:
ORA-10997: another startup/shutdown operation of this instance in progress
ORA-09968: unable to lock file
Linux-x86_64 Error: 37: No locks available
Additional information: 64
. For details refer to "(:CLSN00107:)" in "/u01/app/grid/product/<machine_name>/agent/ohasd/oraagent_grid/oraagent_grid.log".
CRS-2674: Start of 'ora.asm' on '<machine name>' failed

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s