‘ora.mgmtdb’-ORA-09925: Unable to create audit trail file

Env : 12.1.0.2
A month ago i had my OS upgraded to OL-6. When you upgrade OS which has RAC cluster running, you may have to detach the GI prior to OS upgrade and add it back to the cluster after the OS upgrade. On this node i think i overlooked at some steps. All this while there were no issues reported and a month later i had to stop the cluster for another OS maintenance and upon a crsctl stop crs, the processes would not stop on Node-1.

[root@dv0013 rdbms]# /u01/app/12.1.0.2/grid/bin/crsctl stop crs
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'dv0013'
CRS-2673: Attempting to stop 'ora.crsd' on 'dv0013'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'dv0013'
CRS-2673: Attempting to stop 'ora.mgmtdb' on 'dv0013'
ORA-09925: Unable to create audit trail file
Linux-x86_64 Error: 13: Permission denied
Additional information: 9925
ORA-09925: Unable to create audit trail file
Linux-x86_64 Error: 13: Permission denied
Additional information: 9925
CRS-2675: Stop of 'ora.mgmtdb' on 'dv0013' failed
CRS-2673: Attempting to stop 'ora.mgmtdb' on 'dv0013'
^C

I then checked the file permissions under the GI home and *all* of them were with root:oinstall (ideally only few of them should be owned by root).

The mgmtdB audit destination is under GI/rdbms/ and that too is owned by root, meanwhile the SA pulled the cable for maintenance work, and later the cluster would not start.

2016-07-02 03:03:29.411 [ORAAGENT(18361)]CRS-5818: Aborted command 'start' for resource 'ora.evmd'. Details at (:CRSAGF00113:) {0:0:2} in /u01/app/oracle/diag/crs/dv0013/crs/trace/ohasd_oraagent_oracle.trc.
2016-07-02 03:03:29.945 [ORAAGENT(18361)]CRS-5017: The resource action "ora.evmd start" encountered the following error:
2016-07-02 03:03:29.945+Start action for daemon aborted. For details refer to "(:CLSN00107:)" in "/u01/app/oracle/diag/crs/dv0013/crs/trace/ohasd_oraagent_oracle.trc".
2016-07-02 03:03:29.945 [ORAAGENT(18361)]CRS-5818: Aborted command 'start' for resource 'ora.mdnsd'. Details at (:CRSAGF00113:) {0:0:2} in /u01/app/oracle/diag/crs/dv0013/crs/trace/ohasd_oraagent_oracle.trc.
2016-07-02 03:03:30.945 [ORAAGENT(18361)]CRS-5017: The resource action "ora.mdnsd start" encountered the following error:
2016-07-02 03:03:30.945+Start action for daemon aborted. For details refer to "(:CLSN00107:)" in "/u01/app/oracle/diag/crs/dv0013/crs/trace/ohasd_oraagent_oracle.trc".
2016-07-02 03:03:33.413 [OHASD(29537)]CRS-2757: Command 'Start' timed out waiting for response from the resource 'ora.evmd'. Details at (:CRSPE00163:) {0:0:2} in /u01/app/oracle/diag/crs/dv0013/crs/trace/ohasd.trc.
2016-07-02 03:03:33.947 [OHASD(29537)]CRS-2757: Command 'Start' timed out waiting for response from the resource 'ora.mdnsd'. Details at (:CRSPE00163:) {0:0:2} in /u01/app/oracle/diag/crs/dv0013/crs/trace/ohasd.trc.
2016-07-02 03:03:34.011 [ORAAGENT(23911)]CRS-8500: Oracle Clusterware ORAAGENT process is starting with operating system process ID 23911
2016-07-02 03:03:34.096 [ORAAGENT(23924)]CRS-8500: Oracle Clusterware ORAAGENT process is starting with operating system process ID 23924
2016-07-02 03:03:34.175 [GPNPD(23935)]CRS-8500: Oracle Clusterware GPNPD process is starting with operating system process ID 23935
2016-07-02 03:03:34.204 [GPNPD(23935)]CRS-2329: GPNPD on node dv0013 shut down.
2016-07-02 03:03:34.204 [GPNPD(23935)]CRS-8504: Oracle Clusterware GPNPD process with operating system process ID 23935 is exiting
2016-07-02 03:13:34.130 [ORAAGENT(23924)]CRS-5818: Aborted command 'start' for resource 'ora.gpnpd'. Details at (:CRSAGF00113:) {0:0:2} in /u01/app/oracle/diag/crs/dv0013/crs/trace/ohasd_oraagent_oracle.trc.
2016-07-02 03:13:34.625 [ORAAGENT(23924)]CRS-5017: The resource action "ora.gpnpd start" encountered the following error:
2016-07-02 03:13:34.625+Start action for daemon aborted. For details refer to "(:CLSN00107:)" in "/u01/app/oracle/diag/crs/dv0013/crs/trace/ohasd_oraagent_oracle.trc".
2016-07-02 03:13:38.132 [OHASD(29537)]CRS-2757: Command 'Start' timed out waiting for response from the resource 'ora.gpnpd'. Details at (:CRSPE00163:) {0:0:2} in /u01/app/oracle/diag/crs/dv0013/crs/trace/ohasd.trc.
2016-07-02 03:13:38.188 [ORAAGENT(29342)]CRS-8500: Oracle Clusterware ORAAGENT process is starting with operating system process ID 29342

Now that i need to fix the permission issue, else i would not be able to start the cluster.

1. Stop CRS (as root)
/u01/app/12.1.0.2/grid/crs/install/rootcrs.pl -unlock

2. Modify permissions (as root)
cd /u01/app/12.1.0.2/
chown -R oracle:oinstall grid

3. Relink GI (as oracle)
export ORACLE_HOME=/u01/app/12.1.0.2/grid/
time $ORACLE_HOME/bin/relink all

4. start crs using(as root) 
/u01/app/12.1.0.2/grid/crs/install/rootcrs.pl -patch

Bingo !!!, it started the cluster.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s