CRS-0223: Resource has placement error while using SRVCTL

Few days back in the early wee hours, one of my RAC node db instance crashed.
Env: 10.2.0.4 RAC on Solaris 5.10 SPARC
alert.log said:

:
Errors in file /u01/app/oracle/admin/SHCL1/bdump/shcl1n02_j004_27414.trc:
ORA-07445: exception encountered: core dump [kglobcl()+412] [SIGSEGV] [Address not mapped to object] [0x49415C002] [] []
Tue Nov  3 00:00:27 2009
Trace dumping is performing id=[cdmp_20091103000027]
Tue Nov  3 00:00:37 2009
Errors in file /u01/app/oracle/admin/SHCL1/bdump/shcl1n02_pmon_3278.trc:
ORA-07445: [kglobcl()+412] [SIGSEGV] [Address not mapped to object] [0x49415C002] [] []
Tue Nov  3 00:00:51 2009
:
:
Tue Nov  3 00:00:54 2009
MMAN: terminating instance due to error 472
Instance terminated by MMAN, pid = 3368
Wed Nov  4 10:44:47 2009
:

I checked

$ crsctl check crs
OK (running successfully)
$ srvctl status nodeapps -n myjpsuolicdbd02
OK (all resources running successfully)
$ srvctl status database -d SHCL1_PRMY
Instance SHCL1N01 is running on node myjpsuolicdbd01
PRKO-2015 : Error in checking condition of instance on node: myjpsuolicdbd02

$

Since only instance 2 was down, i tried to start it and i got the below error.

$ srvctl start instance -d SHCL1_PRMY -i SHCL1N02
PRKP-1001 : Error starting instance SHCL1N02 on node myjpsuolicdbd02
CRS-1028: Dependency analysis failed because of:
CRS-0223: Resource ‘ora.SHCL1_prmy.SHCL1N02.inst’ has placement error.

$

The crsd.log file too didn’t have much(to the point) information.

2009-11-04 10:18:30.627: [  CRSRES][2970821] CRS-1028: Dependency analysis failed because of:'Resource in UNKNOWN state: ora.SHCL1_prmy.SHCL1N02.inst'

2009-11-04 10:21:23.269: [  CRSRES][2970843] StopResource: setting CLI values
2009-11-04 10:21:23.340: [  CRSRES][2970843] Attempting to stop `ora.SHCL1_prmy.SHCL1N02.inst` on member `myjpsuolicdbd02`
2009-11-04 10:21:30.478: [  CRSAPP][2970843] StopResource error for ora.SHCL1_prmy.SHCL1N02.inst error code = 1
2009-11-04 10:21:30.502: [  CRSRES][2970843] Stop of `ora.SHCL1_prmy.SHCL1N02.inst` on member `myjpsuolicdbd02` succeeded.
2009-11-04 10:21:49.867: [  CRSRES][2970861] startRunnable: setting CLI values
2009-11-04 10:21:49.895: [  CRSRES][2970861] Attempting to start `ora.SHCL1_prmy.SHCL1N02.inst` on member `myjpsuolicdbd02`
2009-11-04 10:21:55.019: [  CRSAPP][2970861] StartResource error for ora.SHCL1_prmy.SHCL1N02.inst error code = 1
2009-11-04 10:22:00.583: [  CRSAPP][2970861] StopResource error for ora.SHCL1_prmy.SHCL1N02.inst error code = 1
2009-11-04 10:22:00.592: [  CRSRES][2970861] X_OP_StopResourceFailed : Stop Resource failed
(File: rti.cpp, line: 1803

2009-11-04 10:22:00.593: [  CRSRES][2970861][ALERT] `ora.SHCL1_prmy.SHCL1N02.inst` on member `myjpsuolicdbd02` has experienced an unrecoverable failure.
2009-11-04 10:22:00.593: [  CRSRES][2970861] Human intervention required to resume its availability.

Clusterware and nodeapps were up and running but db instance on node 2.

I could startup the instance from sqlplus, but this would not start the db services etc ora.SHCL1_prmy.SHCL1N02.inst was not ONLINE and would be of no use in a RAC.
Didn’t have any clue, until I came across the logs under /u01/app/oracle/product/10.2.0/db_1/log/<hostname>/racg/imonSHCL1_prmy.log and imon_SHCL1_prmy.log.

The files imonSHCL1_prmy.log and imon_SHCL1_prmy.log contains details of the starting up of the DB using SRVCTL command.

SQL*Plus: Release 10.2.0.4.0 - Production on Wed Nov 4 18:59:48 2009
Copyright (c) 1982, 2007, Oracle.  All Rights Reserved.

Enter user-name: ERROR:
ORA-01031: insufficient privileges

Enter user-name: SP2-0306: Invalid option.

2009-11-04 18:59:48.968: [    RACG][176] [28813][176][ora.SHCL1_prmy.SHCL1N02.inst]: Usage: CONN[ECT] [logon] [AS {SYSDBA|SYSOPER}]
where <logon>  ::= <username>[/<password>][@<connect_identifier>] | /
Enter user-name: Enter password:
ERROR:
ORA-01005: null password given; logon denied

2009-11-04 18:59:48.968: [    RACG][176] [28813][176][ora.SHCL1_prmy.SHCL1N02.inst]: SP2-0157: unable to CONNECT to ORACLE after 3 attempts, exiting SQL*Plus
2009-11-04 18:59:48.968: [    RACG][176] [28813][176][ora.SHCL1_prmy.SHCL1N02.inst]: clsrcexecut: env _USR_ORA_PFILE=
2009-11-04 18:59:48.968: [    RACG][176] [28813][176][ora.SHCL1_prmy.SHCL1N02.inst]: clsrcexecut: cmd = /u01/app/oracle/product/10.2.0/db_1/bin/racgeut -e _USR_ORA_DEBUG=0 -e ORACLE_SID=SHCL1N02 520 /u01/app/oracle/product/10.2.0/db_1/bin/racgmdb -d abort

 

CAUSE:

Due to security concern, recently i added “ SQLNET.AUTHENTICATION_SERVICES=(NONE) ” to sqlnet.ora and this line would insist on providing the password explicitly. It doesn’t allow to login using “sqlplus / as sysdba”. Hence the SRVCTL failed.

So,

After removing this line from sqlnet.ora, i could start the instance.

 

SOLUTION:

1)Commented the entry in sqlnet.ora
2)crs_stat -u => check for  “ora.SHCL1_prmy.SHCL1N02.inst”, the STATE should be OFFLINE before we ‘start’ again.

NAME=ora.SHCL1_prmy.SHCL1N02.inst
TYPE=application
TARGET=ONLINE
STATE=UNKNOWN on myjpsuolicdbd02

If the STATE is UNKNOWN, ‘start’ing the instance would error out. So stop it by force.

2)crs_stop -f ora.SHCL1_prmy.SHCL1N02.inst
3)crs_stat -u => to confirm that the TARGET is ONLINE and STATE is OFFLINE for ora.SHCL1_prmy.SHCL1N02.inst
4)srvctl start instance -d SHCL1_PRMY -i SHCL1N02
Raised an SR for the ORA 7445 error and let me see what they have in stock for me

Advertisements

2 thoughts on “CRS-0223: Resource has placement error while using SRVCTL

  1. SRVCTL does not work when RACGIMON process cannot connect to the DB [ID 391453.1]

    Thank you,
    Chandrasekar.

  2. Pingback: Index « My confrontations with oracle

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s