Simon Krenger

So today, during the upgrade of a database, all other databases using the same ORACLE_HOME stopped dead in their tracks. The ALERT log reported the following:

2014-09-30 13:34:41.622000 +02:00
Archived Log entry 24398 added for thread 1 sequence 24398 ID 0x2d91d796 dest 1:
2014-09-30 13:47:39.764000 +02:00
Errors in file /u00/app/oracle/diag/rdbms/kdb12345/kdb12345/trace/kdb12345_psp0_57147560.trc:
ORA-07274: spdcr: access error, access to oracle denied.
IBM AIX RISC System/6000 Error: 2: No such file or directory
PSP0 (ospid: 57147560): terminating the instance due to error 7274
2014-09-30 13:47:41.346000 +02:00
System state dump requested by (instance=1, osid=57147560 (PSP0)), summary=[abnormal instance termination].
System State dumped to trace file /u00/app/oracle/diag/rdbms/kdb12345/KDB12345/trace/KDB12345_diag_49938448_20140930134741.trc
2014-09-30 13:47:44.779000 +02:00
Instance terminated by PSP0, pid = 57147560

Hmm, so what to do? In the trace file, we find out a bit more:

error 7274 detected in background process
ORA-07274: spdcr: access error, access to oracle denied.
IBM AIX RISC System/6000 Error: 2: No such file or directory
kjzduptcctx: Notifying DIAG for crash event
----- Abridged Call Stack Trace -----
ksedsts()+240<-kjzdssdmp()+240<-kjzduptcctx()+228<-kjzdicrshnfy()+120<-ksuitm()+5136<-ksbrdp()+4696<-opirip()+1620<-opidrv()+608<-sou2o()+136<-opimai_real()+188<-ssthrdmain()+276<-main()+204<-__start()+112
----- End of Abridged Call Stack Trace -----

*** 2014-09-30 13:47:39.764
PSP0 (ospid: 57147560): terminating the instance due to error 7274
ksuitm: waiting up to [5] seconds before killing DIAG(49938448)

It turns out, during our upgrade procedure we disable the Data Vault option of the database using chopt disable dv. This modification somehow failed and as a result, deleted the faulty oracle binary. So in our ORACLE_HOME, no oracle binary could be found:

oracle@pandora:/u00/app/oracle/product/11.2.0/db_1/bin/ [KDB12345] ll oracle
ls: 0653-341 The file oracle does not exist.

After restoring the oracle binary, we were able to restart the databases:

oracle@pandora:/u00/app/oracle/product/11.2.0/db_1/bin/ [KDB12345] ll oracle
-rwsr-s--x    1 oracle   oinstall  309600886 Sep 30 14:15 oracle*

So if you get the above error, check if the oracle binary still exists in your $ORACLE_HOME/bin folder