EM won’t start after upgrade
After upgrading a database from 10.2.0.1 to 10.2.0.5, I was unable to start the Oracle Enterprise Manager. Whenever I tried to do so, the log showed that it failed to start Database Control. Even when I deconfigured the Enterprise Manager, deleted the repository and started from scratch, I still ran into the same problem. In this post, I will describe the symptoms I encountered and also provide a solution.
Symptoms
So what I saw is basically this:
$ORACLE_HOME/cfgtoollogs/emca/$ORACLE_SID/emca_[time].log:
CONFIG: Starting execution: /u01/app/oracle/product/10.2.0/db_1/bin/emctl start dbconsole
Feb 23, 2011 8:57:36 PM oracle.sysman.emcp.util.PlatformInterface executeCommand
CONFIG: Exit value of 1
Feb 23, 2011 8:57:36 PM oracle.sysman.emcp.util.PlatformInterface executeCommand
CONFIG: TZ set to Europe/Zurich
Oracle Enterprise Manager 10g Database Control Release 10.2.0.5.0
Copyright (c) 1996, 2010 Oracle Corporation. All rights reserved.
Starting Oracle Enterprise Manager 10g Database Control ............................................................................................. failed.
------------------------------------------------------------------
Logs are generated in directory /u01/app/oracle/product/10.2.0/db_1/db01.dev.foo.ch_orcl/sysman/log
Feb 23, 2011 8:57:36 PM oracle.sysman.emcp.util.PlatformInterface executeCommand
WARNING: Error executing /u01/app/oracle/product/10.2.0/db_1/bin/emctl start dbconsole
Feb 23, 2011 8:57:36 PM oracle.sysman.emcp.EMConfig perform
SEVERE: Error starting Database Control
Refer to the log file at /u01/app/oracle/product/10.2.0/db_1/cfgtoollogs/emca/ORCL/emca_2011-02-23_08-49-51-PM.log for more details.
Feb 23, 2011 8:57:36 PM oracle.sysman.emcp.EMConfig perform
CONFIG: Stack Trace:
oracle.sysman.emcp.exception.EMConfigException: Error starting Database Control
at oracle.sysman.emcp.EMDBPostConfig.performConfiguration(EMDBPostConfig.java:649)
at oracle.sysman.emcp.EMDBPostConfig.invoke(EMDBPostConfig.java:227)
at oracle.sysman.emcp.EMDBPostConfig.invoke(EMDBPostConfig.java:196)
at oracle.sysman.emcp.EMConfig.perform(EMConfig.java:184)
at oracle.sysman.emcp.EMConfigAssistant.invokeEMCA(EMConfigAssistant.java:494)
at oracle.sysman.emcp.EMConfigAssistant.performConfiguration(EMConfigAssistant.java:1161)
at oracle.sysman.emcp.EMConfigAssistant.statusMain(EMConfigAssistant.java:478)
at oracle.sysman.emcp.EMConfigAssistant.main(EMConfigAssistant.java:426)Furthermore, there are some ugly Java exceptions showing up in the EM OMS log:
$ORACLE_HOME/$HOSTNAME_$ORACLE_SID/sysman/log/emoms.log:
2011-02-23 22:06:10,684 [EMUI_22_06_10_/console/database/instance/sitemap] ERROR perf.sitemap initializeGeneral.400 - Failed to initialize object: ORA-20206:
ORA-06512: at "SYSMAN.EMD_DATABASE", line 1165
ORA-01403: no data found
ORA-06512: at line 1
2011-02-23 22:10:36,584 [Orion Launcher] ERROR app.ContextInitializer contextInitialized.272 - Integration Class not found: oracle.sysman.ias.ias.IASIntegration
2011-02-23 22:10:36,951 [Orion Launcher] ERROR app.ContextInitializer contextInitialized.272 - Integration Class not found: oracle.sysman.eml.target.slb.common.SLBIntegration
2011-02-23 22:10:37,070 [Orion Launcher] ERROR app.ContextInitializer contextInitialized.272 - Integration Class not found: oracle.sysman.eml.ssl.intg.SSLIntegration
2011-02-23 22:10:37,198 [Orion Launcher] ERROR app.ContextInitializer contextInitialized.272 - Integration Class not found: oracle.tip.oem.central.domain.ProcessConnectDomainIntg
2011-02-23 22:10:37,202 [Orion Launcher] ERROR app.ContextInitializer contextInitialized.272 - Integration Class not found: oracle.tip.oem.central.instance.ProcessConnectInstanceIntg
2011-02-23 22:10:37,204 [Orion Launcher] ERROR app.ContextInitializer contextInitialized.272 - Integration Class not found: oracle.webdb.admin.em.PortalIntegration
2011-02-23 22:10:37,206 [Orion Launcher] ERROR app.ContextInitializer contextInitialized.272 - Integration Class not found: oracle.webdb.admin.em.SSOIntegration
2011-02-23 22:10:37,207 [Orion Launcher] ERROR app.ContextInitializer contextInitialized.272 - Integration Class not found: oracle.reports.em.RepIntg
2011-02-23 22:10:37,209 [Orion Launcher] ERROR app.ContextInitializer contextInitialized.272 - Integration Class not found: oracle.sysman.ocs.mntr.target.OcsEmailIntegration
2011-02-23 22:10:37,211 [Orion Launcher] ERROR app.ContextInitializer contextInitialized.272 - Integration Class not found: oracle.sysman.ocs.mntr.target.OcsOidIntegration
2011-02-23 22:10:37,213 [Orion Launcher] ERROR app.ContextInitializer contextInitialized.272 - Integration Class not found: oracle.sysman.ocs.mntr.target.OcsOvfIntegration
2011-02-23 22:10:37,215 [Orion Launcher] ERROR app.ContextInitializer contextInitialized.272 - Integration Class not found: oracle.sysman.ocs.mntr.target.OcsWebconfIntegration
2011-02-23 22:10:37,218 [Orion Launcher] ERROR app.ContextInitializer contextInitialized.272 - Integration Class not found: oracle.sysman.ocs.mntr.target.OcsWirelessIntg
2011-02-23 22:10:37,220 [Orion Launcher] ERROR app.ContextInitializer contextInitialized.272 - Integration Class not found: oracle.sysman.ocs.mntr.target.OcsCalGrpIntegrationAlso, there seemed to be a problem with the SSL connection:
$ORACLE_HOME/$HOSTNAME_$ORACLE_SID/sysman/log/emdctl.trc
2011-02-23 22:43:23 Thread-4135134912 ERROR http: 5: Unable to initialize ssl connection with server, aborting connection attempt
2011-02-23 22:44:55 Thread-4134893248 ERROR ssl: Open wallet failed, ret = 28750
2011-02-23 22:44:55 Thread-4134893248 ERROR ssl: nmehlenv_openWallet failed
2011-02-23 22:44:55 Thread-4134893248 ERROR http: 5: Unable to initialize ssl connection with server, aborting connection attempt
2011-02-23 22:46:26 Thread-4135548608 ERROR ssl: Open wallet failed, ret = 28750
2011-02-23 22:46:26 Thread-4135548608 ERROR ssl: nmehlenv_openWallet failed
2011-02-23 22:46:26 Thread-4135548608 ERROR http: 5: Unable to initialize ssl connection with server, aborting connection attempt
2011-02-23 22:47:27 Thread-4135274176 WARN http: snmehl_connect: connect failed to (db01.dev.foo.ch:1160): Connection refused (error = 111)
2011-02-23 22:47:29 Thread-4135511744 WARN http: snmehl_connect: connect failed to (db01.dev.foo.ch:1835): Connection refused (error = 111)
2011-02-23 22:54:42 Thread-4134741696 WARN http: snmehl_connect: connect failed to (db01.dev.foo.ch:1835): Connection refused (error = 111)Solution
When searching through My Oracle Support, we then stumbled upon Note 1217493.1, which tells us:
In Enterprise Manager Database Control with Oracle Database 10.2.0.4 and 10.2.0.5, the root certificate used to secure communications via the Secure Socket Layer (SSL) protocol will expire on 31-Dec-2010 00:00:00. The certificate expiration will cause errors if you attempt to configure Database Control on or after 31-Dec-2010.
Oh wow, so this explains the SSL connection warnings seen above. So to get rid of these errors, we will have to apply Patch 8350262 (mentioned in the MOS note):
unzip p8350262_10205_Generic.zip
cd 8350262/
opatch applyI then chose to uninstall the Enterprise Manager configuration and remove the repository from the database. This might not be necessary and should only be done when you know what you are doing:
emca -deconfig dbcontrol db -repos drop
emca -config all db -repos create -DBCONTROL_HTTP_PORT 1158Voilà . Now I was able to start the Enterprise Manager without any errors.