Simon Krenger

When managing a large database environment with Oracle Enterprise Manager Cloud Control 12c, it sometimes happens that a database or another target gets stuck in the status “Pending”. This often happens after a database was restarted (for cold backups for example). So in this article I want to show how to get the target working again.

Check the agent status

The first thing to do is to actually ensure that the agent is working as expected. To do this, log into the machine where the pending target is located and issue the command emctl status agent. The output should look something like this:

$ emctl status agent
Oracle Enterprise Manager Cloud Control 12c Release 3
Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Agent Version     : 12.1.0.3.0
OMS Version       : 12.1.0.3.0
Protocol Version  : 12.1.0.1.0
Agent Home        : /u01/app/oracle/oemagent/agent_inst
Agent Binaries    : /u01/app/oracle/oemagent/core/12.1.0.3.0
Agent Process ID  : 12714132
Parent Process ID : 34996554
Agent URL         : https://pandora.krenger.ch:3872/emd/main/
Repository URL    : https://control.krenger.ch:4903/empbs/upload
Started at        : 2014-01-09 15:23:07
Started by user   : oracle
Last Reload       : (none)
Last successful upload                       : 2014-01-09 15:43:40
Last attempted upload                        : 2014-01-09 15:43:40
Total Megabytes of XML files uploaded so far : 93.07
Number of XML files pending upload           : 0
Size of XML files pending upload(MB)         : 0
Available disk space on upload filesystem    : 33.45%
Collection Status                            : Collections enabled
Heartbeat Status                             : Ok
Last attempted heartbeat to OMS              : 2014-01-09 15:45:33
Last successful heartbeat to OMS             : 2014-01-09 15:45:33
Next scheduled heartbeat to OMS              : 2014-01-09 15:46:33
---------------------------------------------------------------
Agent is Running and Ready

Note that it says that “Agent is Running and Ready“. Also note the following attributes:

• Last successful upload: Tells you when the last successful upload of metric data has taken place.
  This should be somewhere in the last half hour or hour. Refer to the section “Issue a manual upload” to see how you can manually trigger an upload of metric data.
• Total Megabytes of XML files uploaded so far: Amount of data the agent has uploaded since the last restart.
  Unless you just restarted the agent recently, this number should be larger than 0.
• Number of XML files pending upload: This is the number of files that have not yet been uploaded and should be 0 at best. However, when many targets are configured, this number might be quite large.
  Refer to the section “Issue a manual upload” to see how you can manually trigger an upload of metric data.
• Available disk space on upload filesystem: Make sure this is not 0% and there is adequate space available.
• Collection Status: Should be “Collections enabled”. Refer to My Oracle Support Note 1418873.1 if this is not the case.
• Heartbeat to OMS: As with the attribute “Last successful upload”, these timestamps should be current (within the last 15 minutes). Make sure the heartbeat works as expected.

Issue a manual upload

Given that the checks above were all successful, try to start a manual upload of metric data to the OMS:

$ emctl upload agent

After issuing this command, the agent should upload the pending metric data to the OMS. After the upload is done, issue emctl status agent again and make sure the “Last successful upload” timestamp was refreshed. Then, re-check the target status in OEM.

Restart the agent

When the above does not resolve the issue and the target is still “Pending”, it is often a good idea to try to restart the agent before continuing with further investigations. Often, this solves the problem (even My Oracle Support Note 1418873.1 says so!):

$ emctl stop agent
[..]
$ emctl start agent
[..]

After restarting the agent, re-check the target status in OEM.

Agent: clearstate

One last quick trick is to manually clear the agent status.
Note that this removes all state data from the agent and Cloud Control might regenerate alarms for issues that you previously acknowledged for targets managed by this agent! To clear the agent state, issue the following command:

$ emctl clearstate agent

After that, re-check the target status in OEM. Also, deal with any alerts that might have been generated through the use of this command.

Further troubleshooting

If these steps all fail, continue troubleshooting using the My Oracle Support Note 1418873.1. Using this note, I was able to fix all “Target Pending” issues I faced in the past.