Error 404--Not Found in OEM cloud control 13C

ERROR:

Error 404--Not Found in oms 13C

CAUSES:

It might be due to several reasons like "database was not reachable" or "Admin Server might be down" or others

FINDING THE ROOT CAUSE:

Step 1: confirm whether database and listener are up.

Step 2: confirm whether the agent and oms servers are up.

[orawls@devs02 bin]$ export ORACLE_HOME=/u01/orawls/product/middleware/

[orawls@devs02 bin]$ export AGENT_HOME=/u01/orawls/product/Agent/agent_inst

[orawls@devs02 bin]$ $ORACLE_HOME/bin/emctl status oms            

Oracle Enterprise Manager Cloud Control 13c Release 1

Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
WebTier is Up
Oracle Management Server is intializing.
Context not fully initialized yet.
Check EM Server log file for details: /u01/orawls/product/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/logs/EMGC_OMS1.out
JVMD Engine is Up
BI Publisher Server is Up
[orawls@devs02 bin]$

[orawls@devs02 bin]$ $AGENT_HOME/bin/emctl status agent

Oracle Enterprise Manager Cloud Control 13c Release 1

Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Agent Version          : 13.1.0.0.0
OMS Version            : (unknown)
Protocol Version       : 12.1.0.1.0
Agent Home             : /u01/orawls/product/Agent/agent_inst
Agent Log Directory    : /u01/orawls/product/Agent/agent_inst/sysman/log
Agent Binaries         : /u01/orawls/product/Agent/agent_13.1.0.0.0
Core JAR Location      : /u01/orawls/product/Agent/agent_13.1.0.0.0/jlib
Agent Process ID       : 13138
Parent Process ID      : 13101
Agent URL              : https://devs02.local.com:3872/emd/main/
Local Agent URL in NAT : https://devs02.local.com:3872/emd/main/
Repository URL         : https://devs02.local.com:4903/empbs/upload
Started at             : 2016-07-11 15:53:51
Started by user        : orawls
Operating System       : Linux version 2.6.32-504.el6.x86_64 (amd64)
Number of Targets      : 54
Last Reload            : (none)
Last successful upload                       : (none)
Last attempted upload                        : (none)
Total Megabytes of XML files uploaded so far : 0
Number of XML files pending upload          : 1,650
Size of XML files pending upload(MB)        : 2.29
Available disk space on upload filesystem    : 36.67%
Collection Status                                            : Collections enabled
Heartbeat Status                                             : OMS response is missing
Last attempted heartbeat to OMS                  : 2016-07-11 16:07:25
Last successful heartbeat to OMS                 : (none)

Next scheduled heartbeat to OMS                 : 2016-07-11 16:07:55

Step 3: Confirm whether all the ports are accessible

[orawls@devs02 logs]$ $ORACLE_HOME/bin/emctl status oms –details
Oracle Enterprise Manager Cloud Control 13c Release 1
Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
Enter Enterprise Manager Root (SYSMAN) Password :
Console Server Host        : devs02.local.com
HTTP Console Port          : 7788
HTTPS Console Port         : 7802
HTTP Upload Port           : 4889
HTTPS Upload Port          : 4903
EM Instance Home           : /u01/orawls/product/gc_inst/em/EMGC_OMS1
OMS Log Directory Location : /u01/orawls/product/gc_inst/em/EMGC_OMS1/sysman/log
OMS is not configured with SLB or virtual hostname
Agent Upload is locked.
OMS Console is locked.
Active CA ID: 1
Console URL: https://devs02.local.com:7802/em
Upload URL: https://devs02.local.com:4903/empbs/upload

WLS Domain Information
Domain Name            : GCDomain
Admin Server Host      : devs02.local.com
Admin Server HTTPS Port: 7102
Admin Server is FAILED_NOT_RESTARTABLE
Admin Server Port is occupied by another process. Please stop process to free port.

Oracle Management Server Information
Managed Server Instance Name: EMGC_OMS1
Oracle Management Server Instance Host: devs02.bma.com
WebTier is Up
Oracle Management Server is intializing.
Context not fully initialized yet.
Check EM Server log file for details: /u01/orawls/product/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/logs/EMGC_OMS1.out
JVMD Engine is Up

BI Publisher Server Information
BI Publisher Managed Server Name: BIP
BI Publisher Server is Up

BI Publisher HTTP Managed Server Port   : 9701
BI Publisher HTTPS Managed Server Port  : 9803
BI Publisher HTTP OHS Port              : 9788
BI Publisher HTTPS OHS Port             : 9851
BI Publisher is locked.
BI Publisher Server named 'BIP' running at URL: https://devs02.local.com:9851/xmlpserver
BI Publisher Server Logs: /u01/orawls/product/gc_inst/user_projects/domains/GCDomain/servers/BIP/logs/
BI Publisher Log        : /u01/orawls/product/gc_inst/user_projects/domains/GCDomain/servers/BIP/logs/bipublisher/bipublisher.log

SOLUTION:

Step 1: Stop all the oms services

[orawls@devs02 logs]$ $ORACLE_HOME/bin/emctl stop oms -all

Step 2: Get the process id of leftover processes if any.

ps -ef | grep EMGC_ADMINSERVER

ps -ef | grep EMGC_OMS1

ps -ef | grep java

ps -ef | grep opmn

Step 3: Kill those processes

$ kill -9 <PID>

Step 4: Start the node manager, Admin server & managed server related to OMS.

Step 5: Start OMS service and AGENT service.

$ $ORACLE_HOME/bin/emctl status oms

$ $AGENT_HOME/bin/emctl status agent