opmn/apache is exiting with status 2 in EBS R12.2

Issue :

bash-3.2$ ./adopmnctl.sh start

You are running adopmnctl.sh version 120.0.12020000.2

Starting Oracle Process Manager (OPMN) ...

adopmnctl.sh: exiting with status 2

adopmnctl.sh: check the logfile /a01/appsclone/CLONE/fs2/inst/apps/CLONE_ahqtest/logs/appl/admin/log/adopmnctl.txt for more information ...


What was found in log?

Log file :

/a01/appsclone/CLONE/fs2/FMW_Home/webtier/instances/EBS_web_CLONE_OHS1/diagnostics/logs/OPMN/opmn/opmn.log


Error found in log :

[2019-01-01T10:42:16+03:00] [opmn] [NOTIFICATION:1] [90] [ons-internal] ONS server initiated
[2019-01-01T10:42:16+03:00] [opmn] [TRACE:32] [] [internal] Host: ahqtest; Remote Port: 6233; Local Port: 6133; Pid: 15885; 11.1.1.9.0
[2019-01-01T10:42:16+03:00] [opmn] [TRACE:1] [522] [pm-internal] PM state directory exists: /a01/appsclone/CLONE/fs2/FMW_Home/webtier/instance
s/EBS_web_CLONE_OHS1/config/OPMN/opmn/states
[2019-01-01T10:42:16+03:00] [opmn] [NOTIFICATION:1] [675] [pm-internal] OPMN server ready. Request handling enabled
[2019-01-01T10:42:16+03:00] [opmn] [ERROR:1] [17] [ons-listener] 127.0.0.1;6133: BIND (Address already in use)
[2019-01-01T10:43:05+03:00] [opmn] [TRACE:32] [] [internal] ORACLE_HOME: /a01/appsclone/CLONE/fs2/FMW_Home/webtier
[2019-01-01T10:43:05+03:00] [opmn] [TRACE:32] [] [internal] ORACLE_INSTANCE: /a01/appsclone/CLONE/fs2/FMW_Home/webtier/instances/EBS_web_CLONE
_OHS1

Fix :

To resolve the issue test the following steps in a development instance and then migrate accordingly:

1. Stop all services.

2. Rename the directory: OPMN/opmn/states  (eg. /u01/EBS/1225/fs1/FMW_Home/webtier/instances/EBS_1/config/OPMN/opmn/states in the example above.)

bash-3.2$ cd -

/a01/appsclone/CLONE/fs2/FMW_Home/webtier/instances/EBS_web_CLONE_OHS1/config/OPMN/opmn

mv states states_orig

rm -rf states

3. Restart the services and confirm the error no longer occurs.

bash-3.2$ ls

instance.properties  opmn.xml_orig        states               wallet

opmn.xml             ports.prop           states_orig

bash-3.2$ ./adopmnctl.sh start

You are running adopmnctl.sh version 120.0.12020000.2

Starting Oracle Process Manager (OPMN) ...

adopmnctl.sh: exiting with status 0

adopmnctl.sh: check the logfile /a01/appsclone/CLONE/fs2/inst/apps/CLONE_ahqtest/logs/appl/admin/log/adopmnctl.txt for more information ...

bash-3.2$ ./adapcctl.sh start

You are running adapcctl.sh version 120.0.12020000.6

Starting OPMN managed Oracle HTTP Server (OHS) instance ...

adapcctl.sh: exiting with status 0

adapcctl.sh: check the logfile /a01/appsclone/CLONE/fs2/inst/apps/CLONE_ahqtest/logs/appl/admin/log/adapcctl.txt for more information ...

References :

E-Business Suite Technology Stack OPMN Process Fails To Start With Error "Check Adopmnctl.txt" (Doc ID 2174221.1)