Search This Blog

Thursday, September 14, 2017

ADOP issues : FS_CLONE error in 12.2.4

Hello Friends,

This issue have been encountered  while running fs_clone. Handling adop is pretty comfortable if we know where to find the error logs exactly. You can read this post once to know where adop error logs should be checked. 

I am running fs_clone since we apply patch as hotpatch. Its not necessary to run fs_clone if you apply a patch through adop full cycle(Means prepare, apply, finalize, cutover, cleanup). If we apply as hot patch, its recommended to run fs_clone once the patch successful to make sure both run and patch file systems are in sync. Make sure no code movement to patch file system is in progress when you run fs_clone since any code copied to patch file system will be overwritten by fs_clone. 

Command to trigger fs_clone :  adop phase=fs_clone

Error  in adop main session:

Program : txkValidateSetup.pl started @ Thu Sep 14 11:02:48 2017

*** Log File = /u01/oracle/PROD/fs_ne/EBSapps/log/adop/28/fs_clone_20170914_110147/PROD_*********/ADOPValidations_Thu_Sep_14_11_02_48_2017.log

SETUP VALIDATION is in progress. This may take few minutes to complete.
        [UNEXPECTED]Error occurred while executing "perl /u01/oracle/PROD/fs2/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPValidations.pl  -contextfile=/u01/oracle/PROD/fs2/inst/apps/PROD_******/appl/admin/PROD_******.xml -patchctxfile=/u01/oracle/PROD/fs1/inst/apps/PROD_******/appl/admin/PROD_******.xml -phase=fs_clone -logloc=/u01/oracle/PROD/fs_ne/EBSapps/log/adop/28/fs_clone_20170914_110147/PROD_****** -promptmsg=hide"
        [UNEXPECTED]Error 1 occurred while Executing txkADOPValidation script on ******
        Log file: /u01/oracle/PROD/fs_ne/EBSapps/log/adop/28/adop_20170914_110147.log


[STATEMENT] Please run adopscanlog utility, using the command

"adopscanlog -latest=yes"

to get the list of the log files along with snippet of the error message corresponding to each log file.
  
adop exiting with status = 1 (Fail)

 Nothing in detail in the adop session. So i had to dig into adop logs folder. navigate $ADOP_LOG_HOME where adop writes all of its log. we can find the logs for each session id. 

-bash-3.2$ cd $ADOP_LOG_HOME 

-bash-3.2$ ls -ltr --> for latest session. 


drwxr-xr-x   5 applmgr  oinstall     512 May 11 10:30 26 --> for session id 26
drwxr-xr-x   3 applmgr  oinstall     512 May 30 09:50 27 --> for session id 27
-rw-r--r--   1 applmgr  oinstall    1122 Sep  4 16:54 adop_20170904_165405.log --> adop  main session log
-rw-r--r--   1 applmgr  oinstall    1122 Sep  5 09:23 adop_20170905_092239.log
-rw-r--r--   1 applmgr  oinstall   10133 Sep 14 11:02 CheckWLSAdminPwd.log
drwxr-xr-x   5 applmgr  oinstall     512 Sep 14 11:02 28

As my latest adop session was 28 i navigated into folder named '28' and look for the logs. If you are not sure about the session id, look for the adop screen where it will show the session id while it begins. Alternatively you can query ad_adop_sessions table for more session detail.  

-bash-3.2$ cd 28
  
ls -ltr 

drwxr-xr-x   3 applmgr  oinstall     512 Sep  5 10:18 fs_clone_20170905_101716
-rw-r--r--   1 applmgr  oinstall    4878 Sep  5 10:19 adop_20170905_101716.log
drwxr-xr-x   3 applmgr  oinstall     512 Sep 14 10:59 fs_clone_20170914_105815
-rw-r--r--   1 applmgr  oinstall    6910 Sep 14 10:59 adop_20170914_105815.log
drwxr-xr-x   3 applmgr  oinstall     512 Sep 14 11:02 fs_clone_20170914_110147
-rw-r--r--   1 applmgr  oinstall    5040 Sep 14 11:03 adop_20170914_110147.log

Look for the latest directory.

-bash-3.2$ cd fs_clone_20170914_110147
  
-bash-3.2$ ls
ADOPValidations_Thu_Sep_14_11_02_48_2017.log

There were only one log file since this issue was encountered at the beginning stage itself.   
-bash-3.2$ vi ADOPValidations_Thu_Sep_14_11_02_48_2017.log
  
Nodes with context files in the FND_OAM_CONTEXT_FILES table on both run and patch file systems: NONE
Nodes without context files in the FND_OAM_CONTEXT_FILES table on either/or run and patch file systems: ******
Corrective Action:
- If the run file system context file for a node is missing, run AutoConfig on the run file system of that node to sync with the value with the database.
- If the patch file system context file of a node is missing, run AutoConfig on the patch file system of that node with the -syncctx option as follows to sync with the
value with the database.

On UNIX:
 sh <AD_TOP>/bin/adconfig.sh contextfile=<CONTEXT_FILE> -syncctx



Above is the error logged in the log. As soon as i see this i queried the table to see whether both xml files (run and patch) are present in db. 

select * from apps.FND_OAM_CONTEXT_FILES;






 Results clearly showed me there is no xml file for patch fs in the db. So i ran the below to generate xml file for patch file system. 


-bash-3.2$ sh /u01/oracle/PROD/fs1/EBSapps/appl/ad/12.0.0/bin/adconfig.sh contextfile=/u01/oracle/PROD/fs1/inst/apps/PROD_******/appl/admin/PROD_******.xml -syncctx

Enter the APPS user password:
The log file for this session is located at: /u01/oracle/PROD/fs1/inst/apps/PROD_******/admin/log/09141113/adconfig.log

Option specified      : Synchronize context file
Only context file synchronization will be performed

        Classpath                   : /u01/oracle/PROD/fs1/FMW_Home/Oracle_EBS-app1/shared-libs/ebs-appsborg/WEB-INF/lib/ebsAppsborgManifest.jar:/u01/oracle/PROD/fs1/EBSapps/comn/java/classes

        Using ContextFile     : /u01/oracle/PROD/fs1/inst/apps/PROD_******/appl/admin/PROD_******.xml

        Synchronizing the context file ......COMPLETED

AutoConfig completed successfully.

I again queried to see the changes. Look into #8 row, we can see xml file row has been populated for patch file system now. 




fs_clone completed successfully after this change. 


Find FMW version in EBS R12.2.4


Please follow these steps to find out FMW version of your oracle application.

- Login as application owner (ex : applmgr)
- Source run fs env file

Then do the below ,

-bash-3.2$ cd $FMW_HOME

-bash-3.2$ cd Oracle_EBS-app1/

-bash-3.2$ pwd
/u01/oracle/PROD/fs2/FMW_Home/Oracle_EBS-app1

-bash-3.2$ export ORACLE_HOME=/u01/oracle/PROD/fs2/FMW_Home/Oracle_EBS-app1

-bash-3.2$ ./opatch lsinv
-bash: ./opatch: No such file or directory


-bash-3.2$ ls
META-INF            ccr                 common              install             oraInst.loc         shared-libs
OPatch              cfgtoollogs         deployment_plans    inventory           oui                 srvm
applications        clone               diagnostics         jdk                 setEBSDomainEnv.sh  sysman

-bash-3.2$ cd OPatch/

-bash-3.2$ ./opatch lsinv
Invoking OPatch 11.2.0.1.2

Oracle Interim Patch Installer version 11.2.0.1.2
Copyright (c) 2010, Oracle Corporation.  All rights reserved.


Oracle Home       : /u01/oracle/PROD/fs2/FMW_Home/Oracle_EBS-app1
Central Inventory : /u01/oracle/PROD/oraInventory
   from           : /var/opt/oracle/oraInst.loc
OPatch version    : 11.2.0.1.2
OUI version       : 11.2.0.3.0
OUI location      : /u01/oracle/PROD/fs2/FMW_Home/Oracle_EBS-app1/oui
Log file location : /u01/oracle/PROD/fs2/FMW_Home/Oracle_EBS-app1/cfgtoollogs/opatch/opatch2017-09-14_10-48-56AM.log

Patch history file: /u01/oracle/PROD/fs2/FMW_Home/Oracle_EBS-app1/cfgtoollogs/opatch/opatch_history.txt


OPatch detects the Middleware Home as "/u01/oracle/PROD/fs2/FMW_Home"

Lsinventory Output file location : /u01/oracle/PROD/fs2/FMW_Home/Oracle_EBS-app1/cfgtoollogs/opatch/lsinv/lsinventory2017-09-14_10-48-56AM.txt

--------------------------------------------------------------------------------
Installed Top-level Products (1):

Oracle E-Business Suite Component                                    11.1.1.6.0 --> FMW version
There are 1 products installed in this Oracle Home.


There are no Interim patches installed in this Oracle Home.


--------------------------------------------------------------------------------

OPatch succeeded.

Monday, September 4, 2017

How to change weblogic password if lost or forgotten : java s_nm_jvm_startup_properties with example


If the Admin Password of an EBS WebLogic Domain is lost or forgotten :

1. Shut down all running services. Since the AdminServer password is not known, the
servers cannot be stopped from the console and so must be killed as follows.

--1. Connect to the Oracle E-Business Suite instance and source the application tier
environment file.

--2. Identify the PIDs of Node Manager, AdminServer, and all running Managed
Servers:
$ ps -ef | grep "NodeManager"
$ ps -ef | grep "weblogic.Name=AdminServer"
$ ps -ef | grep "weblogic.Name=forms-c4ws_server"
$ ps -ef | grep "weblogic.Name=forms_server"
$ ps -ef | grep "weblogic.Name=oafm_server"
$ ps -ef | grep "weblogic.Name=oacore_server"

3. Kill all these processes, starting with Node Manager and followed by the
Managed Servers.

2. Back up these folders, and then delete them:
<EBS_DOMAIN_HOME>/security/ DefaultAuthenticatorInit.ldift
<EBS_DOMAIN_HOME>/servers/<server_name>/data/ldap
<EBS_DOMAIN_HOME>/servers/<server_name>/security/boot.properties
<EBS_DOMAIN_HOME>/servers/<server_name>/data/nodemanager/boot.
properties
Where:
• <EBS_DOMAIN_HOME> is the absolute path of the EBS WebLogic domain
• <server_name> is the name of the server directory under
<EBS_DOMAIN_HOME>.

If the password is not reset correctly, the backed up files and folders can be
restored.
Note: For certain servers, the boot.properties file may be present in
only one location of the two specified above. In such a case, back it
up and then delete it.

--3. Set up a new environment to change the WLS AdminServer password.
1. Start a new session and connect to the Oracle E-Business Suite instance.
2. Do not source the application tier environment file.
3. Run the following command to source the WebLogic Server domain
environment:
$ cd <EBS_DOMAIN_HOME>/bin
$ source setDomainEnv.sh
4. Run the following commands:
$ cd <EBS_DOMAIN_HOME>/security
$ java weblogic.security.utils.AdminAccount <wls_adminuser>
<wls_admin_new_password> .
Where:
• <wls_adminuser> is the same as the value of context variable
s_wls_admin_user
• <wls_admin_new_password> is the new WLS AdminServer password you
wish to set.
Note: Do not omit the trailing period ('.') in the above
command: it is needed to specify the current domain directory.


--4. Start AdminServer from the command line. You will be prompted for the WebLogic
Server username and password, so that the AdminServer boot.properties file
can be generated.
1. Go to the EBS Domain Home:
$ cd <EBS_DOMAIN_HOME>
2. Start AdminServer:
$ java <s_nm_jvm_startup_properties> -Dweblogic.system.
StoreBootIdentity=true -Dweblogic.Name=AdminServer weblogic.
Server

• <s_nm_jvm_startup_properties> is the same as the value of context variable
ss_nm_jvm_startup_properties

Example:

MAke sure to remove trailing space and line separators.

$ java -d64 -Xms1024m -Xmx1024m -XX:CompileThreshold=8000 -XX:PermSize=128m -XX:MaxPermSize=256m -Djava.security.policy=/u01/oracle/PROD/fs2/FMW_Home/wlserver_10.3/server/lib/weblogic.policy -Xverify:none -da -Dplatform.home=/u01/oracle/PROD/fs2/FMW_Home/wlserver_10.3 -Dwls.home=/u01/oracle/PROD/fs2/FMW_Home/wlserver_10.3/server -Dweblogic.home=/u01/oracle/PROD/fs2/FMW_Home/wlserver_10.3/server -Dcommon.components.home=/u01/oracle/PROD/fs2/FMW_Home/oracle_common -Djrf.version=11.1.1 -Dorg.apache.commons.logging.Log=org.apache.commons.logging.impl.Jdk14Logger -Ddomain.home=/u01/oracle/PROD/fs2/FMW_Home/user_projects/domains/EBS_domain_PROD -Doracle.server.config.dir=/u01/oracle/PROD/fs2/FMW_Home/user_projects/domains/EBS_domain_PROD/config/fmwconfig/servers/AdminServer -Doracle.domain.config.dir=/u01/oracle/PROD/fs2/FMW_Home/user_projects/domains/EBS_domain_PROD/config/fmwconfig -Digf.arisidbeans.carmlloc=/u01/oracle/PROD/fs2/FMW_Home/user_projects/domains/EBS_domain_PROD/config/fmwconfig/carml -Digf.arisidstack.home=/u01/oracle/PROD/fs2/FMW_Home/user_projects/domains/EBS_domain_PROD/config/fmwconfig/arisidprovider -Doracle.security.jps.config=/u01/oracle/PROD/fs2/FMW_Home/user_projects/domains/EBS_domain_PROD/config/fmwconfig/jps-config.xml -Doracle.deployed.app.dir=/u01/oracle/PROD/fs2/FMW_Home/user_projects/domains/EBS_domain_PROD/servers/AdminServer/tmp/_WL_user -Doracle.deployed.app.ext=/- -Dweblogic.alternateTypesDirectory=/u01/oracle/PROD/fs2/FMW_Home/oracle_common/modules/oracle.ossoiap_11.1.1,/u01/oracle/PROD/fs2/FMW_Home/oracle_common/modules/oracle.oamprovider_11.1.1 -Djava.protocol.handler.pkgs=oracle.mds.net.protocol -Dweblogic.jdbc.remoteEnabled=false -Dportlet.oracle.home=/u01/oracle/PROD/fs2/FMW_Home/oracle_common -Dem.oracle.home=/u01/oracle/PROD/fs2/FMW_Home/oracle_common -Dweblogic.management.discover=true -Dwlw.iterativeDev= -Dwlw.testConsole= -Dwlw.logErrorsToConsole= -Dweblogic.ext.dirs=/u01/oracle/PROD/fs2/FMW_Home/patch_wls1036/profiles/default/sysext_manifest_classpath -Dweblogic.system.StoreBootIdentity=true -Dweblogic.Name=AdminServer weblogic.Server

The above command prompts for the WebLogic Server username and password:
Enter username to boot WebLogic server:
Enter password to boot WebLogic server:

Provide the same credentials as you provided in Step 3.

--5. Change Node Manager password
1. Log in to the WebLogic Administration console.
2. Click the 'Lock & Edit' button.
3. In the left panel, click on the EBS Domain link.
4. Select the 'Security' tab.
5. Click on the 'Advanced' link.
6. Edit the 'Node Manager password' field and set it to the new WebLogic Server
password. The password should be same as set in Step 3.
7. Edit the 'Confirm Node Manager Password' field and set it to the new
WebLogic Server password. The password should be same as set in Step 3.
8. Save and activate the changes.

--6. The first time, AdminServer has to be stopped from the Admin console. Follow
these steps:
1. Log in to the WebLogic Administration console.
2. Shut down AdminServer.

--7. Set up your environment to start AdminServer again. AdminServer should now be
started using the normal AD script, which will also start Node Manager using the
new password.
1. Launch a new session and connect to the Oracle E-Business Suite instance.
2. Source the application tier environment file.
3. Start AdminServer with the following command:
$ $ADMIN_SCRIPTS_HOME/adadminsrvctl.sh

--8. Start the Managed Servers. For the first time, all Managed Servers should be started
from the WebLogic Server Admin console. This step will create boot.properties
files for the respective Managed Servers. Follow these steps:
1. Log in to the WebLogic Server Administration Console
2. Start all Managed Servers, one at a time

--9. Shut down all the Managed Servers. This is so the new credentials will be picked up
at the next startup. Follow these steps:
1. Log in to the WebLogic AdminServer console.
2. Shut down all Managed Servers.
3. Shut down AdminServer.

--10. Shut down Node Manager using the normal AD script.
$ $ADMIN_SCRIPTS_HOME/adnodemgrctl.sh stop

--11. Copy the boot.properties file for each Managed Server.
WebLogic Server native scripts use the boot.properties file. The above steps
have created the boot.properties file under
<EBS_DOMAIN_HOME>/servers/<Managed Server
name>/data/nodemanager, which is used by Node Manager. For each Managed
Server, copy the newly-generated boot.properties file from
<EBS_DOMAIN_HOME>/servers/<Managed Servername>/data/nodemanager to <EBS_DOMAIN_HOME>/servers/<ManagedServer name>/security.
The EBS WebLogic Server domain password has now been changed, and all servers
can now be started using the normal AD scripts.
To start AdminServer:
$ADMIN_SCRIPTS_HOME/adadminsrvctl.sh start
To start the Managed Servers:
$ $ADMIN_SCRIPTS_HOME/admanagedsrvctl.sh start <managed_server_name>

--12. The above steps have changed the Oracle WebLogic AdminServer password on the
run file system. You now need to perform an fs_clone operation, to change the
WebLogic EBS Domain password on the patch file system:
1. Launch a new session and connect to the Oracle E-Business Suite instance.
2. Source the application tier environment file.
3. Run the command:
3-42 Oracle E-Business Suite Setup Guide
$ adop phase=fs_clone

You cannot complete this task because one of the following events caused a loss of page data: A system failure has occurred "Front end is locked after 12.2 upgrade"

  If you have upgraded to Oracle E-Business Suite 12.2.10 Release Update Pack from Oracle E-Business Suite 12.2.6 Release Update Pack (or ea...