Search This Blog

Thursday, September 14, 2017

ADOP issues : FS_CLONE error in 12.2.4

Hello Friends,

This issue have been encountered  while running fs_clone. Handling adop is pretty comfortable if we know where to find the error logs exactly. You can read this post once to know where adop error logs should be checked. 

I am running fs_clone since we apply patch as hotpatch. Its not necessary to run fs_clone if you apply a patch through adop full cycle(Means prepare, apply, finalize, cutover, cleanup). If we apply as hot patch, its recommended to run fs_clone once the patch successful to make sure both run and patch file systems are in sync. Make sure no code movement to patch file system is in progress when you run fs_clone since any code copied to patch file system will be overwritten by fs_clone. 

Command to trigger fs_clone :  adop phase=fs_clone

Error  in adop main session:

Program : txkValidateSetup.pl started @ Thu Sep 14 11:02:48 2017

*** Log File = /u01/oracle/PROD/fs_ne/EBSapps/log/adop/28/fs_clone_20170914_110147/PROD_*********/ADOPValidations_Thu_Sep_14_11_02_48_2017.log

SETUP VALIDATION is in progress. This may take few minutes to complete.
        [UNEXPECTED]Error occurred while executing "perl /u01/oracle/PROD/fs2/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPValidations.pl  -contextfile=/u01/oracle/PROD/fs2/inst/apps/PROD_******/appl/admin/PROD_******.xml -patchctxfile=/u01/oracle/PROD/fs1/inst/apps/PROD_******/appl/admin/PROD_******.xml -phase=fs_clone -logloc=/u01/oracle/PROD/fs_ne/EBSapps/log/adop/28/fs_clone_20170914_110147/PROD_****** -promptmsg=hide"
        [UNEXPECTED]Error 1 occurred while Executing txkADOPValidation script on ******
        Log file: /u01/oracle/PROD/fs_ne/EBSapps/log/adop/28/adop_20170914_110147.log


[STATEMENT] Please run adopscanlog utility, using the command

"adopscanlog -latest=yes"

to get the list of the log files along with snippet of the error message corresponding to each log file.
  
adop exiting with status = 1 (Fail)

 Nothing in detail in the adop session. So i had to dig into adop logs folder. navigate $ADOP_LOG_HOME where adop writes all of its log. we can find the logs for each session id. 

-bash-3.2$ cd $ADOP_LOG_HOME 

-bash-3.2$ ls -ltr --> for latest session. 


drwxr-xr-x   5 applmgr  oinstall     512 May 11 10:30 26 --> for session id 26
drwxr-xr-x   3 applmgr  oinstall     512 May 30 09:50 27 --> for session id 27
-rw-r--r--   1 applmgr  oinstall    1122 Sep  4 16:54 adop_20170904_165405.log --> adop  main session log
-rw-r--r--   1 applmgr  oinstall    1122 Sep  5 09:23 adop_20170905_092239.log
-rw-r--r--   1 applmgr  oinstall   10133 Sep 14 11:02 CheckWLSAdminPwd.log
drwxr-xr-x   5 applmgr  oinstall     512 Sep 14 11:02 28

As my latest adop session was 28 i navigated into folder named '28' and look for the logs. If you are not sure about the session id, look for the adop screen where it will show the session id while it begins. Alternatively you can query ad_adop_sessions table for more session detail.  

-bash-3.2$ cd 28
  
ls -ltr 

drwxr-xr-x   3 applmgr  oinstall     512 Sep  5 10:18 fs_clone_20170905_101716
-rw-r--r--   1 applmgr  oinstall    4878 Sep  5 10:19 adop_20170905_101716.log
drwxr-xr-x   3 applmgr  oinstall     512 Sep 14 10:59 fs_clone_20170914_105815
-rw-r--r--   1 applmgr  oinstall    6910 Sep 14 10:59 adop_20170914_105815.log
drwxr-xr-x   3 applmgr  oinstall     512 Sep 14 11:02 fs_clone_20170914_110147
-rw-r--r--   1 applmgr  oinstall    5040 Sep 14 11:03 adop_20170914_110147.log

Look for the latest directory.

-bash-3.2$ cd fs_clone_20170914_110147
  
-bash-3.2$ ls
ADOPValidations_Thu_Sep_14_11_02_48_2017.log

There were only one log file since this issue was encountered at the beginning stage itself.   
-bash-3.2$ vi ADOPValidations_Thu_Sep_14_11_02_48_2017.log
  
Nodes with context files in the FND_OAM_CONTEXT_FILES table on both run and patch file systems: NONE
Nodes without context files in the FND_OAM_CONTEXT_FILES table on either/or run and patch file systems: ******
Corrective Action:
- If the run file system context file for a node is missing, run AutoConfig on the run file system of that node to sync with the value with the database.
- If the patch file system context file of a node is missing, run AutoConfig on the patch file system of that node with the -syncctx option as follows to sync with the
value with the database.

On UNIX:
 sh <AD_TOP>/bin/adconfig.sh contextfile=<CONTEXT_FILE> -syncctx



Above is the error logged in the log. As soon as i see this i queried the table to see whether both xml files (run and patch) are present in db. 

select * from apps.FND_OAM_CONTEXT_FILES;






 Results clearly showed me there is no xml file for patch fs in the db. So i ran the below to generate xml file for patch file system. 


-bash-3.2$ sh /u01/oracle/PROD/fs1/EBSapps/appl/ad/12.0.0/bin/adconfig.sh contextfile=/u01/oracle/PROD/fs1/inst/apps/PROD_******/appl/admin/PROD_******.xml -syncctx

Enter the APPS user password:
The log file for this session is located at: /u01/oracle/PROD/fs1/inst/apps/PROD_******/admin/log/09141113/adconfig.log

Option specified      : Synchronize context file
Only context file synchronization will be performed

        Classpath                   : /u01/oracle/PROD/fs1/FMW_Home/Oracle_EBS-app1/shared-libs/ebs-appsborg/WEB-INF/lib/ebsAppsborgManifest.jar:/u01/oracle/PROD/fs1/EBSapps/comn/java/classes

        Using ContextFile     : /u01/oracle/PROD/fs1/inst/apps/PROD_******/appl/admin/PROD_******.xml

        Synchronizing the context file ......COMPLETED

AutoConfig completed successfully.

I again queried to see the changes. Look into #8 row, we can see xml file row has been populated for patch file system now. 




fs_clone completed successfully after this change. 


Find FMW version in EBS R12.2.4


Please follow these steps to find out FMW version of your oracle application.

- Login as application owner (ex : applmgr)
- Source run fs env file

Then do the below ,

-bash-3.2$ cd $FMW_HOME

-bash-3.2$ cd Oracle_EBS-app1/

-bash-3.2$ pwd
/u01/oracle/PROD/fs2/FMW_Home/Oracle_EBS-app1

-bash-3.2$ export ORACLE_HOME=/u01/oracle/PROD/fs2/FMW_Home/Oracle_EBS-app1

-bash-3.2$ ./opatch lsinv
-bash: ./opatch: No such file or directory


-bash-3.2$ ls
META-INF            ccr                 common              install             oraInst.loc         shared-libs
OPatch              cfgtoollogs         deployment_plans    inventory           oui                 srvm
applications        clone               diagnostics         jdk                 setEBSDomainEnv.sh  sysman

-bash-3.2$ cd OPatch/

-bash-3.2$ ./opatch lsinv
Invoking OPatch 11.2.0.1.2

Oracle Interim Patch Installer version 11.2.0.1.2
Copyright (c) 2010, Oracle Corporation.  All rights reserved.


Oracle Home       : /u01/oracle/PROD/fs2/FMW_Home/Oracle_EBS-app1
Central Inventory : /u01/oracle/PROD/oraInventory
   from           : /var/opt/oracle/oraInst.loc
OPatch version    : 11.2.0.1.2
OUI version       : 11.2.0.3.0
OUI location      : /u01/oracle/PROD/fs2/FMW_Home/Oracle_EBS-app1/oui
Log file location : /u01/oracle/PROD/fs2/FMW_Home/Oracle_EBS-app1/cfgtoollogs/opatch/opatch2017-09-14_10-48-56AM.log

Patch history file: /u01/oracle/PROD/fs2/FMW_Home/Oracle_EBS-app1/cfgtoollogs/opatch/opatch_history.txt


OPatch detects the Middleware Home as "/u01/oracle/PROD/fs2/FMW_Home"

Lsinventory Output file location : /u01/oracle/PROD/fs2/FMW_Home/Oracle_EBS-app1/cfgtoollogs/opatch/lsinv/lsinventory2017-09-14_10-48-56AM.txt

--------------------------------------------------------------------------------
Installed Top-level Products (1):

Oracle E-Business Suite Component                                    11.1.1.6.0 --> FMW version
There are 1 products installed in this Oracle Home.


There are no Interim patches installed in this Oracle Home.


--------------------------------------------------------------------------------

OPatch succeeded.

Monday, September 4, 2017

How to change weblogic password if lost or forgotten : java s_nm_jvm_startup_properties with example


If the Admin Password of an EBS WebLogic Domain is lost or forgotten :

1. Shut down all running services. Since the AdminServer password is not known, the
servers cannot be stopped from the console and so must be killed as follows.

--1. Connect to the Oracle E-Business Suite instance and source the application tier
environment file.

--2. Identify the PIDs of Node Manager, AdminServer, and all running Managed
Servers:
$ ps -ef | grep "NodeManager"
$ ps -ef | grep "weblogic.Name=AdminServer"
$ ps -ef | grep "weblogic.Name=forms-c4ws_server"
$ ps -ef | grep "weblogic.Name=forms_server"
$ ps -ef | grep "weblogic.Name=oafm_server"
$ ps -ef | grep "weblogic.Name=oacore_server"

3. Kill all these processes, starting with Node Manager and followed by the
Managed Servers.

2. Back up these folders, and then delete them:
<EBS_DOMAIN_HOME>/security/ DefaultAuthenticatorInit.ldift
<EBS_DOMAIN_HOME>/servers/<server_name>/data/ldap
<EBS_DOMAIN_HOME>/servers/<server_name>/security/boot.properties
<EBS_DOMAIN_HOME>/servers/<server_name>/data/nodemanager/boot.
properties
Where:
• <EBS_DOMAIN_HOME> is the absolute path of the EBS WebLogic domain
• <server_name> is the name of the server directory under
<EBS_DOMAIN_HOME>.

If the password is not reset correctly, the backed up files and folders can be
restored.
Note: For certain servers, the boot.properties file may be present in
only one location of the two specified above. In such a case, back it
up and then delete it.

--3. Set up a new environment to change the WLS AdminServer password.
1. Start a new session and connect to the Oracle E-Business Suite instance.
2. Do not source the application tier environment file.
3. Run the following command to source the WebLogic Server domain
environment:
$ cd <EBS_DOMAIN_HOME>/bin
$ source setDomainEnv.sh
4. Run the following commands:
$ cd <EBS_DOMAIN_HOME>/security
$ java weblogic.security.utils.AdminAccount <wls_adminuser>
<wls_admin_new_password> .
Where:
• <wls_adminuser> is the same as the value of context variable
s_wls_admin_user
• <wls_admin_new_password> is the new WLS AdminServer password you
wish to set.
Note: Do not omit the trailing period ('.') in the above
command: it is needed to specify the current domain directory.


--4. Start AdminServer from the command line. You will be prompted for the WebLogic
Server username and password, so that the AdminServer boot.properties file
can be generated.
1. Go to the EBS Domain Home:
$ cd <EBS_DOMAIN_HOME>
2. Start AdminServer:
$ java <s_nm_jvm_startup_properties> -Dweblogic.system.
StoreBootIdentity=true -Dweblogic.Name=AdminServer weblogic.
Server

• <s_nm_jvm_startup_properties> is the same as the value of context variable
ss_nm_jvm_startup_properties

Example:

MAke sure to remove trailing space and line separators.

$ java -d64 -Xms1024m -Xmx1024m -XX:CompileThreshold=8000 -XX:PermSize=128m -XX:MaxPermSize=256m -Djava.security.policy=/u01/oracle/PROD/fs2/FMW_Home/wlserver_10.3/server/lib/weblogic.policy -Xverify:none -da -Dplatform.home=/u01/oracle/PROD/fs2/FMW_Home/wlserver_10.3 -Dwls.home=/u01/oracle/PROD/fs2/FMW_Home/wlserver_10.3/server -Dweblogic.home=/u01/oracle/PROD/fs2/FMW_Home/wlserver_10.3/server -Dcommon.components.home=/u01/oracle/PROD/fs2/FMW_Home/oracle_common -Djrf.version=11.1.1 -Dorg.apache.commons.logging.Log=org.apache.commons.logging.impl.Jdk14Logger -Ddomain.home=/u01/oracle/PROD/fs2/FMW_Home/user_projects/domains/EBS_domain_PROD -Doracle.server.config.dir=/u01/oracle/PROD/fs2/FMW_Home/user_projects/domains/EBS_domain_PROD/config/fmwconfig/servers/AdminServer -Doracle.domain.config.dir=/u01/oracle/PROD/fs2/FMW_Home/user_projects/domains/EBS_domain_PROD/config/fmwconfig -Digf.arisidbeans.carmlloc=/u01/oracle/PROD/fs2/FMW_Home/user_projects/domains/EBS_domain_PROD/config/fmwconfig/carml -Digf.arisidstack.home=/u01/oracle/PROD/fs2/FMW_Home/user_projects/domains/EBS_domain_PROD/config/fmwconfig/arisidprovider -Doracle.security.jps.config=/u01/oracle/PROD/fs2/FMW_Home/user_projects/domains/EBS_domain_PROD/config/fmwconfig/jps-config.xml -Doracle.deployed.app.dir=/u01/oracle/PROD/fs2/FMW_Home/user_projects/domains/EBS_domain_PROD/servers/AdminServer/tmp/_WL_user -Doracle.deployed.app.ext=/- -Dweblogic.alternateTypesDirectory=/u01/oracle/PROD/fs2/FMW_Home/oracle_common/modules/oracle.ossoiap_11.1.1,/u01/oracle/PROD/fs2/FMW_Home/oracle_common/modules/oracle.oamprovider_11.1.1 -Djava.protocol.handler.pkgs=oracle.mds.net.protocol -Dweblogic.jdbc.remoteEnabled=false -Dportlet.oracle.home=/u01/oracle/PROD/fs2/FMW_Home/oracle_common -Dem.oracle.home=/u01/oracle/PROD/fs2/FMW_Home/oracle_common -Dweblogic.management.discover=true -Dwlw.iterativeDev= -Dwlw.testConsole= -Dwlw.logErrorsToConsole= -Dweblogic.ext.dirs=/u01/oracle/PROD/fs2/FMW_Home/patch_wls1036/profiles/default/sysext_manifest_classpath -Dweblogic.system.StoreBootIdentity=true -Dweblogic.Name=AdminServer weblogic.Server

The above command prompts for the WebLogic Server username and password:
Enter username to boot WebLogic server:
Enter password to boot WebLogic server:

Provide the same credentials as you provided in Step 3.

--5. Change Node Manager password
1. Log in to the WebLogic Administration console.
2. Click the 'Lock & Edit' button.
3. In the left panel, click on the EBS Domain link.
4. Select the 'Security' tab.
5. Click on the 'Advanced' link.
6. Edit the 'Node Manager password' field and set it to the new WebLogic Server
password. The password should be same as set in Step 3.
7. Edit the 'Confirm Node Manager Password' field and set it to the new
WebLogic Server password. The password should be same as set in Step 3.
8. Save and activate the changes.

--6. The first time, AdminServer has to be stopped from the Admin console. Follow
these steps:
1. Log in to the WebLogic Administration console.
2. Shut down AdminServer.

--7. Set up your environment to start AdminServer again. AdminServer should now be
started using the normal AD script, which will also start Node Manager using the
new password.
1. Launch a new session and connect to the Oracle E-Business Suite instance.
2. Source the application tier environment file.
3. Start AdminServer with the following command:
$ $ADMIN_SCRIPTS_HOME/adadminsrvctl.sh

--8. Start the Managed Servers. For the first time, all Managed Servers should be started
from the WebLogic Server Admin console. This step will create boot.properties
files for the respective Managed Servers. Follow these steps:
1. Log in to the WebLogic Server Administration Console
2. Start all Managed Servers, one at a time

--9. Shut down all the Managed Servers. This is so the new credentials will be picked up
at the next startup. Follow these steps:
1. Log in to the WebLogic AdminServer console.
2. Shut down all Managed Servers.
3. Shut down AdminServer.

--10. Shut down Node Manager using the normal AD script.
$ $ADMIN_SCRIPTS_HOME/adnodemgrctl.sh stop

--11. Copy the boot.properties file for each Managed Server.
WebLogic Server native scripts use the boot.properties file. The above steps
have created the boot.properties file under
<EBS_DOMAIN_HOME>/servers/<Managed Server
name>/data/nodemanager, which is used by Node Manager. For each Managed
Server, copy the newly-generated boot.properties file from
<EBS_DOMAIN_HOME>/servers/<Managed Servername>/data/nodemanager to <EBS_DOMAIN_HOME>/servers/<ManagedServer name>/security.
The EBS WebLogic Server domain password has now been changed, and all servers
can now be started using the normal AD scripts.
To start AdminServer:
$ADMIN_SCRIPTS_HOME/adadminsrvctl.sh start
To start the Managed Servers:
$ $ADMIN_SCRIPTS_HOME/admanagedsrvctl.sh start <managed_server_name>

--12. The above steps have changed the Oracle WebLogic AdminServer password on the
run file system. You now need to perform an fs_clone operation, to change the
WebLogic EBS Domain password on the patch file system:
1. Launch a new session and connect to the Oracle E-Business Suite instance.
2. Source the application tier environment file.
3. Run the command:
3-42 Oracle E-Business Suite Setup Guide
$ adop phase=fs_clone

Monday, August 21, 2017

12.2 apps tier cloning fails - EBSapps/comn/clone/FMW/pasteBinary.sh ERROR: Script failed, exit code 255

Adcfgclone is referring to production inventory in test server due to which cloning is returning into the error. I have provided the error and action plan in this post.

Error :

Started ApplyAppsTier at Thu Aug 17 08:20:01 AST 2017
Version:
        ApplyAppsTier.java      :       120.11.12020000.6
#############################################################

START: Creating the FMW Home from ApplyAppsTier
FMW JDK version in use: 1.7.0_25

START: Creating FMW Home.
Running /d02/DEV/appltest/fs2/EBSapps/comn/clone/FMW/pasteBinary.sh -javaHome /d02/DEV/appltest/fs2/EBSapps/comn/util/jdk -al /d02/DEV/appltest/fs2/EBSapps/comn/clone/FMW/FMW_Home.jar -tl /d02/DEV/appltest/fs2/FMW_Home -invPtrLoc /var/opt/oracle/oraInst.loc -ldl /d02/DEV/appltest/fs2/inst/apps/DEV_ahqtest/admin/log/clone/fmwT2PApply -silent true -debug true -executeSysPrereqs false
Script Executed in 655369 milliseconds, returning status 255
ERROR: Script failed, exit code 255


I found the error from the logs below. Error is while creating fmw and i was able to figure out the rror in the folder below.

/d02/DEV/appltest/fs2/inst/apps/DEV_ahqtest/admin/log/clone/fmwT2PApply

Please look out for *.error file where errors will be logged. 

bash-3.2$ ls -ltr

CLONE2017-08-20_03-47-24PM.error  --> Error will be logged here. 
CLONE2017-08-20_03-47-24PM.log

Error logged :

"CLONE2017-08-17_08-20-03AM.error" 19 lines, 1981 characters
SEVERE : Aug 17, 2017 8:30:57 AM - ERROR - CLONE-20218   Cloning is not successful.
SEVERE : Aug 17, 2017 8:30:57 AM - CAUSE - CLONE-20218   An internal operation failed.
SEVERE : Aug 17, 2017 8:30:57 AM - ACTION - CLONE-20218   Provide the clone log and error file for investigation.
java.lang.Exception: Invalid Oracle home as its inventory pointer file /d02/DEV/appltest/fs2/FMW_Home/oracle_common/oraInst.loc has invalid entry for Oracle inventory.
        at oracle.as.clone.util.OracleHomeDepRefUtil.setInvPtrLocInInventoryManager(OracleHomeDepRefUtil.java:327)
        at oracle.as.clone.util.OracleHomeDepRefUtil.backupAndSetInvPtrLocBeforeInstallEngineInvoke(OracleHomeDepRefUtil.java:269)
        at oracle.as.clone.util.OracleHomeDepRefUtil.preSetupBeforeInstallEngineInvoke(OracleHomeDepRefUtil.java:207)
        at oracle.as.clone.util.OracleHomeDepRefUtil.setOHProperties(OracleHomeDepRefUtil.java:465)
        at oracle.as.clone.util.OracleHomeDepRefUtil.setOracleHomeProperties(OracleHomeDepRefUtil.java:424)
        at oracle.as.clone.mwhome.cloner.OracleHomeApplyCloner.updateOracleHomeInfo(OracleHomeApplyCloner.java:1904)
        at oracle.as.clone.mwhome.cloner.NormalOracleHomeApplyCloner.doPostApplyClone(NormalOracleHomeApplyCloner.java:127)
        at oracle.as.clone.request.ApplyCloneRequest.performFirstPostoperation(ApplyCloneRequest.java:248)
        at oracle.as.clone.request.ApplyCloneRequest.applyArchive(ApplyCloneRequest.java:219)
        at oracle.as.clone.request.ApplyCloneRequest._clone(ApplyCloneRequest.java:69)
        at oracle.as.clone.process.CloningExecutionProcess.execute(CloningExecutionProcess.java:131)
        at oracle.as.clone.process.CloningExecutionProcess.execute(CloningExecutionProcess.java:114)
        at oracle.as.clone.client.CloningClient.executeT2PCommand(CloningClient.java:238)
        at oracle.as.clone.client.CloningClient.main(CloningClient.java:115)
SEVERE : Invalid Oracle home as its inventory pointer file /d02/DEV/appltest/fs2/FMW_Home/oracle_common/oraInst.loc has invalid entry for Oracle inventory.

I found the production inventory location in the file above. Corrected the inventory location in the test server in the file below.

bash-3.2$ find ./ -name '*ora*.loc'
$RUN_FS/EBSapps/10.1.2/oraInst.loc

You can also do the below,

Change All orinst.loc
mv /var/opt/oracle/oraInventory  /var/opt/oracle/oraInventory.org
Remove all files under /tmp
Re-run adcfclone




  


Wednesday, May 10, 2017

Oracle EBS login page is not working

Error :

The system has encountered an error when servicing the request, Please try again. If the error persists, please contact the system administrator.

Just i wanted to share the issue i encountered today. We are unable to login to application due to the error above. We got the error instead of login page.

What i did?

- As soon as i get the error in login page, i checked the apache logs. I can see some chain failed error but it was tripping all the time (Not only today), So i decided it should nt be cause of the today error. 
- I logged into admin console and checked all managed servers since we are in 12.2 (Everything was runnig fine). 
-I bounced apache once but no use
- I went to db tier and connected to sqlplus where i found the actual issue. When i connected to sqlplus i got the error below :

ERROR:

ORA-00020: maximum number of processes (200) exceeded

I found the cause of the issue. Application was unable to create db connection since limit is overloaded. 

How i narrowed down the issue?

I did how many processed connected to db.

ps -ef|grep oratest (Make sure you have only one instance running under OS user, I had only one instance running under oratest)

I checked how many connections coming from outside,

ps -ef|grep -i local=no|grep oratest

I found out totally 170 sessions were created from outside. 

How i fixed?

I decided to kill all local=no processes to reduce number of processes.  My environment was test and i knew the impact. 

****Please double check and check with your manager or your peers before you kill any such process in your environment.****

ps -ef|grep -i local=no|grep -i oratest|grep -v grep|awk '{print $2}'|xargs kill -9

Above command will get all the process id's of local=no processes and kill them. 

Within a minute, limit is freed up and i was able to connect to sqlplus. As soon as application were able to connect to db, we got the login page.

Thanks!
Sikky


Sunday, May 7, 2017

11.2.0.3 to 12.1.0.2 upgrade for E-biz R12.2

Hello Friends,

I have  tested the below in my environment and i was able to upgrade the database to 12.1.0.2. This upgrade done for oracle E-biz R12.2 so it covers lot additional steps for application tier as well. Please refer these steps only if you are upgrading for E-business suite database.

a) Take a backup using RMAN
             (or)
b)  Perform Cold Backup  ( if  your database is in NOARCHIVELOG mode)

--> db_backup_directory >> Location of the Database backup.
--> controlfile_backup_directory >> Location of the Controlfile backup.

• Ensure that all database components/objects provided by Oracle are VALID in the source database prior to starting the upgrade.
• Ensure that you do not have duplicate objects in the SYS and SYSTEM schema.

SQL> SELECT dbms_stats.get_prefs('CONCURRENT') from dual;
When 'concurrent statistics gathering' is not not set to 'FALSE', change the value to 'FALSE before the upgrade.
BEGIN
DBMS_STATS.SET_GLOBAL_PREFS('CONCURRENT','FALSE');
END;
/


Scripts placed at /a02/upgrade_files_12c
cd <location of the script>
$ sqlplus / as sysdba
sql> alter session set nls_language='American';
sql> @dbupgdiag.sql
sql> exit

Review the report
$ORACLE_HOME/rdbms/admin/utlrp.sql

Please run health check scripts as recommended by oracle. Note 136697.1 hcheck.sql" script to check for known problems in Oracle8i, Oracle9i, Oracle10g and Oracle 11g

Script - /a02/upgrade_files_12c

Note 884522.1 How to Download and Run Oracle's Database Pre-Upgrade Utility
@$ORACLE_HOME/rdbms/admin/preupgrd.sql
Review the report and fix the errors.

Take a backup of db links. Copy the output and keep it safer.

SQL> SELECT 'CREATE '||DECODE(U.NAME,'PUBLIC','public ')||'DATABASE LINK '||CHR(10)
||DECODE(U.NAME,'PUBLIC',Null, 'SYS','',U.NAME||'.')|| L.NAME||chr(10)
  2    3  ||'CONNECT TO ' || L.USERID || ' IDENTIFIED BY "'||L.PASSWORD||'" USING
'''||L.HOST||''''
  4    5  ||chr(10)||';' TEXT
  6  FROM SYS.LINK$ L, SYS.USER$ U
  7  WHERE L.OWNER# = U.USER#;



TIMESTAMP WITH TIME ZONE Data Type –Conditional

DST 18 upgrade. This Can be done as part of post upgrade.

Ref : 1665676.1

Optimizer Statistics
EXEC DBMS_STATS.GATHER_DICTIONARY_STATS;

Before upgrading Oracle Database, you must wait until all materialized views have completed refreshing.

Run the following query to determine if there are any materialized view refreshes still in progress:

SQL> select s.obj#,o.obj#,s.containerobj#,lastrefreshdate,pflags,xpflags,o.name,o.owner#, bitand(s.mflags, 8) from obj$ o, sum$ s
where o.obj# = s.obj# and o.type# = 42 AND bitand(s.mflags, 8) = 8;

Ensure That No Files Need Media Recovery Before Upgrading

SELECT * FROM v$recover_file;

Ensure That No Files Are in Backup Mode Before Upgrading, make sure no files are in backup mode.

SQL> SELECT * FROM v$backup WHERE status != 'NOT ACTIVE';


Purge the Database Recycle Bin Before Upgrading :

SQL> PURGE DBA_RECYCLEBIN

NOTE : The database recycle bin must be empty during the upgrade process to avoid possible ORA-00600 errors and to minimize the upgrade time.

Verify SYS and SYSTEM Default tablespace
Ensure the users SYS and SYSTEM have 'SYSTEM' as their default tablespace.
You must have sufficient space in the 'SYSTEM' tablespace or set extents to unlimited.
SQL> SELECT username, default_tablespace
     FROM dba_users
     WHERE username in ('SYS','SYSTEM');


Check whether database has any externally authenticated SSL users

select name from sys.user$ where ext_username is not null and password='GLOBAL';

Location of datafiles, redo logs and control files
Note the location of datafiles, redo logs and control files. Also take a backup of all configuration files like listener.ora, tnsnames.ora, etc. from $ORACLE_HOME.

SQL> SELECT name FROM v$controlfile;
SQL> SELECT file_name FROM dba_data_files;
SQL> SELECT group#, member FROM v$logfile;

Review and Remove any unnecessary hidden/underscore parameters

SQL> SELECT name, value from SYS.V$PARAMETER WHERE name LIKE '\_%' ESCAPE '\' order by name;

Check the XDB ACLs has start_date and end_date ACE attributes
Before upgrading the database to 12c, please run the below query as SYS:

SQL> select aclid, start_date, end_date from xds_ace where start_date is not null;

Download and Install Oracle 12c Release 1 in a new Oracle Home and make sure there are no relinking errors.
Install database examples which is mandatory step for ebs. You wont have oracle database examples with 12c stage, you need to download examples through patch p21419221_121020_platform_9of10.zip.
Install the latest available Patchset from My Oracle Support (if available).
Install the latest opatch available for your platform and database version (if available).
Install the latest available Critical Patch Update (if available).

Copy following  configuration files from the $ORACLE_HOME of the database being upgraded to the new Oracle Home for Oracle Database 12c
          - Parameter file (spfile or pfile)
          - Password file  (orapwsid)

Remove or comment out obsolete and deprecated initialization parameters.

 Stop the listener for the database.

$ lsnrctl stop

After 12c oracle home installation

Execute the below in source:
@?/rdbms/admin/utluppkg.sql
    SET SERVEROUTPUT ON;
    exec dbms_preup.run_fixup_and_report('INVALID_SYS_TABLEDATA')
    SET SERVEROUTPUT OFF;
Save Database Control Files and Data with the emdwgrd Utility
$ emdwgrd -save -sid old_SID -path save_directory


if upgrading database from 11.2.0.3
It is recommended to apply the patch 21550777 on 12.1.0.2 ORACLE_HOME before starting the upgrade to Materialized views related issues.
/d02/DEV/oratest/12.1.0/OPatch/opatch apply

21550777
18966843
19291380
19393542
19472320
19627012
19649152
19779059
19835133
19896336
20093776
20181016
20204035
20294666
20798891
20830911
20887355
21153266
22223463
18793246
19591608
25643931 (19908836)
20766180
21321429
21864513
21904072
21967332
Footnote 2 - If a conflict is reported with Patch 18893947 roll back the conflicting patch before applying Patch 21967332.
22098146
22338374
Footnote 3 - If a conflict is reported with Patch 21188532 roll back the conflicting patch before applying Patch 22338374.
22496904
22731026
23089357
23595848
24481723
24560906
FOotnote 4 - If a conflict is reported with Patch 21286665 roll back the conflicting patch before applying Patch 24560906.
24911308
Footnote 5 - If a conflict is reported with Patch 23745950 roll back the conflicting patch before applying Patch 24911308.
20369415
20880215

Apply Jan cpu 2015

Rollback the patches below

20369415
20766180
21321429
21967332
22731026 (Not in oh)
24684434 (not in oh)
25154882 (Not in oh)
24326444 (Not in oh)
25203714 (Not in oh)
23595848
24481723
24732088
24911308
25643931

Applied Jan 2017 cpu

24732088
cd /a02/patches/12c_upg_files/24917987/24968615/24732088
opatch apply

25101514
24917972 JVM

Applied rolled back patches


20369415
20766180
21321429
21967332
22731026
24684434
25154882
24326444
25203714

$ perl $ORACLE_HOME/nls/data/old/cr9idata.pl


Install JRE 6 or 7 (Copy appsutil directory from source to target oracle home)
Ensure Applications patching cycle is complete
Shut down Applications server processes and database listener
Drop SYS.ENABLED$INDEXES (conditional)
Remove the MGDSYS schema (conditional)

• Create a new listener for Oracle Database 12c Release 1 (12.1) database
• Shutdown the database.

$ sqlplus "/as sysdba"
SQL> shutdown immediate;

bash-3.2$ export ORACLE_HOME=/d02/DEV/oratest/12.1.0/
bash-3.2$ export PATH=$ORACLE_HOME/bin:$PATH
bash-3.2$ export ORACLE_BASE=/d02/DEV/oratest

• Update the oratab entry to set the new ORACLE_HOME pointing to ORCL and disable automatic startup.

Sample : cat /etc/oratab

            #orcl:/opt/oracle/product/11.2/db_1:N
            orcl:/opt/oracle/product/12.2/db_1:N


$ cd $ORACLE_HOME/rdbms/admin
$ sqlplus "/ as sysdba"
SQL> startup UPGRADE
SQL> exit

Run the catctl.pl script from the new Oracle home.
In this release, the new Upgrade Utility, catctl.pl, replaces catupgrd.sql.
To run catctl.pl on Linux:
Example: Where parallelism is 6 ( n=6)

cd $ORACLE_HOME/rdbms/admin
$ORACLE_HOME/perl/bin/perl catctl.pl -n  6 -l $ORACLE_HOME/diagnostics catupgrd.sql



SQL> select count(*) from dba_objects where status='INVALID';

  COUNT(*)
----------
    103861


Post upgrade


Please create stats on fixed objects two weeks after the upgrade using the command:
   EXECUTE DBMS_STATS.GATHER_FIXED_OBJECTS_STATS;


 You should follow that with the execution of rdbms/admin/utlrp.sql, and a
 comparison of invalid objects before and after the upgrade using
 rdbms/admin/utluiobj.sql



DST 18 upgrade

After the upgrade to 12.1.0.2 you can:
* (recommended) update the 12.1.0.2 database(s) to DSTv18 (standard DST version of 12.1.0.2) by :
• or using the upg_tzv_check.sql and upg_tzv_apply.sql scripts in note 1585343.1 Scripts to automatically update the RDBMS DST (timezone) version in an 11gR2 or 12cR1 database .
• or following note 1509653.1 Updating the RDBMS DST version in 12c Release 1 (12.1.0.1 and up) using DBMS_DST from step 3a) using "18" as (<the new DST version number>) in that note.
* (optional) update to a higher DST than 18 version if this is needed.
The latest DST patch and related note on how to apply this is found in Note 412160.1 Updated Time Zones in Oracle Time Zone File patches under "C) Notes covering the current DST available updates".
Simply follow the note for the latest DST update


Conn / as sysdba
-- this gives the current RDBMS DST version
SELECT version FROM v$timezone_file;
-- check also
SELECT PROPERTY_NAME, SUBSTR(property_value, 1, 30) value
FROM DATABASE_PROPERTIES
WHERE PROPERTY_NAME LIKE 'DST_%'
ORDER BY PROPERTY_NAME;

Conn / as sysdba
spool countstatsTSTZ.log
@countstatsTSTZ.sql
spool off
For most databases the biggest amount of data that is affected by DST updates will be in DBMS_SCHEDULER tables.
Conn / as sysdba
exec dbms_scheduler.purge_log;
Conn / as sysdba
-- check current nr of rows in HISTHEAD / HISTGRM
select count(*) from SYS.WRI$_OPTSTAT_HISTGRM_HISTORY;
select count(*) from SYS.WRI$_OPTSTAT_HISTHEAD_HISTORY;
-- check the current retention of stats
-- the default value is 31
select systimestamp - dbms_stats.get_stats_history_availability from dual;
-- now disable stats retention
exec dbms_stats.alter_stats_history_retention(0);
-- remove all stats
exec DBMS_STATS.PURGE_STATS(systimestamp);
-- check result of purge
select count(*) from SYS.WRI$_OPTSTAT_HISTGRM_HISTORY;
select count(*) from SYS.WRI$_OPTSTAT_HISTHEAD_HISTORY;
-- AFTER the DST update you can set the retention back to the or
Conn / as sysdba
spool upg_tzv_check.log
@upg_tzv_check.sql
spool off
A succesfull run will show at the end:
 INFO: A newer RDBMS DST version than the one currently used is found.
 INFO: Note that NO DST update was yet done.
 INFO: Now run upg_tzv_apply.sql to do the actual RDBMS DST update.
 INFO: Note that the upg_tzv_apply.sql script will
 INFO: restart the database 2 times WITHOUT any confirmation or prompt.

Conn / as sysdba
spool upg_tzv_apply.log
@upg_tzv_apply.sql
spool off
A succesfull run will show at the end:
 INFO: The RDBMS DST update is successfully finished.
 INFO: Make sure to exit this sqlplus session.
 INFO: Do not use it for timezone related selects.
To see what's happening during upg_tzv_check.sql or upg_tzv_apply.sql one can use:
CONN / as sysdba

set PAGES 1000
select TARGET, TO_CHAR(START_TIME,'HH24:MI:SS - DD-MM-YY'), TIME_REMAINING, SOFAR,
TOTALWORK, SID, SERIAL#, OPNAME from V$SESSION_LONGOPS
where sid in
(select SID from V$SESSION where CLIENT_INFO = 'upg_tzv')
and SOFAR < TOTALWORK
order by START_TIME;

select S.SID, S.SERIAL#, S.SQL_ID, S.PREV_SQL_ID,
S.EVENT#, S.EVENT, S.P1TEXT, S.P1, S.P2TEXT, S.P2, S.P3TEXT, S.P3, S.TIME_REMAINING_MICRO,
S.SEQ#, S.BLOCKING_SESSION, BS.PROGRAM "Blocking Program",
Q1.SQL_TEXT "Current SQL", Q2.SQL_TEXT "Previous SQL"
from V$SESSION S, V$SQLAREA Q1, V$SQLAREA Q2, V$SESSION BS
where S.SQL_ID = Q1.SQL_ID(+)
and S.PREV_SQL_ID = Q2.SQL_ID(+)
and S.BLOCKING_SESSION = BS.SID(+)
and S.CLIENT_INFO = 'upg_tzv';


Fix for ORA-01804: failure to initialize timezone information:

This is resolved by editing the template ad8idbux.env in location .../oracle/db/tech_st/11.2.0/appsutil/template changing:

From:
set ORA_TZFILE=%s_db_oh%\oracore\zoneinfo\timezlrg.dat

To:
set ORA_TZFILE=%s_db_oh%\oracore\zoneinfo\timezlrg_17.dat



exec dbms_stats.alter_stats_history_retention(31);
After the upgrade, run utluiobj.sql from $ORACLE_HOME/rdbms/admin/ to identify/compare any new invalid objects due to the upgrade.
@$ORACLE_HOME/rdbms/admin/utluiobj.sql

20880215
AND perform one of the following as the table is already logically corrupted
a. Reinitialize the columns in the table:
alter table <owner>.<table_name> modify ( <column_name> null);
alter table <owner>.<table_name> modify ( <column_name> not null);
OR
b. Drop and recreate the affected table(s) ("schema"."table")

JAN CPU 2017 - Patch post installation instructions.

Is it enough if you run datapatch once for all patches. Though catctl.pl will cover this step. We can execute datapatch once if needed.

% ./datapatch –verbose


1. If the OJVM PSU is also installed, you may see invalid objects after execution of datapatch in the previous step. If this is the case, run utlrp.sql to revalidate these objects.
cd $ORACLE_HOME/rdbms/admin
    sqlplus /nolog
2. SQL> CONNECT / AS SYSDBA
3. SQL> @utlrp.sql
4. This patch now includes the OJVM Mitigation patch (Patch:19721304). If an OJVM PSU is installed or planned to be installed, no further actions are necessary. Otherwise, the workaround of using the OJVM Mitigation patch can be activated. As SYSDBA do the following from the admin directory:
5. SQL > @dbmsjdev.sql
6. SQL > exec dbms_java_dev.disable
For more information on the OJVM mitigation patch, see Document 1929745.1 Oracle Recommended Patches -- "Oracle JavaVM Component Database PSU" (OJVM PSU) Patches.

2.6.3 Upgrade Oracle Recovery Manager Catalog
If you are using the Oracle Recovery Manager, the catalog needs to be upgraded. Enter the following command to upgrade it:
$ rman catalog username/password@alias
RMAN> UPGRADE CATALOG;
RMAN> UPGRADE CATALOG;

SELECT TO_CHAR(action_time, 'DD-MON-YYYY HH24:MI:SS') AS action_time,
       action,
             version,
status,
      bundle_series from sys.dba_registry_sqlpatch order by action_time;

Copy $APPL_TOP/admin/adgrants.sql (adgrants_nt.sql for Windows) from the administration server node to the database server node.

sqlplus "/ as sysdba" @adgrants.sql (or adgrants_nt.sql)
    [APPS schema name]


SQL> conn / as sysdba
Connected.
SQL> @?/rdbms/admin/adgrants.sql APPS
Connected.

Set CTXSYS parameter

$ sqlplus "/ as sysdba"
SQL> exec ctxsys.ctx_adm.set_parameter('file_access_role', 'public');

Validate Workflow ruleset
On the administration server node, use SQL*Plus to connect to the database as APPS and run the $FND_TOP/patch/115/sql/wfaqupfix.sql script using the following command:
$ sqlplus [APPS user]/[APPS password] @wfaqupfix.sql\
  [APPLSYS user] [APPS user]

Deregister the current database server (conditional)
If you plan to change the database port, host, or database name parameter on the database server, you must also update AutoConfig on the database tier and deregister the current database server node.

$ sqlplus apps/[APPS password]
SQL> exec fnd_conc_clone.setup_clean;

Implement and run AutoConfig
Enabling AutoConfig on a New Oracle Home

 Install Java Runtime Environment (JRE) on the Database tier
** copy entire appsutil directory from source to target oracle home
The JRE resides in the <ORACLE_HOME>/appsutil/jre directory on the database
tier.

3. Generate the Database Context File
$ perl <RDBMS_ORACLE_HOME>/appsutil/bin/adbldxml.pl

4. Run AutoConfig on the Database tier
Run AutoConfig on the database tier by executing one of the following commands:
On UNIX:
$ <RDBMS_ORACLE_HOME>/appsutil/bin/adconfig.sh \
contextfile=<context_file>


References :
Complete Checklist for Manual Upgrades to Oracle Database 12c Release 1 (12.1) (Doc ID 1503653.1)
Interoperability Notes Oracle EBS 12.2 with Oracle Database 12c Release 1 (Doc ID 1926201.1)

Saturday, May 6, 2017

ORA-01804: failure to initialize timezone information

This error encountered during 12c upgrade. My source database was running in 11.2.0.3 with DSTv17 and i was upgrading database to 12.1.0.2. I had to updgrade DST to v18 as part of the upgrade. AFter DST upgrade i was getting the error below while connecting to sqlplus and while running autoconfig as well.

Error :

ORA-01804: failure to initialize timezone information

SP2-0152: ORACLE may not be functioning properly

The first thing to do when you hit ORA-1804 is to check which timezone file Oracle attempts to use:
If the ORA_TZFILE environment variable is set then that points to the file that will be used.
If ORA_TZFILE is not set then until Oracle 9.2.0.5 $ORACLE_HOME/oracore/zoneinfo/timezone.dat is the default file that is used, since the 9.2.0.5 patchset (and all higher versions up to 11.1.0.7) the default file is $ORACLE_HOME/oracore/zoneinfo/timezlrg.dat
Since 11.2 the .dat files have a prefix indicating the DST version. Hence there are NO timezone.dat or timezlrg.dat in 11.2 and higher . 

For example timezlrg_4.dat is the DSTv4 "large" file, timezlrg_11.dat is the DSTv11 "large" file.
The first thing to do is to check if ORA_TZFILE is set and if it is, remove it and restart the database and listener. Oracle will then try to load $ORACLE_HOME/oracore/zoneinfo/timezlrg.dat or, from 11.2 onwards, the $ORACLE_HOME/oracore/zoneinfo/timezlrg_XX.dat file with XX being the higher number available in the $ORACLE_HOME/oracore/zoneinfo/ directory.

If for some reason the file that Oracle wants to use is not available then this is the cause of the problem.
If the file is present then an alternative reason for the error could be that Oracle cannot load the file because of permission problems.

If all above is correct then it might be a .dat file that is corrupt.

In my scenario, timezlrg.dat file which is set to ORA_TZFILE was not present. From 11g onwards we have version introduced in file name. So i updated the file name as below in ad8idbux.env as mentioned below :

bash-3.2$ echo $ORA_TZFILE
/d02/DEV/oratest/12.1.0/oracore/zoneinfo/timezlrg.dat
bash-3.2$ ls -ltr /d02/DEV/oratest/12.1.0/oracore/zoneinfo/timezlrg.dat
/d02/DEV/oratest/12.1.0/oracore/zoneinfo/timezlrg.dat: No such file or directory

You can find timezone files present under the location /d02/DEV/oratest/12.1.0/oracore/zoneinfo. Update the highest version available. I updated timezlrg_17.dat

---------------------In ad8idbux.env file---------------------

if test "%s_database%" = "db112" -o "%s_database%" = "db121" ; then
     ORA_TZFILE=""
else
     ORA_TZFILE="$ORACLE_HOME/oracore/zoneinfo/timezlrg.dat"
fi
export ORA_TZFILE


1. This is resolved by editing the template ad8idbux.env in location .../oracle/db/tech_st/11.2.0/appsutil/template changing:

From:
set ORA_TZFILE=%s_db_oh%\oracore\zoneinfo\timezlrg.dat

To:
set ORA_TZFILE=%s_db_oh%\oracore\zoneinfo\timezlrg_17.dat

2. Running AutoConfig after this change means the environment file is created with the ORA_TZFILE value as timezlrg_14.dat and AutoConfig completes successfully


References :
Autoconfig Failed On 12c Database With "ORA-01804: failure to initialize timezone information" (Doc ID 1908836.1)
AutoConfig Encounters A ORA-1804 Error: Failure To Initialize Timezone Information (Doc ID 1917923.1)
AutoConfig Encounters A ORA-1804 Error: Failure To Initialize Timezone Information (Doc ID 1917923.1)




Note : these .dat files are only read by Oracle, never written to, so if these files become corrupt then the reason for they becoming corrupt is at OS level or I/O subsystem.
So while copying / reinstalling those files might solve the symptom (the ORA-1804) , the root cause is non-Oracle and a deep diagnostic of the filesystem/ I/O subsystem is strongly advised.
Also there is a big chance these .dat files are not the only files who became corrupt.

Because these timezone files are static and can only be updated by Oracle development the solution is the same in both cases:
Find a different database (same RDBMS and (!) DST version & platform) that does not have these problems (check for example by selecting from V$TIMEZONE_NAMES)
Copy the *.dat files from that system over to the failing system
Restart all Oracle processes
or Reinstall the Oracle software.

You cannot complete this task because one of the following events caused a loss of page data: A system failure has occurred "Front end is locked after 12.2 upgrade"

  If you have upgraded to Oracle E-Business Suite 12.2.10 Release Update Pack from Oracle E-Business Suite 12.2.6 Release Update Pack (or ea...