Search This Blog

Wednesday, August 8, 2018

Step by Step - How to install pre-install rpm in RHEL 7 for oracle ebs R12.2

Dear DBA's,

Installing prerequisite packages are one of the tedious tasks in any implementation or migration. Though this is one time task, its so mundane to check each rpm. Oracle has provided pre-install rpm which will save lot of our time on preparation tasks. All we need to do is configure yum repository and install the single package. Rest of the tasks will be taken care by yum and it will install all the needed pre-requisite packages for both apps and db tier.

Lets see how this can be done.

Go to the directory below. any file you see here with *.repo extension are repositories. YUM will refer this file and look for the packages to be installed. You can either provide any link to download the packages or we can provide the location  (if you have packages in local disk ).

I am going to install preinstall rpm on rhel 7.5 64 bit, So i would use the correspoding ol7 pre install rpm for this installation.

As root user :

# cd /etc/yum.repos.d




Enter the below command as root user and it will create the repository file for preinstall RPM. 

wget http://public-yum.oracle.com/public-yum-ol7.repo



AS you see the above pic public-yum-ol7.repo have been created. Keep only this repository and move other repositories temporarily. 

Open the repository in Vi file and updated enabled=0 to enabled=1 for the items below. You will see lot of items in the repository but only the below are recommended. Items which enabled=1 will be installed by YUM.

[ol6_latest]

name=Oracle Linux $releasever Latest ($basearch)

baseurl=http://public-yum.oracle.com/repo/OracleLinux/OL6/latest/$basearch/

gpgkey=http://public-yum.oracle.com/RPM-GPG-KEY-oracle-ol6

gpgcheck=1

enabled=1


[ol6_addons]

name=Oracle Linux $releasever Add ons ($basearch)

baseurl=http://public-yum.oracle.com/repo/OracleLinux/OL6/addons/$basearch/

gpgkey=http://public-yum.oracle.com/RPM-GPG-KEY-oracle-ol6

gpgcheck=1

enabled=1



[ol6_UEK_latest]

name=Latest Unbreakable Enterprise Kernel for Oracle Linux $releasever ($basearch)

baseurl=http://public-yum.oracle.com/repo/OracleLinux/OL6/UEK/latest/$basearch/

gpgkey=http://public-yum.oracle.com/RPM-GPG-KEY-oracle-ol6

gpgcheck=1


enabled=1

# yum update


Type yes "Y"



Now we can install the packages with the below command and Yum will take care of the rest.

# yum install oracle-ebs-server-R12-preinstall








Thanks!
~Sikky


Couldn't open file /etc/pki/rpm-gpg/RPM-GPG-KEY-oracle error while installing Pre-install RPM

Hi,

OS : RHEL 7,5 64-bit
Preinstall RPM : OEL 7

I got this error while installing Pre-install RPM . I have shared the steps in this post to fix the same. This fix worked perfectly for me and i never publish a fix unless i tested it.

Error :

GPG key retrieval failed: [Errno 14] curl#37 - "Couldn't open file /etc/pki/rpm-gpg/RPM-GPG-KEY-oracle"

Reason :

GPG key is missing.

Solution :

Download the GPG key.


[root@***** ~]# wget http://public-yum.oracle.com/RPM-GPG-KEY-oracle-ol7 -O /etc/pki/rpm-gpg/RPM-GPG-KEY-oracle
--2018-08-07 10:58:53--  http://public-yum.oracle.com/RPM-GPG-KEY-oracle-ol7
Resolving public-yum.oracle.com (public-yum.oracle.com)... 23.34.86.200
Connecting to public-yum.oracle.com (public-yum.oracle.com)|23.34.86.200|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1011 [text/plain]
Saving to: ‘/etc/pki/rpm-gpg/RPM-GPG-KEY-oracle’

100%[====================================================================================================>] 1,011       --.-K/s   in 0s

2018-08-07 10:58:54 (138 MB/s) - ‘/etc/pki/rpm-gpg/RPM-GPG-KEY-oracle’ saved [1011/1011]

Now re-run the installation.

Thanks!
~Sikky

Step by Step R12.2 cloning (Application and DB)


My dearest DBA's,

** I will be happy if you leave a comment or share this post **

Hope this post will be helpful for freshers who does cloning for first time in EBS. 

1.                Pre-Clone

- You can take a backup of the files below (Conditional) :
- /u01/app/oracle/ebzd/R122/EBSapps.env (save actual copy, not symlink)
- $CONTEXT_FILE (both tiers)
- $TNS_ADMIN (both tiers)
- $ORACLE_HOME/dbs (exclude any logfiles)
- Create pfile from spfile
- Create controlfile from trace
- - $<Custom_TOP> (All the files under custom top)

Note :

I have commented few items as conditional since they are environment specific. Please double check whether it actually needed for your environment.

2.                Prepare  Source Instance

Check for in-progress online patching cycle:
On the Source system, there should not be any in-progress online patching cycle. Online
patching should be done till cleanup stage and also fs_clone should be run to synchronize with
the application file system RUN and PATCH.

You can check it easily by below command:

SQL> select ADOP_SESSION_ID, PREPARE_STATUS, APPLY_STATUS, FINALIZE_STATUS,
CUTOVER_STATUS, CLEANUP_STATUS, ABORT_STATUS, STATUS, ABANDON_FLAG,
NODE_NAME from AD_ADOP_SESSIONS order by ADOP_SESSION_ID;

 Or check with adop -status and make sure all the phases are completed properly. If any outstanding patch session, Either complete it or abort before cloning.

2.1 OS and Patch Level

                    Source and Target must be on the same Operating System (OS) and patch level

2.2 Check Disk Space on the Source system and Target

                    Ensure the Source System has enough free disk space. Oracle Fusion Middleware cloning tools require 6GB in /tmp and 6GB under $COMMON_TOP. On the target system file system space should match as that of source.

2.3       AD/TXK Patches

                    Apply the latest AD/TXK patches to the Source system.

                   You can check the lastet version of source instance by below SQL statement.

                SQL> SELECT abbreviation,codelevel FROM AD_TRACKABLE_ENTITIES WHERE abbreviation in (‘txk’,’ad’);

 Output :
                    ABBREVIA CODELEVEL

                    --------        -------------------------

                     ad              C.7

                     txk             C.7


Conditional note :

               If we are not on latest version of AD/TXK patches then we need to apply (Not mandatory step)

Note: Ensure all the services are up in the Source System before proceeding with the following two       steps

3.    Prepare Source Tier

3.1            DB Tier

- You can refresh the db through any of the method as per your convenient.
- If the db is running already in target server, you can simply drop the db and refresh database from source.

Methods can be followed for refresh :

- Cold backup
- Hot backup
- Rman backup

Please read these posts for RMAN methods :

RMAN active db duplication  Read
RMAN backup based duplication Read

3.1.1    Run adpreclone.pl script

cd $ORACLE_HOME/appsutil/scripts/${CONTEXT_NAME}

perl adpreclone.pl dbTier

Note :  $CONTEXT_NAME is nothing but <instance-name>_<hostname>

3.2         Apps Tier

                   Log on to the Application node of the Source System as the applmgr user.

                   Source the environment file of the Run Edition File system.
          
                   $. EBSapps.env run

Important note :

When you do cloning make to sure to note down the run file system in the source. If your source has fs1 has the run file system then we must refresh fs1 as run file system in the target. If source has fs2 in run file system then we must refresh fs2 as run file system in the target.



For example : If my run file system is in fs2 in source then i would do the below.

 - Make fs2 in target (Run file system first)

 - Make fs1 from fs2 (PAtch file system)

3.2.1        Run adpreclone.pl script

                             Check edition , should run file system

                             echo $FILE_EDITION

                             Execute adpreclone

                             $cd $ADMIN_SCRIPTS_HOME

                             $perl adpreclone.pl appsTier


3.2.2        Tar Application RUN File system

                              $. EBSapps.env run

                              Check edition , should return run file system

                              echo $FILE_EDITION

                             $cd $RUN_BASE

                              Ex: /u01/app/oracle/ebz2/R122/fs2

                             $nohup tar –czvf  ebz2_runfs.tar.gz  EBSapps  >  ebz2_runfs.log &

Note: Our run file system is  fs2, so we are compressing fs2 directory.  Its enough if we only copy EBSapps directory under run filesystem. Rest of the directories will be created in target through adcfgclone.


4      Prepare Target System

Backup Context & Config Files (please see Pre Clone steps; you should have saved off the necessary information then; if not, please do so now)


4.1        Backup  files in Target Db Tier/

Take the backup of inti<sid>.ora and $CONTEXT_FILE using  “cp” command.

4.2        Backup  file in Target APPS Tier CONTEXT_FILE and custom env file

Take the backup of CONTEXT_FILE and CUSTOM ENV file usins “cp” command.

 Note : Make sure both apps and DB services are down in target instance before proceeding with below steps

 5         Cleanup Target Application File System

Remove application directory structure from target application instance under $RUN_BASE  
And $PATCH_BASE in the target application tier system.

                   $. EBSapps.env run

bash-3.2$ cd /a01/apps/CLONE/

bash-3.2$ ls

EBSapps.env   L2866958.log  L2866963.log  fs1           fs2           fs_ne         scripts

$ pwd
/a01/apps/CLONE/
$ rm -rf *

Once everything is deleted make a empty directories for run and patch file system.

mkdir -p /a01/apps/CLONE/fs1
mkdir - /a01/apps/CLONE/fs2
Clean old inventory files from local inventory in application tier. Please check local inventory location in /etc/oraInst.loc (other wise you will get error saying oraInventory already exists)


[applmgr@xxxxxxxxxxx ~]$ cat /etc/oraInst.loc

inventory_loc=/u01/app/oracle/ebz2/oraInventory

 cd /u01/app/oracle/ebz2/
mv oraInventory oraInventory_old
mkdir oraInventory

 Cleanup Target Database File

Drop destination DB (If already DB exists)

                    $sqlplus / as sysdba

                    SQL> shutdown immediate;

                    SQL> startup mount exclusive restrict;

                    SQL> drop database;

 You can also go to the location where db files present and clear using rm -rf command.

 Cleanup DB files in ASM (conditional)

Note : If Target DB files are placed under ASM, we need to connect ASM and remove control  files, Data files and redo log files.

Go to server xxxxdbxt02
Sudo –iu grid
 [grid@xxxxdbxt02 ~]$ cat /etc/oratab
+ASM2:/u01/app/12.1.0.2/grid:N          # line added by Agent
Use the below to source the env,
+ASM2 – instance name
/u01/app/12.1.0.2/grid – Grid_Oracle_HOME

 [grid@xxxxdbxt02 ~]$ . oraenv

   ORACLE_SID = [grid] ? +ASM2

   The Oracle base has been set to /u01/app/grid

 [grid@xxxxdbxt02 ~]$ asmcmd

   ASMCMD> lsdg
It will list the available disk groups, go to releavant disk group and delete your DB files under Target sid.


ASMCMD> cd DATA/Target_sid

                  ASMCMD> ls

                  CONTROLFILE/

                  DATAFILE/

                  TEMPFILE/


Go to each directory and execute the below,

 ASMCMD> rm –rf *


Performe the following steps if you are not reusing the old OHOME (Conditional):

Manual DB drop

 Delete all the data files from target database instance. Also check control_file & log_file          

   Location and remove them too.
Use simple rm –rf command to delete these database files.


Deregister old OHOME (Conditional)

cd $ORACLE_HOME/oui/bin

./runInstaller -silent -detachHome -invPtrLoc /etc/oraInst.loc ORACLE_HOME="/u01/app/oracle/product/11.2.0.4/ebzd"

Copy oracle_home from the source to target.

 Verify Inventory & oratab entry


4.1         DB Tier Inventory


                    $cat /etc/oraInst.loc

                    Note: DB Inventory is always set to global inventory “/u01/app/oraInventory”


4.2         Oratab entry - - Only for new instances


                      $vi /etc/oratab

                      Add new instance oracle home entry



4.3         ADCFGCLONE

 Its always recommended to copy source oracle home to target and run adcfgclone.pl

Clone DB Tier DataBase

 Note : For building a new instance (Oracle_Home), Steps to clone DB oracle_home from source to    target given here


11.1     Run adcfgclone in DB Tier – Only for New Instances/ Oracle Home binaries copied


                   a) Log on to the Target Instance as the oracle user(DB binaries owner).

                   b) Source the Database tier environment file.

                   c) Execute the following commands:

                       $ORACLE_HOME/appsutil/clone/bin

                       $perl adcfgclone.pl dbTechStack



Disable RAC mode (Conditional):

cd $ORACLE_HOME/rdbms/lib

make -f ins_rdbms.mk rac_off

make -f ins_rdbms.mk ioracle





adcfgclone options for DB tier :



adcfgclone.pl dbTier - This option will be used specifically for cold backup. This option will create and configure oracle home, config files and database in the target.

adcfgclone.pl dbTechStack - This option will be used mainly if we are duplicating database thru rman or any other hot backup method. We must need to manually copy the source oracle home to target. dbTechStack option will create all the config files (ex : xml file) and configure target oracle home.


11.2     Create context file for DB – if previous context file is not available (Conditional)


                Go to $ORACLE_HOME/appsutil/bin

                [oracle@xxxxdbxt02 bin]$ perl adbldxml.pl


               Answer the questions according to your Target DB server and create context file for DB tier.



4.4         Pre Rman Dupicate setup:


Copy the backed up init<sid>.ora and CONTEXT FILE to auxiliary server.

Create a password file in target DB using “orapw” and copy the same in Auxiliary DB under       

                     $ORACLE_HOME/dbs location

Source the environment using <context_name>.env

Start instance in nomount

Choose any of below RMAN scenario to to restore

- Cold backup
- Hot backup
- Rman backup


6      Post cloning DB activity

6.1         Run EBS Technology Codelevel Checker (ETCC) In DB Tier

Download the latest version from MOS

                      sh checkDBpatch.sh

                      Enter the full path to the DB Context file

                       :/u01/app/oracle/product/11.2.0.4/ebzclone/appsutil/ebzclone_xxxxdbxt02.xml



6.2         Node table clean up

Connect as apps user

                     Sqlplus apps/<Source_apps_pwd>

Exec the following :

                    Select node_name from fnd_nodes;

                    It will display source apps and DB nodes.

Cleanup fnd_nodes table :

                    Sql>exec fnd_conc_clone.setup_clean;

                    Sql>commit;


Select node_name from fnd_nodes;

                 It will retrun no rows.


6.3         Edit template for listener (Conditional)


Edit file: /u01/app/oracle/product/11.2.0.4/ebzd/appsutil/template/ad8ilsnr.ora
Remove this line:
(ADDRESS = (PROTOCOL = TCP)(HOST = %s_virtual_hostname%.%s_domainname%)(PORT = %s_dbport%))


Add these lines:
(ADDRESS_LIST =

(ADDRESS = (PROTOCOL = TCP)(HOST = %s_virtual_hostname%.%s_domainname%)(PORT = %s_dbport%)(IP = FIRST)))

(ADDRESS_LIST =

(ADDRESS = (PROTOCOL = TCP)(HOST = %s_dbhost%)(PORT = %s_dbport%)(IP = FIRST)))

6.4         Run autoconfig in db tier


                     Source target environment file

                     Execute adautocfg.sh

                    $ORACLE_HOME/appsutil/scripts/$CONTEXT_NAME/adautocfg.sh

                     Connect as apps user

                    Sqlplus apps/<Source_apps_pwd>


                   Exec the following :

                   Select node_name from fnd_nodes;

                   It will display only Target DB node.



Restore wallet information to sqlnet ifile (conditional)
Place Pending jobs on hold (Conditional)


update APPLSYS.FND_CONCURRENT_REQUESTS set HOLD_FLAG = 'Y' where PHASE_CODE  = 'P';



7      APPS Tier Cloning


7.1         Copy APPS Tier Run File System


                     Login to Target application server as applmgr user

                    Copy the apps filesystem binaries from source to destination.


Using scp command


Login to Target(ebzclone) application system using applmgr user

                    $scp –rp <ad_id>@<src_db_server>:/scratch/ebz2_runfs.tar.gz  /<stage_location>


 Note :

We need to copy this zip file under fs2 where we intially directoty.

Using sftp

Login to Target(ebzclone) application system using applmgr user

                cd <stage_location>

                ftp username@<source(ebz2) instance name>

               cd /scratch/

              mget ebz2_runfs.tar.gz

After finishing copy of backup,comeout from ftp session by entering “quit” at command prompt

Create apps filesystem directories – Only for New Instance

              $mkdir –p /u01/app/oracle/ebzclone/R122/fs1 /u01/app/oracle/ebzclone/R122/fs2

Untar apps file system

Note: In destination server Set $RUN_BASE and $PATCH_BASE based on source run and patch file system.

If run filesystem is fs2 in source instance, set same fs2 as run in destination instance
and untar.

              $export $RUN_BASE=/u01/app/oracle/ebzclone/R122/fs2

           $cd /u01/app/oracle/ebzclone/R122/fs2

           $nohup tar –xvzf  /<stage_location>/ebz2_runfs.tar.gz &

           $export PATCH_BASE= /u01/app/oracle/ebzclone/R122/fs1

                                                                               

Configure Target Apps Tier


Check the oraInst.loc file entry ,it must contain the entry for target instance


        App Tier Inventory
$cat /etc/oraInst.loc
inventory_loc=/u01/app/oracle/ebzclone/oraInventory
inst_group=oinstall


8.1         Run adcfgclone in Apps Tier

Make sure you are running adcfgclone from the same file system as your source

For example : If my run file system is in fs2 in source then i would do the below.

 - Make fs2 in target (Run file system first)

 - Make fs1 from fs2 (PAtch file system)


$ cd /u01/app/oracle/ebzclone/R122/fs2/EBSapp/comn/clone/bin
$ perl adcfgclone.pl appsTier </path/to/XML/file>

If you do not have the XML file, use the following method:

[applmgr@xxxxvebsxd03 bin]$ perl adcfgclone.pl appsTier

Enter the APPS password :

Do you want to add a node (yes/no) [no] :

Target System File Edition type [run] :


Provide the values required for creation of the new APPL_TOP Context file.

Target System Hostname (virtual or normal) [xxxxVEBSXD03] :

Target System Database SID : ebzclone

Target System Database Server Node [xxxxxxx] : xxxxxx

Target System Database Domain Name [xxxxx.com] : -- Hit Enter

Target System Base Directory : /u01/app/oracle/ebzclone/R122

Target System Base Directory set to /u01/app/oracle/ebzclone/R122

Target System Current File System Base set to /u01/app/oracle/ebzclone/R122/fs2



Target System Other File System Base set to /u01/app/oracle/ebzclone/R122/fs1

Target System Fusion Middleware Home set to /u01/app/oracle/ebzclone/R122/fs2/FMW_Home

Target System Web Oracle Home set to /u01/app/oracle/ebzclone/R122/fs2/FMW_Home/webtier

Target System Appl TOP set to /u01/app/oracle/ebzclone/R122/fs2/EBSapps/appl

Target System COMMON TOP set to /u01/app/oracle/ebzcnv1/R122/fs2/EBSapps/comn

Target System Instance Home Directory [/u01/app/oracle/ebzclone/R122] :  --Hit Enter

Target System Instance Top set to /u01/app/oracle/ebzcnv1/R122/fs2/inst/apps/ebzclone_xxxxvebsxd03

Do you want to preserve the Display [xxxxvebsxd03:0.0] (y/n)  : n

Target System Display [xxxxvebsxd03:0.0] :

 Target System Root Service [enabled] :

Target System Web Administration [enabled] :

Target System Web Entry Point Services [enabled] :

Target System Web Application Services [enabled] :

Target System Batch Processing Services [enabled] :

Target System Other Services [disabled] :

Enter port pool value[0] : 37 (standard port value for ebzclone instance)

-

-

UTL_FILE_DIR on database tier consists of the following directories.



1. /usr/tmp

2. /archive/ebz/inbound

3. /archive/ebz/outbound

4. /apps9/oracle_cifs_mount/operations/shipping_label

Choose a value which will be set as APPLPTMP value on the target node [2] :


 File system clean-up


                     After finishing adcfgclone in run file system,

                     Run the correct ENV file

                     cd /u01/app/oracle/ebzclone/R122/

                    . EBSapps.env  run



                    Make sure you have sourced the run file system


                    echo $FILE_EDITION

                    It should be RUN



                   Delete .env files belong to source(ebz2) instance

                   cd $APPL_TOP

                  rm –rf *ebz2*.env


Copy the Target Run file system to Patch file system:


Do the below in Target server :

fs2 - run file system
fs1- patch file system

                       cd /u01/app/oracle/ebzclone/R122/fs2/

                       cp –r EBSapp /u01/app/oracle/ebzclone/R122/fs1/

 Note :only do copy of “EBSapps” directory,No need to copy FMW_Home and inst directories.
Adcfglone will create those directories

        Run adpreclone on target run file system

 Check edition , should run file system

                             echo $FILE_EDITION

                             Execute adpreclone

                             $cd $ADMIN_SCRIPTS_HOME

                             $perl adpreclone.pl appsTier

  
         Copy run fs to patch fs
         Run adcfgclone on Patch file System(Target):


                      cd /u01/app/oracle/ebzclone/R122/fs1/EBSapp/comn/clone/bin

$echo $CONTEXT_FILE (copy the run file system context file)

perl adcfgclone.pl appsTier


  Target System File Edition type [run] : patch

  Note : It will prompt you for a context file,make sure you will give the fullpath of Run file system 
Context file  (DO NOT use the one you saved off, use the one from the RUN file system)

Location of Run System Context File: <Full patch to run fs context file>

                       -

                       -

      Target System Port Pool [0-99] : 37 if patch is fs1, 38 if patch is fs2

  After finshing adcfgclone start all the services on run file system

 cd /u01/app/oracle/ebzcnv1/R122/fs1/EBSapp/

                       .EBSapps.env run

                           echo $FILE_EDITION

                           it should be RUN


cd $ADMIN_SCRIPTS_HOME

                          ./adstrtal.sh <apps_username>/<apps_password>

  Validate the front end
  
Login into the front end using sysadmin user and perform the validate steps and validate
The application to ensure everything is fine.


Post cloning activities in R12.2 :

Stop application services


                     Go to $ADMIN_SCRIPTS_HOME

                     $ ./adstpall.sh apps/<apps_pwd>



 Reset apps & other passwords:


                      Reset sys & system password:

  $sqlplus / as sysdba

       SQL>alter profile APPS_PROFILE limit PASSWORD_VERIFY_FUNCTION null ;

                       SQL>altyer user sys identified by <newpassword>;

                       SQL>alter user system identified by <newpassword>;

   ALLORACLE

  FNDCPASS apps/<old_password> 0 Y system/system123 ALLORACLE
 <newpassword>

 SYSADMIN --> FNDCPASS apps/not2share123 0 Y system/<password> USER sysadmin

                                                 <new_password>

APPS--> FNDCPASS apps/<old_password> 0 Y system/mock2system SYSTEM APPLSYS

                                     <new password>

 Note: Oracle recommends to use AFPASSWD utility to change the passwords over 
FNDCPASS.

 SQL>alter profile APPS_PROFILE limit PASSWORD_VERIFY_FUNCTION VERIFYPASSWORD14 ;

  Change Apps password in R12.2.4 (Weblogic Console)

                      Start AdminServer using the adadminsrvctl.sh script from your RUN filesystem

                      Do not start any other application tier services.

                      Update the “apps” password in WLS Datasource as follows:

                      Log in to WLS Administration Console.

                      Click Lock & Edit in Change Center.

                      In the Domain Structure tree, expand Services, then select Data Sources.

                      On the “Summary of JDBC Data Sources” page, select EBSDataSource.

                      On the “Settings for EBSDataSource” page, select the Connection Pool tab.

                        Enter the new password in the “Password” field.

                       Enter the new  password in the “Confirm Password” field.

                       Click Save.

                       Click Activate Changes in Change Center.

                       Start all the application tier services using the adstrtal.sh script.

 Validate the changed passwords

                        Try to login with changed passwords and validate the functioning



9      FS_CLONE (Conditional)


Run the fs_clone in application instance and make sure it should be completed without errors.



cd /u01/app/oracle/ebzclone/R122/

.EBSapps.env run

echo $FILE_EDITION

it should be RUN



]$ adop phase=fs_clone



Check the log files for errors if comes and resolve them after that again run fs_clone














You cannot complete this task because one of the following events caused a loss of page data: A system failure has occurred "Front end is locked after 12.2 upgrade"

  If you have upgraded to Oracle E-Business Suite 12.2.10 Release Update Pack from Oracle E-Business Suite 12.2.6 Release Update Pack (or ea...