Oracle Exadata Patching

 

Oracle Exadata Patching (Oracle Homes)

Task#1: patch the Grid Infrastructure (GI) and Oracle RDBMS homes

Target Patches

  • GI & RDBMS Home Patch: p37641958_190000_Linux-x86-64.zip
  • OJVM Patch (if required): p37499406_190000_Linux-x86-64.zip
  • Recommended DB Patches:
    • p34774667_1927000DBRU_Linux-x86-64.zip
    • p34672698_1927000DBRU_Linux-x86-64.zip

Step 1: Configure SSH Equivalence from node1.

## dcli -g /u01/stage/target_Nodes -l root -k -s '-o StrictHostKeyChecking=no'

## dcli -g /u01/stage/target_Nodes -l root hostname


Step 2: Prepare and Copy Patches

## dcli -g /u01/stage/target_Nodes -l root mkdir -p /u01/stage/GI_DB_OJVM_19c/

dcli -g /u01/stage/target_Nodes -l root -f /u01/stage/GI_DB_OJVM_19c/p37641958_190000_Linux-x86-64.zip -d /u01/stage/GI_DB_OJVM_19c/

## dcli -g /u01/stage/target_Nodes -l root -f /u01/stage/GI_DB_OJVM_19c/p37499406_190000_Linux-x86-64.zip -d /u01/stage/GI_DB_OJVM_19c/

## dcli -g /u01/stage/target_Nodes -l root -f /u01/stage/GI_DB_OJVM_19c/p34774667_1927000DBRU_Linux-x86-64.zip -d /u01/stage/GI_DB_OJVM_19c/

## dcli -g /u01/stage/target_Nodes -l root -f /u01/stage/GI_DB_OJVM_19c/p34672698_1927000DBRU_Linux-x86-64.zip -d /u01/stage/GI_DB_OJVM_19c/

## dcli -g /u01/stage/target_Nodes -l root chown -R oracle:oinstall /u01/stage/GI_DB_OJVM_19c/

## dcli -g /u01/stage/target_Nodes -l root chmod -R 666 /u01/stage/GI_DB_OJVM_19c/

 

# Unzip patches:

## dcli -g /u01/stage/target_Nodes -l grid "unzip -oq /u01/stage/GI_DB_OJVM_19c/p37641958_190000_Linux-x86-64.zip -d /u01/stage/GI_DB_OJVM_19c/"

## dcli -g /u01/stage/target_Nodes -l oracle "unzip -oq /u01/stage/GI_DB_OJVM_19c/p37499406_190000_Linux-x86-64.zip -d /u01/stage/GI_DB_OJVM_19c/"

## dcli -g /u01/stage/target_Nodes -l oracle "unzip -oq /u01/stage/GI_DB_OJVM_19c/p34774667_1927000DBRU_Linux-x86-64.zip -d /u01/stage/GI_DB_OJVM_19c/"

## dcli -g /u01/stage/target_Nodes -l oracle "unzip -oq /u01/stage/GI_DB_OJVM_19c/p34672698_1927000DBRU_Linux-x86-64.zip -d /u01/stage/GI_DB_OJVM_19c/"

 

## Pre-Checks

## dcli -g /u01/stage/target_Nodes -l grid /u01/app/19.0.0.0/grid/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /u01/stage/GI_DB_OJVM_19c/37641958/37642901

 

#Check required space for the patch

cat /tmp/patch_list_gihome.txt

/u01/stage/GI_DB_OJVM_19c/37641958/37642901

/u01/stage/GI_DB_OJVM_19c/37641958/37654975

/u01/stage/GI_DB_OJVM_19c/37641958/37643161

/u01/stage/GI_DB_OJVM_19c/37641958/37762426

/u01/stage/GI_DB_OJVM_19c/37641958/36758186

 

# dcli -g /u01/stage/target_Nodes -l root -f /tmp/patch_list_gihome.txt -d /tmp/

# dcli -g /u01/stage/target_Nodes -l root chown grid:oinstall /tmp/patch_list_gihome.txt

# dcli -g /u01/stage/target_Nodes -l grid /tmp/patch_list_gihome.txt prereq CheckSystemSpace -phBaseFile /tmp/patch_list_gihome.txt

 

Step 4: Stop Oracle Homes

$ srvctl stop home -oraclehome /u01/app/oracle/product/19.0.0.0/dbhome_1 -statefile /tmp/orahome1 -node Target_Node_1 -stopoption immediate

$ srvctl stop home -oraclehome /u01/app/oracle/product/19.0.0.0/dbhome_1 -statefile /tmp/orahome2 -node Target_Node_2 -stopoption immediate

 

Step 5: Apply the Patches

# As root, Apply GI Home Patch

/u01/app/19.0.0.0/grid/OPatch/opatchauto apply /u01/stage/GI_DB_OJVM_19c/37641958 -oh /u01/app/19.0.0.0/grid

 

# As root, apply RDBMS Home patch

/u01/app/oracle/product/19.0.0.0/dbhome_1/OPatch/opatchauto apply /u01/stage/GI_DB_OJVM_19c/37641958 -oh /u01/app/oracle/product/19.0.0.0/dbhome_1

 

Step 6: Verify Applied Patches

# /u01/app/19.0.0.0/grid/OPatch/opatch lsinventory -oh /u01/app/19.0.0.0/grid | grep “applied on”

# /u01/app/oracle/product/19.0.0.0/dbhome_1/OPatch/opatch lsinventory -oh /u01/app/oracle/product/19.0.0.0/dbhome_1 | grep “applied”

 

Step 7: Apply OJVM Patch

$ cd /u01/stage/GI_DB_OJVM_19c/37499406

$ /u01/app/oracle/product/19.0.0.0/dbhome_1/OPatch/opatch apply

 

Step 8: Post-Patch Activities

## Open all the DBs and all the pluggable DBs

SQL> startup

SQL> alter pluggable database all open;

 

## Run datapatch

export ORACLE_HOME=/u01/app/oracle/product/19.0.0.0/dbhome_1

 cd $ORACLE_HOME/OPatch

./datapatch -sanity_checks

./datapatch -verbose

 

## Check data dictionary

SQL> select PATCH_ID,ACTION,STATUS,ACTION_TIME,TARGET_VERSION from dba_registry_sqlpatch;

  

Patch Exadata RoCE and Cisco Management Switches

RoCE Switches Upgrade (Patch Version: 24.1.2.0.0.240711)

 

Step 1: Verify Targeted RoCE Switches

cat /EXAVMIMAGES/patches/stage/roceswitches.lst

exaprd01sw-rocea0:leaf

exaprd01sw-roceb0:leaf

 

Step 2: Run Pre-Check

cd /${PATCH_LOCATION}/roceswitches/patch_switch_24.1.2.0.0.240711

./patchmgr --roceswitches /EXAVMIMAGES/patches/stage/roceswitches.lst \

--upgrade --roceswitch-precheck --log_dir auto

 

Step 3: Apply the RoCE Switch Patch

screen -S Sw_RoCE_patch

./patchmgr --roceswitches roceswitches.lst --upgrade --log_dir auto

 

## Cisco Management Switch Upgrade

Step 1: Verify Targeted Cisco Switch

cat /EXAVMIMAGES/patches/stage/adminswitch.lst

exa-Ciscosw-adm0

 

Step 2: Run Pre-Check

./patchmgr --adminswitches /EXAVMIMAGES/patches/stage/adminswitch.lst \

  --upgrade --<strong>adminswitch-precheck</strong> --log_dir auto

 

Step 3: Apply the Cisco Admin Switch Patch

screen -S Mngmnt_SW

./patchmgr --adminswitches adminswitch.lst --<strong>upgrade</strong> --log_dir auto

 

Patch Exadata Cell Servers (Storage Servers)

Pre-Steps:

## Make Sure No Issues in Storage Disks

dcli -l root -g /EXAVMIMAGES/patches/stage/cell_group 'cellcli -e list physicaldisk' | grep -v normal

dcli -l root -g /EXAVMIMAGES/patches/stage/cell_group 'cellcli -e list celldisk' | grep -v normal

dcli -l root -g /EXAVMIMAGES/patches/stage/cell_group 'cellcli -e list griddisk attributes name,asmmodestatus,asmdeactivationoutcome' | grep -vi yes

 

## Drop Old Alert History

dcli -l root -g cell_group drop alerthistroy all

 

## Set disk_repair_time for ASM Disk Groups

-- Check COMPATIBLE.ASM

SQL>column dg format a15

SQL>column attribute format a30

SQL>column value format a15

SQL>select dg.name dg,a.name attribute,a.value from v$asm_diskgroup dg, v$asm_attribute a where dg.group_number=a.group_number and a.name='compatible.asm' order by dg;

                                          

-- Check COMPATIBLE.ADVM

SQL>column dg format a15

SQL>column attribute format a30

SQL>column value format a15

SQL>select dg.name dg,a.name attribute,a.value from v$asm_diskgroup dg, v$asm_attribute a where dg.group_number=a.group_number and a.name='compatible.advm' order by dg;

 

-- Check FAILGROUP_REPAIR_TIME  (min 24h)

SQL>column dg format a15

SQL>column attribute format a30

SQL>column value format a15

SQL>select dg.name dg,a.name attribute,a.value from v$asm_diskgroup dg, v$asm_attribute a where dg.group_number=a.group_number and a.name='failgroup_repair_time' order by dg;

 

-- Check DISK_REPAIR_TIME (min 8.5h)

SQL>column dg format a15

SQL>column attribute format a30

SQL>column value format a15

SQL>select dg.name dg,a.name attribute,a.value from v$asm_diskgroup dg, v$asm_attribute a where dg.group_number=a.group_number and a.name='disk_repair_time' order by dg;

 

-- For each diskgroup set disk_repair_time to 24h

alter diskgroup XXXXX SET ATTRIBUTE 'disk_repair_time' = '24h' ;

 

-- set ASM_POWER_LIMIT

alter system set ASM_POWER_LIMIT = 8 scope=both;

 

Check if there are any operation (resync) running on the disks

SQL> select * from gv$asm_operation;

 

## Patch Precheck

cd ${PATCH_LOCATION}/cells/patch_24.1.2.0.0.240711

./patchmgr -cells /EXAVMIMAGES/patches/stage/cell_group -patch_check_prereq -rolling -log_dir auto

 

## Apply the Patch

screen -S Patch_cell

./patchmgr -cells /EXAVMIMAGES/patches/stage/cell_group -patch -rolling -log_dir auto

 

## After Patching

·       Check the logs generated in the patchmgr log directory

·       Run standard Exadata health checks to verify the patch success

·       Reset changed parameters to the default value

 

Patch Exadata KVM Host Servers

 

Step 1: Setup Passwordless from VM node3

cat /EXAVMIMAGES/patches/stage/KVM_Hosts_1_2

EXA1-KVM1

EXA1-KVM2


$ dcli -g /EXAVMIMAGES/patches/stage/KVM_Hosts_1_2 -l root -k -s '-o StrictHostKeyChecking=no'

$ dcli -g /EXAVMIMAGES/patches/stage/KVM_Hosts_1_2 -l root hostname

 

Step 2: Run patchmgr precheck

$ cd /EXAVMIMAGES/patches/dbnodeupdate/

$ unzip -q p21634633_241200_Linux-x86-64.zip

$ cd /EXAVMIMAGES/patches/dbnodeupdate/dbserver_patch_240719/

 

$ ./patchmgr --dbnodes /EXAVMIMAGES/patches/stage/KVM_Hosts_1_2 --precheck --repo \ /EXAVMIMAGES/patches/YUM_KVM/p36684576_241000_Linux-x86-64.zip --target_version 24.1.2.0.0.240711 --log_dir auto

 

Step 3: Backup 

$ cd /u01/stage/Exadata_24.1.2.0.0/dbserver_patch_240719

$ ./patchmgr --dbnodes /EXAVMIMAGES/patches/stage/KVM_HOSTS_3_4 --backup --repo /EXAVMIMAGES/patches/YUM_KVM/p36684576_241000_Linux-x86-64.zip --target_version 24.1.2.0.0.240711 [--allow_active_network_mounts]

 

Step 4: Stop CRS and Agents

dcli -g /EXAVMIMAGES/patches/stage/ALL_VM_1_2 -l root "/u01/app/19.0.0.0/grid/bin/crsctl disable crs"

dcli -g /EXAVMIMAGES/patches/stage/ALL_VM_1_2  -l root "/u01/app/19.0.0.0/grid/bin/crsctl stop crs"

 

## Validate CRS is down:

dcli -g /EXAVMIMAGES/patches/stage/ALL_VM_1_2  -l root "ps -ef | grep grid | grep -v grep"

dcli -g /EXAVMIMAGES/patches/stage/ALL_VM_1_2  -l root "/u01/app/19.0.0.0/grid/bin/crsctl check crs"

 

## Stop Enterprise Manager agents:

dcli -g /EXAVMIMAGES/patches/stage/ALL_VM_1_2 -l oracle /u01/app/oracle/agent/agent_13.5.0.0.0/bin/emctl stop agent

 

Step 5: Shutdown all Guest in KVM_HOSTS_1_2

From each KVM hosts:

vm_maker --list-domains

vm_maker --stop-domain --all

vm_maker --list-domains

 

Step 6: Apply the Patch From KVM 3

$ cd /EXAVMIMAGES/patches/dbnodeupdate/dbserver_patch_240719/

./patchmgr --dbnodes /EXAVMIMAGES/patches/stage/KVM_HOSTS_1_2 --upgrade --nobackup --repo \ /EXAVMIMAGES/patches/YUM_KVM/p36684576_241000_Linux-x86-64.zip \

--target_version 24.1.2.0.0.240711 --log_dir auto

 

# Cleanup (if required):
./patchmgr –dbnodes /EXAVMIMAGES/patches/stage/KVM_HOSTS_3_4 –precheck -cleanup –log_dir auto

# Enable CRS and Agents

Comments

Post a Comment

Popular posts from this blog

 Exadata  Oracle Exadata Smart Flash Cache * Exadata cell -> Flash LUN -> Cell Disk -> Flash Cache/Flash Log  * CELL_FLASH_CACHE - – DEFAULT (LRU), KEEP, NONE * ALTER TABLE <TABLE_NAME> STORAGE(FLASH_CACHE [NONE|DEFAULT|KEEP]) * ALTER INDEX <INDEX NAME> STORAGE(FLASH_CACHE [NONE|DEFAULT|KEEP]) Smart Flash Cache Mode: 1. Write Through Flash Cache Mode (Default):  Write on Disk -> Copy to Flash Cache  * excellent for absorbing repeated random reads  * commonly found in OLTP applications. * dcli -g cell_servers -l celladmin cellcli -e "list cell attributes flashcachemode" * Steps to enable Write-Through Mode CellCLI> ALTER FLASHCACHE ALL FLUSH CellCLI> DROP FLASHCACHE CellCLI> ALTER CELL SHUTDOWN SERVICES CELLSRV CellCLI> ALTER CELL flashCacheMode=WriteThrough CellCLI> ALTER CELL STARTUP SERVICES CELLSRV CellCLI> CREATE FLASHCACHE ALL 2. Write Back Flash Cache Mode:  Write on Flash -> Copy to Disk  * best ...
Read more

Database Network : Configuration Parameters

LSNRCTL> command [listener_name] lsnrctl @file_name  #to execute in batch mode. REM or # to identify comments in the batch script Listener.ora file parameters: ·          PASSWORDS_listener_name  #an unencrypted password ·          LSNRCTL> change_password #to establish an encrypted password ·          LSNRCTL> SAVE_CONFIG  #Any changes are stored in listener.ora ·          LSNRCTL> SET PASSWORD  #If a password is set, then issue SET PASSWORD command before changing the password. Use the command SET PASSWORD prior to privileged Listener Control utility commands ·          Reload : the database services, instances, service handlers, and listening endpoints that were dynamically registered with the listener will be unregistered and subseque...
Read more

Undo Segments : Drop Problematic Undo Segments

ERROR about to recover undo segment 98 status:6 inst:0 mark undo segment 98 as available status:6 ret:1  about to recover undo segment 46 status:6 inst:0 mark undo segment 46 as available status:6 ret:1   SQL> select * from V$FAST_START_TRANSACTIONS; USN SLT SEQ STATE UNDOBLOCKSDONE UNDOBLOCKSTOTAL PID CPUTIME PARENTUSN PARENTSLT PARENTSEQ XID PXID RCVSERVERS CON_ID ---------- ---------- ---------- ---------------- -------------- --------------- ---------- ---------- ---------- ---------- 46 46 2313064 RECOVERING 505 24992423 77 5586 0 0 0 10001000684B2300 0000000000000000 1 0 98 25 1352150 RECOVERING 0 226231 78 5586 0 0 0 30001900D6A11400 0000000000000000 1 0   SQL> SELECT segment_name, tablespace_name  FROM dba_rollback_segs  WHERE segment_id IN (98, 46); SEGMENT_NAME               TABLESPACE_NAME ------------------------------ ----------------------------...
Read more