RAC - Cheatsheet

RAC Cheatsheet This is a quick and dirty cheatsheet cheatsheet on Oracle RA C 10g, as my experience with RAC grows I will update this section, below is a b eginners guide on the commands and information that you will require to administer Oracle RAC. Acronyms Acronyms GCS

Gl ob obal Ca Cache Se Se rv rvice s

in me me mo mor y database co containing cur re ren t l oc ocks an and aw awaiting l oc ocks, al so so kn ow own as as PC PCM

GES

Gl ob obal En qu qu eu eu e S er er vi vice s

coor di din at ate s th e r eq equ es est s of al l gl ob ob al al e nq nqu eu eu es es u se se s t he he GCS , al so so kn ow own as n on on -P -P CM CM

GRD

Global Resource Directory

all resources available to the cluster (formed and managed by GCS and GES), see GRD for more details

GRM

Gl ob obal Re so sou rc rce M an an ag age r

h el el ps ps t o coor di din at at e an d commu ni nicate t he he l oc ocks r eq eq ue ue st st s b et etwe en en Or ac acl e pr oc oce ss ss es es

GSD

Global Services Daemon

runs on each node with one GSD process per node. The GSD coordinates with the cluster manager to receive requests from clients such as the DBCA, EM, and the SRVCTL utility to execute administrative job tasks such as instance startup or shutdown. The GSD is not an Oracle instance background process and is therefore not started with the Oracle instance

PCM (IDLM)

Para Pa rall llel el Cach che e Ma Mana nag gem emen entt

form rmly ly kno now w as (i (int nteg egra rate ted) d) Dist stri ribu bute ted d Lo Lock Ma Mana nag ger er,, its ano noth ther er na nam me fo for GC GCS

Resource

n/a

it is a identifiable entity it basically has a name or a reference, it can be a area in memory, a disk file or an abstract entity a resource that can be accessed by all the nodes within the cluster exam examples ples would be the following

Resource (Global)

n/a

LVB

Lock Val ue Bl ock

con tain s a smal l amou nt of data re gar din g th e l ock

TRFC

Tr affic Control l er

con tr ol s th e DLM traffic be twe en in stan ce s ( messaging tickets) tickets )

Data Buffer Cache Block Transaction Enqueue Database Data Structures

Files and D irectories irectories Files and Directories $ORA_CRS_HOME/cdata/

OCR backups (default l ocati ocation) on)

$ORA_HOME/log//client/ocrconfig_.log

OCR command log file

$ORA_CRS_HOME/crs/log

contains trace files for the CRS resources

$ORA_CRS_HOME/crs/init

contains trace files for the CRS daemon during startup, a good place to start

$ORA_CRS_HOME/css/log

contains cluster reconfigurations, missed check-ins, connects and disconnects from the client CSS listener. Look here to obtain when reboots occur

$ORA_CRS_HOME/css/init

contains core dumps from the cluster synchronization service daemon ( OCSd OCSd))

$ORA_CRS_HOME/evm/log

logfiles for the event volume manager and eventlogger daemon

$ORA_CRS_HOME/evm/init

pid and lock files for EVM

$ORA_CRS_HOME/srvm/log

logfiles for Oracle Cluster Registry (OCR ( OCR))

$ORA_CRS_HOME/log

log fles for Oracle clusterware which contains diagnostic messages at the Oracle cluster level

Useful Views/Tables GCS and Cache Fusion Diagnostics v$cache

contains information about every cached block in the buffer cache

v$cache_transfer

contains information from the block headers in SGA that have been pinged at least once

v$instance_cache_transfer

contains information about the transfer of cache blocks through the interconnect

v$cr_block_server

contains statistics about CR block transfer across the instances

v$current_block_server

contains statistics about current block transfer across the instances

v$gc_element

contains one-to-one information for each global cache resource used by the buffer cache

GES diagnostics v$lock

contains information about locks held within a database and outstanding requests for locks and latches

v$ges_blocking_enqueue

contains information about locks that are being blocked or blocking others and locks that are known to the lock manager

v$enqueue_statistics

contains details about enqueue statistics in the instance

v$resource_limits

display enqueue statistics

v$locked_object

contains information about DML locks acquired by different transactions in databases with their mode held

v$ges_statistics

contains miscellaneous statistics for for GES

v$ges_convert_local

contains information about all local GES operations

v$ges_convert_remote

contains information about all remote GES operations

v$ges_resource

contains information about all resources known to the lock manager

v$ges_misc

contains information about messaging traffic information

v$ges_traffic_controller

contains information about th e mes sage ticket usage

Dynamic Resource Remastering v$hvmaster_info

contains information about current and previous master instances of GES resources in relation to hash value ID of resource

v$gcshvmaster_info

the same as above but globally

v$gcspfmaster_info

conatins information about current and previous masters about GCS resources belonging to files mapped to a particular master, including the number of times the resource has remastered

v$cluster_interconnects

contains information about interconnects that are being used for cluster communication

v$configured_interconnects

same as above but also contains interconnects that AC is aware off that are not being used

Cluster Interconnect

Miscellanous v$service

services running on an instance

x$kjmsdp

display LMS daemon statistics

x$kjmddp

display LMD daemon statistics

Useful Parameters

Parameters cluster_interconnects

specify a specific IP address to use for the inetrconnect

 _gcs_fast_config

enables fast reconfiguration for gcs locks (true|false)

 _lm_master_weight

controls which instance will hold or (re)master more resources than others

 _gcs_resources

controls the number of resources an instance will master at a time

 _lm_tickets

controls the number of message tickets

 _lm_ticket_active_sendback

controls the number of message tickets (aggressive messaging)

 _db_block_max_cr_dba

limits the number of CR copies per DBA on the buffer cache (see grd)

 _fairness_threshold

used when too many CR requested arrive for a particular buffer and the block becomes disowned (see grd)

 _gc_affinity_time

specifies interval minutes for reamstering

 _gc_affinity_limit

defines the number of times a instance access the resource before remastering

 _gc_affinity_minimum

defines the minimum number of times a instance access the resource before remastering

 _lm_file_affinity

disables dynamic remastering for the objects belonging to those files

 _lm_dynamic_remastering

enable or disable remastering

 _gc_defer_time

define the time by which an instance deferred downgrading a lock (see Cache Fusion)

 _lgwr_async_broadcast

change the SCN boardcast method (see troubleshooting)

Processes

Oracle RAC Daemons and Processes

EVMd OCSSd

Event Management Cluster Synchronization Services

spawns a child process event logger and generates callouts basic node membership, group services, basic locking

CRSd

Cluster Ready Services

resource monitoring, failover and node recovery

LMSn

Lock Manager Server process - GCS

this is the cache fusion part, it handles the consistent copies of blocks that are tranferred between instances. It receives requests from LMD to perform lock requests. I rools back any uncommitted transactions. There can be upto ten LMS processes running and can be started dynamically if demand requires it. they manage lock manager service requests for GCS resources and send them to a service queue to be handled by the LMSn process. It also handles global deadlock detection and monitors for l ock conversion timeouts.

LMON

this process manages the GES, it maintains consistency of GCS memory in case of process death. It is also responsible for cluster reconfiguration and locks reconfiguration (node joining or leaving), it checks for instance deaths and listens for local messaging.

Lock Monitor Process - GES

A detailed log file is created that tracks any reconfigurations that have happened. LMD

Lock Manager Daemon - GES

this manages the enqueue manager service requests for the GCS. It also handles deadlock detention and remote resource requests from other instances.

LCK0

Lock Process - GES

manages instance resource requests and cross-instance call operations for shared resources. It builds a list of invalid lock elements and validates lock elements during recovery.

DIAG

Diagnostic Daemon

This is a lightweight process, it uses the DIAG framework to monitor the healt of the cluster. It captures information for later diagnosis in the event of failures. It will perform any neccessary recovery if an operational hang is detected.

General Administration Managing the Cluster /etc/init.d/init.crs start

starting crsctl start crs /etc/init.d/init.crs stop

stopping crsctl stop crs /etc/init.d/init.crs enable /etc/init.d/init.crs disable

enable/disable at boot time crsctl enable crs crsctl disable crs

Managing the database configuration with SRVCTL srvctl start database -d -o Note: starts listeners if not already running, you can use the -o option to specify startup/shutdown options

start all instances force open mount nomount srvctl stop database -d -o Note: the listeners are not stopped, you can use the -o option to specify startup/shutdown options

stop all instances immediate abort normal transactional

start/stop particular instance

srvctl [start|stop] database -d -i ,

display the registered databases

srvctl config database

status

srvctl srvctl srvctl srvctl srvctl

status status status status status

srvctl srvctl srvctl srvctl srvctl

stop stop stop stop stop

srvctl srvctl srvctl srvctl

start start start start

database -d instance -d -i , service -d nodeapps -n asm -n

database -d instance -d -i , service -d -s , -i , nodeapps -n asm -n

stopping/starting database -d instance -d -i , service -d -s , -i , nodeapps -n

adding/removing

srvctl srvctl srvctl srvctl

add add add add

instance -d -i -n service -d -s -r nodeapps -n -o -A /network asm -n -i -o

srvctl srvctl srvctl srvctl srvctl

remove database -d -o remove instance -d -i -n remove service -d -s -r remove nodeapps -n -o -A /network asm remove -n

OCR utilities log file

$ORA_HOME/log//client/ocrconfig_.log ocrcheck

checking

Note: will return the OCR version, total space allocated, space used, free space, location of each device and the result of the integrity check ocrdump -backupfile

dump contents Note: by default it dumps the contents into a file named OCRDUMP in the current directory ocrconfig -export

export/import ocrconfig -restore # show backups ocrconfig -showbackup # to change the location of the backup, you can even specify a ASM disk ocrconfig -backuploc

backup/restore

# perform a backup, will use the location specified by the -backuploc location ocrconfig -manualbackup # perform a restore ocrconfig -restore # delete a backup orcconfig -delete Note: there are many more option so see the ocrconfig man page ## add/relocate the ocrmirror file to the specified location ocrconfig -replace ocrmirror '/ocfs2/ocr2.dbf'

add/remove/replace

## relocate an existing OCR file ocrconfig -replace ocr '/ocfs1/ocr_new.dbf' ## remove the OCR or OCRMirror file ocrconfig -replace ocr ocrconfig -replace ocrmirror

CRS Administration

CRS Administration

starting

## Starting CRS using Oracle 10g R1 not possible ## Starting CRS using Oracle 10g R2 $ORA_CRS_HOME/bin/crsctl start crs

stopping

## Stopping CRS using Oracle 10g R1 srvctl stop database -d srvctl stop asm -n srvctl stop nodeapps -n /etc/init.d/init.crs stop ## Stopping CRS using Oracle 10g R2 $ORA_CRS_HOME/bin/crsctl stop crs ## use to stop CRS restarting after a reboot

disabling/enabling

## Oracle 10g R1 /etc/init.d/init.crs [disable|enable] ## Oracle 10g R2 $ORA_CRS_HOME/bin/crsctl [disable|enable] crs

checking

$ORA_CRS_HOME/bin/crsctl $ORA_CRS_HOME/bin/crsctl $ORA_CRS_HOME/bin/crsctl $ORA_CRS_HOME/bin/crsctl $ORA_CRS_HOME/bin/crsctl

check check check check check

crs evmd cssd crsd install -wait 600

Resource Applications (CRS Utilities) status

$ORA_CRS_HOME/bin/crs_stat

create profile

$ORA_CRS_HOME/bin/crs_profile

register/unregister application

$ORA_CRS_HOME/bin/crs_register $ORA_CRS_HOME/bin/crs_unregister

Start/Stop an application

$ORA_CRS_HOME/bin/crs_start $ORA_CRS_HOME/bin/crs_stop

Resource permissions

$ORA_CRS_HOME/bin/crs_getparam

Relocate a resource

Nodes

member number/name

olsnodes -n

local node name

olsnodes -l

activates l ogging

olsnodes -g

Oracle Interfaces display

oifcfg getif

delete

oicfg delig -global

set

oicfg setif -global /:public oicfg setif -global /:cluster_interconnect

Global Services Daemon Control starting

gsdctl start

stopping

gsdctl stop

status

gsdctl status

Cluster Configuration (clscfg is used during installation) create a new configuration

clscfg -install

upgrade or downgrade and existing configuration

clscfg -upgrade clscfg -downgrade

add or delete a node from the configuration

clscfg -add clscfg -delete

create a special single-node configuration for ASM

clscfg -local

brief listing of terminology used in the other nodes

clscfg -concepts

used for tracing

clscfg -trace

help

clscfg -h

Cluster Name Check cemulto -n

print cluster name Note: in Oracle 9i the ulity was called "cemutls" cemulto -w

print the clusterware version Note: in Oracle 9i the ulity was called "cemutls"

Node Scripts addnode.sh

Add Node Note: see adding and deleting nodes deletenode.sh

Delete Node Note: see adding and deleting nodes

Enqueues

displaying statistics

SQL> SQL> SQL> SQL>

column column column column

current_utilization heading current max_utilization heading max_usage initial_allocation heading initial resource_limit format a23;

SQL> select * from v$resource_limit;

Messaging (tickets) ticket usage

select local_nid local, remote_nid remote, tckt_avail avail, tckt_limit limit, snd_q_len send_queue, tckt_wait waiting from v$ges_traffic_controller;

dump ticket information

SQL> oradebug setmypid SQL> oradebug unlimit SQL> oradebug lkdebug -t

Lighwork Rule and Fairness Threshold select cr_requests, light_works, data_requests, fairness_down_converts from v$cr_block_server;

downconvert

Note: lower the _fairness_threshold if the ratio goes above 40%, set to 0 if the instance is a query only instance.

Remastering ## Obtain the OBJECT_ID form the below table SQL> select * from v$gcspfmaster_info;

force dynamic remastering (DRM)

## Determine who masters it SQL> oradebug setmypid SQL> oradebug lkdebug -a ## Now remaster the resource SQL> oradebug setmypid SQL> oradebug lkdebug -m pkey

GRD, SRVCTL, GSD and SRVCONFIG Tracing

Disable tracing

$ export SRVM_TRACE=""

Voting Disk adding

crsctl add css votedisk

deleting

crsctl delete css votedisk

querying

crsctl query css votedisk