Real Application Clusters (RAC)

RAC for Beginners

Nanda's ORACLE DBA & APPS DBA Blog

http://nandakumarappsdba.blogspot.com/
Real Application Clusters (RAC)


Oracle RAC, introduced with Oracle9i, is the successor to Oracle Parallel Server (OPS). Oracle RAC allows multiple instances to access the same database (storage) simultaneously. RAC provides fault tolerance, load balancing, and performance benefits by allowing the system to scale out, and at the same time since all nodes access the same database, the failure of one instance will not cause the loss of access to the database.

Oracle RAC 10g is a shared disk subsystem. All nodes in the cluster must be able to access all of the data, redo log files, control files and parameter files for all nodes in the cluster. The data disks must be globally available in order to allow all nodes to access the database. Each node has its own redo log file(s) and UNDO tablespace, but the other nodes must be able to access them (and the shared control file) in order to recover that node in the event of a system failure.

The difference between Oracle RAC and OPS is the addition of Cache Fusion. With OPS a request for data from one node to another required the data to be written to disk first, then the requesting node can read that data. With cache fusion, data is passed along a high-speed interconnect using a sophisticated locking algorithm.

With Oracle RAC 10g, the data files, redo log files, control files, and archived log files reside on shared storage on raw-disk devices, a NAS, ASM, or on a clustered file system

Oracle RAC is composed of two or more database instances.
They are composed of Memory structures and background processes same as the single instance database.

Oracle RAC instances use two processes

==> GES(Global Enqueue Service)
==> GCS(Global Cache Service) this enable cache fusion.

Oracle RAC instances are composed of following background processes:

ACMS—Atomic Controlfile to Memory Service (ACMS)
GTX0-j—Global Transaction Process
LMON—Global Enqueue Service Monitor
LMD—Global Enqueue Service Daemon
LMS—Global Cache Service Process
LCK0—Instance Enqueue Process
RMSn—Oracle RAC Management Processes (RMSn)
RSMN—Remote Slave Monitor


LMON

The background Global Enqueue Service Monitor (LMON) monitors the entire cluster to manage global resources. LMON manages instance and process failures and the associated recovery for the Global Cache Service (GCS) and Global Enqueue Service (GES). In particular, LMON handles the part of recovery associated with global resources. LMON-provided services are also known as cluster group services (CGS)

This process monitors global enques and resources across the cluster and performs global enqueue recovery operations.This is called as Global Enqueue Service Monitor.


LCKx

The LCK process manages instance global enqueue requests and cross-instance call operations. Workload is automatically shared and balanced when there are multiple Global Cache Service Processes (LMSx).

This process is called as Instance enqueue process.This process manages non-cache fusion resource requests such as libry and row cache requests.

LMSx

The Global Cache Service Processes (LMSx) are the processes that handle remote Global Cache Service (GCS) messages. Current Real Application Clusters software provides for up to 10 Global Cache Service Processes. The number of LMSx varies depending on the amount of messaging traffic among nodes in the cluster. The LMSx handles the acquisition interrupt and blocking interrupt requests from the remote instances for Global Cache Service resources. For cross-instance consistent read requests, the LMSx will create a consistent read version of the block and send it to the requesting instance. The LMSx also controls the flow of messages to remote instances.

This process is called as Global Cache service process.This process maintains statuses of datafiles and each cahed block by recording information in a Global Resource Dectory(GRD).This process also controls the flow of messages to remote instances and manages global data block access and transmits block images between the buffer caches of different instances.This processing is a part of cache fusion feature.


LMDx

The Global Enqueue Service Daemon (LMD) is the resource agent process that manages Global Enqueue Service (GES) resource requests. The LMD process also handles deadlock detection Global Enqueue Service (GES) requests. Remote resource requests are requests originating from another instance.

This process is called as global enqueue service daemon. This process manages incoming remote resource requests within each instance.

DIAG

The diagnose daemon is a Real Application Clusters background process that captures diagnostic data on instance process failures. No user control is required for this demo.

ACMS

ACMS stands for Atomic Controlfile Memory Service.In an Oracle RAC environment ACMS is an agent that ensures a distributed SGA memory update(ie)SGA updates are globally committed on success or globally aborted in event of a failure.

GTX0-j

The process provides transparent support for XA global transactions in a RAC environment.The database autotunes the number of these processes based on the workload of XA global transactions.

RMSn

This process is called as Oracle RAC management process.These pocesses perform managability tasks for Oracle RAC.Tasks include creation of resources related Oracle RAC when new instances are added to the cluster.

RSMN

This process is called as Remote Slave Monitor.This process manages background slave process creation andd communication on remote instances. This is a background slave process.This process performs tasks on behalf of a co-ordinating process running in another instance.

CRS

CRS (Cluster Ready Services) is a new feature for 10g Real Application Clusters that provides a standard cluster interface on all platforms and performs new high availability operations not available in previous versions. CRS manages cluster database functions including node membership, group services, global resource management, and high availability. CRS serves as the clusterware software for all platforms. It can be the only clusterware or run on top of vendor clusterware such as Sun Cluster, HP Serviceguard, etc.


CRS automatically starts the following resources:
· Nodeapps
o Virtual Internet Protocol(VIP) address for each node
o Global Services Daemon
o Oracle Net Listeners
o Oracle Network Services (ONS)
· Database Instance
· Services


Oracle Clusterware (Cluster Ready Services in 10g/ Cluster Manager in 9i) - provides infrastructure that binds multiple nodes that then operate as single server. Clusterware monitors all components like instances and listeners. There are two important components in Oracle clusterware, Voting Disk and OCR (Oracle Cluster Registry)

OCR & Voting Disk

Oracle, 10g RAC, provided its own cluster-ware stack called CRS. The main file components of CRS are the Oracle Cluster Repository (OCR) and the Voting Disk.
The OCR contains cluster and database configuration information for RAC and Cluster Ready Services (CRS). Some of this information includes the cluster node list, cluster database instance-to-node mapping information, and the CRS application resource profiles. The OCR contains configuration details for the cluster database and for high availability resources such as services, Virtual Inerconnect Protocoal (VIP) addresses.

The Voting Disk is used by the Oracle cluster manager in various layers. The Node Monitor (NM) uses the Voting Disk for the Disk Hearbeat, which is essential in the detection and resolution of cluster "split brain".


Cache Fusion:-

Oracle RAC is composed of two or more instances. When a block of data is read from datafile by an instance within the cluster and another instance is in need of the same block,it is easy to get the block image from the insatnce which has the block in its SGA rather than reading from the disk. To enable inter instance communication Oracle RAC makes use of interconnects. The Global Enqueue Service(GES) monitors and Instance enqueue process manages the cahce fusion

Cache Fusion and Global Cache Service (GCS)
Memory-to-memory copies between buffer caches over high-speed interconnects

· fast remote access times
· memory transfers for write or read access
· transfers for all types (e.g data, index, undo, headers )
· Cache coherency across the cluster
· globally managed access permissions to cached data
· GCS always knows whether and where a data block is cached
· a local cache miss may result in remote cache hit or disk read

Compare two table Datas

from the forum by

John Spencer

thread:HOWTO: Compare two tables

TO FIND RECORDS IN TABLE1 WHICH DO NOT EXIST IN TABLE2

SELECT column_list
FROM table1
WHERE column_list NOT IN (SELECT column_list
                         FROM table2)

SELECT column_list
FROM table1
MINUS
SELECT column_list
FROM table2;

SELECT column_list
FROM table1
WHERE NOT EXISTS (SELECT 1
                 FROM table2
                 WHERE table1.column1 = table2.column1 and
                       table1.column2 = table2.column2 and
                       ...);

SELECT table1.column_list
FROM table1, table2
WHERE table1.column1 = table2.column1(+) and
     table1.column2 = table2.column2(+) and
     ...  and
     (table2.column1 IS NULL or
      table2.column2 IS NULL or
      ...)
-- Use and instead of or if you are sure that there is no case where
-- the two column1's will match, but the two column2's will not.

TO FIND RECORDS IN TABLE1 WHICH EXIST IN TABLE2.

SELECT column_list
FROM table1
WHERE column_list IN (SELECT column_list
                     FROM table2)

SELECT column_list
FROM table1
INTERSECT
SELECT column_list
FROM table2;

SELECT column_list
FROM table1
WHERE EXISTS (SELECT 1
             FROM table2
             WHERE table1.column1 = table2.column1 and
                   table1.column2 = table2.column2 and
                   ...);

SELECT table1.column_list
FROM table1, table2
WHERE table1.column1 = table2.column1 and
     table1.column2 = table2.column2 and
     ...

Which is better, depends on a number of factors. Some of the most important factors are:

1. Absloute sizes of the two tables
2. The relative sizes of the two tables
3. Cost Based Optimizer versus Rule Based Optimizer
4. Version of Oracle
5. Number and type columns that are being compared.

Number 5 may need a little clarification. If the question is find all Primary Key's in table1 which exist or do not exist in table2, then using EXISTS or a JOIN is probably best. However, if the question is find entire rows in table1 that do or do not have a corresponding entire row in table2 then INTERSECT or MINUS are likely to be best.

In all of the queries above, column_list represents the columns that you want to find the matches or mismatches for. This can be anything from a single column to * (all columns). column1, column2 ... represent each of the columns in your column list.

If we assume that there are no other selection criteria used against either table (which might affect the choice of indexes), then all methods will require at least one full table scan (at best a fast full index scan), and sometimes two. So, the chief factor affecting performance is the method Oracle uses to get the rows in table1 that do or do not exist in table2. On a conceptual basis, this is how I understand the various approaches operate. This is almost certainly not the exact algorithm that Oracle uses.

IN / NOT IN will do a full scan of table2 to get a list of values, then for each row of table1 scan this LOV to see if it finds a match (IN) or does not (NOT IN). If table2 is absolutely small (e.g. state_code table with 50 records) then this may well be the fastest approach since scanning the LOV should be fast in memory.

INTERSECT/ MINUS does a full scan of both tables, sorts each, then scans the two lists together to identify matches or mismatches. If the two tables are relatively similar in size, then this is likely to be one of the faster methods. The optimizer may choose to use index scans if the columns being compared are all indexed. If you are trying to compare all columns, or a large (unindexed) subset, then this is likely to be the fastest method. It may also be faster if you expect a relatively large number of matched or mis-matched records.

In a NOT EXISTS (or EXISTS), for each row in table 1, the correlated sub-query is executed against table2. If the all of the indentity columns in the sub-query are indexed, then this is pretty quick. The relative sizes of the tables do not matter much, but if both are large, this may be a little slower. Also, it is not as efficient when you need to look at values in several columns. If table2 is not appropriately indexed, and it is large, then this could be incredibly slow since it will require a full scan of table2 for each row in table1.

The [OUTER] JOIN approach uses SORT JOIN - MERGE JOIN or SORT JOIN - NESTED LOOP (depending on the Oracle version and optimizer mode). This is pretty efficient, particularly if the two tables are large, and relativley equally sized, and you expect a relativley small number of mismatches, or a relatively large number of matches (INNER JOIN).

There is also a fifth approach to finding records in table1 that do not exist in table2.

In sqlplus run $ORACLE_HOME/rdbms/admin/utlexcpt.sql then

ALTER TABLE table1
ADD constraint t1_t2_fk
FOREIGN KEY (column_list) REFERENCES table2 (column list)
EXCEPTIONS INTO exceptions

The rowid of any rows that do not pass validation will be stored in a table called exception (created by the script). You can then use the rowids to identify and deal with the rows. NOTE: This requires a unique index on column_list in table2.

Probably not useful for a one-off excercise, but it will prevent it happening on an on-going basis.

Caveat emptor

This posting is the work of the Rogue Moderators. It is posted with the best of intentions but is not guaranteed in any way, shape or form. In particular, the code is presented "as is" and you assume full responsibility for running it on your system.

reference:
http://forums.oracle.com/forums/thread.jspa?threadID=2996

Terminal time for Linux

Linux only: Do you spend most of your day working at the command line? Either in a terminal editor or full screen text? Then how about embedding a clock into the top right hand corner of your terminal so you know exactly how long you have been working. There are many ways to achieve this (or similar) effects:

Sergio Gonzalez Duran on Linux.com offers this longish but effective way round:

Save the clock.sh script into your home directory: clock.sh

Chmod clock.sh’s status to 755:

chmod 755 clock.sh

Then in a terminal run:

./clock.sh&

and you’ll have a small clock in the top right hand of your terminal window. It works in xterm and apparently in many other emulators as well. There are also variations on this script at the original article on Linux.com.

For a simpler and more elegant solution if you regularly use full screen text (ie no graphical interface at all) try this on Ubuntu (and many other distros):

sudo -b vcstime&

and you’ll have a simple clock in the right-top corner of your screen.

[Via Linux.com]

http://www.tectonic.co.za/2008/01/terminal-time-for-linux/comment-page-1/#comment-976175

Oracle 10g RAC Cluster interconnects

Oracle 10g RAC Cluster interconnects

Posted by Arul Ramachandran on January 3, 2008

Yet another post on … say, you’ve just joined a new client, you are in a new RAC environment and you are trying to figure out the different components of the cluster.

This time it is on the Cluster private interconnects.

The private interconnects used in a RAC are typically configured during the Clusterware install and would be registered in OCR. However this default config could thereafter be overrided using cluster_interconnects initialization parameter (not recommended though ) for various reasons.

In order to troubleshoot interconnect related issues, a starting point would be to find out what interconnects were originally configured and/or what interconnects are presently in use by the cluster. Here are a few ways to do this.

1. Using the dynamic view gv$cluster_interconnects:

select * from gv$cluster_interconnects ;

INST_ID    NAME            IP_ADDRESS       IS_ SOURCE
---------- --------------- ---------------- --- -------------------------------
1          eth0            192.168.10.1     NO  Oracle Cluster Repository
1          eth1            192.168.11.1     NO  Oracle Cluster Repository
2          eth0            192.168.10.2     NO  Oracle Cluster Repository
2          eth1            192.168.11.2     NO  Oracle Cluster Repository

In the above output, the column SOURCE indicates where the private interconnect info was derived from; this column could be one of OCR, OS dependent software or cluster_interconnects parameter.

2. Using the clusterware command oifcfg:

$oifcfg getif
eth2 10.104.95.0 global public
eth0 192.168.10.0 global cluster_interconnect
eth1 192.168.11.0 global cluster_interconnect

3. Using oradebug ipc:

sqlplus “/ as sysdba”

SQL>oradebug setmypid
Statement processed.
SQL>oradebug ipc
Information written to trace file.

The above command would dump a trace to user_dump_dest. The last few lines of the trace would indicate the IP of the cluster interconnect. Below is a sample output of those lines.

From the trace file on node1:

SSKGXPT 0x5edf558 flags SSKGXPT_READPENDING socket no 9 IP 192.168.11.1 UDP 18852

From the trace file on node2:

SSKGXPT 0x5edf558 flags SSKGXPT_READPENDING socket no 9 IP 192.168.10.2 UDP 38967

Change or switch undo tablespace in Oracle database http://www.oracleflash.com/32/Change-or-switch-undo-tablespace-in-Oracle-database.html

Posted by Zahid Qureshi on July 8, 2010.

This is a mostly asked question on different oracle forums; How do I change the UNDO tablespace for my database and drop the old one? and the answer often is; Its very simple daa. But I don't think its that simple. The complexity comes in if there were pending transactions in your old undo tablespace and you have already switched to the new one and when you try to drop the old one it says "Tablespace is in use".

How to switch the database to a new UNDO tablespace and drop the old one

$ sqlplus / as sysdba

SQL> show parameter undo

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
undo_management                      string      AUTO
undo_retention                       integer     900
undo_tablespace                      string      UNDOTBS1
SQL> 

The current undo tablespace as suggested by the initialization parameter undo_tablespace is UNDOTBS1. Leave this sysdba as is, open another console, log in as user SCOTT and initiate a transaction.

$ sqlplus scott/tiger

update emp set sal = sal + 1000 where empno=7839;

1 row updated.

With an update on emp table we have initiated a transaction. The undo data is written to a segment in the UNDOTBS1 tablespace. Now leave this SCOTT's session intact and go back to the sysdba console without issuing any COMMIT or ROLLBACK.

CREATE UNDO TABLESPACE undotbs2
DATAFILE '/d01/apps/oradata/oraxpo/undotbs201.dbf'
       SIZE 50M AUTOEXTEND ON NEXT 5M;

Tablespace created.

-- We created a new UNDO tablespace named UNDOTBS2

ALTER SYSTEM SET UNDO_TABLESPACE=UNDOTBS2 SCOPE=BOTH;

System altered.

-- Switch the database to the new UNDO tablespace.

SQL> DROP TABLESPACE undotbs1 INCLUDING CONTENTS AND DATAFILES;
DROP TABLESPACE undotbs1 INCLUDING CONTENTS AND DATAFILES
*
ERROR at line 1:
ORA-30013: undo tablespace 'UNDOTBS1' is currently in use

-- Try to drop the tablespace but failed. 

SQL>

With the alter system set undo_tablespace=UNDOTBS2, the database UNDO tablespace is changed and any new transaction's undo data will go to the new tablespace i.e. UNDOTBS2. But the undo data for already pending transaction (e.g. the one initiated by SCOTT before the database UNDO tablespace switch) is still in the old tablespace with a status of PENDING OFFLINE. As far as it is there you cannot drop the old tablespace.

set lines 10000
column name format a10

SELECT a.name,b.status
FROM   v$rollname a,v$rollstat b
WHERE  a.usn = b.usn
AND    a.name IN (
   SELECT segment_name
   FROM dba_segments
   WHERE tablespace_name = 'UNDOTBS1'
  );

NAME       STATUS
---------- ---------------
_SYSSMU8$  PENDING OFFLINE

The above query shows the name of the UNDO segment in the UNDOTBS1 tablespace and its status. Now lets see which users/sessions are running this pending transaction.

column username format a6

SELECT a.name,b.status , d.username , d.sid , d.serial#
FROM   v$rollname a,v$rollstat b, v$transaction c , v$session d
WHERE  a.usn = b.usn
AND    a.usn = c.xidusn
AND    c.ses_addr = d.saddr
AND    a.name IN (
   SELECT segment_name
   FROM dba_segments
   WHERE tablespace_name = 'UNDOTBS1'
  );

NAME       STATUS          USERNA        SID    SERIAL#
---------- --------------- ------ ---------- ----------
_SYSSMU8$  PENDING OFFLINE SCOTT         147          4

So this is SCOTT with SID=147 and SERIAL#=4. Since we know now the user, we can go to him/her and request to end the transaction gracefully i.e. issue a ROLLBACK or COMMIT. However, if this is not possible (say the user initiated the transaction and left for annual leave :) and trust me this happens) you may go ahead and kill the session to release the undo segments in the UNDOTBS1 tablespace.

SQL> alter system kill session '147,4';

System altered.

SELECT a.name,b.status , d.username , d.sid , d.serial#
FROM   v$rollname a,v$rollstat b, v$transaction c , v$session d
WHERE  a.usn = b.usn
AND    a.usn = c.xidusn
AND    c.ses_addr = d.saddr
AND    a.name IN (
   SELECT segment_name
   FROM dba_segments
   WHERE tablespace_name = 'UNDOTBS1'
  );

no rows selected

As we can see once the session is kills we don't see anymore segments occupied in the UNDOTBS1 tablespace. Lets drop UNDOTBS1.

SQL> DROP TABLESPACE undotbs1 INCLUDING CONTENTS AND DATAFILES;
DROP TABLESPACE undotbs1 INCLUDING CONTENTS AND DATAFILES
*
ERROR at line 1:
ORA-30013: undo tablespace 'UNDOTBS1' is currently in use

If you are retaining undo data then you still won't be able to drop the tablespace because it is still in use by undo_retention. Let the UNDO_RETENTION time pass and then try to drop the tablespace. In my case it is 900 seconds i.e. 15 minutes.

-- After 15 minutes.
SQL> DROP TABLESPACE undotbs1 INCLUDING CONTENTS AND DATAFILES;

Tablespace dropped.

from webpage:
http://www.oracleflash.com/32/Change-or-switch-undo-tablespace-in-Oracle-database.html

RAC IMPORTANT NOTES

What are Oracle Clusterware processes for 10g on Unix and Linux

Cluster Synchronization Services (ocssd) — Manages cluster node membership and runs as the oracle user; failure of this process results in cluster restart.

Cluster Ready Services (crsd) — The crs process manages cluster resources (which could be a database, an instance, a service, a Listener, a virtual IP (VIP) address, an application process, and so on) based on the resource's configuration information that is stored in the OCR. This includes start, stop, monitor and failover operations. This process runs as the root user

Event manager daemon (evmd) —A background process that publishes events that crs creates.

Process Monitor Daemon (OPROCD) —This process monitor the cluster and provide I/O fencing. OPROCD performs its check, stops running, and if the wake up is beyond the expected time, then OPROCD resets the processor and reboots the node. An OPROCD failure results in Oracle Clusterware restarting the node. OPROCD uses the hangcheck timer on Linux platforms.

RACG (racgmain, racgimon) —Extends clusterware to support Oracle-specific requirements and complex resources. Runs server callout scripts when FAN events occur.

What are Oracle database background processes specific to RAC

•LMS—Global Cache Service Process

•LMD—Global Enqueue Service Daemon

•LMON—Global Enqueue Service Monitor

•LCK0—Instance Enqueue Process

To ensure that each Oracle RAC database instance obtains the block that it needs to satisfy a query or transaction, Oracle RAC instances use two processes, the Global Cache Service (GCS) and the Global Enqueue Service (GES). The GCS and GES maintain records of the statuses of each data file and each cached block using a Global Resource Directory (GRD). The GRD contents are distributed across all of the active instances.

What are Oracle Clusterware Components

Voting Disk — Oracle RAC uses the voting disk to manage cluster membership by way of a health check and arbitrates cluster ownership among the instances in case of network failures. The voting disk must reside on shared disk.

Oracle Cluster Registry (OCR) — Maintains cluster configuration information as well as configuration information about any cluster database within the cluster. The OCR must reside on shared disk that is accessible by all of the nodes in your cluster

How do you troubleshoot node reboot

Please check metalink ...

Note 265769.1 Troubleshooting CRS Reboots
Note.559365.1 Using Diagwait as a diagnostic to get more information for diagnosing Oracle Clusterware Node evictions.

How do you backup the OCR

There is an automatic backup mechanism for OCR. The default location is : $ORA_CRS_HOME\cdata\"clustername"\

To display backups :
#ocrconfig -showbackup
To restore a backup :
#ocrconfig -restore

With Oracle RAC 10g Release 2 or later, you can also use the export command:
#ocrconfig -export -s online, and use -import option to restore the contents back.
With Oracle RAC 11g Release 1, you can do a manaual backup of the OCR with the command:
# ocrconfig -manualbackup

How do you backup voting disk

#dd if=voting_disk_name of=backup_file_name

How do I identify the voting disk location

#crsctl query css votedisk

How do I identify the OCR file location

check /var/opt/oracle/ocr.loc or /etc/ocr.loc ( depends upon platform)
or
#ocrcheck

Is ssh required for normal Oracle RAC operation ?

"ssh" are not required for normal Oracle RAC operation. However "ssh" should be enabled for Oracle RAC and patchset installation.

What is SCAN?

Single Client Access Name (SCAN) is s a new Oracle Real Application Clusters (RAC) 11g Release 2 feature that provides a single name for clients to access an Oracle Database running in a cluster. The benefit is clients using SCAN do not need to change if you add or remove nodes in the cluster.

Click here for more details from Oracle

What is the purpose of Private Interconnect ?

Clusterware uses the private interconnect for cluster synchronization (network heartbeat) and daemon communication between the the clustered nodes. This communication is based on the TCP protocol.
RAC uses the interconnect for cache fusion (UDP) and inter-process communication (TCP). Cache Fusion is the remote memory mapping of Oracle buffers, shared between the caches of participating nodes in the cluster.

Why do we have a Virtual IP (VIP) in Oracle RAC?

Without using VIPs or FAN, clients connected to a node that died will often wait for a TCP timeout period (which can be up to 10 min) before getting an error. As a result, you don't really have a good HA solution without using VIPs.
When a node fails, the VIP associated with it is automatically failed over to some other node and new node re-arps the world indicating a new MAC address for the IP. Subsequent packets sent to the VIP go to the new node, which will send error RST packets back to the clients. This results in the clients getting errors immediately.

What do you do if you see GC CR BLOCK LOST in top 5 Timed Events in AWR Report?

This is most likely due to a fault in interconnect network.
Check netstat -s
if you see "fragments dropped" or "packet reassemblies failed" , Work with your system administrator find the fault with network.

How many nodes are supported in a RAC Database?

10g Release 2, support 100 nodes in a cluster using Oracle Clusterware, and 100 instances in a RAC database.

Srvctl cannot start instance, I get the following error PRKP-1001 CRS-0215, however sqlplus can start it on both nodes? How do you identify the problem?

Set the environmental variable SRVM_TRACE to true.. And start the instance with srvctl. Now you will get detailed error stack.

what is the purpose of the ONS daemon?

The Oracle Notification Service (ONS) daemon is an daemon started by the CRS clusterware as part of the nodeapps. There is one ons daemon started per clustered node.
The Oracle Notification Service daemon receive a subset of published clusterware events via the local evmd and racgimon clusterware daemons and forward those events to application subscribers and to the local listeners.

This in order to facilitate:

a. the FAN or Fast Application Notification feature or allowing applications to respond to database state changes.
b. the 10gR2 Load Balancing Advisory, the feature that permit load balancing accross different rac nodes dependent of the load on the different nodes. The rdbms MMON is creating an advisory for distribution of work every 30seconds and forward it via racgimon and ONS to listeners and applications.

from blog:
http://dbaregistry.blogspot.com/2010/02/oracle-rac-interview-questions.html

Oracle Memory Components and Background Processes

Oracle Memory Components and Background Processes

Memory Components		Comments	Dynamic Auto Tune MMAN SGA_TARGET
PGA =%20*SGA		In general ,in Dedicated connection PGA contains · Private SQL area · Cursor and SQL area · Work area · Session memory that holds session specific variables, such as logon information. PGA Contains data and control information for a server process. If the session connects to a dedicated Oracle server, PGA is automatically managed. If oracle is using shared servers, this memory area is shared. The database administrator just needs to set the PGA_AGGREGATE_TARGET initialization parameter to the maximum amount of memory he/she wants Oracle to use for client processes. Oracle will ensure that the total PGA allocated to all processes will never exceed this value.
SGA	Default DB Buffer Cache DB_CACHE_SIZE	copies of data files data blocks · Free/Unused buffers: · Clean buffers: These are buffers that do not contain any useful data, and, thus, the database can reuse them to hold new data it reads from disk. · Pinned buffers: These are data buffers that are currently in active use by user sessions. · Dirty buffers: These contain data that was read from disk and then modified, but hasn’t yet been written to the data files on disk.	YES
	Redo Log buffer LOG_BUFFER
	Shared Pool SHARED_POOL_SIZE	Data dictionary cache Library cache(SQL and PL/SQL parsed code) Shared SQL Area(Parse tree , SQL Execution Plans) Private SQL Area(Actual Data Values) Locks Session Data Data Dictionary Cache(a roadmap to structure and layout of the database)	YES
	O p t i o n a l
	Large Pool LARGE_POOL_SIZE=[2GB - 2TB ]	XA Interface I/O Server RMAN Backup/Restore buffers Use by Parallel processing Shared Servers	YES
	Java Pool JAVA_POOL_SIZE		YES
	Streams Pool STREAM_POOL_SIZE	Oracle Streams buffer
	N e w
	Keep Buffer Cache DB_KEEP_CACHE_SIZE
	Recycle Buffer Cache DB_RECYCLE_CACHE_SIZE
	nK Block Size Buffer Caches DB_nK_CACHE_SIZE

The SGA isn’t a homogeneous entity; rather, it’s a combination of several memory structures. The following are the main components of the SGA:

· Database buffer cache: Holds copies of data blocks read from data files.

· Shared pool: Contains the library cache for storing SQL and PL/SQL parsed code in order to share it among users. It also contains the data dictionary cache, which holds key data dictionary information.

· Redo log buffer: Contains the information necessary to reconstruct changes made to the database by DML operations. This information is then recorded in the redo logs by the log writer.

· Java pool: Keeps the state of Java program execution.

· Large pool: Stores large memory allocations, such as RMAN backup buffers.

· Streams pool: Supports the Oracle Streams feature.

Background Processes

Server Side (Mandatory)	RAC	ASM	Other
1. PMON - Process Monitor 2. DBWR, DBWn - Database Writer (n=0-9,a-j) 3. LGWR, LGWn - Online Redo Log Writer 4. CKPT - Checkpoint 5. SMON - System Monitor	1. CM Cluster Manager 2. GCS Global Cache Service. 3. GES Global Enqueue Service Daemon 4. GMS Group Membership Service 5. GSD Global Services Daemon 6. LCKn Shared Resource Locking 7. LMDn Global Enqueue Service Daemon 8. LMON Global Enqueue Service Monitor 9. LMSn Global Cache Service Processes(Lock Manager Server) 10. NM Node Monitor 11. NPIC Network Inter-Process Communication 12. PSPn a PL-SQL plug in. · Oracle Parallel Fail Safe in 8ià Oracle Real Application Clusters Guard in 9i · DLMà GES+ GCS · OPSDàGSD · PCMàGCS NOTE:CGS Cluster Group Services (it is really a layer not process)	ASM Instance 1. ARBn , A00n 2. ORBn 3. RBAL ASM Clients (RDBMS instances) 1. ASMB 2. OSMB 3. RBAL	1. AQ 2. DMON 3. LNSV 4. MPR 5. SNPnnn 6. TRWR 7. WMON
Server Side (Optinal)			UnDocumented
1. RECO Distributed Transaction Recovery, Remote DB 2. CJQn Job Coordinator Process 3. QMNn 4. ARCH, ARCn Archiver(Archive Redo Log Files) 5. CTWR Block Change Tracking Writer 6. Dnnn Dispatcher 7. Jnnn Jobb Queue 8. MMAN Memory Manager 9. MMNL Metrics Monitor, session history, metrics computation 10. MMON Metrics Monitor, statistics, snapshots 11. ORAnnn Dedicated Server(User Process) 12. RVWR Recovery Writer(Flash Back Database) 13. Snnn Share Server			· Server Side (Extra) 1. CM 2. DIAG 3. EMNO 4. FAL Server 5. FMON 6. NSV0 7. RSM0 · Standby Node 1. FAL Client 2. LSPn 3. MRP0 4. RFS

Oracle Processes

• AQ Time Manager
• ARBn,A00n A group of slave processes establish connections to the ASM instance,
• ARCH The Archiver Process archives redo log files if ARCHIVELOG is enabled.
• ASMB This process contacts CSS using the diskgroup name, and acquires the associated ASM connect string. This connect string is then used to connect into ASM instance.
• CGS The software layer that provides mechanisms for instances to exchange configuration information, interprocess communication (IPC) port identifiers, and other kinds of meta data needed for Real Application Clusters operation. Cluster group services are also known as LMON-provided services.
• CKPT The Checkpoint Process reqularly initiates a checkpoint which uses DBWR to write all dirty blocks back to the datafiles, thus synchronizing the database. Since a Checkpoint records the current SCN, in a recovery only redo records with a SCN higher than that of the last checkpoint need to be applied.
• DBWR The Database Writer writes dirty blocks from the database buffer to the datafiles. How many DBWn Processes are started is determined by the initialization parameter DB_WRITER_PROCESSES. DBWR also writes the actual SCN with the Block.
• DMON The Data Guard Broker process.
• Dnnn The Dispatcher Process is used in a shared server environment.
• LCKnnn Inter-instance locking process.

· LMSn The Global Cache Service Processes (LMSn) are the processes that handle remote Global Cache Service (GCS) messages. Current Real Application Clusters software provides for up to 10 Global Cache Service Processes. The number of LMSn varies depending on the amount of messaging traffic among nodes in the cluster. The LMSn handle the acquisition interrupt and blocking interrupt requests from the remote instances for Global Cache Service resources. For cross-instance consistent read requests, the LMSn will create a consistent read version of the block and send it to the requesting instance. The LMSn also control the flow of messages to remote instances.

• LGWR The Log Writer writes the redo log buffer from the SGA to the online redo log file.

· GCS The Global Cache Service is the controlling process that implements Cache Fusion. It maintains the block mode for blocks in the global role. It is responsible for block transfers between instances. The Global Cache Service employs various background processes such as the Global Cache Service Processes (LMSn) and Global Enqueue Service Daemon (LMD).

· GES This service coordinates enqueues that are shared globally.

· GSD The Global Services Daemon (GSD) is a component that receives requests from SRVCTL to execute administrative job tasks, such as startup or shutdown. The command is executed locally on each node, and the results are sent back to SRVCTL. The daemon is installed on the nodes by default. It should not be deleted.

• LMDn Global Enqueue Service Daemon (LMD) The Global Enqueue Service Daemon (LMD) is the resource agent process that manages Global Enqueue Service (GES) resource requests. The LMD process also handles deadlock detection Global Enqueue Service (GES) requests. Remote resource requests are requests originating from another instance.
• LMON monitors the health of the Global Cache Service (GCS), registers and de-registers from the Cluster Manager CM. The background Global Enqueue Service Monitor (LMON) monitors the entire cluster to manage global resources. LMON manages instance and process expirations and the associated recovery for the Global Cache Service (GCS) and Global Enqueue Service (GES). In particular, LMON handles the part of recovery associated with global resources. LMON-provided services are also known as cluster group services (CGS).
• MMAN The memory manager
• MMON New background process in Oracle 10g.
• MRP Managed recovery process: the process that applies archived redo log to the standby database.
• ORBn These perform the actual rebalance data extent movements
• OSMB Any database instance that is using an ASM disk group will contain a background process called OSMB. The OSMB process is responsible for communicating with the ASM instance. The RBAL background process within in database instance, called RBAL (just like in the ASM Instance) performs a global open on ASM disks. A global open means that more than one database instance can be accessing the ASM disks at a time.
• PMON The Process Monitor checks if a user process fails and if so, does all cleaning up of resources that the user process has aquired.
• QMNn
• RBAL (ASM Clients)
• RBAL (ASM Instance) Rebalancer, Coordinator, This process performs global opens of all the disks in the disk groups
• RECO The Distributed Transaction Recovery Process finds pending (distributed) transaction and resolves them.
• RFS The remote file server process on the standby database receives archived redo logs from the primary database.
• SMON The System Monitor carries out a crash recovery when a crashed insance is started up again. It also cleans temporary segments.
• Snnn The Shared Server Process is used in a shared server environment.
• SNPnnn The snapshot process.
• TRWR Trace writer
• WMON The wakeup monitor process.

Table 4-1 Oracle Database Background Processes

Process Name	Description
CKPT Checkpoint	A checkpoint is the conclusion of the process of writing a buffer of a dirty block to stable storage. At specific times, all modified database buffers in the system global area are written to the datafiles by DBWn. This event is called a checkpoint. The checkpoint process is responsible for signalling DBWn at checkpoints and updating all the datafiles and control files of the database to indicate the most recent checkpoint. Checkpoint Process (CKPT) When a checkpoint occurs, Oracle must update the headers of all datafiles to record the details of the checkpoint. This is done by the CKPT process. The CKPT process does not write blocks to disk; DBWn always performs that work. The statistic DBWR checkpoints displayed by the System_Statistics monitor in Enterprise Manager indicates the number of checkpoint requests completed. Control files also record information about checkpoints. Every 3 seconds, the checkpoint process (CKPT) records information in the control file about the checkpoint position in the redo log. This information is used during database recovery to tell Oracle that all redo entries recorded before this point in the redo log group are not necessary for database recovery; they were already written to the datafiles. A checkpoint performs the following operations: 1. Every dirty block in the SGA buffer cache is written to the files. That is, it synchronizes the datablocks in the buffer cache with the datafiles on disk. · LGWR Flushing the contents of the redo log buffers to the redo log files · DBWR Flushing the contents of the database buffer cache to disk 2. Writing a checkpoint record to the redo log file. 3. The latest SCN is written (updated) into the: · Datafile header. · Controlfiles. NOTE: The update of the datafile headers and the control files is done by the LGWR(CKPT if CKPT is enabled). As of version 8.0, CKPT is enabled by default. Events that trigger a checkpoint The following events trigger a checkpoint. 1. Online Redo log switch 2. LOG_CHECKPOINT_TIMEOUT has expired 3. LOG_CHECKPOINT_INTERVAL has been reached 4. DBA requires so (ALTER SYSTEM CHECKPOINT ;) 5. In RAC environment, after releasing the lock of each data block that have been asked to accessed by another instance. 6. Every 3 seconds
PMON Process monitor	· PMON(Process Monitor) is in charge to perform process recovery when a user process fails (also when killed). · PMON is responsible for cleaning up the cache and freeing resources that the process was using. · PMON also checks on the Dispatcher Processes and Server Processes and restarts them if they have failed. · As a part of service registration, PMON registers instance information with the listener. Service registration is a feature which allows an Oracle Server to register itself with a listener. That means that the Oracle Service needs not be configured in the listener.ora configuration file. Service registration provides the listener with the following information: o The service names for each running instance of the database o Instance names of the database o Service handlers (dispatchers and dedicated servers) available for each instance This allows the listener to direct a client's request appropriately. o Dispatcher, instance, and node load information. This load information allows the listener to determine which dispatcher can best handle a client connection's request. If all dispatchers are blocked, then the listener can spawn a dedicated server for the connection. However, if there is no listener at startup, PMON can obviously not register those information. Therefore, PMON tries then periodically to register with the listener, which might take up to 60 seconds. It is possible to force this registration with ALTER SYSTEM REGISTER. Service registration offers the following benefits: o Simplified configuration: Service registration reduces the need for the SID_LIST_listener_name parameter setting, which specifies information about the databases served by the listener, in the listener.ora file. Note: The SID_LIST_listener_name parameter is still required if you are using Oracle Enterprise Manager to manage the database. o Connect-time failover: Because the listener always knows the state of the instances, service registration facilitates automatic failover of the client connect request to a different instance if one instance is down. In a static configuration model, a listener would start a dedicated server upon receiving a client request. The server would later find out that the instance is not up, causing an "Oracle not available" error message. o Runtime connection load balancing: Service registration enables the listener to forward client connect requests to the least loaded instance and dispatcher or dedicated server. Service registration balances the load across the service handlers and nodes. · in shared server environment , PMON watch MAX_SHARED_SERVER value and check : o if the load of system increased , it will spawn new shared server process . o if the load of system decreased, it will kill the extra shared server process to reach the SHARED_SERVER value. · PMON is responsible to report the traffic of dispatchers to listener. Choosing the dispatcher is up to listener. · in RAC environment , During reconfiguration, the process monitor (PMON) process records reconfiguration information with all the listeners (LOCAL_LISTENER and REMOTE_LISTENER) in the cluster for cross-instance listener registration.
DBWn Database writer	The database writer writes modified blocks from the database buffer cache to the datafiles. Oracle Database allows a maximum of 20 database writer processes (DBW0-DBW9 and DBWa-DBWj). The DB_WRITER_PROCESSES initialization parameter specifies the number of DBWn processes. The database selects an appropriate default setting for this initialization parameter or adjusts a user-specified setting based on the number of CPUs and the number of processor groups.
LGWR Log writer	The log writer process writes redo log entries to disk. Redo log entries are generated in the redo log buffer of the system global area (SGA). LGWR writes the redo log entries sequentially into a redo log file. If the database has a multiplexed redo log, then LGWR writes the redo log entries to a group of redo log files. LGWR begins writing to copy entries from the redo log buffer to the online redo log if any of the following are true: · The log buffer becomes one third full. · Each 3 seconds. · more than 1MB of log buffers be exist. · LGWR is posted by a server process performing a COMMIT or ROLLBACK. · DBWR posts LGWR to do so.
SMON System monitor	· The system monitor performs recovery when a failed instance starts up again. In a RAC database, the SMON process of one instance can perform instance recovery for other instances that have failed. · SMON also cleans up temporary segments that are no longer in use and recovers dead transactions skipped during system failure and instance recovery because of file-read or offline errors. These transactions are eventually recovered by SMON when the tablespace or file is brought back online.
ARCn Archiver	One or more archiver processes copy the redo log files to archival storage when they are full or a log switch occurs.
RECO Recoverer	The recoverer process is used to resolve distributed transactions that are pending because of a network or system failure in a distributed database. At timed intervals, the local RECO attempts to connect to remote databases and automatically complete the commit or rollback of the local portion of any pending distributed transactions.
Dnnn Dispatcher	Dispatchers are optional background processes, present only when the shared server configuration is used.
LMS Global Cache Service	In a Real Application Clusters environment, this process manages resources and provides inter-instance resource control.

http://www.jamsahar.com/Oracle/Oracle.Mem.Process.V12.htm