Oracle Core Help: Basic RAC Management Commands (Quick Reference)

Overview of Basic RAC Management Commands

The commands we will use are listed below. Remember that this document is a quick reference, and not an exhaustive list of all commands for managing your RAC environment.

Cluster Related Commands
crs_stat -t	Shows HA resource status (hard to read)
crsstat	Ouptut of crs_stat -t formatted nicely
ps -ef\|grep d.bin	crsd.bin evmd.bin ocssd.bin
crsctl check crs	CSS,CRS,EVM appears healthy
crsctl stop crs	Stop crs and all other services
crsctl disable crs*	Prevents CRS from starting on reboot
crsctl enable crs*	Enables CRS start on reboot
crs_stop -all	Stops all registered resources
crs_start -all	Starts all registered resources

* These commands update the file /etc/oracle/scls_scr/<node>/root/crsstart which contains the string “enable” or “disable” as appropriate.

Database Related Commands
srvctl start instance -d <db_name> -i <inst_name>	Starts an instance
srvctl start database -d <db_name>	Starts all instances
srvctl stop database -d <db_name>	Stops all instances, closes database
srvctl stop instance -d <db_name> -i <inst_name>	Stops an instance
srvctl start service -d <db_name> -s <service_name>	Starts a service
srvctl stop service -d <db_name> -s <service_name>	Stops a service
srvctl status service -d <db_name>	Checks status of a service
srvctl status instance -d <db_name> -i <inst_name>	Checks an individual instance
srvctl status database -d <db_name>	Checks status of all instances
srvctl start nodeapps -n <node_name>	Starts gsd, vip, listener, and ons
srvctl stop nodeapps -n <node_name>	Stops gsd, vip and listener

*Keep in mind that some resources will not start unless other resources are already online. We will now look at the general dependency list in greater detail.

There are three main background processes you can see when doing a ps –ef|grep d.bin. They are normally started by init during the operating system boot process. They can be started and stopped manually by issuing the command /etc/init.d/init.crs {start|stop|enable|disable}

/etc/rc.d/init.d/init.evmd
/etc/rc.d/init.d/init.cssd
/etc/rc.d/init.d/init.crsd

Once the above processes are running, they will automatically start the following services in the following order if they are enabled. This list assumes you are using ASM and have a service set up for TAF/load balancing.

The nodeapps (gsd, VIP, ons, listener) are brought online.
The ASM instances are brought online.
The database instances are brought online.
Any defined services are brought online.

Basic RAC Management Commands

Now that we know the dependency tree and have some commands at our disposal, let’s have a look at them one at a time, starting with the cluster commands and processes.

crs_stat -t

This command shows us the status of each registered resource in the cluster. I generally avoid this command because its output is hard to read since the names are truncated as you can see in the sample output below. You can download a helpful script called crsstat from http://www.dbspecialists.com/specialists/specialist2007-05.html to make it easy on your eyes.

[oracle@green ~]$ crs_stat -t
Name           Type           Target    State     Host
------------------------------------------------------------
ora....SM1.asm application    ONLINE    ONLINE    green
ora....EN.lsnr application    ONLINE    ONLINE    green
ora.green.gsd  application    ONLINE    ONLINE    green
ora.green.ons  application    ONLINE    ONLINE    green
ora.green.vip  application    ONLINE    ONLINE    green
ora.....RAC.cs application    ONLINE    ONLINE    red
ora....cl1.srv application    ONLINE    ONLINE    green
ora....cl2.srv application    ONLINE    ONLINE    red
ora.orcl.db    application    ONLINE    ONLINE    red
ora....l1.inst application    ONLINE    ONLINE    green
ora....l2.inst application    ONLINE    ONLINE    red
ora....SM2.asm application    ONLINE    ONLINE    red
ora....ED.lsnr application    ONLINE    ONLINE    red
ora.red.gsd    application    ONLINE    ONLINE    red
ora.red.ons    application    ONLINE    ONLINE    red
ora.red.vip    application    ONLINE    ONLINE    red
[oracle@green ~]$

crsstat

The output of this script is much better. You can learn more about this script and download it at http://www.dbspecialists.com/specialists/specialist2007-05.html.

[oracle@green ~]$ crsstat
HA Resource                                 Target     State
-----------                                 ------     -----
ora.green.ASM1.asm                          ONLINE     ONLINE on green
ora.green.LISTENER_GREEN.lsnr               ONLINE     ONLINE on green
ora.green.gsd                               ONLINE     ONLINE on green
ora.green.ons                               ONLINE     ONLINE on green
ora.green.vip                               ONLINE     ONLINE on green
ora.orcl.RAC.cs                             ONLINE     ONLINE on red
ora.orcl.RAC.orcl1.srv                      ONLINE     ONLINE on green
ora.orcl.RAC.orcl2.srv                      ONLINE     ONLINE on red
ora.orcl.db                                 ONLINE     ONLINE on red
ora.orcl.orcl1.inst                         ONLINE     ONLINE on green
ora.orcl.orcl2.inst                         ONLINE     ONLINE on red
ora.red.ASM2.asm                            ONLINE     ONLINE on red
ora.red.LISTENER_RED.lsnr                   ONLINE     ONLINE on red
ora.red.gsd                                 ONLINE     ONLINE on red
ora.red.ons                                 ONLINE     ONLINE on red
ora.red.vip                                 ONLINE     ONLINE on red
[oracle@green ~]$

ps -ef|grep d.bin

We can use this command to verify that the CRS background processes are actually running. It is implicit that they are running if the crs_stat command and crsstat script work. If they do not work, you will want to verify the background processes are really running.

[root@green ~]# ps -ef|grep d.bin
oracle    5335  3525  0 Jul11 ?        00:00:05 /u01/app/oracle/product/10.2.0/crs/bin/evmd.bin
root      5487  3817  0 Jul11 ?        00:00:00 /u01/app/oracle/product/10.2.0/crs/bin/crsd.bin reboot
oracle    5932  5392  0 Jul11 ?        00:00:00 /u01/app/oracle/product/10.2.0/crs/bin/ocssd.bin
root     30486 30177  0 18:23 pts/1    00:00:00 grep d.bin
[root@green ~]#

crsctl check crs

This command verifies that the above background daemons are functioning.

[oracle@green ~]$ crsctl check crs
CSS appears healthy
CRS appears healthy
EVM appears healthy
[oracle@green ~]$

crsctl stop crs

We’ll need to be logged onto the server as the root user to run this command. It will stop all HA resources on the local node, and it will also stop the above mentioned background daemons.

[oracle@green ~]$ crsctl stop crs
Insufficient user privileges.
[oracle@green ~]$ su
Password:
[root@green oracle]# crsctl stop crs
Stopping resources. This could take several minutes.
Successfully stopped CRS resources.
Stopping CSSD.
Shutting down CSS daemon.
Shutdown request successfully issued.
[root@green oracle]#

crsctl disable crs

This command will prevent CRS from starting on a reboot. Note there is no return output from the command.

[root@green oracle]# crsctl disable crs
[root@green oracle]#

We did a reboot after this and verified that CRS did not come back online because we wanted to do some operating system maintenance. Let’s check the status by running some of the commands we’ve just discussed.

[oracle@green ~]$ crsstat
HA Resource                                   Target     State
-----------                                   ------     -----
error connecting to CRSD at [(ADDRESS=(PROTOCOL=ipc)(KEY=ora_crsqs))] clsccon 184

[oracle@green ~]$ crsctl check crs
Failure 1 contacting CSS daemon
Cannot communicate with CRS
Cannot communicate with EVM

[oracle@green ~]$ ps -ef|grep d.bin
oracle    6149  5582  0 15:54 pts/1    00:00:00 grep d.bin
[oracle@green ~]$

Everything appears to be down on this node as expected.
Now let’s start everything back up. We will need to be root for this, unless you have been given permissions or sudo to run crsctl start crs.

[root@green oracle]# crsctl start crs
Attempting to start CRS stack
The CRS stack will be started shortly
[root@green oracle]#

After a few minutes the registered resources for this node should come online. Let’s check to be sure:

[oracle@green ~]$ crsstat
HA Resource                                   Target     State
-----------                                   ------     -----
ora.green.ASM1.asm                            ONLINE     ONLINE on green
ora.green.LISTENER_GREEN.lsnr                 ONLINE     ONLINE on green
ora.green.gsd                                 ONLINE     ONLINE on green
ora.green.ons                                 ONLINE     ONLINE on green
ora.green.vip                                 ONLINE     ONLINE on green
ora.orcl.RAC.cs                               ONLINE     ONLINE on red
ora.orcl.RAC.orcl1.srv                        ONLINE     ONLINE on green
ora.orcl.RAC.orcl2.srv                        ONLINE     ONLINE on red
ora.orcl.db                                   ONLINE     ONLINE on red
ora.orcl.orcl1.inst                           ONLINE     ONLINE on green
ora.orcl.orcl2.inst                           ONLINE     ONLINE on red
ora.red.ASM2.asm                              ONLINE     ONLINE on red
ora.red.LISTENER_RED.lsnr                     ONLINE     ONLINE on red
ora.red.gsd                                   ONLINE     ONLINE on red
ora.red.ons                                   ONLINE     ONLINE on red
ora.red.vip                                   ONLINE     ONLINE on red
[oracle@green ~]$

Let’s not forget to enable CRS on reboot:

[root@green oracle]# crsctl enable crs

crs_stop -all

This is a handy script that stops the registered resources and leaves the CRS running. This includes all services in the cluster, so it will bring down all registered resources on all nodes.

[oracle@green ~]$ crs_stop -all
Attempting to stop `ora.green.gsd` on member `green`
Attempting to stop `ora.orcl.RAC.orcl2.srv` on member `red`
Stop of `ora.orcl.TEST.orcl1.srv` on member `green` succeeded.
Attempting to stop `ora.orcl.RAC.orcl1.srv` on member `green`
Attempting to stop `ora.green.ons` on member `green`
Attempting to stop `ora.orcl.RAC.cs` on member `red`
Stop of `ora.green.gsd` on member `green` succeeded.
Stop of `ora.orcl.RAC.orcl1.srv` on member `green` succeeded.
Stop of `ora.orcl.RAC.orcl2.srv` on member `red` succeeded.
Stop of `ora.orcl.TEST.orcl2.srv` on member `red` succeeded.
Stop of `ora.green.ons` on member `green` succeeded.
--snip--
CRS-0216: Could not stop resource 'ora.orcl.orcl2.inst'.
[oracle@green ~]$

Occasionally you will get the CRS-0216 error message shown above. This is usually bogus, but you should re-check with crsstat and ps –ef|grep smon or similar to be sure everything has died off.
Let’s verify that crs_stop -all worked as expected:

[root@green oracle]# crsstat
HA Resource                                   Target     State
-----------                                   ------     -----
ora.green.ASM1.asm                            OFFLINE    OFFLINE
ora.green.LISTENER_GREEN.lsnr                 OFFLINE    OFFLINE
ora.green.gsd                                 OFFLINE    OFFLINE
ora.green.ons                                 OFFLINE    OFFLINE
ora.green.vip                                 OFFLINE    OFFLINE
ora.orcl.RAC.cs                               OFFLINE    OFFLINE
ora.orcl.RAC.orcl1.srv                        OFFLINE    OFFLINE
ora.orcl.RAC.orcl2.srv                        OFFLINE    OFFLINE
ora.orcl.db                                   OFFLINE    OFFLINE
ora.orcl.orcl1.inst                           OFFLINE    OFFLINE
ora.orcl.orcl2.inst                           OFFLINE    OFFLINE
ora.red.ASM2.asm                              OFFLINE    OFFLINE
ora.red.LISTENER_RED.lsnr                     OFFLINE    OFFLINE
ora.red.gsd                                   OFFLINE    OFFLINE
ora.red.ons                                   OFFLINE    OFFLINE
ora.red.vip                                   OFFLINE    OFFLINE
[root@green oracle]#

Let’s move on to working with srvctl and managing individual resources. We will begin with the crs background daemons already running, and all registered resources being offline from the last step above. We will first start the nodeapps, then the ASM instances, followed by the database instances, and lastly the services for TAF and load balancing. This is the dependency order in our particular environment. You may or may not have ASM or TAF and load balancing services to start in your environment.

srvctl start nodeapps -n (node)

This will bring up the gsd, ons, listener, and vip. The same command can shut down the nodeapps by replacing start with stop.

[oracle@green ~]$ srvctl start nodeapps -n green
[oracle@green ~]$ srvctl start nodeapps -n red

Now we will check with crsstat again to be sure the nodeapps have started.

[oracle@green ~]$ crsstat
HA Resource                               Target     State
-----------                               ------     -----
ora.green.ASM1.asm                        OFFLINE    OFFLINE
ora.green.LISTENER_GREEN.lsnr             ONLINE     ONLINE on green
ora.green.gsd                             ONLINE     ONLINE on green
ora.green.ons                             ONLINE     ONLINE on green
ora.green.vip                             ONLINE     ONLINE on green
ora.orcl.RAC.cs                           OFFLINE    OFFLINE
ora.orcl.RAC.orcl1.srv                    OFFLINE    OFFLINE
ora.orcl.RAC.orcl2.srv                    OFFLINE    OFFLINE
ora.orcl.db                               OFFLINE    OFFLINE
ora.orcl.orcl1.inst                       OFFLINE    OFFLINE
ora.orcl.orcl2.inst                       OFFLINE    OFFLINE
ora.red.ASM2.asm                          OFFLINE    OFFLINE
ora.red.LISTENER_RED.lsnr                 ONLINE     ONLINE on red
ora.red.gsd                               ONLINE     ONLINE on red
ora.red.ons                               ONLINE     ONLINE on red
ora.red.vip                               ONLINE     ONLINE on red
[root@green oracle]#

Now we need to start our ASM instances before we bring up our database and services.

srvctl start asm -n (node)

This will bring up our ASM instances on nodes green and red. Again, the same command will stop the ASM instances by replacing start with stop.

[oracle@green ~]$ srvctl start asm -n green
[oracle@green ~]$ srvctl start asm -n red
[oracle@green ~]$ crsstat
HA Resource                               Target     State
-----------                               ------     -----
ora.green.ASM1.asm                        ONLINE     ONLINE on green
ora.green.LISTENER_GREEN.lsnr             ONLINE     ONLINE on green
ora.green.gsd                             ONLINE     ONLINE on green
ora.green.ons                             ONLINE     ONLINE on green
ora.green.vip                             ONLINE     ONLINE on green
ora.orcl.RAC.cs                           OFFLINE    OFFLINE
ora.orcl.RAC.orcl1.srv                    OFFLINE    OFFLINE
ora.orcl.RAC.orcl2.srv                    OFFLINE    OFFLINE
ora.orcl.db                               OFFLINE    OFFLINE
ora.orcl.orcl1.inst                       OFFLINE    OFFLINE
ora.orcl.orcl2.inst                       OFFLINE    OFFLINE
ora.red.ASM2.asm                          ONLINE     ONLINE on red
ora.red.LISTENER_RED.lsnr                 ONLINE     ONLINE on red
ora.red.gsd                               ONLINE     ONLINE on red
ora.red.ons                               ONLINE     ONLINE on red
ora.red.vip                               ONLINE     ONLINE on red
[oracle@green ~]$

Now let’s bring up our orcl1 and orcl2 instances, and verify they are up with crsstat. Once more we can replace start with stop and shutdown an individual instance if we so choose.

srvctl start instance -d (database) -I (instance)

[oracle@green ~]$ srvctl start instance –d orcl –i orcl1
[oracle@green ~]$ srvctl start instance -d orcl -i orcl2
[oracle@green ~]$ crsstat
HA Resource                               Target     State
-----------                               ------     -----
ora.green.ASM1.asm                        ONLINE     ONLINE on green
ora.green.LISTENER_GREEN.lsnr             ONLINE     ONLINE on green
ora.green.gsd                             ONLINE     ONLINE on green
ora.green.ons                             ONLINE     ONLINE on green
ora.green.vip                             ONLINE     ONLINE on green
ora.orcl.RAC.cs                           OFFLINE    OFFLINE
ora.orcl.RAC.orcl1.srv                    OFFLINE    OFFLINE
ora.orcl.RAC.orcl2.srv                    OFFLINE    OFFLINE
ora.orcl.db                               ONLINE     ONLINE on red
ora.orcl.orcl1.inst                       ONLINE     ONLINE on green
ora.orcl.orcl2.inst                       ONLINE     ONLINE on red
ora.red.ASM2.asm                          ONLINE     ONLINE on red
ora.red.LISTENER_RED.lsnr                 ONLINE     ONLINE on red
ora.red.gsd                               ONLINE     ONLINE on red
ora.red.ons                               ONLINE     ONLINE on red
ora.red.vip                               ONLINE     ONLINE on red
[oracle@green ~]$

srvctl start service -d (database) -s (service)

Now we will finish up by bringing our load balanced/TAF service named RAC online.

[oracle@green ~]$ srvctl start service -d orcl -s RAC
[oracle@green ~]$ crsstat
HA Resource                               Target     State
-----------                               ------     -----
ora.green.ASM1.asm                        ONLINE     ONLINE on green
ora.green.LISTENER_GREEN.lsnr             ONLINE     ONLINE on green
ora.green.gsd                             ONLINE     ONLINE on green
ora.green.ons                             ONLINE     ONLINE on green
ora.green.vip                             ONLINE     ONLINE on green
ora.orcl.RAC.cs                           ONLINE     ONLINE on red
ora.orcl.RAC.orcl1.srv                    ONLINE     ONLINE on green
ora.orcl.RAC.orcl2.srv                    ONLINE     ONLINE on red
ora.orcl.db                               ONLINE     ONLINE on red
ora.orcl.orcl1.inst                       ONLINE     ONLINE on green
ora.orcl.orcl2.inst                       ONLINE     ONLINE on red
ora.red.ASM2.asm                          ONLINE     ONLINE on red
ora.red.LISTENER_RED.lsnr                 ONLINE     ONLINE on red
ora.red.gsd                               ONLINE     ONLINE on red
ora.red.ons                               ONLINE     ONLINE on red
ora.red.vip                               ONLINE     ONLINE on red
[oracle@green ~]$

There we have it; all of our resources are now online. The next steps would be to verify you can connect via SQL*Plus or your favorite application.

*******************************************************************************************************************

Oracle Core Help

Thursday, August 9, 2012

Basic RAC Management Commands (Quick Reference)

No comments: