Oracle RAC Installation error: root.sh failed when installing 11gR2 Grid Infrastructure

While executing root.sh we are seeing below error
root@rac1:>/u01/app/11.2.0.3/grid/root.sh

Running Oracle 11g root script...The following environment variables are set as:    ORACLE_OWNER= grid    
ORACLE_HOME=  /u01/app/11.2.0.3/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.
Entries will be added to the /etc/oratab file as needed by Database Configuration Assistant when a database is created.
Finished running generic part of root script.
Now product-specific root actions will be performed.Using configuration parameter file: /u01/app/11.2.0.3/grid/crs/install/crsconfig_params/u01/app/11.2.0.3/grid/crs/install/crsconfig_lib.pm line 6763./u01/app/11.2.0.3/grid/perl/bin/perl -I/u01/app/11.2.0/grid/perl/lib -I/u01/app/11.2.0/grid/crs/install /u01/app/11.2.0.3/grid/crs/install/rootcrs.pl execution failed

Checking cfgtoollogs file shows $ORACLE_HOME/cfgtoollogs/crsconfig/rootcrs_rac1.log

root@rac1:>

cd /u01/app/11.2.0.3/grid/cfgtoollogs/crsconfig

tail -10 rootcrs_rac1.log

2011-07-21 16:02:26: Executing /u01/app/11.2.0.3/grid/bin/crsctl create scr oracle

2011-07-21 16:02:26: Executing cmd:/u01/app/11.2.0.3/grid/bin/crsctl create scr oracle

2011-07-21 16:02:26: Command output:

>  CRS-4046: Invalid Oracle Clusterware configuration.

>  CRS-4000: Command Create failed, or completed with errors.

>End Command output

Solution:-

1st Node:-

[root rac1] /u01/app/11.2.0.3/grid/crs/install# ./rootcrs.pl -verbose -deconfig -force

Using configuration parameter file: ./crsconfig_params
****Unable to retrieve Oracle Clusterware home.
Start Oracle Clusterware stack and try again.
CRS-4046: Invalid Oracle Clusterware configuration.
CRS-4000: Command Stop failed, or completed with errors.
CRS-4046: Invalid Oracle Clusterware configuration.
CRS-4000: Command Stop failed, or completed with errors.
################################################################
# You must kill processes or reboot the system to properly #
# cleanup the processes started by Oracle clusterware          #
################################################################
ACFS-9313: No ADVM/ACFS installation detected.
Failure in execution (rc=-1, 256, No such file or directory) for command /etc/ohasd deinstall
Successfully deconfigured Oracle clusterware stack on this node
[root rac1] /u01/app/11.2.0.3/grid/crs/install# ps -ef|grep crs
    grid  8388648        1   0 05:14:05      -  2:39 /tmp/OraInstall2014-01-20_05-14-02AM/jdk/jre/bin/java -Doracle.installer.library_loc=/tmp/OraInstall2014-01-20_05-14-02AM/oui/lib/aix -Doracle.installer.oui_loc=/tmp/OraInstall2014-01-20_05-14-02AM/oui -Doracle.installer.bootstrap=TRUE -Doracle.installer.startup_location=/u01/app/Oracle_Software/grid/install -Doracle.installer.jre_loc=/tmp/OraInstall2014-01-20_05-14-02AM/jdk/jre -Doracle.installer.nlsEnabled="TRUE" -Doracle.installer.prereqConfigLoc=/tmp/OraInstall2014-01-20_05-14-02AM/prereq  -Doracle.installer.unixVersion=1 -Doracle.install.setup.workDir=/u01/app/Oracle_Software/grid -mx150m -Xverify:none -Dice.pilots.html4.defaultEncoding=ISO8859_1 -cp /tmp/OraInstall2014-01-20_05-14-02AM::/tmp/OraInstall2014-01-20_05-14-02AM/ext/jlib/OraPrereq.jar:/tmp/OraInstall2014-01-20_05-14-02AM/ext/jlib/OraPrereqChecks.jar:/tmp/OraInstall2014-01-20_05-14-02AM/ext/jlib/cvu.jar:/tmp/OraInstall2014-01-20_05-14-02AM/ext/jlib/installcommons_1.0.0b.jar:/tmp/OraInstall2014-01-20_05-14-02AM/ext/jlib/instcommon.jar:/tmp/OraInstall2014-01-20_05-14-02AM/ext/jlib/instcrs.jar:/tmp/OraInstall2014-01-20_05-14-02AM/ext/jlib/jsch.jar:/tmp/OraInstall2014-01-20_05-14-02AM/ext/jlib/ojdbc5.jar:/tmp/OraInstall2014-01-20_05-14-02AM/ext/jlib/orai18n-mapping.jar:/tmp/OraInstall2014-01-20_05-14-02AM/ext/jlib/orai18n-utility.jar:/tmp/OraInstall2014-01-20_05-14-02AM/ext/jlib/prov_fixup.jar:/tmp/OraInstall2014-01-20_05-14-02AM/ext/jlib/remoteinterfaces.jar:/tmp/OraInstall2014-01-20_05-14-02AM/ext/jlib/ssh.jar:/tmp/OraInstall2014-01-20_05-14-02AM/oui/jlib/OraInstaller.jar:/tmp/OraInstall2014-01-20_05-14-02AM/oui/jlib/oneclick.jar:/tmp/OraInstall2014-01-20_05-14-02AM/oui/jlib/xmlparserv2.jar:/tmp/OraInstall2014-01-20_05-14-02AM/oui/jlib/share.jar:/tmp/OraInstall2014-01-20_05-14-02AM/oui/jlib/OraInstallerNet.jar:/tmp/OraInstall2014-01-20_05-14-02AM/oui/jlib/emCfg.jar:/tmp/OraInstall2014-01-20_05-14-02AM/oui/jlib/emocmutl.jar:/tmp/OraInstall2014-01-20_05-14-02AM/oui/jlib/OraPrereq.jar:/tmp/OraInstall2014-01-20_05-14-02AM/oui/jlib/jsch.jar:/tmp/OraInstall2014-01-20_05-14-02AM/oui/jlib/ssh.jar:/tmp/OraInstall2014-01-20_05-14-02AM/oui/jlib/remoteinterfaces.jar:/tmp/OraInstall2014-01-20_05-14-02AM/oui/jlib/http_client.jar:/tmp/OraInstall2014-01-20_05-14-02AM/oui/jlib/OraSuiteInstaller.jar:../stage/Components/oracle.swd.opatch/11.2.0.3.0/1/DataFiles/jlib/opatch.jar:../stage/Components/oracle.swd.opatch/11.2.0.3.0/1/DataFiles/jlib/opatchactions.jar:../stage/Components/oracle.swd.opatch/11.2.0.3.0/1/DataFiles/jlib/opatchprereq.jar:../stage/Components/oracle.swd.opatch/11.2.0.3.0/1/DataFiles/jlib/opatchutil.jar:/tmp/OraInstall2014-01-20_05-14-02AM/oui/jlib/OraCheckPoint.jar:/tmp/OraInstall2014-01-20_05-14-02AM/oui/jlib/InstImages.jar:/tmp/OraInstall2014-01-20_05-14-02AM/oui/jlib/InstHelp.jar:/tmp/OraInstall2014-01-20_05-14-02AM/oui/jlib/InstHelp_de.jar:/tmp/OraInstall2014-01-20_05-14-02AM/oui/jlib/InstHelp_es.jar:/tmp/OraInstall2014-01-20_05-14-02AM/oui/jlib/InstHelp_fr.jar:/tmp/OraInstall2014-01-20_05-14-02AM/oui/jlib/InstHelp_it.jar:/tmp/OraInstall2014-01-20_05-14-02AM/oui/jlib/InstHelp_ja.jar:/tmp/OraInstall2014-01-20_05-14-02AM/oui/jlib/InstHelp_ko.jar:/tmp/OraInstall2014-01-20_05-14-02AM/oui/jlib/InstHelp_pt_BR.jar:/tmp/OraInstall2014-01-20_05-14-02AM/oui/jlib/InstHelp_zh_CN.jar:/tmp/OraInstall2014-01-20_05-14-02AM/oui/jlib/InstHelp_zh_TW.jar:/tmp/OraInstall2014-01-20_05-14-02AM/oui/jlib/oracle_ice.jar:/tmp/OraInstall2014-01-20_05-14-02AM/oui/jlib/help4.jar:/tmp/OraInstall2014-01-20_05-14-02AM/oui/jlib/help4-nls.jar:/tmp/OraInstall2014-01-20_05-14-02AM/oui/jlib/ewt3.jar:/tmp/OraInstall2014-01-20_05-14-02AM/oui/jlib/ewt3-swingaccess.jar:/tmp/OraInstall2014-01-20_05-14-02AM/oui/jlib/ewt3-nls.jar:/tmp/OraInstall2014-01-20_05-14-02AM/oui/jlib/swingaccess.jar::/tmp/OraInstall2014-01-20_05-14-02AM/oui/jlib/jewt4.jar:/tmp/OraInstall2014-01-20_05-14-02AM/oui/jlib/jewt4-nls.jar:/tmp/OraInstall2014-01-20_05-14-02AM/oui/jlib/orai18n-collation.jar:/tmp/OraInstall2014-01-20_05-14-02AM/oui/jlib/orai18n-mapping.jar:/tmp/OraInstall2014-01-20_05-14-02AM/oui/jlib/ojmisc.jar:/tmp/OraInstall2014-01-20_05-14-02AM/oui/jlib/xml.jar:/tmp/OraInstall2014-01-20_05-14-02AM/oui/jlib/srvm.jar:/tmp/OraInstall2014-01-20_05-14-02AM/oui/jlib/srvmasm.jar oracle.install.ivw.crs.driver.CRSInstaller -scratchPath /tmp/OraInstall2014-01-20_05-14-02AM -sourceLoc /u01/app/Oracle_Software/grid/install/../stage/products.xml -sourceType network -timestamp 2014-01-20_05-14-02AM -formCluster

Now retry running root.sh 

[root rac1] /u01/app/11.2.0.3/grid/crs/install# /u01/app/11.2.0.3/grid/root.sh
Performing root user operation for Oracle 11g

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /u01/app/11.2.0.3/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/app/11.2.0.3/grid/crs/install/crsconfig_params
User ignored Prerequisites during installation
User grid is missing the following capabilities required to run CSSD in realtime:
  CAP_NUMA_ATTACH,CAP_BYPASS_RAC_VMM,CAP_PROPAGATE
To add the required capabilities, please run:
   /usr/bin/chuser capabilities=CAP_NUMA_ATTACH,CAP_BYPASS_RAC_VMM,CAP_PROPAGATE grid
CSS cannot be run in realtime mode at /u01/app/11.2.0.3/grid/crs/install/crsconfig_lib.pm line 11423.
/u01/app/11.2.0.3/grid/perl/bin/perl -I/u01/app/11.2.0.3/grid/perl/lib -I/u01/app/11.2.0.3/grid/crs/install /u01/app/11.2.0.3/grid/crs/install/rootcrs.pl execution failed

again thrown the error? execute below command

[root rac1] /u01/app/11.2.0.3/grid/crs/install#  /usr/bin/chuser capabilities=CAP_NUMA_ATTACH,CAP_BYPASS_RAC_VMM,CAP_PROPAGATE grid

Now retry executing root.sh

[root rac1] /u01/app/11.2.0.3/grid/crs/install# /u01/app/11.2.0.3/grid/root.sh

Performing root user operation for Oracle 11g

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /u01/app/11.2.0.3/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/app/11.2.0.3/grid/crs/install/crsconfig_params
User ignored Prerequisites during installation
User grid has the required capabilities to run CSSD in realtime mode
OLR initialization - successful
  root wallet
  root wallet cert
  root cert export
  peer wallet
  profile reader wallet
  pa wallet
  peer wallet keys
  pa wallet keys
  peer cert request
  pa cert request
  peer cert
  pa cert
  peer root cert TP
  profile reader root cert TP
  pa root cert TP
  peer pa cert TP
  pa peer cert TP
  profile reader pa cert TP
  profile reader peer cert TP
  peer user cert
  pa user cert
Adding Clusterware entries to inittab
CRS-2672: Attempting to start 'ora.mdnsd' on 'rac1'
CRS-2676: Start of 'ora.mdnsd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rac1'
CRS-2676: Start of 'ora.gpnpd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac1'
CRS-2672: Attempting to start 'ora.gipcd' on 'rac1'
CRS-2676: Start of 'ora.gipcd' on 'rac1' succeeded
CRS-2676: Start of 'ora.cssdmonitor' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rac1'
CRS-2672: Attempting to start 'ora.diskmon' on 'rac1'
CRS-2676: Start of 'ora.diskmon' on 'rac1' succeeded
CRS-2676: Start of 'ora.cssd' on 'rac1' succeeded

ASM created and started successfully.

Disk Group OCR_VOTE created successfully.

clscfg: -install mode specified
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'system'..
Operation successful.
CRS-4256: Updating the profile
Successful addition of voting disk 8ee5a0b0f1474fcbbf3127724549a1cd.
Successful addition of voting disk 4f89d99d73594fecbfcb12a92c1c3c7e.
Successful addition of voting disk 11855ece9be44fe9bfe031dd507f163b.
Successfully replaced voting disk group with +OCR_VOTE.
CRS-4256: Updating the profile
CRS-4266: Voting file(s) successfully replaced
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   8ee5a0b0f1474fcbbf3127724549a1cd (/dev/rhdisk4) [OCR_VOTE]
 2. ONLINE   4f89d99d73594fecbfcb12a92c1c3c7e (/dev/rhdisk5) [OCR_VOTE]
 3. ONLINE   11855ece9be44fe9bfe031dd507f163b (/dev/rhdisk6) [OCR_VOTE]
Located 3 voting disk(s).
CRS-2672: Attempting to start 'ora.asm' on 'rac1'
CRS-2676: Start of 'ora.asm' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.OCR_VOTE.dg' on 'rac1'
CRS-2676: Start of 'ora.OCR_VOTE.dg' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.registry.acfs' on 'rac1'
CRS-2676: Start of 'ora.registry.acfs' on 'rac1' succeeded
Configure Oracle Grid Infrastructure for a Cluster ... succeeded



2nd Node:-

[root rac2] /u01/app/11.2.0.3/grid/crs/install# ./rootcrs.pl -verbose -deconfig -force
Using configuration parameter file: ./crsconfig_params
****Unable to retrieve Oracle Clusterware home.
Start Oracle Clusterware stack and try again.
CRS-4047: No Oracle Clusterware components configured.
CRS-4000: Command Stop failed, or completed with errors.
CRS-4047: No Oracle Clusterware components configured.
CRS-4000: Command Stop failed, or completed with errors.
################################################################
# You must kill processes or reboot the system to properly #
# cleanup the processes started by Oracle clusterware          #
################################################################
ACFS-9313: No ADVM/ACFS installation detected.
Either /etc/oracle/olr.loc does not exist or is not readable
Make sure the file exists and it has read and execute access
Either /etc/oracle/olr.loc does not exist or is not readable
Make sure the file exists and it has read and execute access
Failure in execution (rc=-1, 256, No such file or directory) for command /etc/ohasd deinstall
Successfully deconfigured Oracle clusterware stack on this node

Retry the execution of root.sh

[root rac2] /u01/app/11.2.0.3/grid/crs/install# /u01/app/11.2.0.3/grid/root.sh
Performing root user operation for Oracle 11g

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /u01/app/11.2.0.3/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/app/11.2.0.3/grid/crs/install/crsconfig_params
User ignored Prerequisites during installation
User grid is missing the following capabilities required to run CSSD in realtime:
  CAP_NUMA_ATTACH,CAP_BYPASS_RAC_VMM,CAP_PROPAGATE
To add the required capabilities, please run:
   /usr/bin/chuser capabilities=CAP_NUMA_ATTACH,CAP_BYPASS_RAC_VMM,CAP_PROPAGATE grid
CSS cannot be run in realtime mode at /u01/app/11.2.0.3/grid/crs/install/crsconfig_lib.pm line 11423.
/u01/app/11.2.0.3/grid/perl/bin/perl -I/u01/app/11.2.0.3/grid/perl/lib -I/u01/app/11.2.0.3/grid/crs/install /u01/app/11.2.0.3/grid/crs/install/rootcrs.pl execution failed


Again thrown the error? No problem execute below step

[root rac2] /u01/app/11.2.0.3/grid/crs/install# /usr/bin/chuser capabilities=CAP_NUMA_ATTACH,CAP_BYPASS_RAC_VMM,CAP_PROPAGATE grid

Now retry the execution of root.sh

[root rac2] /u01/app/11.2.0.3/grid/crs/install# /u01/app/11.2.0.3/grid/root.sh
Performing root user operation for Oracle 11g

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /u01/app/11.2.0.3/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:

The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/app/11.2.0.3/grid/crs/install/crsconfig_params
User ignored Prerequisites during installation
User grid has the required capabilities to run CSSD in realtime mode
OLR initialization - successful
Adding Clusterware entries to inittab
CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node rac1, number 1, and is terminating
An active cluster was found during exclusive startup, restarting to join the cluster
Configure Oracle Grid Infrastructure for a Cluster ... succeeded