Oracle RAC Error: CRS-2805: Unable to start 'ora.LISTENER.lsnr' because it has a 'hard' dependency on resource type 'ora.cluster_vip_net1.type' and no resource of that type can satisfy the dependency

In one of our RAC setups Listener (ora.LISTENER.lsnr) status was OFFLINE in the CRS. When I tried to start it as root user with the command "crs_start of ora.LISTENER.lsnr" it thrown below error and listener did not come up.

"CRS-2805: Unable to start 'ora.LISTENER.lsnr' because it has a 'hard' dependency on resource type 'ora.cluster_vip_net1.type' and no resource of that type can satisfy the dependency"

After doing some research I found that VIP of the node is not pinging even though CRS is up and running. Then learnt that VIP of the node is not registered with CRS which caused the listener to not to come up.

[root@rac1 bin]# ./srvctl config nodeapps -a
Network exists: 1/192.168.2.0/255.255.255.0/eth0, type static

VIP exists: /rac2-vip/192.168.2.112/192.168.2.0/255.255.255.0/eth0, hosting node rac2

The above output shows that only 2nd node rac2-vip is registered. 1st node VIP rac1-vip is not registered.
So, I registered the VIP of the node to CRS as root user with below command.

[root@rac1 bin]# ./srvctl add vip -n rac1 -A rac1-vip/255.255.255.0/eth0 -k 1

Now both the VIPs are registered.

[root@rac1 bin]# ./srvctl config nodeapps -a
Network exists: 1/192.168.2.0/255.255.255.0/eth0, type static
VIP exists: /rac1-vip/192.168.2.111/192.168.2.0/255.255.255.0/eth0, hosting node rac1

VIP exists: /rac2-vip/192.168.2.112/192.168.2.0/255.255.255.0/eth0, hosting node rac2

Start rac1-vip now

[root@rac1 bin]# ./crs_start -f ora.rac1.vip
Attempting to start `ora.rac1.vip` on member `rac1`
Start of `ora.rac1.vip` on member `rac1` succeeded.
Attempting to start `ora.LISTENER.lsnr` on member `rac1`

Start of `ora.LISTENER.lsnr` on member `rac1` succeeded.

Now the VIP is registered and it is reachable now. ora.LISTENER.lsnr status become ONLINE now.

Oracle RAC Error: Warning "ora.oc4j has 1 instances in INTERMEDIATE State" on one of the RAC nodes

I have 11gR2 2-node RAC setup in Windows servers. 

Got the alerts from OEM grid with the warning message "ora.oc4j has 1 instances in INTERMEDIATE State".

The status of CRS in both the boxes is up and running. But the output of "crsctl stat res -t" has displayed the warnings with OC4J and SCAN listeners status was showing "INTERMEDIATE".

It is a known bug 14593380 of 11gR2 RAC in Windows servers and the solution is reboot the affected node.

After rebooting the affected node, we did not see any issues and all the components came up cleanly.

11gR2 RAC Database Fails to Start With Error ORA-00132: syntax error or unresolved network name ':1521'

We were seeing below error when trying to start an instance.

ORA-00119: invalid specification for system parameter REMOTE_LISTENER
ORA-00132: syntax error or unresolved network name '<scanname>:1521'

After doing some research I found that an entry of SQLNET.ORA was not properly set.

The location of SQLNET.ORA is <DB Home>/network/admin.

NAMES.DIRECTORY_PATH entry of SQLNET.ORA was wrong. 

It had the entry "names.directory_path = (TNSNAMES)" instead of "names.directory_path = (TNSNAMES,EZCONNECT)"

So, I made the change as "names.directory_path = (TNSNAMES,EZCONNECT)"

After making the above change I was able to start the instance of RAC without any errors.

Oracle RAC Installation error: root.sh failed when installing 11gR2 Grid Infrastructure

While executing root.sh we are seeing below error
root@rac1:>/u01/app/11.2.0.3/grid/root.sh

Running Oracle 11g root script...The following environment variables are set as:    ORACLE_OWNER= grid    
ORACLE_HOME=  /u01/app/11.2.0.3/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.
Entries will be added to the /etc/oratab file as needed by Database Configuration Assistant when a database is created.
Finished running generic part of root script.
Now product-specific root actions will be performed.Using configuration parameter file: /u01/app/11.2.0.3/grid/crs/install/crsconfig_params/u01/app/11.2.0.3/grid/crs/install/crsconfig_lib.pm line 6763./u01/app/11.2.0.3/grid/perl/bin/perl -I/u01/app/11.2.0/grid/perl/lib -I/u01/app/11.2.0/grid/crs/install /u01/app/11.2.0.3/grid/crs/install/rootcrs.pl execution failed

Checking cfgtoollogs file shows $ORACLE_HOME/cfgtoollogs/crsconfig/rootcrs_rac1.log

root@rac1:>

cd /u01/app/11.2.0.3/grid/cfgtoollogs/crsconfig

tail -10 rootcrs_rac1.log

2011-07-21 16:02:26: Executing /u01/app/11.2.0.3/grid/bin/crsctl create scr oracle

2011-07-21 16:02:26: Executing cmd:/u01/app/11.2.0.3/grid/bin/crsctl create scr oracle

2011-07-21 16:02:26: Command output:

>  CRS-4046: Invalid Oracle Clusterware configuration.

>  CRS-4000: Command Create failed, or completed with errors.

>End Command output

Solution:-

1st Node:-

[root rac1] /u01/app/11.2.0.3/grid/crs/install# ./rootcrs.pl -verbose -deconfig -force

Using configuration parameter file: ./crsconfig_params
****Unable to retrieve Oracle Clusterware home.
Start Oracle Clusterware stack and try again.
CRS-4046: Invalid Oracle Clusterware configuration.
CRS-4000: Command Stop failed, or completed with errors.
CRS-4046: Invalid Oracle Clusterware configuration.
CRS-4000: Command Stop failed, or completed with errors.
################################################################
# You must kill processes or reboot the system to properly #
# cleanup the processes started by Oracle clusterware          #
################################################################
ACFS-9313: No ADVM/ACFS installation detected.
Failure in execution (rc=-1, 256, No such file or directory) for command /etc/ohasd deinstall
Successfully deconfigured Oracle clusterware stack on this node
[root rac1] /u01/app/11.2.0.3/grid/crs/install# ps -ef|grep crs
    grid  8388648        1   0 05:14:05      -  2:39 /tmp/OraInstall2014-01-20_05-14-02AM/jdk/jre/bin/java -Doracle.installer.library_loc=/tmp/OraInstall2014-01-20_05-14-02AM/oui/lib/aix -Doracle.installer.oui_loc=/tmp/OraInstall2014-01-20_05-14-02AM/oui -Doracle.installer.bootstrap=TRUE -Doracle.installer.startup_location=/u01/app/Oracle_Software/grid/install -Doracle.installer.jre_loc=/tmp/OraInstall2014-01-20_05-14-02AM/jdk/jre -Doracle.installer.nlsEnabled="TRUE" -Doracle.installer.prereqConfigLoc=/tmp/OraInstall2014-01-20_05-14-02AM/prereq  -Doracle.installer.unixVersion=1 -Doracle.install.setup.workDir=/u01/app/Oracle_Software/grid -mx150m -Xverify:none -Dice.pilots.html4.defaultEncoding=ISO8859_1 -cp /tmp/OraInstall2014-01-20_05-14-02AM::/tmp/OraInstall2014-01-20_05-14-02AM/ext/jlib/OraPrereq.jar:/tmp/OraInstall2014-01-20_05-14-02AM/ext/jlib/OraPrereqChecks.jar:/tmp/OraInstall2014-01-20_05-14-02AM/ext/jlib/cvu.jar:/tmp/OraInstall2014-01-20_05-14-02AM/ext/jlib/installcommons_1.0.0b.jar:/tmp/OraInstall2014-01-20_05-14-02AM/ext/jlib/instcommon.jar:/tmp/OraInstall2014-01-20_05-14-02AM/ext/jlib/instcrs.jar:/tmp/OraInstall2014-01-20_05-14-02AM/ext/jlib/jsch.jar:/tmp/OraInstall2014-01-20_05-14-02AM/ext/jlib/ojdbc5.jar:/tmp/OraInstall2014-01-20_05-14-02AM/ext/jlib/orai18n-mapping.jar:/tmp/OraInstall2014-01-20_05-14-02AM/ext/jlib/orai18n-utility.jar:/tmp/OraInstall2014-01-20_05-14-02AM/ext/jlib/prov_fixup.jar:/tmp/OraInstall2014-01-20_05-14-02AM/ext/jlib/remoteinterfaces.jar:/tmp/OraInstall2014-01-20_05-14-02AM/ext/jlib/ssh.jar:/tmp/OraInstall2014-01-20_05-14-02AM/oui/jlib/OraInstaller.jar:/tmp/OraInstall2014-01-20_05-14-02AM/oui/jlib/oneclick.jar:/tmp/OraInstall2014-01-20_05-14-02AM/oui/jlib/xmlparserv2.jar:/tmp/OraInstall2014-01-20_05-14-02AM/oui/jlib/share.jar:/tmp/OraInstall2014-01-20_05-14-02AM/oui/jlib/OraInstallerNet.jar:/tmp/OraInstall2014-01-20_05-14-02AM/oui/jlib/emCfg.jar:/tmp/OraInstall2014-01-20_05-14-02AM/oui/jlib/emocmutl.jar:/tmp/OraInstall2014-01-20_05-14-02AM/oui/jlib/OraPrereq.jar:/tmp/OraInstall2014-01-20_05-14-02AM/oui/jlib/jsch.jar:/tmp/OraInstall2014-01-20_05-14-02AM/oui/jlib/ssh.jar:/tmp/OraInstall2014-01-20_05-14-02AM/oui/jlib/remoteinterfaces.jar:/tmp/OraInstall2014-01-20_05-14-02AM/oui/jlib/http_client.jar:/tmp/OraInstall2014-01-20_05-14-02AM/oui/jlib/OraSuiteInstaller.jar:../stage/Components/oracle.swd.opatch/11.2.0.3.0/1/DataFiles/jlib/opatch.jar:../stage/Components/oracle.swd.opatch/11.2.0.3.0/1/DataFiles/jlib/opatchactions.jar:../stage/Components/oracle.swd.opatch/11.2.0.3.0/1/DataFiles/jlib/opatchprereq.jar:../stage/Components/oracle.swd.opatch/11.2.0.3.0/1/DataFiles/jlib/opatchutil.jar:/tmp/OraInstall2014-01-20_05-14-02AM/oui/jlib/OraCheckPoint.jar:/tmp/OraInstall2014-01-20_05-14-02AM/oui/jlib/InstImages.jar:/tmp/OraInstall2014-01-20_05-14-02AM/oui/jlib/InstHelp.jar:/tmp/OraInstall2014-01-20_05-14-02AM/oui/jlib/InstHelp_de.jar:/tmp/OraInstall2014-01-20_05-14-02AM/oui/jlib/InstHelp_es.jar:/tmp/OraInstall2014-01-20_05-14-02AM/oui/jlib/InstHelp_fr.jar:/tmp/OraInstall2014-01-20_05-14-02AM/oui/jlib/InstHelp_it.jar:/tmp/OraInstall2014-01-20_05-14-02AM/oui/jlib/InstHelp_ja.jar:/tmp/OraInstall2014-01-20_05-14-02AM/oui/jlib/InstHelp_ko.jar:/tmp/OraInstall2014-01-20_05-14-02AM/oui/jlib/InstHelp_pt_BR.jar:/tmp/OraInstall2014-01-20_05-14-02AM/oui/jlib/InstHelp_zh_CN.jar:/tmp/OraInstall2014-01-20_05-14-02AM/oui/jlib/InstHelp_zh_TW.jar:/tmp/OraInstall2014-01-20_05-14-02AM/oui/jlib/oracle_ice.jar:/tmp/OraInstall2014-01-20_05-14-02AM/oui/jlib/help4.jar:/tmp/OraInstall2014-01-20_05-14-02AM/oui/jlib/help4-nls.jar:/tmp/OraInstall2014-01-20_05-14-02AM/oui/jlib/ewt3.jar:/tmp/OraInstall2014-01-20_05-14-02AM/oui/jlib/ewt3-swingaccess.jar:/tmp/OraInstall2014-01-20_05-14-02AM/oui/jlib/ewt3-nls.jar:/tmp/OraInstall2014-01-20_05-14-02AM/oui/jlib/swingaccess.jar::/tmp/OraInstall2014-01-20_05-14-02AM/oui/jlib/jewt4.jar:/tmp/OraInstall2014-01-20_05-14-02AM/oui/jlib/jewt4-nls.jar:/tmp/OraInstall2014-01-20_05-14-02AM/oui/jlib/orai18n-collation.jar:/tmp/OraInstall2014-01-20_05-14-02AM/oui/jlib/orai18n-mapping.jar:/tmp/OraInstall2014-01-20_05-14-02AM/oui/jlib/ojmisc.jar:/tmp/OraInstall2014-01-20_05-14-02AM/oui/jlib/xml.jar:/tmp/OraInstall2014-01-20_05-14-02AM/oui/jlib/srvm.jar:/tmp/OraInstall2014-01-20_05-14-02AM/oui/jlib/srvmasm.jar oracle.install.ivw.crs.driver.CRSInstaller -scratchPath /tmp/OraInstall2014-01-20_05-14-02AM -sourceLoc /u01/app/Oracle_Software/grid/install/../stage/products.xml -sourceType network -timestamp 2014-01-20_05-14-02AM -formCluster

Now retry running root.sh 

[root rac1] /u01/app/11.2.0.3/grid/crs/install# /u01/app/11.2.0.3/grid/root.sh
Performing root user operation for Oracle 11g

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /u01/app/11.2.0.3/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/app/11.2.0.3/grid/crs/install/crsconfig_params
User ignored Prerequisites during installation
User grid is missing the following capabilities required to run CSSD in realtime:
  CAP_NUMA_ATTACH,CAP_BYPASS_RAC_VMM,CAP_PROPAGATE
To add the required capabilities, please run:
   /usr/bin/chuser capabilities=CAP_NUMA_ATTACH,CAP_BYPASS_RAC_VMM,CAP_PROPAGATE grid
CSS cannot be run in realtime mode at /u01/app/11.2.0.3/grid/crs/install/crsconfig_lib.pm line 11423.
/u01/app/11.2.0.3/grid/perl/bin/perl -I/u01/app/11.2.0.3/grid/perl/lib -I/u01/app/11.2.0.3/grid/crs/install /u01/app/11.2.0.3/grid/crs/install/rootcrs.pl execution failed

again thrown the error? execute below command

[root rac1] /u01/app/11.2.0.3/grid/crs/install#  /usr/bin/chuser capabilities=CAP_NUMA_ATTACH,CAP_BYPASS_RAC_VMM,CAP_PROPAGATE grid

Now retry executing root.sh

[root rac1] /u01/app/11.2.0.3/grid/crs/install# /u01/app/11.2.0.3/grid/root.sh

Performing root user operation for Oracle 11g

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /u01/app/11.2.0.3/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/app/11.2.0.3/grid/crs/install/crsconfig_params
User ignored Prerequisites during installation
User grid has the required capabilities to run CSSD in realtime mode
OLR initialization - successful
  root wallet
  root wallet cert
  root cert export
  peer wallet
  profile reader wallet
  pa wallet
  peer wallet keys
  pa wallet keys
  peer cert request
  pa cert request
  peer cert
  pa cert
  peer root cert TP
  profile reader root cert TP
  pa root cert TP
  peer pa cert TP
  pa peer cert TP
  profile reader pa cert TP
  profile reader peer cert TP
  peer user cert
  pa user cert
Adding Clusterware entries to inittab
CRS-2672: Attempting to start 'ora.mdnsd' on 'rac1'
CRS-2676: Start of 'ora.mdnsd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rac1'
CRS-2676: Start of 'ora.gpnpd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac1'
CRS-2672: Attempting to start 'ora.gipcd' on 'rac1'
CRS-2676: Start of 'ora.gipcd' on 'rac1' succeeded
CRS-2676: Start of 'ora.cssdmonitor' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rac1'
CRS-2672: Attempting to start 'ora.diskmon' on 'rac1'
CRS-2676: Start of 'ora.diskmon' on 'rac1' succeeded
CRS-2676: Start of 'ora.cssd' on 'rac1' succeeded

ASM created and started successfully.

Disk Group OCR_VOTE created successfully.

clscfg: -install mode specified
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'system'..
Operation successful.
CRS-4256: Updating the profile
Successful addition of voting disk 8ee5a0b0f1474fcbbf3127724549a1cd.
Successful addition of voting disk 4f89d99d73594fecbfcb12a92c1c3c7e.
Successful addition of voting disk 11855ece9be44fe9bfe031dd507f163b.
Successfully replaced voting disk group with +OCR_VOTE.
CRS-4256: Updating the profile
CRS-4266: Voting file(s) successfully replaced
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   8ee5a0b0f1474fcbbf3127724549a1cd (/dev/rhdisk4) [OCR_VOTE]
 2. ONLINE   4f89d99d73594fecbfcb12a92c1c3c7e (/dev/rhdisk5) [OCR_VOTE]
 3. ONLINE   11855ece9be44fe9bfe031dd507f163b (/dev/rhdisk6) [OCR_VOTE]
Located 3 voting disk(s).
CRS-2672: Attempting to start 'ora.asm' on 'rac1'
CRS-2676: Start of 'ora.asm' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.OCR_VOTE.dg' on 'rac1'
CRS-2676: Start of 'ora.OCR_VOTE.dg' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.registry.acfs' on 'rac1'
CRS-2676: Start of 'ora.registry.acfs' on 'rac1' succeeded
Configure Oracle Grid Infrastructure for a Cluster ... succeeded



2nd Node:-

[root rac2] /u01/app/11.2.0.3/grid/crs/install# ./rootcrs.pl -verbose -deconfig -force
Using configuration parameter file: ./crsconfig_params
****Unable to retrieve Oracle Clusterware home.
Start Oracle Clusterware stack and try again.
CRS-4047: No Oracle Clusterware components configured.
CRS-4000: Command Stop failed, or completed with errors.
CRS-4047: No Oracle Clusterware components configured.
CRS-4000: Command Stop failed, or completed with errors.
################################################################
# You must kill processes or reboot the system to properly #
# cleanup the processes started by Oracle clusterware          #
################################################################
ACFS-9313: No ADVM/ACFS installation detected.
Either /etc/oracle/olr.loc does not exist or is not readable
Make sure the file exists and it has read and execute access
Either /etc/oracle/olr.loc does not exist or is not readable
Make sure the file exists and it has read and execute access
Failure in execution (rc=-1, 256, No such file or directory) for command /etc/ohasd deinstall
Successfully deconfigured Oracle clusterware stack on this node

Retry the execution of root.sh

[root rac2] /u01/app/11.2.0.3/grid/crs/install# /u01/app/11.2.0.3/grid/root.sh
Performing root user operation for Oracle 11g

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /u01/app/11.2.0.3/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/app/11.2.0.3/grid/crs/install/crsconfig_params
User ignored Prerequisites during installation
User grid is missing the following capabilities required to run CSSD in realtime:
  CAP_NUMA_ATTACH,CAP_BYPASS_RAC_VMM,CAP_PROPAGATE
To add the required capabilities, please run:
   /usr/bin/chuser capabilities=CAP_NUMA_ATTACH,CAP_BYPASS_RAC_VMM,CAP_PROPAGATE grid
CSS cannot be run in realtime mode at /u01/app/11.2.0.3/grid/crs/install/crsconfig_lib.pm line 11423.
/u01/app/11.2.0.3/grid/perl/bin/perl -I/u01/app/11.2.0.3/grid/perl/lib -I/u01/app/11.2.0.3/grid/crs/install /u01/app/11.2.0.3/grid/crs/install/rootcrs.pl execution failed


Again thrown the error? No problem execute below step

[root rac2] /u01/app/11.2.0.3/grid/crs/install# /usr/bin/chuser capabilities=CAP_NUMA_ATTACH,CAP_BYPASS_RAC_VMM,CAP_PROPAGATE grid

Now retry the execution of root.sh

[root rac2] /u01/app/11.2.0.3/grid/crs/install# /u01/app/11.2.0.3/grid/root.sh
Performing root user operation for Oracle 11g

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /u01/app/11.2.0.3/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:

The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/app/11.2.0.3/grid/crs/install/crsconfig_params
User ignored Prerequisites during installation
User grid has the required capabilities to run CSSD in realtime mode
OLR initialization - successful
Adding Clusterware entries to inittab
CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node rac1, number 1, and is terminating
An active cluster was found during exclusive startup, restarting to join the cluster
Configure Oracle Grid Infrastructure for a Cluster ... succeeded

Oracle RAC ASM Error: Device checks for ASM failed during 11.2.0.3 Grid Infrastructure setup

I had seen "Device checks for ASM failed" warning while I was installing 11.2.0.3 Grid Infrastructure (GI) setup in AIX servers.

Actually I uninstalled an earlier setup of GI and trying to re-install it for a test purpose. So, I thought "Device checks for ASM failed" must be because of the improper cleanup of old GI and it's ASM disks.

rhdisk4, rhdisk5, rhdisk6, rhdisk7 and rhdisk7 are the shared storage I used earlier for the OCR and DATA disk groups.

I cancelled the installation of GI; cleaned up the disks and gave the proper permissions as shown in below steps.

Execute the steps as root user:-

1st Node (rac1):-

#cd /dev

[root rac1] /dev# chdev -l hdisk4 -a pv=clear
hdisk4 changed
[root rac1] /dev# chdev -l hdisk5 -a pv=clear
hdisk5 changed
[root rac1] /dev# chdev -l hdisk6 -a pv=clear
hdisk6 changed
[root rac1] /dev# chdev -l hdisk7 -a pv=clear
hdisk7 changed
[root rac1] /dev# chdev -l hdisk8 -a pv=clear
hdisk8 changed
[root rac1] /dev# chmod 660 rhdisk4
[root rac1] /dev# chmod 660 rhdisk5
[root rac1] /dev# chmod 660 rhdisk6
[root rac1] /dev# chmod 660 rhdisk7
[root rac1] /dev# chmod 660 rhdisk8

[root rac1] /dev# chown grid:asmadmin rhdisk4
[root rac1] /dev# chown grid:asmadmin rhdisk5
[root rac1] /dev# chown grid:asmadmin rhdisk6
[root rac1] /dev# chown grid:asmadmin rhdisk7
[root rac1] /dev# chown grid:asmadmin rhdisk8


2nd Node (rac2):-

#cd /dev

[root rac2] /dev# chdev -l hdisk4 -a pv=clear
hdisk4 changed
[root rac2] /dev# chdev -l hdisk5 -a pv=clear
hdisk5 changed
[root rac2] /dev# chdev -l hdisk6 -a pv=clear
hdisk6 changed
[root rac2] /dev# chdev -l hdisk7 -a pv=clear
hdisk7 changed
[root rac2] /dev# chdev -l hdisk8 -a pv=clear
hdisk8 changed
[root rac2] /dev# chmod 660 rhdisk4
[root rac2] /dev# chmod 660 rhdisk5
[root rac2] /dev# chmod 660 rhdisk6
[root rac2] /dev# chmod 660 rhdisk7
[root rac2] /dev# chmod 660 rhdisk8

[root rac2] /dev# chown grid:asmadmin rhdisk4
[root rac2] /dev# chown grid:asmadmin rhdisk5
[root rac2] /dev# chown grid:asmadmin rhdisk6
[root rac2] /dev# chown grid:asmadmin rhdisk7

[root rac2] /dev# chown grid:asmadmin rhdisk8

Now restart the GI setup, you will not see the error.

Oracle RAC Error: PRVF-4007 : User equivalence check failed for user "grid"

I setup the SSH user equivalence between 2 RAC nodes and was able to ssh without password to other nodes. But still my runcluvfy.sh was failing with below message:-

./runcluvfy.sh stage -pre crsinst -n rac1,rac2

Performing pre-checks for cluster services setup

Checking node reachability...
Node reachability check passed from node "rac1"

Checking user equivalence...
PRVF-4007 : User equivalence check failed for user "grid"
Check failed on nodes:
rac1

WARNING:
User equivalence is not set for nodes:
rac1
Verification will proceed with nodes:
rac2
Version of exectask could not be retrieved from node "rac2"

ERROR:
Framework setup check failed on all the nodes
Verification cannot proceed

Pre-check for cluster services setup was unsuccessful on all the nodes.


Solution:-

It seems the problem is with the software. I downloaded new fresh software and re-ran runcluvfy.sh.

But again it had thrown me the error /usr/bin/scp is not found, but actually it was there.

So, I deleted /usr/bin/scp file from both the nodes and recreated the link from /opt/pware64/bin/scp to /usr/bin/scp and re-executed runcluvfy.sh.

It went well without any warnings now.