Wednesday, August 21, 2013

Oracle RACcheck - RAC Configuration Audit Tool

This blogpost provides details of the RACcheck (RAC Configuration Audit Tool) . I have tried to run this wonderful tool on my RAC database on my laptop and wanted to share the output of the tool received, The detailed output from this tool is self explanatory and requires no narration. This tool is brought by RAC assurance team of Oracle. The tool is designed to audit various important configuration settings within an RAC System. The tool audits configuration settings within the following categories:

1. OS kernel parameters
2. OS packages
3. Many other OS configuration settings important to RAC.
4. CRS/Grid Infrastructure
5. RDBMS
6. ASM
7. Database parameters
8. Many other database configuration settings important to RAC.
9. 11.2.0.3 Upgrade Readiness assessment

The scope of the RACcheck health assessment tool is RAC databases servers, Grid Infrastructure, RAC databases, hardware, Operating System and RAC software. Starting with RACcheck 2.2.0, RACcheck functionality has been extended to Oracle Single Instance Databases, Oracle Restart Systems as well as RAC One node configurations.


ajithpathiyil1:/home/oracle/RACCHECK[RAC2]$ ./raccheck

CRS stack is running and CRS_HOME is not set. Do you want to set CRS_HOME to /u01/grid/oracle/product/11.2.0/grid_1?[y/n][y]


Checking ssh user equivalency settings on all nodes in cluster


Node ajithpathiyil2 is configured for ssh user equivalency for oracle user



Searching for running databases . . . . .


.

List of running databases registered in OCR
1. RAC
2. None of above

Select databases from list for checking best practices. For multiple databases, select 1 for All or comma separated number like 1,2 etc [1-2][1].

. .


Checking Status of Oracle Software Stack - Clusterware, ASM, RDBMS


. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

-------------------------------------------------------------------------------------------------------
                                                 Oracle Stack Status
-------------------------------------------------------------------------------------------------------
Host Name  CRS Installed  ASM HOME       RDBMS Installed  CRS UP    ASM UP    RDBMS UP  DB Instance Name
-------------------------------------------------------------------------------------------------------
ajithpathiyil1 Yes             Yes             Yes             Yes        Yes      Yes      RAC2
ajithpathiyil2 Yes             Yes             Yes             Yes        Yes      Yes      RAC1
-------------------------------------------------------------------------------------------------------

74 of the included audit checks require root privileged data collection . If sudo is not configured or the root password is not available, audit checks which  require root privileged data collection can be skipped.



1. Enter 1 if you will enter root password for each  host when prompted


2. Enter 2 if you have sudo configured for oracle user to execute root_raccheck.sh script


3. Enter 3 to skip the root privileged collections


4. Enter 4 to exit and work with the SA to configure sudo  or to arrange for root access and run the tool later.


Please indicate your selection from one of the above options[1-4][1]:- 1


*** Checking Best Practice Recommendations (PASS/WARNING/FAIL) ***




Log file for collections and audit checks are at

/home/oracle/RACCHECK/raccheck_082113_224408/raccheck.log

Running raccheck in serial mode because expect(/usr/bin/expect) is not available to supply root passwords on remote nodes


NOTICE:  Installing the expect utility (/usr/bin/expect) will allow raccheck to gather root passwords at the beginning of the process and execute raccheck on all nodes in parallel speeding up the entire process. For more info - http://www.nist.gov/el/msid/expect.cfm.  Expect is available for all major platforms.  See User Guide for more details.


=============================================================

                    Node name - ajithpathiyil1
=============================================================
Collecting - ASM DIsk I/O stats
Collecting - ASM Disk Groups
Collecting - ASM Diskgroup Attributes
Collecting - ASM disk partnership imbalance
Collecting - ASM diskgroup attributes
Collecting - ASM initialization parameters
Collecting - Active sessions load balance for RAC database
Collecting - Archived Destination Status for RAC database
Collecting - Cluster Interconnect Config for RAC database
Collecting - Database Archive Destinations for RAC database
Collecting - Database Files for RAC database
Collecting - Database Instance Settings for RAC database
Collecting - Database Parameters for RAC database
Collecting - Database Properties for RAC database
Collecting - Database Registry for RAC database
Collecting - Database Sequences for RAC database
Collecting - Database Undocumented Parameters for RAC database
Collecting - Database Workload Services for RAC database
Collecting - Dataguard Status for RAC database
Collecting - Files not opened by ASM
Collecting - Log Sequence Numbers for RAC database
Collecting - Percentage of asm disk  Imbalance
Collecting - Process for shipping Redo to standby for RAC database
Collecting - Redo Log information for RAC database
Collecting - Standby redo log creation status before switchover for RAC database
Collecting - CPU Information
Collecting - CRS active version
Collecting - CRS oifcfg
Collecting - CRS software version
Collecting - CSS Reboot time
Collecting - CSS disktimout
Collecting - Cluster interconnect (clusterware)
Collecting - Clusterware OCR healthcheck
Collecting - Clusterware Resource Status
Collecting - Huge pages configuration
Collecting - Kernel parameters
Collecting - Linux module config.
Collecting - Maximum number of semaphore sets on system
Collecting - Maximum number of semaphores on system
Collecting - Maximum number of semaphores per semaphore set
Collecting - Memory Information
Collecting - OS Packages
Collecting - Operating system release information and kernel version
Collecting - Oracle Executable Attributes
Collecting - Patches for Grid Infrastructure
Collecting - Patches for RDBMS Home
Collecting - Shared memory segments
Collecting - Table of file system defaults
Collecting - Voting disks (clusterware)
Collecting - number of semaphore operations per semop system call
Preparing to run root privileged commands  ajithpathiyil1.  Please enter root password when prompted.
root@ajithpathiyil1's password:
Collecting - ACFS and ASM driver version comparison [ACFS]
Collecting - CRS user time zone check
Collecting - Custom rc init scripts (rc.local)
Collecting - Generic ACFS health [ACFS]
Collecting - Grid Infastructure user shell limits configuration
Collecting - Health of the mounted ACFS file systems [ACFS]
Collecting - Health of unmounted ACFS file systems [ACFS]
Collecting - Interconnect interface config
Collecting - Network interface stats
Collecting - Number of RDBMS LMS running in real time
Collecting - OCFS2 disks
Collecting - Root Open File Limit
Collecting - Verify ioctl to advm [ACFS]
Collecting - Volume list for unmount ACFS file system [ACFS]
Collecting - ocsf status
Collecting - root time zone check


Data collections completed. Checking best practices on ajithpathiyil1.

--------------------------------------------------------------------------------------


 INFO =>    CSS disktimeout is not set to the default value of 200

 WARNING => OCR is NOT being backed up daily
 INFO =>    $CRS_HOME/log/hostname/client directory has too many older log files.
 INFO =>    user_dump_dest has trace files older than 30 days for RAC
 INFO =>    background_dump_dest has files older than 30 days for RAC
 INFO =>    At some times checkpoints are not being completed for RAC
 INFO =>    audit_file_dest has audit files older than 30 days for RAC
 WARNING => /tmp is NOT on a dedicated filesystem
 WARNING => Open file limit for root user (ulimit -n) is NOT >= 65536 or unlimited
 INFO =>    CSS misscount is not set to the default value of 30
 WARNING => pam_limits NOT configured properly for shell limits
 INFO =>    Number of SCAN listeners is NOT equal to the recommended number of 3.
 WARNING => NIC bonding is not configured for interconnect
 WARNING => NIC bonding is NOT configured for public network (VIP)
 WARNING => OSWatcher is not running as is recommended.
 INFO =>    Jumbo frames (MTU >= 8192) are not configured for interconnect
 WARNING => NTP is not running with correct setting
 FAIL =>    Database parameter DB_BLOCK_CHECKSUM is NOT set to recommended value on RAC2 instance
 FAIL =>    Database parameter DB_LOST_WRITE_PROTECT is NOT set to recommended value on RAC2 instance
 WARNING => Database parameter DB_BLOCK_CHECKING on PRIMARY is NOT set to the recommended value. for RAC
 FAIL =>    Flashback on PRIMARY is not configured for RAC
 INFO =>    Operational Best Practices
 INFO =>    Consolidation Database Practices
 INFO =>    Computer failure prevention best practices
 INFO =>    Data corruption prevention best practices
 INFO =>    Logical corruption prevention best practices
 INFO =>    Database/Cluster/Site failure prevention best practices
 INFO =>    Client failover operational best practices
 WARNING => fast_start_mttr_target has NOT been changed from default on RAC2 instance

 INFO =>    IMPORTANT: Oracle Database SCN Headroom. See Details for More Info /u01/app/oracle/product/11.2.0/db_1

 INFO =>    Information about hanganalyze and systemstate dump
 FAIL =>    Your Configuration is NOT in Compliance with Oracle Security Alert for CVE-2012-1675 /u01/grid/oracle/product/11.2.0/grid_1
 FAIL =>    Your Configuration is NOT in Compliance with Oracle Security Alert for CVE-2012-1675 /u01/app/oracle/product/11.2.0/db_1
 INFO =>    Database failure prevention best practices
 WARNING => Database Archivelog Mode should be set to ARCHIVELOG for RAC
 FAIL =>    Primary database is NOT protected with Data Guard (standby database) for real-time data protection and availability for RAC
 WARNING => ASM memory_target is < recommended value
 INFO =>    Parallel Execution Health-Checks and Diagnostics Reports for RAC


Best Practice checking completed.Checking recommended patches on ajithpathiyil1.

---------------------------------------------------------------------------------


Collecting patch inventory on  CRS HOME /u01/grid/oracle/product/11.2.0/grid_1

Collecting patch inventory on ORACLE_HOME /u01/app/oracle/product/11.2.0/db_1
---------------------------------------------------------------------------------
1 Recommended CRS patches for 112010 from /u01/grid/oracle/product/11.2.0/grid_1 on ajithpathiyil1
---------------------------------------------------------------------------------
Patch#   CRS  ASM    RDBMS RDBMS_HOME                              Patch-Description
---------------------------------------------------------------------------------
9655006  yes          no  /u01/app/oracle/product/11.2.0/db_1GI PSU 11.2.0.1.2 (INCLUDES DATABASE PS
---------------------------------------------------------------------------------


---------------------------------------------------------------------------------

26 Recommended RDBMS patches for 112010 from /u01/app/oracle/product/11.2.0/db_1 on ajithpathiyil1
---------------------------------------------------------------------------------
Patch#   RDBMS    ASM     type                Patch-Description
---------------------------------------------------------------------------------
9218789   6/26           N-APPLY             N-APPLY BUNDLE PATCH FOR RDBMS 11.2.0.1
---------------------------------------------------------------------------------
---------------------------------------------------------------------------------
---------------------------------------------------------------------------------


---------------------------------------------------------------------------------

              Clusterware patches summary report
---------------------------------------------------------------------------------
Total patches  Applied on CRS Applied on RDBMS Applied on ASM
---------------------------------------------------------------------------------
1              1              0                0
---------------------------------------------------------------------------------


---------------------------------------------------------------------------------

              RDBMS homes patches summary report
---------------------------------------------------------------------------------
Total patches  Applied on RDBMS Applied on ASM ORACLE_HOME
---------------------------------------------------------------------------------
 26             6              0                /u01/app/oracle/product/11.2.0/db_1
---------------------------------------------------------------------------------



=============================================================

                    Node name - ajithpathiyil2
=============================================================
Collecting - CPU Information
Collecting - CRS active version
Collecting - CRS oifcfg
Collecting - CRS software version
Collecting - Cluster interconnect (clusterware)
Collecting - Huge pages configuration
Collecting - Kernel parameters
Collecting - Linux module config.
Collecting - Maximum number of semaphore sets on system
Collecting - Maximum number of semaphores on system
Collecting - Maximum number of semaphores per semaphore set
Collecting - Memory Information
Collecting - OS Packages
Collecting - Operating system release information and kernel version
Collecting - Oracle Executable Attributes
Collecting - Patches for Grid Infrastructure
Collecting - Patches for RDBMS Home
Collecting - Shared memory segments
Collecting - Table of file system defaults
Collecting - number of semaphore operations per semop system call
Preparing to run root privileged commands  ajithpathiyil2.  Please enter root password when prompted.
root@ajithpathiyil2's password:
Collecting - ACFS and ASM driver version comparison [ACFS]
Collecting - CRS user time zone check
Collecting - Generic ACFS health [ACFS]
Collecting - Grid Infastructure user shell limits configuration
Collecting - Health of the mounted ACFS file systems [ACFS]
Collecting - Health of unmounted ACFS file systems [ACFS]
Collecting - Interconnect interface config
Collecting - Network interface stats
Collecting - Number of RDBMS LMS running in real time
Collecting - OCFS2 disks
Collecting - Root Open File Limit
Collecting - Verify ioctl to advm [ACFS]
Collecting - Volume list for unmount ACFS file system [ACFS]
Collecting - ocsf status
Collecting - root time zone check


Data collections completed. Checking best practices on ajithpathiyil2.

--------------------------------------------------------------------------------------


 INFO =>    $CRS_HOME/log/hostname/client directory has too many older log files.

 INFO =>    user_dump_dest has trace files older than 30 days for RAC
 INFO =>    background_dump_dest has files older than 30 days for RAC
 INFO =>    audit_file_dest has audit files older than 30 days for RAC
 WARNING => /tmp is NOT on a dedicated filesystem
 WARNING => Open file limit for root user (ulimit -n) is NOT >= 65536 or unlimited
 WARNING => pam_limits NOT configured properly for shell limits
 INFO =>    Number of SCAN listeners is NOT equal to the recommended number of 3.
 WARNING => NIC bonding is not configured for interconnect
 WARNING => NIC bonding is NOT configured for public network (VIP)
 WARNING => OSWatcher is not running as is recommended.
 INFO =>    Jumbo frames (MTU >= 8192) are not configured for interconnect
 WARNING => NTP is not running with correct setting
 FAIL =>    Database parameter DB_BLOCK_CHECKSUM is NOT set to recommended value on RAC1 instance
 FAIL =>    Database parameter DB_LOST_WRITE_PROTECT is NOT set to recommended value on RAC1 instance
 WARNING => Database parameter DB_BLOCK_CHECKING on PRIMARY is NOT set to the recommended value. for RAC
 WARNING => fast_start_mttr_target has NOT been changed from default on RAC1 instance

 INFO =>    IMPORTANT: Oracle Database SCN Headroom. See Details for More Info /u01/app/oracle/product/11.2.0/db_1

 FAIL =>    Your Configuration is NOT in Compliance with Oracle Security Alert for CVE-2012-1675 /u01/grid/oracle/product/11.2.0/grid_1
 FAIL =>    Your Configuration is NOT in Compliance with Oracle Security Alert for CVE-2012-1675 /u01/app/oracle/product/11.2.0/db_1
 WARNING => ASM memory_target is < recommended value


Best Practice checking completed.Checking recommended patches on ajithpathiyil2.

---------------------------------------------------------------------------------


Collecting patch inventory on  CRS HOME /u01/grid/oracle/product/11.2.0/grid_1

Collecting patch inventory on ORACLE_HOME /u01/app/oracle/product/11.2.0/db_1
---------------------------------------------------------------------------------
1 Recommended CRS patches for 112010 from /u01/grid/oracle/product/11.2.0/grid_1 on ajithpathiyil2
---------------------------------------------------------------------------------
Patch#   CRS  ASM    RDBMS RDBMS_HOME                              Patch-Description
---------------------------------------------------------------------------------
9655006  yes    no    no  /u01/app/oracle/product/11.2.0/db_1GI PSU 11.2.0.1.2 (INCLUDES DATABASE PS
---------------------------------------------------------------------------------


---------------------------------------------------------------------------------

26 Recommended RDBMS patches for 112010 from /u01/app/oracle/product/11.2.0/db_1 on ajithpathiyil2
---------------------------------------------------------------------------------
Patch#   RDBMS    ASM     type                Patch-Description
---------------------------------------------------------------------------------
9218789   6/26           N-APPLY             N-APPLY BUNDLE PATCH FOR RDBMS 11.2.0.1
---------------------------------------------------------------------------------
---------------------------------------------------------------------------------
---------------------------------------------------------------------------------


---------------------------------------------------------------------------------

              Clusterware patches summary report
---------------------------------------------------------------------------------
Total patches  Applied on CRS Applied on RDBMS Applied on ASM
---------------------------------------------------------------------------------
1              1              0                0
---------------------------------------------------------------------------------


---------------------------------------------------------------------------------

              RDBMS homes patches summary report
---------------------------------------------------------------------------------
Total patches  Applied on RDBMS Applied on ASM ORACLE_HOME
---------------------------------------------------------------------------------
 26             6              0                /u01/app/oracle/product/11.2.0/db_1
---------------------------------------------------------------------------------





---------------------------------------------------------------------------------

                      CLUSTERWIDE CHECKS
---------------------------------------------------------------------------------
---------------------------------------------------------------------------------

Detailed report (html) - /home/oracle/RACCHECK/raccheck_ajithpathiyil1_RAC_082113_224408/raccheck_ajithpathiyil1_RAC_082113_224408.html



UPLOAD(if required) - /home/oracle/RACCHECK/raccheck_ajithpathiyil1_RAC_082113_224408.zip





ajithpathiyil1:/home/oracle/RACCHECK[RAC2]$ ls -ltr

total 24936
-rw-r--r-- 1 oracle oinstall      304 Jun 18 03:09 UserGuide.txt
-rw-rw-r-- 1 oracle oinstall  4134812 Jun 18 03:09 rules.dat
-rw-rw-r-- 1 oracle oinstall 15540712 Jun 18 03:09 collections.dat
-rwxr-xr-x 1 oracle oinstall  1035340 Aug  9 22:27 raccheck
-rwxr-xr-x 1 oracle oinstall  3379414 Aug 21 22:24 raccheck.zip
-rw-r--r-- 1 oracle oinstall     2222 Aug 21 22:41 readme.txt
-rw-r--r-- 1 oracle oinstall     9603 Aug 21 22:51 sql_shared_cursor.sql
-rw-r--r-- 1 oracle oinstall     5243 Aug 21 22:51 pxhcdr.log
-rw-r--r-- 1 oracle oinstall   503808 Aug 21 22:52 pxhcdr_RAC_ajithpathiyil1_11.2.0.1.0_20130821_225131_main.html
drwxr-xr-x 6 oracle oinstall    57344 Aug 21 23:17 raccheck_ajithpathiyil1_RAC_082113_224408
-rw-r--r-- 1 oracle oinstall   794987 Aug 21 23:18 raccheck_ajithpathiyil1_RAC_082113_224408.zip

ajithpathiyil1:/home/oracle/RACCHECK[RAC2]$

The output zip file can be downloaded here
For the RACSIG webinar on "Troubleshooting and Diagnosing RAC and GI" click here

HAPPY LEARNING!