REFRESHING ORACLE TIMESTEN DATASTORE USING PRODUCTION BACKUPS

Today I was searching for some old document , But saw this document created by me to refresh the timesten database and thought of posting it, This might be helpful for somebody who is searching for timesten refresh procedure.

Note:- I do not remember anything about timesten database these days, this procedure was created by me atleast 4 years ago, When I first learned to timesten refresh.

Very Important Note:- I created this document and following this document one of my team members lost his job, Don't get scared.....It was not because the document was wrong, but the person refreshed production from dev environment instead of reverse.

(1) SOURCE DATASTORE(PROD)
    ======================
(1.1) Identify from where timesten binaries are running
      =================================================
ajithpathiyil1% ps -ef|grep -i timesten
timesten  1458  1455   0   Sep 26 ?           0:58 /export/MYTTINST/TimesTen/7.0.5_64/bin/timestensubd -verbose -userlog /export/CHE
timesten  1459  1455   0   Sep 26 ?           2:49 /export/MYTTINST/TimesTen/7.0.5_64/bin/timestensubd -verbose -userlog /export/CHE
timesten  1455     1   0   Sep 26 ?          33:06 /export/MYTTINST/TimesTen/7.0.5_64/bin/timestend -initfd 13
timesten  1460  1455   0   Sep 26 ?           0:57 /export/MYTTINST/TimesTen/7.0.5_64/bin/timestensubd -verbose -userlog /export/CHE
timesten  1461  1455   0   Sep 26 ?         587:39 /export/MYTTINST/TimesTen/7.0.5_64/bin/timestensubd -verbose -userlog /export/CHE
timesten  1462  1455   0   Sep 26 ?           1:17 /export/MYTTINST/TimesTen/7.0.5_64/bin/ttcserver -verbose -userlog /export/MYTTINST
timesten 22251 21439   0 03:19:13 pts/8       0:00 grep -i timesten
timesten  3566     1   0   Sep 30 ?           3:26 /export/client/app/oracle/product/10.2.0/agent10g/perl/bin/perl /export/client/
timesten 12511  1455   0   Oct 09 ?           9:56 /export/MYTTINST/TimesTen/7.0.5_64/bin/timestenrepd -verbose -userlog /export/CHE
timesten  3583  3566   0   Sep 30 ?          84:07 /export/client/app/oracle/product/10.2.0/agent10g/bin/emagent
timesten  9199  9189   0 03:15:05 ?           0:00 /bin/sh -c /bin/ksh
timesten 20383  9200   3 03:18:38 ?           0:36 gzip /home/edgprod/cheprod/logs/tt_backups/BACKUP_MYTTINSTTLNP1A_121010.0.bac /ho
timesten 21439 21430   0 03:18:59 pts/8       0:00 -csh
timesten 22250 21439   0 03:19:13 pts/8       0:00 ps -ef
timesten  9200  9199   0 03:15:05 ?           0:00 /bin/ksh
timesten 17518     1   0   Oct 08 ?           0:37 direct -SDCTT_AJITHPATHIYIL1
timesten 17551 17518   0   Oct 08 ?           0:00 directc -R3 -W4
ajithpathiyil1%

(1.2) Identify the binary location for the timesten source datastore
      ===============================================================
      In case of multiple datastores as in above case.identify the exact bin folder from where our source datastore is running(SOURCE) 

(1.3) Find the TTSTATUS
      =================
ajithpathiyil1% ./ttstatus
TimesTen status report as of Tue Oct 12 03:42:13 2010

Daemon pid 1455 port 44710 instance 7.0.5_64
TimesTen server pid 1462 started on port 44711
No TimesTen webserver running

------------------------------------------------------------------------
Data store /export/MYTTINSTTLNP1/ttdata/MYTTINSTTLNP1A
There are 119 connections to the data store
Data store is in shared mode
Shared Memory KEY 0x16005815 ID 1 (ISM LOCKED)
Type            PID     Context             Connection Name              ConnID
Process         24843   0x0000000109ace810  java                            108
Process         24843   0x0000000109b28940  java                            109
Process         24843   0x0000000109b82a70  java                            110
Process         24843   0x0000000109bdcba0  java                            111
Process         24843   0x0000000109c36cd0  java                            113
Process         470     0x0000000100a71400  java                              1
Process         470     0x0000000100b843c0  java                              2
Process         952     0x0000000100b4ee70  java                              3
Replication     12511   0x00000001002e8980  REPHOLD                           5
Replication     12511   0x00000001002f9640  REPLISTENER                       6
Replication     12511   0x000000010030a300  LOGFORCE                          7
Replication     12511   0x00000001003cac20  TRANSMITTER                       4
Replication     12511   0x00000001003db8e0  TRANSMITTER                     112
Replication     12511   0x00000001003fc5c0  TRANSMITTER                      10
Replication     12511   0x000000010053d9e0  RECEIVER                          8
Subdaemon       1461    0x000000010015bc40  Worker                         2042
Subdaemon       1461    0x00000001002525e0  Flusher                        2046
Subdaemon       1461    0x00000001002632a0  Checkpoint                     2047
Subdaemon       1461    0x0000000100273f60  Aging                          2043
Subdaemon       1461    0x00000001002c4dc0  HistGC                         2045
Subdaemon       1461    0x0000000100356000  Monitor                        2044
RAM residence policy: Always
Replication policy  : Always
Replication agent is running.
Cache agent policy  : Manual
------------------------------------------------------------------------
Access control enabled.
End of report
ajithpathiyil1%

(2) Identify backups for restoration in TARGET non-prod server
    ===========================================================

(2.1) For NA check the crontab -l for the backup script and its log file to know the backup location
(2.2) For EMEA/APAC check the OS script in the scheduled GRID jobs (Use the target datastore name to find the backup job scheduled for the PROD)

In this example the below OS script was identified from GRID
==============================================FIND BACKUP LOCATION FOR PROD TTDUMPS=====================================================
export DT=`date +'%%d%%m%%y`
export TT_INSTANCE=MYTTINSTTLNP1S
/export/MYTTINST/TimesTen/7.0.5_64/bin/ttbackup -dir /home/edgprod/cheprod/logs/tt_backups -fname BACKUP_MYTTINSTTLNP1S_$DT MYTTINSTTLNP1S
if [ "$?" -ne "0" ]; then
 echo "Backup Failed"
 exit 1
fi
gzip /home/edgprod/cheprod/logs/tt_backups/BACKUP_MYTTINSTTLNP1S_$DT*
if [ "$?" -ne "0" ]; then
 echo "GZip Failed"
 exit 1
fi

find /home/edgprod/cheprod/logs/tt_backups -name 'BACKUP_MYTTINSTTLNP1S*' -mtime +7 -exec rm -f {} \;

if [ "$?" -ne "0" ]; then
 echo "Find/Delete Failed"
 exit 1
fi
============================================================================================================================================


(3) SCP the latest backups to the target to any directory
==========================================================
Push to TARGET host
scp BACKUP_MYTTINSTTLNP1S_121010.0*  timesten@ajithpathiyil2.lab.com:/opt/timesten/REFRESH_BKP
(or)

Pull from TARGET host
scp e396939@ajithpathiyil1.lab.com:/home/edgprod/cheprod/logs/tt_backups/BACKUP_MYTTINSTTLNP1S_121010.0* .


(3) TARGET Datastore
    ================
timesten@MYTTINSTDEV:REFRESH_BKP:[24] cd /opt/timesten/7.0.5.64/solaris/TimesTen/tt70/bin/
timesten@MYTTINSTDEV:bin:[25] ./ttstatus
TimesTen status report as of Tue Oct 12 04:10:44 2010

Daemon pid 4188 port 17001 instance tt70
TimesTen server pid 4193 started on port 17003
No TimesTen webserver running

------------------------------------------------------------------------
Data store /MYTTINST_U1/ttds/UTTDS_CHEOLN5/ds/UTTDS_CHEOLN5
There are no connections to the data store
Replication policy  : Manual
Cache agent policy  : Manual
------------------------------------------------------------------------
Data store /MYTTINST_U1/ttds/UTT70_SUPPLN1/ds/UTT70_SUPPLN1
There are 30 connections to the data store
Data store is in shared mode
Shared Memory KEY 0x16003301 ID 16777266 (ISM LOCKED)
Type            PID     Context             Connection Name              ConnID
Process         12254   0x00000001006186e0  java                             12
Process         12254   0x0000000102d43760  java                              2
Process         12254   0x0000000102e88c60  java                              5
Process         12254   0x0000000103be6ff0  java                             23
Process         12254   0x0000000103c37d30  java                             24
Process         9561    0x00000001007565c0  java                              1
Subdaemon       12193   0x000000010015bc40  Worker                         2042
Subdaemon       12193   0x00000001001ce3f0  Flusher                        2047
Subdaemon       12193   0x00000001001df0b0  Checkpoint                     2043
Subdaemon       12193   0x0000000100252370  Aging                          2045
Subdaemon       12193   0x0000000100263030  HistGC                         2046
Subdaemon       12193   0x0000000100273cf0  Monitor                        2044
RAM residence policy: Always
Replication policy  : Manual
Cache agent policy  : Manual
------------------------------------------------------------------------
Access control enabled.
End of report
timesten@MYTTINSTDEV:bin:[26]


(3.1) Confirm all backup files have been received at TARGET
     ======================================================
timesten@MYTTINSTDEV:REFRESH_BKP:[28]  ls -ltr BACKUP_MYTTINSTTLNP1S_121010*gz|wc -l
      37

(3.2) Unzip or Uncompress the backup files if necessary
      ==================================================

(4) NOW START RESTORE FROM BACKUP USING THE BELOW COMMANDS
============================================================

./ttadmin -rampolicy manual UTTDS_CHPINTNY1 
/apps/timesten/11.2.1.6.1/TimesTen/tt1121/bin/ttdaemonadmin -stop
/apps/timesten/11.2.1.6.1/TimesTen/tt1121/bin/ttdaemonadmin -start
ttdestroy UTTDS_CHPINTNY1 
ttrestore -d . -f filename UTTDS_CHPINTNY1



/apps/timesten/11.2.1.6.1/TimesTen/tt1121/bin/ttadmin -rampolicy manual UTTDS_CHPINTNY1
/apps/timesten/11.2.1.6.1/TimesTen/tt1121/bin/ttadmin -ramunload UTTDS_CHPINTNY1
/apps/timesten/11.2.1.6.1/TimesTen/tt1121/bin/ttdestroy UTTDS_CHPINTNY1
/apps/timesten/11.2.1.6.1/TimesTen/tt1121/bin/ttrestore -fname temp1 -dir . UTTDS_CHPINTNY1


./ttisql UTTDS_CHPINTNY1
repschemes;
drop replication ; 
xladeletebookmark;
xladeletebookmark ;
===============================================================================

(4.1) Destroy the TARGET datastore
      =============================
/opt/timesten/7.0.5.64/solaris/TimesTen/tt70/bin/ttadmin -rampolicy manual UTT70_SUPPLN1 
/opt/timesten/7.0.5.64/solaris/TimesTen/tt70/bin/ttdestroy UTT70_SUPPLN1 

Note:- Using ttstatus command make sure no user connections are existing(Confirm with user if the datastore can be destroyed)

(4.2) Change to the directory that contains the backup files from PROD
      =================================================================
timesten@MYTTINSTDEV:REFRESH_BKP:[109] /opt/timesten/7.0.5.64/solaris/TimesTen/tt70/bin/ttadmin -rampolicy manual UTT70_SUPPLN1
timesten@MYTTINSTDEV:REFRESH_BKP:[110] /opt/timesten/7.0.5.64/solaris/TimesTen/tt70/bin/ttadmin -ramunload UTT70_SUPPLN1
timesten@MYTTINSTDEV:REFRESH_BKP:[111] /opt/timesten/7.0.5.64/solaris/TimesTen/tt70/bin/ttdestroy UTT70_SUPPLN1
timesten@MYTTINSTDEV:REFRESH_BKP:[112] /opt/timesten/7.0.5.64/solaris/TimesTen/tt70/bin/ttrestore -fname BACKUP_MYTTINSTTLNP1S_121010 -dir . UTT70_SUPPLN1
/opt/timesten/7.0.5.64/solaris/TimesTen/tt70/bin/ttisql UTT70_SUPPLN1
repschemes;


xladeletebookmark;

Command> repschemes;

0 replication schemes found.
Command> xladeletebookmark;

Note:-  Use "drop replication ;" 

0 bookmarks found.

Note:- Use "xladeletebookmark ;"

Command> exit
Disconnecting...
Done.



(4.3) Send the TTSTSTAUS to the user
      =================================
timesten@MYTTINSTDEV:REFRESH_BKP:[113] /opt/timesten/7.0.5.64/solaris/TimesTen/tt70/bin/ttstatus
TimesTen status report as of Tue Oct 12 06:23:05 2010

Daemon pid 4188 port 17001 instance tt70
TimesTen server pid 4193 started on port 17003
No TimesTen webserver running

------------------------------------------------------------------------
Data store /MYTTINST_U1/ttds/UTT70_SUPPLN1/ds/UTT70_SUPPLN1
There are 6 connections to the data store
Data store is in shared mode
Shared Memory KEY 0x04003303 ID 16777314 (ISM LOCKED)
Type            PID     Context             Connection Name              ConnID
Subdaemon       12193   0x000000010015bc40  Worker                         2042
Subdaemon       12193   0x00000001001ce3f0  Flusher                        2043
Subdaemon       12193   0x00000001001df0b0  Checkpoint                     2047
Subdaemon       12193   0x00000001003450c0  Aging                          2045
Subdaemon       12193   0x0000000100355d80  HistGC                         2046
Subdaemon       12193   0x0000000100396aa0  Monitor                        2044
RAM residence policy: Always
Replication policy  : Manual
Cache agent policy  : Manual
------------------------------------------------------------------------
Data store /MYTTINST_U1/ttds/UTTDS_CHEOLN5/ds/UTTDS_CHEOLN5
There are no connections to the data store
Replication policy  : Manual
Cache agent policy  : Manual
------------------------------------------------------------------------
Access control enabled.
End of report
timesten@MYTTINSTDEV:REFRESH_BKP:


Note :- If the TTSTATUS does not show the TARGET datastore use the below commands
        /opt/timesten/7.0.5.64/solaris/TimesTen/tt70/bin/ttadmin -rampolicy always UTT70_SUPPLN1
 /opt/timesten/7.0.5.64/solaris/TimesTen/tt70/bin/ttadmin -ramload UTT70_SUPPLN1


POSSIBLE ERRORS DURING REFRESH
##############################################################################################################################
timesten@MYTTINSTDEV:REFRESH_BKP:[65] /opt/timesten/7.0.5.64/solaris/TimesTen/tt70/bin/ttrestore -fname BACKUP_MYTTINSTTLNP1S_121010 -dir . UTT70_SUPPLN1
Restore started ...
Restore failed:
Error 12133: TT12133: Data store file already exists -- file "restore.c", lineno 961, procedure "doRestore"
timesten@MYTTINSTDEV:REFRESH_BKP:[66]

In this case 
===========

timesten@MYTTINSTDEV:timesten:[135] cd /var/TimesTen
timesten@MYTTINSTDEV:TimesTen:[136] ls -ltr
total 312
-rw-r--r--   1 timesten timesten     470 Jul 28  2008 snmp.ini
-rw-r--r--   1 timesten timesten     130 Jul 28  2008 sys.ttconnect.ini
lrwxrwxrwx   1 timesten timesten      23 Jul 30  2008 log -> /MYTTINST_U2/ttlog/server
-rw-r--r--   1 timesten timesten   18944 Sep 19  2008 TT70.tar
-rw-r--r--   1 timesten timesten  117760 Oct 24  2008 tt70-24Oct-prd.tar
-rw-r--r--   1 timesten timesten    3996 Jul  7 08:23 sys.odbc.ini
-rw-r--r--   1 timesten timesten      32 Jul  7 08:26 chris,dat
drwxr-xr-x   2 timesten timesten     512 Oct 12 06:09 tt70
timesten@MYTTINSTDEV:TimesTen:[137] vi sys.odbc.ini
"sys.odbc.ini" 189 lines, 3996 characters
[UTT70_SUPPLN1]
Description=Test TimesTen Data Store
Driver=/opt/timesten/7.0.5.64/solaris/TimesTen/tt70/lib/libtten.so
Authenticate=1
DataStore=/MYTTINST_U1/ttds/UTT70_SUPPLN1/ds/UTT70_SUPPLN1<<<<<<<<<<<<<<<---------------------- font="">
LogDir=/MYTTINST_U2/ttlog/UTT70_SUPPLN1/logs  <<<<<<<<<<<<<<<<<<<<<<<<<<<---------------------- 2="" a="" after="" be="" command="" directories="" empty="" font="" in="" make="" note="" of="" should="" step="" the="" ttdestroy="" which="">
TypeMode=1
DatabaseCharacterSet=US7ASCII
Temporary=0
# Originally 1 to allow for creation
AutoCreate=1
# In MB
LogFileSize=64
# In KB
LogBuffSize=65536
Logging=1
LogPurge=1
MemoryLock=4
Overwrite=0
PermSize=7168
Preallocate=0
SMPOptLevel=1
TempSize=128
DurableCommits=0
ExclAccess=0
Isolation=1
#IsoOverride=0
LockLevel=0
LockWait=120
MatchLogOpts=0
ThreadSafe=1
WaitForConnect=1
CkptFrequency=300
CkptLogVolume=0
Connections=256
PermWarnThreshold=80
TempWarnThreshold=80

Note:- Make sure the following directories get empty after using ttdestroy command in step(4.2)

timesten@MYTTINSTDEV:TimesTen:[139] cd /MYTTINST_U1/ttds/UTT70_SUPPLN1/ds/
timesten@MYTTINSTDEV:ds:[140] ls -ltr
total 23892436
-rw-rw-rw-   1 timesten timesten 6261323072 Oct 12 07:33 UTT70_SUPPLN1.ds0
-rw-rw-rw-   1 timesten timesten 6261325824 Oct 12 07:38 UTT70_SUPPLN1.ds1
timesten@MYTTINSTDEV:ds:[141] cd /MYTTINST_U2/ttlog/UTT70_SUPPLN1/logs
timesten@MYTTINSTDEV:logs:[142] ls -ltr
total 452560
-rw-rw-rw-   1 timesten timesten 67108864 Oct 12 06:09 UTT70_SUPPLN1.res0
-rw-rw-rw-   1 timesten timesten 67108864 Oct 12 06:09 UTT70_SUPPLN1.res1
-rw-rw-rw-   1 timesten timesten 67108864 Oct 12 06:09 UTT70_SUPPLN1.res2
-rw-rw-rw-   1 timesten timesten 29278208 Oct 12 07:38 UTT70_SUPPLN1.log10546
timesten@MYTTINSTDEV:logs:[143]


HAPPY LEARNING!

Creating Oracle EBS R12 MultiNode Lab With iSCSI Shared Storage - Part II

5) Openfiler – NAS/SAN Appliance Download/Install

5.1 Install Openfiler

This section provides a summary of the screens used to install the Openfiler software. For the purpose of this article, We opted to install Openfiler with all default options. The only manual change required was for configuring the local network settings.

Once the install has completed, the server will reboot to make sure all required components, services and drivers are started and recognized. After the reboot, any external hard drives (if connected) will be discovered by the Openfiler server.

Before installing the Openfiler software to the network storage server, you should have both NIC interfaces (cards) installed and any external hard drives connected and turned on (if you will be using external hard drives).

After downloading and burning the Openfiler ISO image (ISO file) to CD, insert the CD into the network storage server (openfiler1 in this example), power it on, and answer the installation screen prompts as noted below.

5.2 Boot Screen

The first screen is the Openfiler boot screen. At the boot: prompt, hit [Enter] to start the installation process.

5.3 Media Test

When asked to test the CD media, tab over to [Skip] and hit [Enter]. If there were any errors, the media burning software would have warned us. After several seconds, the installer should then detect the video card, monitor, and mouse. The installer then goes into GUI mode.

5.4 Welcome to Openfiler NSA

At the welcome screen, click [Next] to continue.

5.5 Keyboard Configuration

The next screen prompts you for the Keyboard settings. Make the appropriate selection for your configuration.

5.6 Disk Partitioning Setup

The next screen asks whether to perform disk partitioning using "Automatic Partitioning" or "Manual Partitioning with Disk Druid". Although the official Openfiler documentation suggests to use Manual Partitioning, We opted to use "Automatic Partitioning" given the simplicity of my example configuration.

Select [Automatically partition] and click [Next] continue.

5.7 Automatic Partitioning

If there were a previous installation of Linux on this machine, the next screen will ask if you want to "remove" or "keep" old partitions. Select the option to [Remove all partitions on this system]. For my example configuration, We selected ONLY the 500GB SATA internal hard drive [sda] for the operating system and Openfiler application installation. We de-selected the 73GB SCSI internal hard drive since this disk will be used exclusively in the next section to create a single "Volume Group" that will be used for all iSCSI based shared disk storage requirements for Oracle RAC 10g.

We also keep the checkbox [Review (and modify if needed) the partitions created] selected. Click [Next] to continue.

You will then be prompted with a dialog window asking if you really want to remove all partitions. Click [Yes] to acknowledge this warning.

5.8 Partitioning

The installer will then allow you to view (and modify if needed) the disk partitions it automatically chose for hard disks selected in the previous screen. In almost all cases, the installer will choose 100MB for /boot, an adequate amount of swap, and the rest going to the root (/) partition for that disk (or disks). In this example, We are satisfied with the installers recommended partitioning for /dev/sda.

The installer will also show any other internal hard disks it discovered. For my example configuration, the installer found the 73GB SCSI internal hard drive as /dev/sdb. For now, We will "Delete" any and all partitions on this drive (there was only one, /dev/sdb1). In the next section, we will create the required partition for this particular hard disk.

5.9 Network Configuration

We made sure to install all NIC interfaces (cards) in the network storage server before starting the Openfiler installation. This screen should have successfully detected each of the network devices.

First, make sure that each of the network devices are checked to [Active on boot]. The installer may choose to not activate eth1 by default.

Second, [Edit] both eth0 and eth1 as follows. You may choose to use different IP addresses for both eth0 and eth1 and that is OK. You must, however, configure eth1 (the storage network) to be on the same subnet you configured for eth1 on linux1 and linux2:

eth0:
- Check OFF the option to [Configure using DHCP]
- Leave the [Activate on boot] checked ON
- IP Address: 192.168.1.195
- Netmask: 255.255.255.0

eth1:
- Check OFF the option to [Configure using DHCP]
- Leave the [Activate on boot] checked ON
- IP Address: 192.168.2.195
- Netmask: 255.255.255.0

Continue by setting your hostname manually. We used a hostname of "openfiler1". Finish this dialog off by supplying your gateway and DNS servers.

Time Zone Selection

The next screen allows you to configure your time zone information. Make the appropriate selection for your location.

Set Root Password

Select a root password and click [Next] to continue.

About to Install

This screen is basically a confirmation screen. Click [Next] to start the installation.

Congratulations

And that's it. You have successfully installed Openfiler on the network storage server. The installer will eject the CD from the CD-ROM drive. Take out the CD and click [Reboot] to reboot the system.

If everything was successful after the reboot, you should now be presented with a text login screen and the URL(s) to use for administering the Openfiler server.

5.10 Modify /etc/hosts File on Openfiler Server

Although not mandatory, We typically copy the contents of the /etc/hosts file from one of the Oracle RAC nodes to the new Openfiler server. This allows convenient name resolution when testing the network for the cluster.

5.11 Configure iSCSI Volumes using Openfiler

Perform the following configuration tasks on the network storage server (openfiler1)!

Openfiler administration is performed using the Openfiler Storage Control Center — a browser based tool over an https connection on port 446. For example:

https://openfiler1.linux5.info:446/

From the Openfiler Storage Control Center home page, log in as an administrator. The default administration login credentials for Openfiler are:

• Username: openfiler
• Password: password

The first page the administrator sees is the [Status] / [System Information] screen.

To use Openfiler as an iSCSI storage server, we have to perform six major tasks; set up iSCSI services, configure network access, identify and partition the physical storage, create a new volume group, create all logical volumes, and finally, create new iSCSI targets for each of the logical volumes.

5.12 Services

To control services, we use the Openfiler Storage Control Center and navigate to [Services] / [Manage Services]:

Enable iSCSI Openfiler Service

To enable the iSCSI service, click on the 'Enable' link under the 'iSCSI target server' service name. After that, the 'iSCSI target server' status should change to 'Enabled'.

The ietd program implements the user level part of iSCSI Enterprise Target software for building an iSCSI storage system on Linux. With the iSCSI target enabled, we should be able to SSH into the Openfiler server and see the iscsi-target service running:

[root@openfiler1 ~]# service iscsi-target status

ietd (pid 3839) is running...

5.13 Network Access Configuration

The next step is to configure network access in Openfiler to identify both Oracle RAC nodes (linux1 and linux2) that will need to access the iSCSI volumes through the storage (private) network. Note that iSCSI volumes will be created later on in this section. Also note that this step does not actually grant the appropriate permissions to the iSCSI volumes required by both Oracle RAC nodes. That will be accomplished later in this section by updating the ACL for each new logical volume.

As in the previous section, configuring network access is accomplished using the Openfiler Storage Control Center by navigating to [System] / [Network Setup]. The "Network Access Configuration" section (at the bottom of the page) allows an administrator to setup networks and/or hosts that will be allowed to access resources exported by the Openfiler appliance. For the purpose of this article, we will want to add both Oracle RAC nodes individually rather than allowing the entire 192.168.2.0 network to have access to Openfiler resources.

When entering each of the Oracle RAC nodes, note that the 'Name' field is just a logical name used for reference only. As a convention when entering nodes, We can use the node name defined for that IP address. Next, when entering the actual node in the 'Network/Host' field, always use its IP address even though its host name may already be defined in your /etc/hosts file or DNS. Lastly, when entering actual hosts in our Class C network, use a subnet mask of 255.255.255.255.

It is important to remember that you will be entering the IP address of the private network (eth1) for each of the RAC nodes in the cluster.

The following image shows the results of adding both Oracle RAC nodes:

Figure 7: Configure Openfiler Network Access for Oracle RAC Nodes

5.14 Physical Storage

In this section, we will be creating the five iSCSI volumes to be used as shared storage by both of the Oracle RAC nodes in the cluster. This involves multiple steps that will be performed on the internal 73GB 15K SCSI hard disk connected to the Openfiler server.

Storage devices like internal IDE/SATA/SCSI/SAS disks, storage arrays, external USB drives, external FireWire drives, or ANY other storage can be connected to the Openfiler server and served to the clients. Once these devices are discovered at the OS level, Openfiler Storage Control Center can be used to set up and manage all of that storage.

In our case, we have a 73GB internal SCSI hard drive for our shared storage needs. On the Openfiler server this drive is seen as /dev/sdb (MAXTOR ATLAS15K2_73SCA). To see this and to start the process of creating our iSCSI volumes, navigate to [Volumes] / [Block Devices] from the Openfiler Storage Control Center:

Openfiler Physical Storage - Block Device Management

5.15 Partitioning the Physical Disk

The first step we will perform is to create a single primary partition on the /dev/sdb internal hard disk. By clicking on the /dev/sdb link, we are presented with the options to 'Edit' or 'Create' a partition. Since we will be creating a single primary partition that spans the entire disk, most of the options can be left to their default setting where the only modification would be to change the 'Partition Type' from 'Extended partition' to 'Physical volume'. Here are the values We specified to create the primary partition on /dev/sdb:

Mode: Primary
Partition Type: Physical volume
Starting Cylinder: 1
Ending Cylinder: 8924

The size now shows 68.36 GB. To accept that we click on the "Create" button. This results in a new partition (/dev/sdb1) on our internal hard disk:

Partition the Physical Volume

5.16 Volume Group Management

The next step is to create a Volume Group. We will be creating a single volume group named rac1 that contains the newly created primary partition.

From the Openfiler Storage Control Center, navigate to [Volumes] / [Volume Groups]. There we would see any existing volume groups, or none as in our case. Using the Volume Group Management screen, enter the name of the new volume group (rac1), click on the checkbox in front of /dev/sdb1 to select that partition, and finally click on the 'Add volume group' button.

After that we are presented with the list that now shows our newly created volume group named "rac1":

5.17 Logical Volumes

We can now create the five logical volumes in the newly created volume group (rac1).

From the Openfiler Storage Control Center, navigate to [Volumes] / [Add Volume]. There we will see the newly created volume group (rac1) along with its block storage statistics. Also available at the bottom of this screen is the option to create a new volume in the selected volume group - (Create a volume in "rac1"). Use this screen to create the following five logical (iSCSI) volumes. After creating each logical volume, the application will point you to the "Manage Volumes" screen. You will then need to click back to the "Add Volume" tab to create the next logical volume until all five iSCSI volumes are created:

iSCSI / Logical Volumes
Volume Name	Volume Description	Required Space (MB)	Filesystem Type
racdb-crs	racdb - Oracle Clusterware	2,048	iSCSI
racdb-asm1	racdb - ASM Volume 1	16,984	iSCSI
racdb-asm2	racdb - ASM Volume 2	16,984	iSCSI
racdb-asm3	racdb - ASM Volume 3	16,984	iSCSI
racdb-asm4	racdb - ASM Volume 4	16,984	iSCSI

In effect we have created five iSCSI disks that can now be presented to iSCSI clients (linux1 and linux2) on the network. The "Manage Volumes" screen should look as follows:

5.18 iSCSI Targets

At this point we have five iSCSI logical volumes. Before an iSCSI client can have access to them, however, an iSCSI target will need to be created for each of these five volumes. Each iSCSI logical volume will be mapped to a specific iSCSI target and the appropriate network access permissions to that target will be granted to both Oracle RAC nodes. For the purpose of this article, there will be a one-to-one mapping between an iSCSI logical volume and an iSCSI target.

There are three steps involved in creating and configuring an iSCSI target; create a unique Target IQN (basically, the universal name for the new iSCSI target), map one of the iSCSI logical volumes created in the previous section to the newly created iSCSI target, and finally, grant both of the Oracle RAC nodes access to the new iSCSI target. Please note that this process will need to be performed for each of the five iSCSI logical volumes created in the previous section.

For the purpose of this article, the following table lists the new iSCSI target names (the Target IQN) and which iSCSI logical volume it will be mapped to:

iSCSI Target / Logical Volume Mappings
Target IQN	iSCSI Volume Name	Volume Description
iqn.2006-01.com.openfiler:racdb.crs	racdb-crs	racdb - Oracle Clusterware
iqn.2006-01.com.openfiler:racdb.asm1	racdb-asm1	racdb - ASM Volume 1
iqn.2006-01.com.openfiler:racdb.asm2	racdb-asm2	racdb - ASM Volume 2
iqn.2006-01.com.openfiler:racdb.asm3	racdb-asm3	racdb - ASM Volume 3
iqn.2006-01.com.openfiler:racdb.asm4	racdb-asm4	racdb - ASM Volume 4

We are now ready to create the five new iSCSI targets - one for each of the iSCSI logical volumes. The example below illustrates the three steps required to create a new iSCSI target by creating the Oracle Clusterware / racdb-crs target (iqn.2006-01.com.openfiler:racdb.crs). This three step process will need to be repeated for each of the five new iSCSI targets listed in the table above.

5.19 Create New Target IQN

From the Openfiler Storage Control Center, navigate to [Volumes] / [iSCSI Targets]. Verify the grey sub-tab "Target Configuration" is selected. This page allows you to create a new iSCSI target. A default value is automatically generated for the name of the new iSCSI target (better known as the "Target IQN"). An example Target IQN is "iqn.2006-01.com.openfiler:tsn.ae4683b67fd3":

 Create New iSCSI Target : Default Target IQN

Prefer to replace the last segment of the default Target IQN with something more meaningful. For the first iSCSI target (Oracle Clusterware / racdb-crs), We will modify the default Target IQN by replacing the string "tsn.ae4683b67fd3" with "racdb.crs" as shown in Fig below: 

Create New iSCSI Target : Replace Default Target IQN

Once you are satisfied with the new Target IQN, click the "Add" button. This will create a new iSCSI target and then bring up a page that allows you to modify a number of settings for the new iSCSI target. For the purpose of this article, none of settings for the new iSCSI target need to be changed.

5.20 LUN Mapping

After creating the new iSCSI target, the next step is to map the appropriate iSCSI logical volumes to it. Under the "Target Configuration" sub-tab, verify the correct iSCSI target is selected in the section "Select iSCSI Target". If not, use the pull-down menu to select the correct iSCSI target and hit the "Change" button.

Next, click on the grey sub-tab named "LUN Mapping" (next to "Target Configuration" sub-tab). Locate the appropriate iSCSI logical volume (/dev/rac1/racdb-crs in this case) and click the "Map" button. You do not need to change any settings on this page. Your screen should look similar to Fig below after clicking the "Map" button for volume /dev/rac1/racdb-crs:

Create New iSCSI Target : Map LUN

5.21 Network ACL

Before an iSCSI client can have access to the newly created iSCSI target, it needs to be granted the appropriate permissions. A while back, we configured network access in Openfiler for two hosts (the Oracle RAC nodes). These are the two nodes that will need to access the new iSCSI targets through the storage (private) network. We now need to grant both of the Oracle RAC nodes access to the new iSCSI target.

Click on the grey sub-tab named "Network ACL" (next to "LUN Mapping" sub-tab). For the current iSCSI target, change the "Access" for both hosts from 'Deny' to 'Allow' and click the 'Update' button:

Create New iSCSI Target : Update Network ACL

Go back to the Create New Target IQN section and perform these three tasks for the remaining four iSCSI logical volumes while substituting the values found in the "iSCSI Target / Logical Volume Mappings" table .

5.22 Configure iSCSI Volumes on Oracle RAC Nodes

Configure the iSCSI initiator on both Oracle RAC nodes in the cluster! Creating partitions, however, should only be executed on one of nodes in the RAC cluster.

An iSCSI client can be any system (Linux, Unix, MS Windows, Apple Mac, etc.) for which iSCSI support (a driver) is available. In our case, the clients are two Linux servers, linux1 and linux2, running CentOS 5.3.

In this section we will be configuring the iSCSI software initiator on both of the Oracle RAC nodes. CentOS 5.3 includes the Open-iSCSI iSCSI software initiator which can be found in the iscsi-initiator-utils RPM. This is a change from previous versions of CentOS (4.x) which included the Linux iscsi-sfnet software driver developed as part of the Linux-iSCSI Project. All iSCSI management tasks like discovery and logins will use the command-line interface iscsiadm which is included with Open-iSCSI.

The iSCSI software initiator will be configured to automatically log in to the network storage server (openfiler1) and discover the iSCSI volumes created in the previous section. We will then go through the steps of creating persistent local SCSI device names (i.e. /dev/iscsi/asm1) for each of the iSCSI target names discovered using udev. Having a consistent local SCSI device name and which iSCSI target it maps to is required in order to know which volume (device) is to be used for OCFS2 and which volumes belong to ASM. Before we can do any of this, however, we must first install the iSCSI initiator software!

5.23 Installing the iSCSI (initiator) service

With CentOS 5.3, the Open-iSCSI iSCSI software initiator does not get installed by default. The software is included in the iscsi-initiator-utils package which can be found on CD #1. To determine if this package is installed (which in most cases, it will not be), perform the following on both Oracle RAC nodes:

# rpm -qa --queryformat "%{NAME}-%{VERSION}-%{RELEASE} (%{ARCH})\n"| grep iscsi-initiator-utils

If the iscsi-initiator-utils package is not installed, load CD #1 into each of the Oracle RAC nodes and perform the following:

# mount -r /dev/cdrom /media/cdrom

# cd /media/cdrom/CentOS

# rpm -Uvh iscsi-initiator-utils-*

# cd /

# eject

5.24 Configure the iSCSI (initiator) service

After verifying that the iscsi-initiator-utils package is installed on both Oracle RAC nodes, start the iscsid service and enable it to automatically start when the system boots. We will also configure the iscsi service to automatically start which logs into iSCSI targets needed at system startup.

# service iscsid start

Turning off network shutdown. Starting iSCSI daemon:       [  OK  ]

                                                           [  OK  ]

# chkconfig iscsid on

# chkconfig iscsi on

Now that the iSCSI service is started, use the iscsiadm command-line interface to discover all available targets on the network storage server. This should be performed on both Oracle RAC nodes to verify the configuration is functioning properly:

# iscsiadm -m discovery -t sendtargets -p openfiler1-priv

192.168.2.195:3260,1 iqn.2006-01.com.openfiler:racdb.asm1

192.168.2.195:3260,1 iqn.2006-01.com.openfiler:racdb.asm2

192.168.2.195:3260,1 iqn.2006-01.com.openfiler:racdb.asm3

192.168.2.195:3260,1 iqn.2006-01.com.openfiler:racdb.asm4

192.168.2.195:3260,1 iqn.2006-01.com.openfiler:racdb.crs

5.25 Manually Log In to iSCSI Targets

At this point the iSCSI initiator service has been started and each of the Oracle RAC nodes were able to discover the available targets from the network storage server. The next step is to manually log in to each of the available targets which can be done using the iscsiadm command-line interface. This needs to be run on both Oracle RAC nodes. Note that we had to specify the IP address and not the host name of the network storage server (openfiler1-priv) - we believe this is required given the discovery (above) shows the targets using the IP address.

# iscsiadm -m node -T iqn.2006-01.com.openfiler:racdb.asm1 -p 192.168.2.195 -l

# iscsiadm -m node -T iqn.2006-01.com.openfiler:racdb.asm2 -p 192.168.2.195 -l

# iscsiadm -m node -T iqn.2006-01.com.openfiler:racdb.asm3 -p 192.168.2.195 -l

# iscsiadm -m node -T iqn.2006-01.com.openfiler:racdb.asm4 -p 192.168.2.195 -l

# iscsiadm -m node -T iqn.2006-01.com.openfiler:racdb.crs -p 192.168.2.195 -l

5.26 Configure Automatic Log In

The next step is to ensure the client will automatically log in to each of the targets listed above when the machine is booted (or the iSCSI initiator service is started/restarted). As with the manual log in process described above, perform the following on both Oracle RAC nodes:

# iscsiadm -m node -T iqn.2006-01.com.openfiler:racdb.asm1 -p 192.168.2.195 --op update -n node.startup -v automatic

# iscsiadm -m node -T iqn.2006-01.com.openfiler:racdb.asm2 -p 192.168.2.195 --op update -n node.startup -v automatic

# iscsiadm -m node -T iqn.2006-01.com.openfiler:racdb.asm3 -p 192.168.2.195 --op update -n node.startup -v automatic

# iscsiadm -m node -T iqn.2006-01.com.openfiler:racdb.asm4 -p 192.168.2.195 --op update -n node.startup -v automatic

# iscsiadm -m node -T iqn.2006-01.com.openfiler:racdb.crs -p 192.168.2.195 --op update -n node.startup -v automatic

5.27 Create Persistent Local SCSI Device Names

In this section, we will go through the steps to create persistent local SCSI device names for each of the iSCSI target names. This will be done using udev. Having a consistent local SCSI device name and which iSCSI target it maps to is required in order to know which volume (device) is to be used for OCFS2 and which volumes belong to ASM.

When either of the Oracle RAC nodes boot and the iSCSI initiator service is started, it will automatically log in to each of the targets configured in a random fashion and map them to the next available local SCSI device name. For example, the target iqn.2006-01.com.openfiler:racdb.asm1 may get mapped to /dev/sda. We can actually determine the current mappings for all targets by looking at the /dev/disk/by-path directory:

# (cd /dev/disk/by-path; ls -l *openfiler* | awk '{FS=" "; print $9 " " $10 " " $11}')

ip-192.168.2.195:3260-iscsi-iqn.2006-01.com.openfiler:racdb.asm1-lun-0 -> ../../sda

ip-192.168.2.195:3260-iscsi-iqn.2006-01.com.openfiler:racdb.asm2-lun-0 -> ../../sdb

ip-192.168.2.195:3260-iscsi-iqn.2006-01.com.openfiler:racdb.asm3-lun-0 -> ../../sdc

ip-192.168.2.195:3260-iscsi-iqn.2006-01.com.openfiler:racdb.asm4-lun-0 -> ../../sdd

ip-192.168.2.195:3260-iscsi-iqn.2006-01.com.openfiler:racdb.crs-lun-0 -> ../../sde

Using the output from the above listing, we can establish the following current mappings:

Current iSCSI Target Name to local SCSI Device Name Mappings
iSCSI Target Name	SCSI Device Name
iqn.2006-01.com.openfiler:racdb.asm1	/dev/sda
iqn.2006-01.com.openfiler:racdb.asm2	/dev/sdb
iqn.2006-01.com.openfiler:racdb.asm3	/dev/sdc
iqn.2006-01.com.openfiler:racdb.asm4	/dev/sdd
iqn.2006-01.com.openfiler:racdb.crs	/dev/sde

This mapping, however, may change every time the Oracle RAC node is rebooted. For example, after a reboot it may be determined that the iSCSI target iqn.2006-01.com.openfiler:racdb.asm1 gets mapped to the local SCSI device /dev/sdd. It is therefore impractical to rely on using the local SCSI device name given there is no way to predict the iSCSI target mappings after a reboot.

What we need is a consistent device name we can reference (i.e. /dev/iscsi/asm1) that will always point to the appropriate iSCSI target through reboots. This is where the Dynamic Device Management tool named udev comes in. udev provides a dynamic device directory using symbolic links that point to the actual device using a configurable set of rules. When udev receives a device event (for example, the client logging in to an iSCSI target), it matches its configured rules against the available device attributes provided in sysfs to identify the device. Rules that match may provide additional device information or specify a device node name and multiple symlink names and instruct udev to run additional programs (a SHELL script for example) as part of the device event handling process.

The first step is to create a new rules file. The file will be named /etc/udev/rules.d/55-openiscsi.rules and contain only a single line of name=value pairs used to receive events we are interested in. It will also define a call-out SHELL script (/etc/udev/scripts/iscsidev.sh) to handle the event.

Create the following rules file /etc/udev/rules.d/55-openiscsi.rules on both Oracle RAC nodes:

/etc/udev/rules.d/55-openiscsi.rules

# /etc/udev/rules.d/55-openiscsi.rules KERNEL=="sd*", BUS=="scsi", PROGRAM="/etc/udev/scripts/iscsidev.sh %b",SYMLINK+="iscsi/%c/part%n"

We now need to create the UNIX SHELL script that will be called when this event is received. Let's first create a separate directory on both Oracle RAC nodes where udev scripts can be stored:

# mkdir -p /etc/udev/scripts

Next, create the UNIX shell script /etc/udev/scripts/iscsidev.sh on both Oracle RAC nodes:

/etc/udev/scripts/iscsidev.sh

#!/bin/sh # FILE: /etc/udev/scripts/iscsidev.sh BUS=${1} HOST=${BUS%%:*} [ -e /sys/class/iscsi_host ] || exit 1 file="/sys/class/iscsi_host/host${HOST}/device/session*/iscsi_session*/targetname" target_name=$(cat ${file}) # This is not an open-scsi drive if [ -z "${target_name}" ]; then    exit 1 fi # Check if QNAP drive check_qnap_target_name=${target_name%%:*} if [ $check_qnap_target_name = "iqn.2004-04.com.qnap" ]; then     target_name=`echo "${target_name%.*}"` fi echo "${target_name##*.}"

After creating the UNIX SHELL script, change it to executable:

# chmod 755 /etc/udev/scripts/iscsidev.sh

Now that udev is configured, restart the iSCSI service on both Oracle RAC nodes:

# service iscsi stop

Logging out of session [sid: 1, target: iqn.2006-01.com.openfiler:racdb.asm1, portal: 192.168.2.195,3260]

Logging out of session [sid: 2, target: iqn.2006-01.com.openfiler:racdb.asm2, portal: 192.168.2.195,3260]

Logging out of session [sid: 3, target: iqn.2006-01.com.openfiler:racdb.asm3, portal: 192.168.2.195,3260]

Logging out of session [sid: 4, target: iqn.2006-01.com.openfiler:racdb.asm4, portal: 192.168.2.195,3260]

Logging out of session [sid: 5, target: iqn.2006-01.com.openfiler:racdb.crs, portal: 192.168.2.195,3260]

Logout of [sid: 1, target: iqn.2006-01.com.openfiler:racdb.asm1, portal: 192.168.2.195,3260]: successful

Logout of [sid: 2, target: iqn.2006-01.com.openfiler:racdb.asm2, portal: 192.168.2.195,3260]: successful

Logout of [sid: 3, target: iqn.2006-01.com.openfiler:racdb.asm3, portal: 192.168.2.195,3260]: successful

Logout of [sid: 4, target: iqn.2006-01.com.openfiler:racdb.asm4, portal: 192.168.2.195,3260]: successful

Logout of [sid: 5, target: iqn.2006-01.com.openfiler:racdb.crs, portal: 192.168.2.195,3260]: successful

Stopping iSCSI daemon:                                     [  OK  ]

# service iscsi start

iscsid dead but pid file exists

Turning off network shutdown. Starting iSCSI daemon:       [  OK  ]

                                                           [  OK  ]

Setting up iSCSI targets: Logging in to [iface: default, target: iqn.2006-01.com.openfiler:racdb.asm2, portal: 192.168.2.195,3260]

Logging in to [iface: default, target: iqn.2006-01.com.openfiler:racdb.asm1, portal: 192.168.2.195,3260]

Logging in to [iface: default, target: iqn.2006-01.com.openfiler:racdb.crs, portal: 192.168.2.195,3260]

Logging in to [iface: default, target: iqn.2006-01.com.openfiler:racdb.asm4, portal: 192.168.2.195,3260]

Logging in to [iface: default, target: iqn.2006-01.com.openfiler:racdb.asm3, portal: 192.168.2.195,3260]

Login to [iface: default, target: iqn.2006-01.com.openfiler:racdb.asm2, portal: 192.168.2.195,3260]: successful

Login to [iface: default, target: iqn.2006-01.com.openfiler:racdb.asm1, portal: 192.168.2.195,3260]: successful

Login to [iface: default, target: iqn.2006-01.com.openfiler:racdb.crs, portal: 192.168.2.195,3260]: successful

Login to [iface: default, target: iqn.2006-01.com.openfiler:racdb.asm4, portal: 192.168.2.195,3260]: successful

Login to [iface: default, target: iqn.2006-01.com.openfiler:racdb.asm3, portal: 192.168.2.195,3260]: successful

                                                           [  OK  ]

Let's see if our hard work paid off:

# ls -l /dev/iscsi/*

/dev/iscsi/asm1:

total 0

lrwxrwxrwx 1 root root 9 Aug 16 00:49 part -> ../../sda

/dev/iscsi/asm2:

total 0

lrwxrwxrwx 1 root root 9 Aug 16 00:49 part -> ../../sdc

/dev/iscsi/asm3:

total 0

lrwxrwxrwx 1 root root 9 Aug 16 00:49 part -> ../../sdb

/dev/iscsi/asm4:

total 0

lrwxrwxrwx 1 root root 9 Aug 16 00:49 part -> ../../sde

/dev/iscsi/crs:

total 0

lrwxrwxrwx 1 root root 9 Aug 16 00:49 part -> ../../sdd

The listing above shows that udev did the job it was supposed to do! We now have a consistent set of local device names that can be used to reference the iSCSI targets. For example, we can safely assume that the device name /dev/iscsi/asm1/part will always reference the iSCSI target iqn.2006-01.com.openfiler:racdb.asm1. We now have a consistent iSCSI target name to local device name mapping which is described in the following table:

iSCSI Target Name to Local Device Name Mappings
iSCSI Target Name	Local Device Name
iqn.2006-01.com.openfiler:racdb.asm1	/dev/iscsi/asm1/part
iqn.2006-01.com.openfiler:racdb.asm2	/dev/iscsi/asm2/part
iqn.2006-01.com.openfiler:racdb.asm3	/dev/iscsi/asm3/part
iqn.2006-01.com.openfiler:racdb.asm4	/dev/iscsi/asm4/part
iqn.2006-01.com.openfiler:racdb.crs	/dev/iscsi/crs/part

5.28 Create Partitions on iSCSI Volumes

We now need to create a single primary partition on each of the iSCSI volumes that spans the entire size of the volume. As mentioned earlier in this article, We will be using Oracle's Cluster File System, Release 2 (OCFS2) to store the two files to be shared for Oracle's Clusterware software. We will then be using Automatic Storage Management (ASM) to create four ASM volumes; two for all physical database files (data/index files, online redo log files, and control files) and two for the Flash Recovery Area (RMAN backups and archived redo log files).

The following table lists the five iSCSI volumes and what file systems they will support:

Oracle Shared Drive Configuration
File System Type	iSCSI Target (short) Name	Size	Mount Point	ASM Diskgroup Name	File Types
OCFS2	crs	2GB	/u02		Oracle Cluster Registry (OCR) File - (~250 MB) Voting Disk - (~20MB)
ASM	asm1	17.8GB	ORCL:VOL1	+RACDB_DATA1	Oracle Database Files
ASM	asm2	17.8GB	ORCL:VOL2	+RACDB_DATA1	Oracle Database Files
ASM	asm3	17.8GB	ORCL:VOL3	+FLASH_RECOVERY_AREA	Oracle Flash Recovery Area
ASM	asm4	17.8GB	ORCL:VOL4	+FLASH_RECOVERY_AREA	Oracle Flash Recovery Area
Total		73.2GB

As shown in the table above, we will need to create a single Linux primary partition on each of the five iSCSI volumes. The fdisk command is used in Linux for creating (and removing) partitions. For each of the five iSCSI volumes, you can use the default values when creating the primary partition as the default action is to use the entire disk. You can safely ignore any warnings that may indicate the device does not contain a valid DOS partition (or Sun, SGI or OSF disklabel).

In this example, We will be running the fdisk command from linux1 to create a single primary partition on each iSCSI target using the local device names created by udev in the previous section:

·         /dev/iscsi/asm1/part

·         /dev/iscsi/asm2/part

·         /dev/iscsi/asm3/part

·         /dev/iscsi/asm4/part

·         /dev/iscsi/crs/part

Creating the single partition on each of the iSCSI volumes must only be run from one of the nodes in the Oracle RAC cluster! (i.e. linux1)

# ---------------------------------------

# fdisk /dev/iscsi/asm1/part

Command (m for help): n

Command action

   e   extended

   p   primary partition (1-4)

p

Partition number (1-4): 1

First cylinder (1-16992, default 1): 1

Last cylinder or +size or +sizeM or +sizeK (1-16992, default 16992): 16992

Command (m for help): p

Disk /dev/iscsi/asm1/part: 17.8 GB, 17817403392 bytes

64 heads, 32 sectors/track, 16992 cylinders

Units = cylinders of 2048 * 512 = 1048576 bytes

               Device Boot      Start         End      Blocks   Id  System

/dev/iscsi/asm1/part1               1       16992    17399792   83  Linux

Command (m for help): w

The partition table has been altered!

Calling ioctl() to re-read partition table.

Syncing disks.

# ---------------------------------------

# fdisk /dev/iscsi/asm2/part

Command (m for help): n

Command action

   e   extended

   p   primary partition (1-4)

p

Partition number (1-4): 1

First cylinder (1-16992, default 1): 1

Last cylinder or +size or +sizeM or +sizeK (1-16992, default 16992): 16992

Command (m for help): p

Disk /dev/iscsi/asm2/part: 17.8 GB, 17817403392 bytes

64 heads, 32 sectors/track, 16992 cylinders

Units = cylinders of 2048 * 512 = 1048576 bytes

               Device Boot      Start         End      Blocks   Id  System

/dev/iscsi/asm2/part1               1       16992    17399792   83  Linux

Command (m for help): w

The partition table has been altered!

Calling ioctl() to re-read partition table.

Syncing disks.

# ---------------------------------------

# fdisk /dev/iscsi/asm3/part

Command (m for help): n

Command action

   e   extended

   p   primary partition (1-4)

p

Partition number (1-4): 1

First cylinder (1-16992, default 1): 1

Last cylinder or +size or +sizeM or +sizeK (1-16992, default 16992): 16992

Command (m for help): p

Disk /dev/iscsi/asm3/part: 17.8 GB, 17817403392 bytes

64 heads, 32 sectors/track, 16992 cylinders

Units = cylinders of 2048 * 512 = 1048576 bytes

               Device Boot      Start         End      Blocks   Id  System

/dev/iscsi/asm3/part1               1       16992    17399792   83  Linux

Command (m for help): w

The partition table has been altered!

Calling ioctl() to re-read partition table.

Syncing disks.

# ---------------------------------------

# fdisk /dev/iscsi/asm4/part

Command (m for help): n

Command action

   e   extended

   p   primary partition (1-4)

p

Partition number (1-4): 1

First cylinder (1-16960, default 1): 1

Last cylinder or +size or +sizeM or +sizeK (1-16960, default 16960): 16960

Command (m for help): p

Disk /dev/iscsi/asm4/part: 17.7 GB, 17783848960 bytes

64 heads, 32 sectors/track, 16960 cylinders

Units = cylinders of 2048 * 512 = 1048576 bytes

               Device Boot      Start         End      Blocks   Id  System

/dev/iscsi/asm4/part1               1       16960    17367024   83  Linux

Command (m for help): w

The partition table has been altered!

Calling ioctl() to re-read partition table.

Syncing disks.

# ---------------------------------------

# fdisk /dev/iscsi/crs/part

Command (m for help): n

Command action

   e   extended

   p   primary partition (1-4)

p

Partition number (1-4): 1

First cylinder (1-1009, default 1): 1

Last cylinder or +size or +sizeM or +sizeK (1-1009, default 1009): 1009

Command (m for help): p

Disk /dev/iscsi/crs/part: 2147 MB, 2147483648 bytes

67 heads, 62 sectors/track, 1009 cylinders

Units = cylinders of 4154 * 512 = 2126848 bytes

              Device Boot      Start         End      Blocks   Id  System

/dev/iscsi/crs/part1               1        1009     2095662   83  Linux

Command (m for help): w

The partition table has been altered!

Calling ioctl() to re-read partition table.

Syncing disks.

5.29 Verify New Partitions

After creating all required partitions from linux1, you should now inform the kernel of the partition changes using the following command as the "root" user account from all remaining nodes in the Oracle RAC cluster (linux2). Note that the mapping of iSCSI target names discovered from Openfiler and the local SCSI device name will be different on both Oracle RAC nodes. This is not a concern and will not cause any problems since we will not be using the local SCSI device names but rather the local device names created by udev in the previous section.

From linux2, run the following commands:

# partprobe

# fdisk -l

Disk /dev/hda: 40.0 GB, 40000000000 bytes

255 heads, 63 sectors/track, 4863 cylinders

Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System

/dev/hda1   *           1          13      104391   83  Linux

/dev/hda2              14        4863    38957625   8e  Linux LVM

Disk /dev/sda: 2147 MB, 2147483648 bytes

67 heads, 62 sectors/track, 1009 cylinders

Units = cylinders of 4154 * 512 = 2126848 bytes

   Device Boot      Start         End      Blocks   Id  System

/dev/sda1               1        1009     2095662   83  Linux

Disk /dev/sdb: 17.7 GB, 17783848960 bytes

64 heads, 32 sectors/track, 16960 cylinders

Units = cylinders of 2048 * 512 = 1048576 bytes

   Device Boot      Start         End      Blocks   Id  System

/dev/sdb1               1       16960    17367024   83  Linux

Disk /dev/sdc: 17.8 GB, 17817403392 bytes

64 heads, 32 sectors/track, 16992 cylinders

Units = cylinders of 2048 * 512 = 1048576 bytes

   Device Boot      Start         End      Blocks   Id  System

/dev/sdc1               1       16992    17399792   83  Linux

Disk /dev/sdd: 17.8 GB, 17817403392 bytes

64 heads, 32 sectors/track, 16992 cylinders

Units = cylinders of 2048 * 512 = 1048576 bytes

   Device Boot      Start         End      Blocks   Id  System

/dev/sdd1               1       16992    17399792   83  Linux

Disk /dev/sde: 17.8 GB, 17817403392 bytes

64 heads, 32 sectors/track, 16992 cylinders

Units = cylinders of 2048 * 512 = 1048576 bytes

   Device Boot      Start         End      Blocks   Id  System

/dev/sde1               1       16992    17399792   83  Linux

As a final step you should run the following command on both Oracle RAC nodes to verify that udev created the new symbolic links for each new partition:

# (cd /dev/disk/by-path; ls -l *openfiler* | awk '{FS=" "; print $9 " " $10 " " $11}')

ip-192.168.2.195:3260-iscsi-iqn.2006-01.com.openfiler:racdb.asm1-lun-0 -> ../../sdc

ip-192.168.2.195:3260-iscsi-iqn.2006-01.com.openfiler:racdb.asm1-lun-0-part1 -> ../../sdc1

ip-192.168.2.195:3260-iscsi-iqn.2006-01.com.openfiler:racdb.asm2-lun-0 -> ../../sdd

ip-192.168.2.195:3260-iscsi-iqn.2006-01.com.openfiler:racdb.asm2-lun-0-part1 -> ../../sdd1

ip-192.168.2.195:3260-iscsi-iqn.2006-01.com.openfiler:racdb.asm3-lun-0 -> ../../sde

ip-192.168.2.195:3260-iscsi-iqn.2006-01.com.openfiler:racdb.asm3-lun-0-part1 -> ../../sde1

ip-192.168.2.195:3260-iscsi-iqn.2006-01.com.openfiler:racdb.asm4-lun-0 -> ../../sdb

ip-192.168.2.195:3260-iscsi-iqn.2006-01.com.openfiler:racdb.asm4-lun-0-part1 -> ../../sdb1

ip-192.168.2.195:3260-iscsi-iqn.2006-01.com.openfiler:racdb.crs-lun-0 -> ../../sda

ip-192.168.2.195:3260-iscsi-iqn.2006-01.com.openfiler:racdb.crs-lun-0-part1 -> ../../sda1

The listing above shows that udev did indeed create new device names for each of the new partitions. We will be using these new device names when configuring the volumes for OCFS2 and ASMlib later in this guide:

·         /dev/iscsi/asm1/part1

·         /dev/iscsi/asm2/part1

·         /dev/iscsi/asm3/part1

·         /dev/iscsi/asm4/part1

·         /dev/iscsi/crs/part1

7) R12 Installation paths

7.1 R12 Installation (Initially 2 mid-tier & 1 DB-tier node)

Installation Requirement :
1) Disk space requirement:
Node                 Space Required
-----------------------------------------------
Mid-Tier                       50GB X 2 (2 nodes)
DB Tier – Vision               250GB

2) Installation is done by root OS user
3) Database OS User created (oracle)

         # mkdir –p /u01/app/oracle

         # chown –R oracle:oinstall /u01/app/oracle

         # chmod –R 775 /u01/app/oracle

At the end of this procedure, you will have the following:

·         /u01 owned by root.

·         /u01/app owned by root.

·         /u01/app/oracle owned by oracle:oinstall with 775 permissions. This ownership and permissions enables the OUI to create the oraInventory directory, in the path /u01/app/oracle/oraInventory.

4) Apps OS User created         (applmgr)
3) OS Utilities like ar, gcc, g++, ld, ksh, make, X Display Server must exist in PATH.

Installation Process:
Below are some screen-shots of Oracle Applications R12 Vision Instance Installation. Well the screens are self-explanatory, but we have also given some explanation with few screens.

Screen1: Welcome Screen: See everything and press Next. 

Screen1: Welcome Screen: See everything and press Next.

Screen 2: Wizard Operation : Choose whether you want a fresh installation or upgrade from previous version.

Screen 3: Oracle Configuration Manager: Accept this if you have metalink account and you want support from Oracle. This will ask for Metalink account and Support Identifier in the next 

Screen 4: Oracle Configuration Manager Details:

Screen 5: Configuration Choice: If you already have configurations file from previous installation Locate the file and load the configurations from that else Create a new configuration. This option is very helpful, if you have got any error during installation and you want to restart the installation again. In that case you can use the previous selected configurations.

Screen 6: Database Node setup : Provide the DB node details e.g. DB SID, HostName, Domain Name, OS, OS User and Group, and Installation Base Directory.
Please note that the installation is done by root user and you need a on’tte user for DB account e.g. oracle with dba group.

  

Screen 7: Primary Apps Node setup : Next is the Primary Application Node setup. Provide the on’tte details.
Notice that it is asking for “Instance Directory” This is the $INST_TOP for this particular node.

Screen 8: Enable/Disable Application Services for Primary Node : As explained earlier, we can have may Appl nodes. We have taken example of two appl nodes (appl_node1 and appl_node2). We have disabled the Batch Processing Services on appl_node1 and will enable it on appl_node2 as explained earlier in :Shared File System Architecutre”.

Screen 9: Node Information: This screen gives the DB and primary node informations. Now click on the Add Server button to add additional Appl node.

Screen 10: Additional node config : This screen shows the additional node setup. Note the Shared File System Checkbox. Check this if you want a single installation and share the installation setup for both the nodes (by NFS mount). And if you want a on’tte $INST_TOP for the additional node, on’t check the box and provide the paths for that.

Screen 11: Additional node application services: Enable Root Service group and batch processing services for the additional Appl node.

Screen 12 : Node Information :  Now this screen shows all the three node information..

Screen 13 : System Status Check

Screen 14: Pre-install Checks : Once all the Checks are passed, proceed for the installation.

Screen 15 : Install in Progress :

Screen 16 : Post install checks : If any of the check failed, see the error by clicking icon near the item and try to remove the errors. Then again check. If every things is fine. Click next and then finish on the next screen.

Screen 17 : Final Screen

Congratulations... Your Oracle Applications R12 Installation is successfully done. Now you can just type the URL in the browser and see beautiful screen of Oracle Apps R12. You can do the initial login with User: SYSADMIN and password : sysadmin. Then create new users with System Administrator Responsibility and PLAY/WORK J.

LOG FILE LOCATIONS FOR VARIOUS TYPES OF LOGS

Log files are useful in troubleshooting issues in Oracle Applications. Here is the list of Log file location in Oracle Applications for Startup/Shutdown, Cloning, Patching, DB & Apps Listener and various components in Apps R12/12i:

C.  Startup/Shutdown Log files for Application Tier in R12

Instance Top is new TOP added in R12. Most of the logs are location in INST_TOP. Below are the location for logs files for Startup/shutdown processes:

–Startup/Shutdown error message text files like adapcctl.txt, adcmctl.txt…
$INST_TOP/apps/$CONTEXT_NAME/logs/appl/admin/log

–Startup/Shutdown error message related to tech stack (10.1.2, 10.1.3 forms/reports/web)
$INST_TOP/apps/$CONTEXT_NAME/logs/ora/  (10.1.2 & 10.1.3)
$INST_TOP/apps/$CONTEXT_NAME/logs/ora/10.1.3/Apache/error_log[timestamp]
$INST_TOP/apps/$CONTEXT_NAME/logs/ora/10.1.3/opmn/ (OC4J~…, oa*, opmn.log)$INST_TOP/apps/$CONTEXT_NAME/logs/ora/10.1.2/network/ (listener log)
$INST_TOP/apps/$CONTEXT_NAME/logs/appl/conc/log  (CM log files)

B. Autoconfig related log files in R12
i) Database Tier Autoconfig log :
$ORACLE_HOME/appsutil/log/$CONTEXT_NAME/MMDDHHMM/adconfig.log
$ORACLE_HOME/appsutil/log/$CONTEXT_NAME/MMDDHHMM/NetServiceHandler.log

ii) Application Tier Autoconfig log - $INST_TOP/apps/$CONTEXT_NAME/admin/log/$MMDDHHMM/adconfig.log

Autoconfig context file location in R12 - $INST_TOP/apps/$CONTEXT_NAME/appl/admin/$CONTEXT_NAME.xml

C. R12 Installation Logs

Database Tier Installation

RDBMS $ORACLE_HOME/appsutil/log/$CONTEXT_NAME/.logRDBMS $ORACLE_HOME/appsutil/log/$CONTEXT_NAME/ApplyDBTechStack_.logRDBMS
$ORACLE_HOME/appsutil/log/$CONTEXT_NAME/ohclone.logRDBMS $ORACLE_HOME/appsutil/log/$CONTEXT_NAME/make_.logRDBMS
$ORACLE_HOME/appsutil/log/$CONTEXT_NAME/installdbf.logRDBMS $ORACLE_HOME/appsutil/log/$CONTEXT_NAME/adcrdb_.log RDBMS
$ORACLE_HOME/appsutil/log/$CONTEXT_NAME/ApplyDatabase_.logRDBMS $ORACLE_HOME/appsutil/log/$CONTEXT_NAME//adconfig.log    RDBMS
$ORACLE_HOME/appsutil/log/$CONTEXT_NAME//NetServiceHandler.log

Application Tier Installation

$INST_TOP/logs/.log
$APPL_TOP/admin/$CONTEXT_NAME/log/ApplyAppsTechStack.log
$INST_TOP/logs/ora/10.1.2/install/make_.log
$INST_TOP/logs/ora/10.1.3/install/make_.log
$INST_TOP/admin/log/ApplyAppsTechStack.log
$INST_TOP/admin/log/ohclone.log
$APPL_TOP/admin/$CONTEXT_NAME/log/installAppl.log
$APPL_TOP/admin/$CONTEXT_NAME/log/ApplyAppltop_.log
$APPL_TOP/admin/$CONTEXT_NAME/log//adconfig.log
$APPL_TOP/admin/$CONTEXT_NAME/log//NetServiceHandler.log

8) R12 DB-Tier RAC Conversion

8.1 Check the hardware setup done previously

Ø  Shared Storage – Already setup is done before R12 Installation

Ø  Network – Already setup is done before R12 Installation

Ø  Configure SSH between both DB nodes – Explained below

Ø  Run Cluster verify utility – PreInstall – runcluvfy

Installation Requirement :
1) Disk space requirement:
Node                 Space Required
-----------------------------------------------
Mid-Tier                       50GB X 2 (2 nodes)
DB Tier – Vision               250GB

2) Installation is done by root OS user
3) Database OS User created (oracle)

         # mkdir -p /u01/app/oracle

         # chown -R oracle:oinstall /u01/app/oracle

         # chmod -R 775 /u01/app/oracle

At the end of this procedure, you will have the following:

·         /u01 owned by root.

·         /u01/app owned by root.

·         /u01/app/oracle owned by oracle:oinstall with 775 permissions. This ownership and permissions enables the OUI to create the oraInventory directory, in the path /u01/app/oracle/oraInventory.

4) Apps OS User created     (applmgr)
3) OS Utilities like ar, gcc, g++, ld, ksh, make, X Display Server must exist in PATH.

8.2 CRS Installation

Ø  Install CRS 10.2.0.1 Using RunInstaller (/u01/apps/oracle/VIS/db/racdb/crs)

Ø  Verify Install - /u01/apps/oracle/VIS/db/racdb/crs/bin/olsnodes

8.3 Install Oracle DB 10.2.0.1 Binaries

8.4 Install Oracle DB Components from Components CD

8.5 Upgrade CRS & Database software to 10.2.0.2

8.6 Upgrade the apps database to 10.2.0.2 (utlu102i.sql)

Ø  Interoperabilty patch note:- 454750.1

8.7 Listener Configuration

Ø  Create vis_lnsr under $ORACLE_HOME/network/admin

Ø  On all cluster nodes run NetCA to create listeners

8.8 Create ASM Instance and ASM Diskgroups

8.9 Run DBCA to configure ASM instances

Ø  Create diskgroup ‘+DATADG’ on ‘/dev/raw/ASM1’

Ø  Verify ASM instance is registered in CRS ($CRS_HOME/bin/crs_stat –t)

8.10 Prepare ConvertToRAC.xml using rconfig

Ø  File Location ($ORACLE_HOME/assistants/rconfig/sampleXMLs

Ø  Edit following parameters

SourceDBHome

TargetDBHome

RAC Nodes

Instance Prefix

Sharedstorage type = ASM

TargetDatabaseArea: ASM Diskgroupname

Convert Verify:”NO”||”ONLY”

8.11 Create a new spfile on ASM for target(RAC) DB_Home

Ø  Create spfile=’+DATADG/spfilevis.ora’ from pfile;

Ø  Link the init.ora to the spfile in the target DB_home

Ø  Startup database from target DB_HOME

Ø  Use NetCA to create local and remote listeners

8.12 RUN rconfig

$ORACLE_HOME/bin/rconfig ConvertToRac.XML

Ø  Migrate the DB to ASM storage; Create DB instances

Ø  Configure Listener & Netservices; Configure/Register CRS

Ø  Start the instance on all nodes

8.13 Enable Autoconfig on database tier

Ø  Generate appsutil.zip from apps tier

o    Configure tnsnames.ora in apps tier to point to 1 DB instance

o    Execute $AD_TOP/bin/admkappsutil.pl to create appsutil.zip

Ø  Create appsutil in RAC ORACLE_HOME in DB Tier

o    Copy appsutil.zip to RAC ORACLE_HOME and unzip it.

o    Copy instance information from OLD ORACLE_HOME

Perl $OLD_ORACLE_HOME/appsutil/scripts//adpreclone.pl

Copy context_file,jre from old ORACLE_HOME to RAC ORACLE_HOME/appsutil

Ø  Create instance specific XML context file

o    De-register the current config: exec find_conc_clone.setup_clean

o    Create pairsfile.txt file under new RAC ORACLE_HOME/appsutil/clone

S_undo_tablespace=UNDOTBS2; s_dbClusterInst=2

S_db_oh=/u01/apps/oracle/VIS/racdb/10/2/0

o    Run perl adclonectx.pl to create instance specific XML context file

8.14 Run Autoconfig on database tier node-1 & node-2

8.15 Check database & nodeapps status using srvctl

8.16 Establish application environment for RAC

Ø  Run adconfig on application nodes

o    Prepare tnsnames.ora to connect to RAC node-1

o    Set jdbc_url in the context_file to the instance-1 of RAC

o    Execute autoconfig ($AD_TOP/bin/autoconfig.sh context_file=<>.xml)

o    Check tnsnames.ora in $INST_TOP/ora/10.1.2 & 10.1.3

o    Verify the DBC file in $FND_SECURE

Ø  Load balancing –edit context_file and set TWO_TASK

Ø  Run autoconfig $AD_TOP/adconfig.sh

Ø  Profile option “Application Database ID” to set the DBC file at $FND_SECURE

NOW YOUR DREAM MACHINE IS READY!!!! A MULTINODE ORACLE EBS INSTANCE............

HAPPY LEARNING!