We're here to help!

Clustered Replication running on ACFS enabled servers.

Follow

Read our docs The below article gives a detailed worked example of using Clustered Dbvisit Replicate between a 2 node RAC Source 11g database and 2 node RAC Target both with ACFS enabled.

 

Environment Details

 

Source System: 2 Node Oracle RAC Target System: 2 Node Oracle RAC
  • Oracle Linux 6.4 x86_64
  • Oracle Database 11.2.0.4 – installed using “oracle” unix account
  • Oracle Grid Infrastructure 11.2.0.4 – installed using “grid” unix account
  • Diskgroups:
    • DATA1 – Database Files
    • DATA2 – Flash/Fast Recovery Area (FRA)
    • DATA3 – Shared storage used for ACFS filesytem
    • Database Name: LAA
    • Node 1 – kiwi81
    • Node 2 – kiwi82
    • Dbvisit Replicate 2.8.04
    • DNS Resolved via server kiwidns
  • Oracle Linux 6.4 x86_64
  • Oracle Database 11.2.0.4 – installed using “oracle” unix account
  • Oracle Grid Infrastructure 11.2.0.4 – installed using “grid” unix account
  • Diskgroups:
    • DATA1 – Database Files
    • DATA2 – Flash/Fast Recovery Area (FRA)
    • DATA3 – Shared storage used for ACFS filesytem
    • Database Name: DEV
    • Node 1 – kiwi91
    • Node 2 – kiwi92
    • Dbvisit Replicate 2.8.04
    • DNS Resolved via server kiwidns

 

Assumptions:

It is assumed the grid binaries have already been pre-installed and that the acfs mount point has been created.This article will proceed from the point of creating a Directory for the replication configuration on mount point /acfs and symbolic links from the oracle user home on each node are created to point to this directory. E.g. on nodes kiwi81/kiwi82


NODE kiwi81 as oracle:
oracle@kiwi81[/home/oracle]: cd /acfs
oracle@kiwi81[/acfs]: mkdir laa_rac
oracle@kiwi81[/acfs]: cd
oracle@kiwi81[/home/oracle]: ln -s /acfs/laa_rac
oracle@kiwi81[/home/oracle]: ls -l
lrwxrwxrwx 1 oracle oinstall 13 Mar 13 12:17 laa_rac -> /acfs/laa_rac

NODE kiwi82 as oracle:
oracle@kiwi82[/home/oracle]: ln -s /acfs/laa_rac
oracle@kiwi82[/home/oracle]: ls -l
lrwxrwxrwx 1 oracle oinstall 13 Mar 13 12:19 laa_rac -> /acfs/laa_rac
 

References:

“ora.registry.acfs” ("ora.drivers.acfs") Resource Was Not Configured Therefore RAC ACFS Filesystem Is Not Mounting During The Reboot. (Doc ID 1486208.1)

https://dbvisit.atlassian.net/wiki/display/ugd8/

http://blog.dbvisit.com/adding-dbvisit-replicate-as-an-oracle-clusterware-resource/ 

https://docs.oracle.com/cd/E18283_01/server.112/e16102/asmfs_util010.htm#CACGHBHI

  

Pre-Steps Performed - Before Replication Setup
1. Ensure the Acfs mount point auto-mounts at startup.  (MOS# 1286208.1) Even though registry was configured correctly, the /acfs was not mounting on startup.  This in turn has a knock on affect to the start of Dbvisit Replicate as the configuration files resided on this drive. The issue can be solved by issuing the following commands on all nodes as the root user.
# /bin/acfsroot install
# /bin/acfsroot enable 

root@kiwi91[/home/oracle/laa_rac]: cd /u01/app/11.2.0/grid/bin
root@kiwi91
[/home/oracle/laa_rac]: . oraenv
ORACLE_SID = [root] ? +ASM1
The Oracle base has been set to /u01/app/grid
root@kiwi91[/u01/app/11.2.0/grid/bin]: acfsroot install
ACFS-9300: ADVM/ACFS distribution files found.
ACFS-9118: oracleacfs.ko driver in use - cannot unload.
ACFS-9312: Existing ADVM/ACFS installation detected.
ACFS-9118: oracleacfs.ko driver in use - cannot unload.
ACFS-9314: Removing previous ADVM/ACFS installation.
ACFS-9315: Previous ADVM/ACFS components successfully removed.
ACFS-9307: Installing requested ADVM/ACFS software.
ACFS-9308: Loading installed ADVM/ACFS drivers.
ACFS-9321: Creating udev for ADVM/ACFS.
ACFS-9323: Creating module dependencies - this may take some time.
ACFS-9327: Verifying ADVM/ACFS devices.
ACFS-9156: Detecting control device '/dev/asm/.asm_ctl_spec'.
ACFS-9156: Detecting control device '/dev/ofsctl'.
ACFS-9309: ADVM/ACFS installation correctness verified.

root@kiwi91[/u01/app/11.2.0/grid/bin]: acfsroot enable
ACFS-9376: Adding ADVM/ACFS drivers resource succeeded.
CRS-2672: Attempting to start 'ora.drivers.acfs' on 'kiwi91'
CRS-2676: Start of 'ora.drivers.acfs' on 'kiwi91' succeeded
ACFS-9380: Starting ADVM/ACFS drivers resource succeeded.
ACFS-9368: Adding ACFS registry resource succeeded.
CRS-2672: Attempting to start 'ora.registry.acfs' on 'kiwi92'
CRS-2672: Attempting to start 'ora.registry.acfs' on 'kiwi91'
CRS-2676: Start of 'ora.registry.acfs' on 'kiwi91' succeeded
CRS-2676: Start of 'ora.registry.acfs' on 'kiwi92' succeeded
ACFS-9372: Starting ACFS registry resource succeeded.

2. Create New dedicated Services on each database for replication to run. These services will be used in the section on TNS entry.

oracle@kiwi81[/home/oracle]: . oraenv 
ORACLE_SID = [oracle] ? LAA1 
The Oracle base remains unchanged with value /u01/app/oracle 
oracle@kiwi81[/home/oracle]: srvctl add service -d LAA -s DBVLAA -r LAA1 -a LAA2 
oracle@kiwi81[/home/oracle]: srvctl status service -d LAA -s DBVLAA 
Service DBVLAA is not running. 
oracle@kiwi81[/home/oracle]: srvctl start service -d LAA -s DBVLAA 
oracle@kiwi81[/home/oracle]: srvctl status service -d LAA -s DBVLAA 
Service DBVLAA is running on instance(s) LAA1

Repeat on Nodes 91/92
oracle@kiwi91[/home/oracle]: srvctl add service -d DEV -s DBVDEV -r DEV1 -a DEV2 
oracle@kiwi91[/home/oracle]: srvctl status service -d DEV -s DBVDEV 
Service DBVDEV is not running. 
oracle@kiwi91[/home/oracle]: srvctl start service -d DEV -s DBVDEV 
oracle@kiwi91[/home/oracle]: srvctl status service -d DEV -s DBVDEV 
Service DBVDEV is running on instance(s) DEV1 

Add the TNS Entries for both in 4 locations.  $ORACLE_HOME/network/admin/tnsnames.ora on each node.

DBVLAA  = 
  (DESCRIPTION = 
    (ADDRESS = (PROTOCOL = TCP)(HOST = kiwi812-scan)(PORT = 1521)) 
    (CONNECT_DATA = 
      (SERVER = DEDICATED) 
      (SERVICE_NAME = DBVLAA) 
    ) 
  ) 

DBVDEV  = 
  (DESCRIPTION = 
    (ADDRESS = (PROTOCOL = TCP)(HOST = kiwi912-scan)(PORT = 1521)) 
    (CONNECT_DATA = 
      (SERVER = DEDICATED) 
      (SERVICE_NAME = DBVDEV) 
    ) 
  )

3. Created new dedicated Virtual IP addresses (used as host name in setup wizard )
root@kiwi81[/home/oracle/laa_rac]: . oraenv
ORACLE_SID = [root] ? +ASM1
The Oracle base has been set to /u01/app/grid
root@kiwi81[/root]: appvipcfg create -network=1 -ip=10.0.2.172 -vipname=dbvrep812-vip 
-user=root
root@kiwi81[/root]: crsctl status resource dbvrep812-vip -p NAME=dbvrep812-vip TYPE=app.appvip_net1.type ACL=owner:root:rwx,pgrp:root:r-x,other::r--,user:root:r-x ACTION_FAILURE_TEMPLATE= ACTION_SCRIPT= ACTIVE_PLACEMENT=1 AGENT_FILENAME=%CRS_HOME%/bin/orarootagent%CRS_EXE_SUFFIX% APPSVIP_FAILBACK=0 AUTO_START=restore CARDINALITY=1 CHECK_INTERVAL=1 CHECK_TIMEOUT=120 DEFAULT_TEMPLATE=PROPERTY(RESOURCE_CLASS=vip) ELEMENT(HOSTING_MEMBERS=%HOSTING_MEMBERS%) DEGREE=1 DESCRIPTION=Application VIP ENABLED=1 FAILOVER_DELAY=0 FAILURE_INTERVAL=0 FAILURE_THRESHOLD=0 GEN_USR_ORA_STATIC_VIP= GEN_USR_ORA_VIP= HOSTING_MEMBERS=kiwi81.oraclekiwi.co.nz LOAD=1 LOGGING_LEVEL=1 NLS_LANG= NOT_RESTARTING_TEMPLATE= OFFLINE_CHECK_INTERVAL=0 PLACEMENT=balanced PROFILE_CHANGE_TEMPLATE= RESTART_ATTEMPTS=0 SCRIPT_TIMEOUT=60 SERVER_POOLS=* START_DEPENDENCIES=hard(ora.net1.network) pullup(ora.net1.network) START_TIMEOUT=120 STATE_CHANGE_TEMPLATE= STOP_DEPENDENCIES=hard(ora.net1.network) STOP_TIMEOUT=0 TYPE_VERSION=2.2 UPTIME_THRESHOLD=7d USR_ORA_ENV= USR_ORA_VIP=10.0.2.172 VERSION=11.2.0.3.0 root@kiwi81[/root]: crsctl setperm resource dbvrep812-vip -u user:oracle:r-x root@kiwi81[/root]: crsctl setperm resource dbvrep812-vip -u user:grid:r-x root@kiwi81[/root]: crsctl start resource dbvrep812-vip -n kiwi81 CRS-2672: Attempting to start 'dbvrep812-vip' on 'kiwi81' CRS-2676: Start of 'dbvrep812-vip' on 'kiwi81' succeeded root@kiwi81[/root]: crsctl status resource dbvrep812-vip NAME=dbvrep812-vip TYPE=app.appvip_net1.type TARGET=ONLINE STATE=ONLINE on kiwi81 Repeat for other Nodes root@kiwi91[/root]: appvipcfg create -network=1 -ip=10.0.2.177 -vipname=dbvrep912-vip
-user=root
root@kiwi91[/root]: crsctl setperm resource dbvrep912-vip -u user:oracle:r-x
root@kiwi91[/root]: crsctl setperm resource dbvrep912-vip -u user:grid:r-x root@kiwi91[/root]: crsctl start resource dbvrep912-vip CRS-2672: Attempting to start 'dbvrep912-vip' on 'kiwi91' CRS-2676: Start of 'dbvrep912-vip' on 'kiwi91' succeeded root@kiwi91[/root]:
4. As we are using a DNS Server. Added the IP addresses to the forward and reverse rules on the
DNS server. Otherwise new IPs would have to be added to the /etc/hosts file on ever node. [root@kiwidns]# cd /var/named [root@kiwidns]# vi reverse.oraclekiwi [root@kiwidns]# cat reverse.oraclekiwi|grep -i 172 dbvrep812-vip   IN A            10.0.2.172 172      IN PTR dbvrep812-vip.oraclekiwi.co.nz. [root@kiwidns]# cat reverse.oraclekiwi|grep -i 177 dbvrep912-vip   IN A            10.0.2.177 177      IN PTR dbvrep912-vip.oraclekiwi.co.nz. [root@kiwidns]# vi forward.oraclekiwi [root@kiwidns]# cat forward.oraclekiwi |grep -i 172 @          IN A          10.0.2.172 dbvrep812-vip   IN A            10.0.2.172 [root@kiwidns]# cat forward.oraclekiwi |grep -i 177 @          IN A          10.0.2.177 dbvrep912-vip   IN A            10.0.2.177 Testing New IPs from kiwi82 oracle@kiwi82[/home/oracle]: nslookup dbvrep812-vip Server:          10.0.2.11 Address:     10.0.2.11#53 Name:     dbvrep812-vip.oraclekiwi.co.nz Address: 10.0.2.172 oracle@kiwi82[/home/oracle]: nslookup dbvrep912-vip Server:          10.0.2.11 Address:     10.0.2.11#53 Name:     dbvrep912-vip.oraclekiwi.co.nz Address: 10.0.2.177
Steps Performed - Replication Setup
 
1. Install replication binaries on all 4 nodes :  Servers kiwi81/82 & kiwi91/92
 
root@kiwi91[/home/oracle]: rpm -ivh dbvisit_replicate-2.8.04_rc1-1.x86_64.rpm Preparing...                ########################################### [100%]    1:dbvisit_replicate      ########################################### [100%] root@kiwi91[/home/oracle]:
 
2. Run from 1 source node (kiwi81) to configure Replication
oracle@kiwi81[/home/oracle]: dbvrep 
Initializing......done 
Dbvisit Replicate version 2.8.04_rc1 
Copyright (C) Dbvisit Software Limited.  All rights reserved. 
No DDC file loaded. 
Run "setup wizard" to start the configuration wizard or try "help" to see all commands available. 
dbvrep> setup wizard 
This wizard configures Dbvisit Replicate. 

The setup wizard creates configuration scripts, which need to be run after the wizard ends. No changes to the databases are made before that. 

The progress is saved every time a list of databases, replications, etc. is shown. It will be re-read if wizard is restarted and the same DDC name and script path is selected. 
Run the wizard now? [Yes] Yes 
Accept end-user license agreement? (View/Yes/No) [View] Yes 

Before starting the actual configuration, some basic information is needed. The DDC name and script path determines where all files created by the wizard go (and where to reread them if 
wizard is rerun) and the license key determines which options are available for this configuration. 
(DDC_NAME) - Please enter a name for this replication: [] laa_rac 
(LICENSE_KEY) - Please enter your license key: [(trial)] 
Which Replicate edition do you want to trial (LTD/XTD/MAX): [MAX] 
(SETUP_SCRIPT_PATH) - Please enter a directory for location of configuration scripts on this machine: []/home/oracle/laa_rac

Network configuration files were detected on this system in these locations: 
/u01/app/11.2.0/grid/network/admin 
/u01/app/oracle/product/11.2.0/db_1/network/admin 
(TNS_ADMIN) - Please enter TNS configuration directory for this machine: []/u01/app/oracle/product/11.2.0/db_1/network/admin

Step 1 - Describe databases 
======================================== 
The first step is to describe databases used in the replication. There are usually two of them (source and target). 
Store SYSDBA and DBA passwords? Passwords only required during setup and initialization? (Yes/No) [Yes] 
Let's configure the database, describing its type, connectivity, user names etc. 
What type of database is this? (Oracle/MySQL/Google Cloud SQL/SQL Server/Oracle AWS RDS/CSV/Hadoop): [Oracle] 
Please enter database TNS alias: [] DBVLAA -- Dedicated Service Name on Source
Please enter SYSDBA user name: [SYS] 
Please enter password for this user: [change_on_install] ******* 
Please enter user with DBA role: [SYSTEM] 
Please enter password for this user: [manager] ******* 

Connecting to database DBVLAA as SYSTEM to query list of tablespaces and to detect ASM (by looking whether any redo logs or archived logs are stored in ASM). 
Enter the Dbvisit Replicate owner (this user will be created by this script): [dbvrep] 
Please enter password for this user: [dbvpasswd] ******* 

Permanent tablespaces detected on the database: USERS. 
Please enter default permanent tablespace for this user: [USERS] 

Temporary tablespaces detected on the database: TEMP. 
Please enter default temporary tablespace for this user: [TEMP] 

Following databases are now configured: 
1: Oracle DBVLAA, SYS/***, SYSTEM/***, dbvrep/***, USERS/TEMP, dbvrep/, ASM:Yes, TZ: +00:00 
Enter the number of the database to modify it, or "add", or "done": [add] 
Let's configure the database, describing its type, connectivity, user names etc. 
What type of database is this? (Oracle/MySQL/Google Cloud SQL/SQL Server/Oracle AWS RDS/CSV/Hadoop): [Oracle] 
Please enter database TNS alias: [] DBVDEV -- Dedicated Service Name on Target
Please enter SYSDBA user name: [SYS] 
Please enter password for this user: [change_on_install] ******* 
Please enter user with DBA role: [SYSTEM] 
Please enter password for this user: [manager] ******* 

Connecting to database DBVDEV as SYSTEM to query list of tablespaces and to detect ASM (by looking whether any redo logs or archived logs are stored in ASM). 
Enter the Dbvisit Replicate owner (this user will be created by this script): [dbvrep] 
Please enter password for this user: [dbvpasswd] ******* 

Permanent tablespaces detected on the database: USERS. 
Please enter default permanent tablespace for this user: [USERS] 

Temporary tablespaces detected on the database: TEMP. 
Please enter default temporary tablespace for this user: [TEMP] 

Following databases are now configured: 
1: Oracle DBVLAA, SYS/***, SYSTEM/***, dbvrep/***, USERS/TEMP, dbvrep/, ASM:Yes, TZ: +00:00 
2: Oracle DBVDEV, SYS/***, SYSTEM/***, dbvrep/***, USERS/TEMP, dbvrep/, ASM:Yes, TZ: +00:00 
Enter the number of the database to modify it, or "add", or "done": [done] 

Step 2 - Replication pairs 
======================================== 
The second step is to set source and targets for each replication pair. 
Let's configure the replication pair, selecting source and target. 
Following databases are described: 
1: DBVLAA (Oracle) 
2: DBVDEV (Oracle) 
Select source database: [1] 1 
Select target database: [2] 2 
Will DDL replication be enabled? (Yes/No) [Yes] 
Use fetcher to offload the mining to a different server? (Yes/No) [No] 
Should where clauses (and Event Streaming) include all columns, not just changed and PK? (Yes/No) [No] 
Would you like to encrypt the data across the network? (Yes/No) [No] 
Would you like to compress the data across the network? (Yes/No) [No] 
How long do you want to set the network timeouts. Recommended range between 60-300 seconds [60] 
Lock and copy the data initially one-by-one or at a single SCN? 
one-by-one : Lock tables one by one and capture SCN 
single-scn : One SCN for all tables 
ddl-only   : Only DDL script for target objects 
resetlogs  : Use SCN from last resetlogs operation (standby activation, rman incomplete recovery) 
no-lock    : Do not lock tables. Captures previous SCN of oldest active transaction. Requires pre-requisite running of pre-all.sh script                                                                            (one-by-one/single-scn/ddl-only/resetlogs/no-lock) [single-scn] 

What data instantiation script to create? 
dp_networklink : Data Pump with network link. No export datafile is created (APPLY.sh) 
dp_exp         : Data Pump with export datafile. Creates an export datafile (APPLY.sh) 
exp            : Pre-datapump exp/imp with export datafile (APPLY.sh) 
ddl_file       : DDL file created (APPLY.sql) 
ddl_run        : DDL is automatically executed on target 
load           : All replicated data is created and loaded automatically 
none                                                                                                                                                                                                                (dp_networklink/dp_exp/exp/ddl_file/ddl_run/load/none) [dp_networklink] 

Following replication pairs are now configured: 
1: DBVLAA (Oracle) ==> DBVDEV (Oracle), DDL: Yes, fetcher: No, process suffix: (no suffix), compression: No, encryption: No, network timeout: 60, prepare type: single-scn, data load: load 
Enter number of replication pair to modify it, or "add", or "done": [done] 

Step 3 - Replicated tables 
======================================== 
The third step is to choose the schemas and tables to be replicated. If the databases are reachable, the tables are checked for existence, datatype support, etc., schemas are queried for tables. Note that all 
messages are merely hints/warnings and may be ignored if issues are rectified before the scripts are actually executed. 

Following tables are defined for replication pairs: 
1: DBVLAA (Oracle) ==> DBVDEV (Oracle), DDL: Yes, suffix: (no suffix), prepare: single-scn 
  No tables defined. 
Enter number of replication pair to modify it, or "done": [1] 
Please enter list of all individual tables to be replicated. Enter schema name(s) only to replicate all tables in that schema. Use comma or space to delimit the entries. 
Enter the tables and schemas: [] scott 
Selected schemas: SCOTT 
Add more tables or schemas? (Yes/No) [No] 

To replicate changes of PL/SQL objects in schema(s), please enter the schemas to be replicated. Note that specifying any entry will cause additional privileges to be granted to dbvrep. Enter through a 
comma or space-delimited list. 
Enter the list of schemas (PL/SQL): [] 

You can also specify some advanced options: 
1. Exclude some tables from schema-level replication 
2. Rename schemas or tables. 
3. Specify filtering conditions. 
4. (Tables only) Configure Event Streaming; this does not maintain a copy of the source table, but logs all operations as separate entries. This is useful for ETL or as an audit trail. This usually 
requires adding of new columns (timestamps, old/new values etc.) to the target table. 
Specify rename name, filter condition or audit for any of the specified schemas? (Yes/No) [No] 
(PREPARE_SCHEMA_EXCEPTIONS) - Specify tables to exclude from PREPARE SCHEMA, if any: [] 

Following tables are defined for replication pairs: 
1: DBVLAA (Oracle) ==> DBVDEV (Oracle), DDL: Yes, suffix: (no suffix), prepare: single-scn 
  SCOTT(tables) 
Enter number of replication pair to modify it, or "done": [done] 

Step 4 - Process configuration 
======================================== 
The fourth step is to configure the replication processes for each replication. 

Following processes are defined: 
1: MINE on DBVLAA 
  Not configured. 
2: APPLY on DBVDEV 
  Not configured. 
Enter number of process to modify it, or "done": [1] 1 
Fully qualified name of the server for the process (usually co-located with the database, unless mine is offloaded using fetcher): [] dbvrep812-vip -- vip for hostname
Server type (Windows/Linux/Unix): [Linux] 
Enable email notifications about problems? (Yes/No) [No] 
Enable SNMP traps/notifications about problems? (Yes/No) [No] 
Directory with DDC file and default where to create log files etc. (recommended: same as global setting, if possible)? [/home/oracle/laa_rac] 

Following settings were pre-filled with defaults or your reloaded settings: 
---------------------------------------- 
[MINE_REMOTE_INTERFACE]: Network remote interface: dbvrep812-vip:7901 
[MINE_DATABASE]: Database TNS: DBVLAA 
[TNS_ADMIN]: tnsnames.ora path: /u01/app/oracle/product/11.2.0/db_1/network/admin 
[ORACLE_HOME]: ASM ORACLE_HOME: /u01/app/11.2.0/grid 
[MINE_PLOG]: Filemask for generated plogs: /home/oracle/laa_rac/mine/%S.%E (%S is sequence, %T thread, %F original filename (stripped extension), %P process type, %N process name, %E default extension) 
[LOG_FILE]: General log file: /home/oracle/laa_rac/log/dbvrep_%N_%D.%E 
[LOG_FILE_TRACE]: Error traces: /home/oracle/laa_rac/log/trace/dbvrep_%N_%D_%I_%U.%E 

Checking that these settings are valid... 
Do you want to change any of the settings? [No] 

Following processes are defined: 
1: MINE on DBVLAA 
  Host: dbvrep812-vip, SMTP: No, SNMP: No 
2: APPLY on DBVDEV 
  Not configured. 
Enter number of process to modify it, or "done": [2] 
Fully qualified name of the server for the process (usually co-located with the database, unless mine is offloaded using fetcher): [] dbvrep912-vip -- Target vip
Server type (Windows/Linux/Unix): [Linux] 
Enable email notifications about problems? (Yes/No) [No] 
Enable SNMP traps/notifications about problems? (Yes/No) [No] 
Directory with DDC file and default where to create log files etc. (recommended: same as global setting, if possible)? [/home/oracle/laa_rac] 

Following settings were pre-filled with defaults or your reloaded settings: 
---------------------------------------- 
[APPLY_REMOTE_INTERFACE]: Network remote interface: dbvrep912-vip:7902 
[APPLY_DATABASE]: Database TNS: DBVDEV 
[TNS_ADMIN]: tnsnames.ora path: /u01/app/oracle/product/11.2.0/db_1/network/admin 
[APPLY_STAGING_DIR]: Directory for received plogs: /home/oracle/laa_rac/apply 
[LOG_FILE]: General log file: /home/oracle/laa_rac/log/dbvrep_%N_%D.%E 
[LOG_FILE_TRACE]: Error traces: /home/oracle/laa_rac/log/trace/dbvrep_%N_%D_%I_%U.%E 

Checking that these settings are valid... 
Do you want to change any of the settings? [No] 

Following processes are defined: 
1: MINE on DBVLAA 
  Host: dbvrep812-vip, SMTP: No, SNMP: No 
2: APPLY on DBVDEV 
  Host: dbvrep912-vip, SMTP: No, SNMP: No
 
3. Run the generated ./laa_rac-all.sh script from 1 of the Target Nodes. 
oracle@kiwi81[/home/oracle/laa_rac]: ./laa_rac-all.sh 
Setting up Dbvisit Replicate configuration 
Configure database DBVLAA... 

This check fails if the DBID is not the expected one... 

Ok, check passed. 
Configure database DBVDEV... 

This check fails if the DBID is not the expected one... 

Ok, check passed. 
Object grants for database DBVLAA... 

Object grants for database DBVDEV... 

Setting up the configuration 
Initializing......done 
DDC loaded from database (0 variables). 
Dbvisit Replicate version 2.8.04_rc1 
Copyright (C) Dbvisit Software Limited.  All rights reserved. 
DDC file /home/oracle/laa_rac/config/laa_rac-onetime.ddc loaded. 
MINE: Cannot determine Dbvisit Replicate dictionary version. (no 
dictionary exists) 
APPLY: Cannot determine Dbvisit Replicate dictionary version. (no 
dictionary exists) 
dbvrep> #clear the no-DDC-DB-available warning 
dbvrep> process clear previous warnings 
dbvrep> set ON_WARNING SKIP 
Variable ON_WARNING set to SKIP for process *. 
dbvrep> set ON_ERROR EXIT 
Variable ON_ERROR set to EXIT for process *. 
dbvrep> 
dbvrep> # Configuring default processes 
dbvrep> choose process MINE 
Process type MINE set to: MINE. 
dbvrep> choose process APPLY 
Process type APPLY set to: APPLY. 
dbvrep> PROCESS SWITCH_REDOLOG 
Redo log switch requested.dbvrep> PROCESS SETUP MINE DROP DICTIONARY 
0 dictionary objects dropped. 
dbvrep> PROCESS SETUP MINE CREATE DICTIONARY 
dbvrep> PROCESS SETUP MINE LOAD DICTIONARY 
Oldest active transaction SCN: 1179659 (no active transaction) 
Supplemental logging on database set. 
Loading dictionary table DBRSCCOL$ 
Loading dictionary table DBRSCDEF$ 
Loading dictionary table DBRSCOL$ 
Loading dictionary table DBRSIND$ 
Loading dictionary table DBRSOBJ$ 
Loading dictionary table DBRSTAB$ 
Loading dictionary table DBRSUSER$ 
Loading dictionary table DBRSV_$DATABASE 
dbvrep> PROCESS SETUP APPLY DROP DICTIONARY 
0 dictionary objects dropped. 
dbvrep> PROCESS SETUP APPLY CREATE DICTIONARY 
dbvrep> PROCESS SETUP APPLY LOAD DICTIONARY 
dbvrep> PROCESS PREPARE_DP SETUP CLEAR 
dbvrep> PROCESS SETUP PAIR MINE AND APPLY 
Applier SCN set (start=1179673, current=1179673). 
dbvrep> SET APPLY.INSTANTIATE_SCN NOW 
Variable INSTANTIATE_SCN set to NOW for process APPLY. 
dbvrep> SET MINE._PREPARE_SUPLOG_TYPE PK 
Variable _PREPARE_SUPLOG_TYPE set to PK for process MINE. 
dbvrep> EXCLUDE CREATE TABLE %.DBMS_TABCOMP_TEMP_UNCMP #Ignore tables 
created by Compression Advisor 
Exclude rule created. 
dbvrep> EXCLUDE CREATE TABLE %.DBMS_TABCOMP_TEMP_CMP #Ignore tables 
created by Compression Advisor 
Exclude rule created. 
dbvrep> EXCLUDE CREATE TABLE %.SCHEDULER$_% #Ignore tables created by 
Oracle scheduler (also used by schema/full expdp/impdp) 
Exclude rule created. 
dbvrep> EXCLUDE CREATE TABLE %.CMP1$% #Ignore tables created by 
Compression Advisor since 11.2.0.4 
Exclude rule created. 
dbvrep> EXCLUDE CREATE TABLE %.CMP2$% #Ignore tables created by 
Compression Advisor since 11.2.0.4 
Exclude rule created. 
dbvrep> EXCLUDE CREATE TABLE %.CMP3$% #Ignore tables created by 
Compression Advisor since 11.2.0.4 
Exclude rule created. 
dbvrep> EXCLUDE CREATE TABLE %.CMP4$% #Ignore tables created by 
Compression Advisor since 11.2.0.4 
Exclude rule created. 
dbvrep> SET PREPARE_SCHEMA_EXCEPTIONS none 
Variable PREPARE_SCHEMA_EXCEPTIONS set to none for process *. 
dbvrep> PROCESS SUPPLEMENTAL LOGGING SCHEMA "SCOTT" ENABLE PRIMARY KEY 
dbvrep> PROCESS SWITCH_REDOLOG 
Redo log switch requested.dbvrep> PROCESS WAIT_SCN_FLIP 
Waited 1 seconds until scn_to_timestamp changed. 
dbvrep> #single-scn instantiation: lock all tables and schemas 
dbvrep> PROCESS LOCK SCHEMAS "SCOTT" 
Locking all schemas. 
...locked 4 of 4 tables from SCOTT schema. 
Lock done. 
dbvrep> #single-scn instantiation: unlock all tables and schemas, but 
keep the SCN 
dbvrep> PROCESS LOCK RELEASE LOCKS 
dbvrep> 
dbvrep> #prepare the tables (we use OFFLINE as neither MINE nor APPLY 
is running; with OFFLINE we won't wait on network timeout) 
dbvrep> PREPARE OFFLINE SCHEMA "SCOTT" 
Table SCOTT.BONUS instantiated at SCN 1187845 
Table SCOTT.DEPT instantiated at SCN 1187845 
Table SCOTT.EMP instantiated at SCN 1187845 
Table SCOTT.SALGRADE instantiated at SCN 1187845 
dbvrep> 
dbvrep> #single-scn instantiation: unlock all tables and schemas, 
forget the SCN (so it does not affect any further PREPARE statements) 
dbvrep> PROCESS LOCK CLEAR SCN 
dbvrep> PROCESS SWITCH_REDOLOG 
Redo log switch requested.dbvrep> #prepare script for instantiation 
dbvrep> PROCESS PREPARE_DP WRITE DP_NETWORKLINK DIRECTORY DATA_PUMP_DIR 
FILE /home/oracle/laa_rac/APPLY.sh DBLINK DBVLAA USERID 
SYSTEM/kiwi123@DBVDEV 
Created Data Pump script /home/oracle/laa_rac/APPLY.sh, using network 
import. 
dbvrep> create ddcdb from ddcfile 
DDC loaded into database (406 variables). 
dbvrep> set ON_WARNING SKIP 
Variable ON_WARNING set to SKIP for process *. 
dbvrep> set ON_ERROR SKIP 
Variable ON_ERROR set to SKIP for process *. 
OK-0: Completed successfully. 
WARN-1850: No DDC DB available, dictionary table does not exist. 
These steps are required after the laa_rac-all.sh script runs: 

1) Create the necessary directory(ies) on the servers: 
dbvrep812-vip: /home/oracle/laa_rac 
dbvrep912-vip: /home/oracle/laa_rac 

2) Copy the DDC files to the server(s) where the processes will run: 
dbvrep912-vip: /home/oracle/laa_rac/laa_rac-APPLY.ddc 
dbvrep812-vip: /home/oracle/laa_rac/laa_rac-MINE.ddc 

3) Review that path to dbvrep executable is correct in the run scripts: 
/home/oracle/laa_rac/laa_rac-run-dbvrep812-vip.sh 
/home/oracle/laa_rac/laa_rac-run-dbvrep912-vip.sh 

4) Copy the run script to the server(s) where the processes will run: 
dbvrep812-vip: /home/oracle/laa_rac/laa_rac-run-dbvrep812-vip.sh 
dbvrep912-vip: /home/oracle/laa_rac/laa_rac-run-dbvrep912-vip.sh 

5) Ensure firewall is open for listen interfaces 0.0.0.0:7902, 0.0.0.0:7901 used by the processes. 

6) Make sure the data on apply are in sync as of time when setup was run. 
Scripts for Data Pump/export/DDL were created as requested: 

Create referenced database links (if any) before running the scripts. 
/home/oracle/laa_rac/APPLY.sh 

7) Start the replication processes on all servers: 
dbvrep812-vip: /home/oracle/laa_rac/laa_rac-run-dbvrep812-vip.sh 
dbvrep912-vip: /home/oracle/laa_rac/laa_rac-run-dbvrep912-vip.sh 

8) Start the console to monitor the progress: 
/home/oracle/laa_rac/start-console.sh 

The above list is stored in /home/oracle/laa_rac/Nextsteps.txt.  
 
4. Perform the actions listed in the /home/oracle/laa_rac/Nextsteps.txt
oracle@kiwi81[/home/oracle/laa_rac]: scp laa_rac-run-dbvrep912-vip.sh oracle@kiwi91:/home/oracle/laa_rac/laa_rac-run-dbvrep912-vip.sh                                                                                                                                            
oracle@kiwi81[/home/oracle/laa_rac]: scp laa_rac-APPLY.ddc oracle@kiwi91:/home/oracle/laa_rac/laa_rac-APPLY.ddc                                                                                                                                    
 
5. Before starting the MINE and APPLY processes, run crsctl to check the locations of the vip and the oracle service. Then start processes on same node.   If the vip and services are running on different nodes, relocate one to be the same as the other.
oracle@kiwi81[/home/oracle]: . oraenv
ORACLE_SID = [LAA1] ? +ASM1
oracle@kiwi81[/home/oracle]: crsctl status resource dbvrep812-vip
NAME=dbvrep812-vip
TYPE=app.appvip_net1.type
TARGET=ONLINE
STATE=ONLINE on kiwi81
oracle@kiwi81[/home/oracle]: crsctl status resource ora.dev.dbvlaa.svc
NAME=ora.dev.dbvlaa.svc
TYPE=ora.service.type
TARGET=ONLINE
STATE=ONLINE on kiwi81
oracle@kiwi81[/home/oracle]: ssh kiwi91
oracle@kiwi91[/home/oracle]: crsctl status resource dbvrep912-vip
NAME=dbvrep912-vip
TYPE=app.appvip_net1.type
TARGET=ONLINE
STATE=ONLINE on kiwi91
oracle@kiwi91[/home/oracle]: crsctl status resource ora.dev.dbvdev.svc
NAME=ora.dev.dbvdev.svc
TYPE=ora.service.type
TARGET=ONLINE
STATE=ONLINE on kiwi91
oracle@kiwi91[/home/oracle]: exit
-- Running on 1 node in each cluster
oracle@kiwi81[/home/oracle]: cd laa_rac
oracle@kiwi81[/home/oracle/laa_rac]: ./laa_rac-run-dbvrep812-vip.sh Initializing......done DDC loaded from database (406 variables). Dbvisit Replicate version 2.8.04_rc1 Copyright (C) Dbvisit Software Limited.  All rights reserved. DDC file /home/oracle/laa_rac/laa_rac-MINE.ddc loaded. Starting process MINE...Created directory /home/oracle/laa_rac/ddc_backup Created directory /home/oracle/laa_rac/log/trace/ Created directory /home/oracle/laa_rac/mine/ started oracle@kiwi81[/home/oracle/laa_rac]: ssh kiwi91 Start Apply Process Running on Node 1 on Target (kiwi91:DEV) oracle@kiwi91[/home/oracle]: cd laa_rac
oracle@kiwi91[/home/oracle/laa_rac]: ./laa_rac-run-dbvrep912-vip.sh
Initializing......done DDC loaded from database (407 variables). Dbvisit Replicate version 2.8.04_rc1 Copyright (C) Dbvisit Software Limited.  All rights reserved. DDC file /home/oracle/laa_rac/laa_rac-APPLY.ddc loaded. Starting process APPLY...Created directory /home/oracle/laa_rac/ddc_backup Created directory /home/oracle/laa_rac/log/ Created directory /home/oracle/laa_rac/log/trace/ Created directory /home/oracle/laa_rac/apply started
 
6. Check replication Status by running the console from the Node running the MINE process
oracle@kiwi81[/home/oracle/laa_rac]: ./start-console.sh 
Initializing......done 
DDC loaded from database (407 variables). 
Dbvisit Replicate version 2.8.04_rc1 
Copyright (C) Dbvisit Software Limited.  All rights reserved. 
/ Dbvisit Replicate 2.8.04_rc1(MAX edition) - Evaluation License expires in 30 days 
MINE is running. Currently at plog 20 (redo sequence 23 [1] 13 [2]) and SCN 1200910 (03/13/2017 15:32:28). 
APPLY is running. Currently at plog 20 and SCN 1200877 (03/13/2017 15:32:24). 
Progress of replication laa_rac:MINE->APPLY: total/this execution 
-------------------------------------------------------------------------------------- 
SCOTT.DEPT:                   100%  Mine:1/1             Unrecov:0/0         Applied:1/1         Conflicts:0/0       Last:13/03/2017 15:30:19/OK 
SCOTT.EMP:                    100%  Mine:1/1             Unrecov:0/0         Applied:1/1         Conflicts:0/0       Last:13/03/2017 15:28:48/OK 
-------------------------------------------------------------------------------------- 
2 tables listed. 
dbvrep> 
 
7. Perform the following steps to relocate the service and vip to the 2nd node (kiwi82). The restart the MINE process from the 2nd node.
  1. Shutdown Mine Process
  2. Relocate Database Service and VIP
  3. Start MINE process from other node
  4. Confirm replication is running 

Step 1 - Shutdown Mine Process ===============================

dbvrep> shutdown mine
Upon restart MINE will go back to redolog 23 (thread 1), going back no logs.
Upon restart MINE will go back to redolog 12 (thread 2), going back 1 log.
Are you sure you want to shutdown? (Yes/No) [No] Yes
Dbvisit Replicate MINE process shutting down.

Step 2 - Relocate Database Service and vip
=========================================== 
Log in as Root and then relocate VIP
oracle@kiwi82[/home/oracle/laa_rac]: su -
root@kiwi81[/root]: crsctl status resource dbvrep812-vip
NAME=dbvrep812-vip
TYPE=app.appvip_net1.type
TARGET=ONLINE STATE=ONLINE on kiwi81
root@kiwi81[/root]: crsctl relocate resource dbvrep812-vip
CRS-2673: Attempting to stop 'dbvrep812-vip' on 'kiwi81'
CRS-2677: Stop of 'dbvrep812-vip' on 'kiwi81' succeeded
CRS-2672: Attempting to start 'dbvrep812-vip' on 'kiwi82'
CRS-2676: Start of 'dbvrep812-vip' on 'kiwi82' succeeded
root@kiwi81[/root]: crsctl status resource dbvrep812-vip
NAME=dbvrep812-vip
TYPE=app.appvip_net1.type
TARGET=ONLINE
STATE=ONLINE on kiwi82

root@kiwi81[/root]: exit

## As the oracle user relocate the service DBVLAA to node 2 (kiwi82)

oracle@kiwi81[/home/oracle/laa_rac]:
srvctl relocate service -d LAA -s DBVLAA -i LAA1 -t LAA2
oracle@kiwi81[/home/oracle/laa_rac]: srvctl status service -d PROD -s DBVLAA
Service DBVLAA is running on instance(s) LAA2


Step 3 - Start MINE process from 2nd node
==========================================
oracle@kiwi82[/home/oracle/laa_rac]: ./laa_rac-run-dbvrep812-vip.sh
Initializing......done
DDC loaded from database (406 variables).
Dbvisit Replicate version 2.8.04_rc1
Copyright (C) Dbvisit Software Limited.  All rights reserved.
DDC file /home/oracle/laa_rac/laa_rac-MINE.ddc loaded.
Starting process MINE...started
 
Step 4 - Confirm replication is running via the Console
========================================================
oracle@kiwi82[/home/oracle/laa_rac]: ./start-console.sh

DDC loaded from database (407 variables). Dbvisit Replicate version 2.8.04_rc1 Copyright (C) Dbvisit Software Limited.  All rights reserved. | Dbvisit Replicate 2.8.04_rc1(MAX edition) - Evaluation License expires in 30 days MINE is running. Currently at plog 25 (redo sequence 25 [1] 16 [2]) and SCN 1207240. APPLY is running. Currently at plog 25 and SCN 1206804 (03/13/2017 16:39:59). Progress of replication laa_rac:MINE->APPLY: total/this execution ------------------------------------------------------------------------------------------------- SCOTT.DEPT:                   150%  Mine:2/0             Unrecov:0/0         Applied:3/3         Conflicts:0/0       Last:13/03/2017 16:34:45/OK SCOTT.EMP:                    100%  Mine:1/0             Unrecov:0/0         Applied:1/1         Conflicts:0/0       Last:13/03/2017 15:28:48/OK dbvrep>
### REPEAT STEPS on NODES KIWI91/92 for APPLY PROCESS.

8. Before adding replication to the cluster resources we need to configure an action script to start/stop the MINE (kiwi81/kiwi82) and APPLY (kiwi91/kiwi92) processes. The scripts for this example are created in the /home/oracle/laa_rac directory on both clusters and are owned by user oracle with execute permissions.
KIWI81/82 --  MINE Action Script


#!/bin/bash
#
# Description:
# This is a sample action script for adding the Dbvisit Replicate Mine process
# as a cluster resource
# Note:
# This script needs to be updated to reflect the required environment details
# and should be tested before running on production systems
#

############################
## Echo usage of program
############################
usage ()
{
program=`basename $0`

cat <<EOF
Usage:
${program} start|stop|check|clean
EOF
exit 1
}

###########################################
# Function to change database environments
###########################################
set_env ()
{
export ORAENV_ASK=NO
export ORACLE_SID=$1
. oraenv >> /dev/null
}

###########################################
# Function to setup parameter variables
###########################################
setup_parameters ()
{
DBVREP=/usr/bin/dbvrep
DDCFILE=/home/oracle/laa_rac/laa_rac-MINE.ddc
PROCESS=MINE
db=LAA
set_env $db
}

###########################################
# Function used to stop replicate process
###########################################
stop_replicate ()
{
echo "Stopping: $DBVREP --ddcfile $DDCFILE shutdown $PROCESS"
$DBVREP --ddcfile $DDCFILE shutdown $PROCESS

for c1 in {1..10}
do
# process might take few seconds to shutdown, check every 2 seconds 10 times
sleep 3
check=`ps -ef | grep $PROCESS | grep -v grep | wc -l`
if [ $check -eq 0 ]
then
echo "Completed: $DBVREP --ddcfile $DDCFILE shutdown $PROCESS"
exit 0
fi
echo "... sleep for 3 sec and check again"
done

# if we get to this stage the above process did not
# stop as expected, exit with status 1
echo "Error: $DBVREP --ddcfile $DDCFILE shutdown $PROCESS"
exit 1
}

###########################################
# Function used to stop replicate process
###########################################
start_replicate ()
{
echo "Starting: $DBVREP --daemon --ddcfile $DDCFILE start $PROCESS"
$DBVREP --daemon --ddcfile $DDCFILE start $PROCESS
}

###########################################
# Function used to stop replicate process
###########################################
check_replicate ()
{
echo "Starting: $DBVREP --ddcfile $DDCFILE healthcheck for $PROCESS"
$DBVREP --ddcfile $DDCFILE healthcheck for $PROCESS
}

#################
## Main Section
#################

## Basic test number of input values
if test $# -ne 1
then
usage
exit 1
fi

# First call function to set the environment or specific global variables
setup_parameters

#
# This is logged to CRSD agent log file
echo "`date` Action script '$_CRS_ACTION_SCRIPT' for resource [$_CRS_NAME] called for action $1"
#

case "$1" in
'start') start_replicate
;;

'stop') stop_replicate
;;

'check') check_replicate
;;

'clean')
exit 0
;;

*) usage
;;

esac oracle@kiwi82[/home/oracle/laa_rac]: chmod u+rwx action_script.scr
-- We can test the action script by using it to stop/start the MINE process on node kiwi82 as dbvrep812-vip
and service DBVLAA are still running from before on this node.

oracle@kiwi82[/home/oracle/laa_rac]: ./action_script.scr
Usage:
action_script.scr start|stop|check|clean
oracle@kiwi82[/home/oracle/laa_rac]: ./action_script.scr start
Wed Apr 5 15:07:37 NZST 2017 Action script '' for resource [] called for action start
Starting: /usr/bin/dbvrep --daemon --ddcfile /home/oracle/laa_rac/laa_rac-MINE.ddc start MINE
Initializing......done
DDC loaded from database (407 variables).
Dbvisit Replicate version 2.8.04_126_g0810a26
Copyright (C) Dbvisit Software Limited. All rights reserved.
DDC file /home/oracle/laa_rac/laa_rac-MINE.ddc loaded.
Starting process MINE...started
oracle@kiwi82[/home/oracle/laa_rac]: ./action_script.scr check
Wed Apr 5 15:07:59 NZST 2017 Action script '' for resource [] called for action check
Starting: /usr/bin/dbvrep --ddcfile /home/oracle/laa_rac/laa_rac-MINE.ddc healthcheck for MINE
Initializing......done
DDC loaded from database (407 variables).
Dbvisit Replicate version 2.8.04_126_g0810a26
Copyright (C) Dbvisit Software Limited. All rights reserved.
DDC file /home/oracle/laa_rac/laa_rac-MINE.ddc loaded.
oracle@kiwi82[/home/oracle/laa_rac]: ps -ef|grep -i dbv
oracle 23448 1 23 15:07 ? 00:00:10 dbvrep MINE laa --daemon --ddcfile /home/oracle/laa_rac/laa_rac-MINE.ddc start MINE
oracle 23812 3842 0 15:08 pts/0 00:00:00 grep -i dbv Repeat on KIWI 91/92 -- APPLY Action Script

#!/bin/bash
#
# Description:
# This is a sample action script for adding the Dbvisit Replicate Mine process
# as a cluster resource
# Note:
# This script needs to be updated to reflect the required environment details
# and should be tested before running on production systems
#

############################
## Echo usage of program
############################
usage ()
{
program=`basename $0`

cat <<EOF
Usage:
${program} start|stop|check|clean
EOF
exit 1
}

###########################################
# Function to change database environments
###########################################
set_env ()
{
export ORAENV_ASK=NO
export ORACLE_SID=$1
. oraenv >> /dev/null
}

###########################################
# Function to setup parameter variables
###########################################
setup_parameters ()
{
DBVREP=/usr/bin/dbvrep
DDCFILE=/home/oracle/laa_rac/laa_rac-APPLY.ddc
PROCESS=APPLY
db=DEV
set_env $db
}

###########################################
# Function used to stop replicate process
###########################################
stop_replicate ()
{
echo "Stopping: $DBVREP --ddcfile $DDCFILE shutdown $PROCESS"
$DBVREP --ddcfile $DDCFILE shutdown $PROCESS

for c1 in {1..10}
do
# process might take few seconds to shutdown, check every 2 seconds 10 times
sleep 3
check=`ps -ef | grep $PROCESS | grep -v grep | wc -l`
if [ $check -eq 0 ]
then
echo "Completed: $DBVREP --ddcfile $DDCFILE shutdown $PROCESS"
exit 0
fi
echo "... sleep for 3 sec and check again"
done

# if we get to this stage the above process did not
# stop as expected, exit with status 1
echo "Error: $DBVREP --ddcfile $DDCFILE shutdown $PROCESS"
exit 1
}

###########################################
# Function used to stop replicate process
###########################################
start_replicate ()
{
echo "Starting: $DBVREP --daemon --ddcfile $DDCFILE start $PROCESS"
$DBVREP --daemon --ddcfile $DDCFILE start $PROCESS
}

###########################################
# Function used to stop replicate process
###########################################
check_replicate ()
{
echo "Starting: $DBVREP --ddcfile $DDCFILE healthcheck for $PROCESS"
$DBVREP --ddcfile $DDCFILE healthcheck for $PROCESS
}

#################
## Main Section
#################

## Basic test number of input values
if test $# -ne 1
then
usage
exit 1
fi

# First call function to set the environment or specific global variables
setup_parameters

#
# This is logged to CRSD agent log file
echo "`date` Action script '$_CRS_ACTION_SCRIPT' for resource [$_CRS_NAME] called for action $1"
#

case "$1" in
'start') start_replicate
;;

'stop') stop_replicate
;;

'check') check_replicate
;;

'clean')
exit 0
;;

*) usage
;;

esac

oracle@kiwi91[/home/oracle/laa_rac]: chmod u+rwx action_script.scr

-- Similarly Testing Stop with action_script on NODE kiwi91

oracle@kiwi91[/home/oracle/laa_rac]: ./action_script.scr stop
Wed Apr 5 15:17:03 NZST 2017 Action script '' for resource [] called for action stop
Stopping: /usr/bin/dbvrep --ddcfile /home/oracle/laa_rac/laa_rac-APPLY.ddc shutdown APPLY
Initializing......done
DDC loaded from database (407 variables).
Dbvisit Replicate version 2.8.04_126_g0810a26
Copyright (C) Dbvisit Software Limited. All rights reserved.
DDC file /home/oracle/laa_rac/laa_rac-APPLY.ddc loaded.
Dbvisit Replicate APPLY process shutting down.
OK-0: Completed successfully.
Completed: /usr/bin/dbvrep --ddcfile /home/oracle/laa_rac/laa_rac-APPLY.ddc shutdown APPLY
oracle@kiwi91[/home/oracle/laa_rac]: ps -ef|grep -i dbv
oracle 6548 5782 0 15:17 pts/0 00:00:00 grep -i dbv

9. Now we can proceed to add the MINE process as a clustered Resource ( dbvMine on NODE kiwi82 as root)
oracle@kiwi82[/home/oracle/laa_rac]: su -
Password:
root@kiwi82[/root]: . oraenv
ORACLE_SID = [root] ? +ASM2
The Oracle base has been set to /u01/app/grid
root@kiwi82[/root]: crsctl add resource dbvMine -type cluster_resource -attr "ACTION_SCRIPT=/home/oracle/laa_rac/action_script.scr, CHECK_INTERVAL=30, START_DEPENDENCIES='hard(dbvrep812-vip,ora.laa.db,ora.asm,ora.laa.dbvlaa.svc) pullup(ora.laa.dbvlaa.svc,dbvrep812-vip)',STOP_DEPENDENCIES='hard(ora.laa.dbvlaa.svc,dbvrep812-vip)' ACL='owner:oracle:rwx,pgrp:oinstall:rwx,other::r--' PLACEMENT='favored' HOSTING_MEMBERS='kiwi81'"
root@kiwi82[/root]: crsctl status resource dbvMine
NAME=dbvMine
TYPE=cluster_resource
TARGET=OFFLINE
STATE=OFFLINE
root@kiwi82[/root]: crsctl start resource dbvMine
CRS-2672: Attempting to start 'dbvMine' on 'kiwi82'
CRS-2676: Start of 'dbvMine' on 'kiwi82' succeeded
root@kiwi82[/root]: ps -ef|grep -i dbv
oracle   32235     1 48 15:32 ?        00:00:16 dbvrep MINE laa --daemon --ddcfile /home/oracle/laa_rac/laa_rac-MINE.ddc start MINE
root     32456 29931  0 15:32 pts/0    00:00:00 grep -i dbv
root@kiwi82[/root]: crsctl stop resource dbvMine
CRS-2673: Attempting to stop 'dbvMine' on 'kiwi82'
CRS-2677: Stop of 'dbvMine' on 'kiwi82' succeeded
root@kiwi82[/root]: ps -ef|grep -i dbv
root     32174 29931  0 15:32 pts/0    00:00:00 grep -i dbv
root@kiwi82[/root]: crsctl start resource dbvMine CRS-2672: Attempting to start 'dbvMine' on 'kiwi82' CRS-2676: Start of 'dbvMine' on 'kiwi82' succeeded root@kiwi82[/root]: crsctl status resource dbvMine
NAME=dbvMine
TYPE=cluster_resource
TARGET=ONLINE
STATE=ONLINE on kiwi82
10. Now we can repeat and add the APPLY process as a clustered Resource
root@kiwi91[/root]: . oraenv
ORACLE_SID = [root] ? +ASM1
The Oracle base has been set to /u01/app/grid
root@kiwi91[/root]: crsctl add resource dbvApply -type cluster_resource -attr "ACTION_SCRIPT=/home/oracle/laa_rac/action_script.scr, CHECK_INTERVAL=30, START_DEPENDENCIES='hard(dbvrep912-vip,ora.dev.db,ora.asm,ora.dev.dbvdev.svc) pullup(ora.dev.dbvdev.svc,dbvrep912-vip)',STOP_DEPENDENCIES='hard(ora.dev.dbvdev.svc,dbvrep912-vip)' ACL='owner:oracle:rwx,pgrp:oinstall:rwx,other::r--' PLACEMENT='favored' HOSTING_MEMBERS='kiwi91'"
root@kiwi91[/root]: crsctl status resource dbvApply
NAME=dbvApply
TYPE=cluster_resource
TARGET=OFFLINE
STATE=OFFLINE
root@kiwi91[/root]: crsctl start resource dbvApply
CRS-2672: Attempting to start 'dbvApply' on 'kiwi91'
CRS-2676: Start of 'dbvApply' on 'kiwi91' succeeded
root@kiwi91[/root]: ps -ef|grep -i dbv
oracle   29109     1 27 17:24 ?        00:00:10 dbvrep APPLY la --daemon --ddcfile /home/oracle/laa_rac/laa_rac-APPLY.ddc start APPLY
root     29368 27631  0 17:24 pts/0    00:00:00 grep -i dbv
root@kiwi91[/root]: crsctl stop resource dbvApply
CRS-2673: Attempting to stop 'dbvApply' on 'kiwi91'
CRS-2677: Stop of 'dbvApply' on 'kiwi91' succeeded
root@kiwi91[/root]: ps -ef|grep -i dbv
root     30218 27631  0 17:26 pts/0    00:00:00 grep -i dbv
root@kiwi91[/root]: crsctl start resource dbvApply
CRS-2672: Attempting to start 'dbvApply' on 'kiwi91'
CRS-2676: Start of 'dbvApply' on 'kiwi91' succeeded
root@kiwi91[/root]: crsctl status resource dbvApply
NAME=dbvApply
TYPE=cluster_resource
TARGET=ONLINE
STATE=ONLINE on kiwi91
root@kiwi91[/root]:
11. Check the Status of Replication and run a few inserts/updates
oracle@kiwi82[/home/oracle/laa_rac]: ./start-console.sh
Initializing......done
DDC loaded from database (407 variables).
Dbvisit Replicate version 2.8.04_rc1
Copyright (C) Dbvisit Software Limited.  All rights reserved.
- Dbvisit Replicate 2.8.04_rc1(MAX edition) - Evaluation License expires in 30 days
MINE is running. Currently at plog 89 (redo sequence 59 [1] 50 [2]) and SCN 1371036 (04/05/2017 17:59:41).
APPLY is running. Currently at plog 89 and SCN 1371002 (04/05/2017 17:59:38).
Progress of replication laa_rac:MINE->APPLY: total/this execution
--------------------------------------------------------------------------------------------------------------------------------------------
SCOTT.DEPT:                   100%  Mine:5/0             Unrecov:0/0         Applied:5/0         Conflicts:0/0       Last:--/--
SCOTT.EMP:                    100%  Mine:33/10           Unrecov:0/0         Applied:33/10       Conflicts:0/0       Last:05/04/2017 17:58:45/OK
SCOTT.SALGRADE:               100%  Mine:5/0             Unrecov:0/0         Applied:5/0         Conflicts:0/0       Last:--/--
--------------------------------------------------------------------------------------------------------------------------------------------
3 tables listed.
12. Testing Failover of the dbvMine and dbvApply resources. Because of hard dependencies on the vip and service the -f (force) option must be used.
 
-- 1. dbvMine Relocate from kiwi82 to kiwi81
root@kiwi82[/root]: . oraenv
ORACLE_SID = [root] ? +ASM2
The Oracle base has been set to /u01/app/grid
root@kiwi82[/root]: crsctl relocate resource dbvMine -n kiwi81 -f
RS-2673: Attempting to stop 'dbvMine' on 'kiwi82'
CRS-2677: Stop of 'dbvMine' on 'kiwi82' succeeded
CRS-2673: Attempting to stop 'dbvrep812-vip' on 'kiwi82'
CRS-2673: Attempting to stop 'ora.laa.dbvlaa.svc' on 'kiwi82'
CRS-2677: Stop of 'ora.laa.dbvlaa.svc' on 'kiwi82' succeeded
CRS-2672: Attempting to start 'ora.laa.dbvlaa.svc' on 'kiwi81'
CRS-2676: Start of 'ora.laa.dbvlaa.svc' on 'kiwi81' succeeded
CRS-2677: Stop of 'dbvrep812-vip' on 'kiwi82' succeeded
CRS-2672: Attempting to start 'dbvrep812-vip' on 'kiwi81'
CRS-2676: Start of 'dbvrep812-vip' on 'kiwi81' succeeded
CRS-2672: Attempting to start 'dbvMine' on 'kiwi81'
CRS-2676: Start of 'dbvMine' on 'kiwi81' succeeded
root@kiwi82[/root]: crsctl status resource dbvMine
NAME=dbvMine
TYPE=cluster_resource
TARGET=ONLINE
STATE=ONLINE on kiwi81
oracle@kiwi81[/home/oracle/laa_rac/sql]: ps -ef|grep -i dbv
oracle    2359     1 15 20:30 ?        00:00:13 dbvrep MINE laa --daemon --ddcfile /home/oracle/laa_rac/laa_rac-MINE.ddc start MINE
oracle@kiwi81[/home/oracle/laa_rac/sql]: . oraenv
ORACLE_SID = [+ASM1] ? LAA1
The Oracle base has been changed from /u01/app/grid to /u01/app/oracle
oracle@kiwi81[/home/oracle/laa_rac]: ./start-console.sh
Initializing......done
DDC loaded from database (407 variables).
Dbvisit Replicate version 2.8.04_rc1
Copyright (C) Dbvisit Software Limited.  All rights reserved.
- Dbvisit Replicate 2.8.04_126_rc1(MAX edition) - Evaluation License expires in 30 days
MINE is running. Currently at plog 141 (redo sequence 85 [1] 76 [2]) and SCN 1441295 (04/05/2017 20:35:10).
APPLY is running. Currently at plog 141 and SCN 1441245 (04/05/2017 20:35:02).
Progress of replication laa_rac:MINE->APPLY: total/this execution
--------------------------------------------------------------------------------------------------------------------------------------------
SCOTT.DEPT:                   100%  Mine:6/0             Unrecov:0/0         Applied:6/1         Conflicts:0/0       Last:05/04/2017 18:04:33/OK
SCOTT.EMP:                    100%  Mine:35/2            Unrecov:0/0         Applied:35/12       Conflicts:0/0       Last:05/04/2017 20:35:02/OK
SCOTT.SALGRADE:               100%  Mine:7/0             Unrecov:0/0         Applied:7/2         Conflicts:0/0       Last:05/04/2017 18:03:47/OK
--------------------------------------------------------------------------------------------------------------------------------------------
3 tables listed.
-- 2. dbvApply Relocate from kiwi91 to kiwi92 continuing to monitor replication in session above.
 
root@kiwi91[/root]: crsctl relocate resource dbvApply -n kiwi92 -f
CRS-2673: Attempting to stop 'dbvApply' on 'kiwi91'
CRS-2677: Stop of 'dbvApply' on 'kiwi91' succeeded
CRS-2673: Attempting to stop 'dbvrep912-vip' on 'kiwi91'
CRS-2673: Attempting to stop 'ora.dev.dbvdev.svc' on 'kiwi91'
CRS-2677: Stop of 'ora.dev.dbvdev.svc' on 'kiwi91' succeeded
CRS-2672: Attempting to start 'ora.dev.dbvdev.svc' on 'kiwi92'
CRS-2676: Start of 'ora.dev.dbvdev.svc' on 'kiwi92' succeeded
CRS-2677: Stop of 'dbvrep912-vip' on 'kiwi91' succeeded
CRS-2672: Attempting to start 'dbvrep912-vip' on 'kiwi92'
CRS-2676: Start of 'dbvrep912-vip' on 'kiwi92' succeeded
CRS-2672: Attempting to start 'dbvApply' on 'kiwi92'
CRS-2676: Start of 'dbvApply' on 'kiwi92' succeeded
root@kiwi91[/root]: crsctl status resource dbvApply
NAME=dbvApply
TYPE=cluster_resource
TARGET=ONLINE
STATE=ONLINE on kiwi92
 
13. Final Tests.  Shutdown/Crash of 1 node from each cluster by running the init 6 command as root. Also continuing to monitor replication status from the console in a shell seperate window. 
-- Test 1 Crash kiwi92
root@kiwi92[/root]: init 6
root@kiwi91[/root]: crsctl status resource dbvApply
NAME=dbvApply
TYPE=cluster_resource
TARGET=ONLINE
STATE=ONLINE on kiwi91
-- Test 2 Crash kiwi81
root@kiwi81[/root]: init 6
root@kiwi82[/root]: . oraenv
ORACLE_SID = [root] ? +ASM1
The Oracle base has been set to /u01/app/grid
root@kiwi82[/root]: crsctl status resource dbvMine
NAME=dbvMine
TYPE=cluster_resource
TARGET=ONLINE
STATE=ONLINE on kiwi82

Conclusion
 
In summary the process is really straight forward and works very well with a slick and seamless failover between the 2 RAC nodes of the SOURCE and also between the 2 RAC nodes of the TARGET.
Have more questions? Submit a request

Comments