We're here to help!

oracle@kiwi81[/home/oracle]: dbvrep 
Initializing......done 
Dbvisit Replicate version 2.8.04_rc1 
Copyright (C) Dbvisit Software Limited.  All rights reserved. 
No DDC file loaded. 
Run "setup wizard" to start the configuration wizard or try "help" to see all commands available. 
dbvrep> setup wizard 
This wizard configures Dbvisit Replicate. 

The setup wizard creates configuration scripts, which need to be run after the wizard ends. No changes to the databases are made before that. 

The progress is saved every time a list of databases, replications, etc. is shown. It will be re-read if wizard is restarted and the same DDC name and script path is selected. 
Run the wizard now? [Yes] Yes 
Accept end-user license agreement? (View/Yes/No) [View] Yes 

Before starting the actual configuration, some basic information is needed. The DDC name and script path determines where all files created by the wizard go (and where to reread them if 
wizard is rerun) and the license key determines which options are available for this configuration. 
(DDC_NAME) - Please enter a name for this replication: [] laa_rac 
(LICENSE_KEY) - Please enter your license key: [(trial)] 
Which Replicate edition do you want to trial (LTD/XTD/MAX): [MAX] 
(SETUP_SCRIPT_PATH) - Please enter a directory for location of configuration scripts on this machine: []/home/oracle/laa_rac

Network configuration files were detected on this system in these locations: 
/u01/app/11.2.0/grid/network/admin 
/u01/app/oracle/product/11.2.0/db_1/network/admin 
(TNS_ADMIN) - Please enter TNS configuration directory for this machine: []/u01/app/oracle/product/11.2.0/db_1/network/admin

Step 1 - Describe databases 
======================================== 
The first step is to describe databases used in the replication. There are usually two of them (source and target). 
Store SYSDBA and DBA passwords? Passwords only required during setup and initialization? (Yes/No) [Yes] 
Let's configure the database, describing its type, connectivity, user names etc. 
What type of database is this? (Oracle/MySQL/Google Cloud SQL/SQL Server/Oracle AWS RDS/CSV/Hadoop): [Oracle] 
Please enter database TNS alias: [] DBVLAA -- Dedicated Service Name on Source
Please enter SYSDBA user name: [SYS] 
Please enter password for this user: [change_on_install] ******* 
Please enter user with DBA role: [SYSTEM] 
Please enter password for this user: [manager] ******* 

Connecting to database DBVLAA as SYSTEM to query list of tablespaces and to detect ASM (by looking whether any redo logs or archived logs are stored in ASM). 
Enter the Dbvisit Replicate owner (this user will be created by this script): [dbvrep] 
Please enter password for this user: [dbvpasswd] ******* 

Permanent tablespaces detected on the database: USERS. 
Please enter default permanent tablespace for this user: [USERS] 

Temporary tablespaces detected on the database: TEMP. 
Please enter default temporary tablespace for this user: [TEMP] 

Following databases are now configured: 
1: Oracle DBVLAA, SYS/***, SYSTEM/***, dbvrep/***, USERS/TEMP, dbvrep/, ASM:Yes, TZ: +00:00 
Enter the number of the database to modify it, or "add", or "done": [add] 
Let's configure the database, describing its type, connectivity, user names etc. 
What type of database is this? (Oracle/MySQL/Google Cloud SQL/SQL Server/Oracle AWS RDS/CSV/Hadoop): [Oracle] 
Please enter database TNS alias: [] DBVDEV -- Dedicated Service Name on Target
Please enter SYSDBA user name: [SYS] 
Please enter password for this user: [change_on_install] ******* 
Please enter user with DBA role: [SYSTEM] 
Please enter password for this user: [manager] ******* 

Connecting to database DBVDEV as SYSTEM to query list of tablespaces and to detect ASM (by looking whether any redo logs or archived logs are stored in ASM). 
Enter the Dbvisit Replicate owner (this user will be created by this script): [dbvrep] 
Please enter password for this user: [dbvpasswd] ******* 

Permanent tablespaces detected on the database: USERS. 
Please enter default permanent tablespace for this user: [USERS] 

Temporary tablespaces detected on the database: TEMP. 
Please enter default temporary tablespace for this user: [TEMP] 

Following databases are now configured: 
1: Oracle DBVLAA, SYS/***, SYSTEM/***, dbvrep/***, USERS/TEMP, dbvrep/, ASM:Yes, TZ: +00:00 
2: Oracle DBVDEV, SYS/***, SYSTEM/***, dbvrep/***, USERS/TEMP, dbvrep/, ASM:Yes, TZ: +00:00 
Enter the number of the database to modify it, or "add", or "done": [done] 

Step 2 - Replication pairs 
======================================== 
The second step is to set source and targets for each replication pair. 
Let's configure the replication pair, selecting source and target. 
Following databases are described: 
1: DBVLAA (Oracle) 
2: DBVDEV (Oracle) 
Select source database: [1] 1 
Select target database: [2] 2 
Will DDL replication be enabled? (Yes/No) [Yes] 
Use fetcher to offload the mining to a different server? (Yes/No) [No] 
Should where clauses (and Event Streaming) include all columns, not just changed and PK? (Yes/No) [No] 
Would you like to encrypt the data across the network? (Yes/No) [No] 
Would you like to compress the data across the network? (Yes/No) [No] 
How long do you want to set the network timeouts. Recommended range between 60-300 seconds [60] 
Lock and copy the data initially one-by-one or at a single SCN? 
one-by-one : Lock tables one by one and capture SCN 
single-scn : One SCN for all tables 
ddl-only   : Only DDL script for target objects 
resetlogs  : Use SCN from last resetlogs operation (standby activation, rman incomplete recovery) 
no-lock    : Do not lock tables. Captures previous SCN of oldest active transaction. Requires pre-requisite running of pre-all.sh script                                                                            (one-by-one/single-scn/ddl-only/resetlogs/no-lock) [single-scn] 

What data instantiation script to create? 
dp_networklink : Data Pump with network link. No export datafile is created (APPLY.sh) 
dp_exp         : Data Pump with export datafile. Creates an export datafile (APPLY.sh) 
exp            : Pre-datapump exp/imp with export datafile (APPLY.sh) 
ddl_file       : DDL file created (APPLY.sql) 
ddl_run        : DDL is automatically executed on target 
load           : All replicated data is created and loaded automatically 
none                                                                                                                                                                                                                (dp_networklink/dp_exp/exp/ddl_file/ddl_run/load/none) [dp_networklink] 

Following replication pairs are now configured: 
1: DBVLAA (Oracle) ==> DBVDEV (Oracle), DDL: Yes, fetcher: No, process suffix: (no suffix), compression: No, encryption: No, network timeout: 60, prepare type: single-scn, data load: load 
Enter number of replication pair to modify it, or "add", or "done": [done] 

Step 3 - Replicated tables 
======================================== 
The third step is to choose the schemas and tables to be replicated. If the databases are reachable, the tables are checked for existence, datatype support, etc., schemas are queried for tables. Note that all 
messages are merely hints/warnings and may be ignored if issues are rectified before the scripts are actually executed. 

Following tables are defined for replication pairs: 
1: DBVLAA (Oracle) ==> DBVDEV (Oracle), DDL: Yes, suffix: (no suffix), prepare: single-scn 
  No tables defined. 
Enter number of replication pair to modify it, or "done": [1] 
Please enter list of all individual tables to be replicated. Enter schema name(s) only to replicate all tables in that schema. Use comma or space to delimit the entries. 
Enter the tables and schemas: [] scott 
Selected schemas: SCOTT 
Add more tables or schemas? (Yes/No) [No] 

To replicate changes of PL/SQL objects in schema(s), please enter the schemas to be replicated. Note that specifying any entry will cause additional privileges to be granted to dbvrep. Enter through a 
comma or space-delimited list. 
Enter the list of schemas (PL/SQL): [] 

You can also specify some advanced options: 
1. Exclude some tables from schema-level replication 
2. Rename schemas or tables. 
3. Specify filtering conditions. 
4. (Tables only) Configure Event Streaming; this does not maintain a copy of the source table, but logs all operations as separate entries. This is useful for ETL or as an audit trail. This usually 
requires adding of new columns (timestamps, old/new values etc.) to the target table. 
Specify rename name, filter condition or audit for any of the specified schemas? (Yes/No) [No] 
(PREPARE_SCHEMA_EXCEPTIONS) - Specify tables to exclude from PREPARE SCHEMA, if any: [] 

Following tables are defined for replication pairs: 
1: DBVLAA (Oracle) ==> DBVDEV (Oracle), DDL: Yes, suffix: (no suffix), prepare: single-scn 
  SCOTT(tables) 
Enter number of replication pair to modify it, or "done": [done] 

Step 4 - Process configuration 
======================================== 
The fourth step is to configure the replication processes for each replication. 

Following processes are defined: 
1: MINE on DBVLAA 
  Not configured. 
2: APPLY on DBVDEV 
  Not configured. 
Enter number of process to modify it, or "done": [1] 1 
Fully qualified name of the server for the process (usually co-located with the database, unless mine is offloaded using fetcher): [] dbvrep812-vip -- vip for hostname
Server type (Windows/Linux/Unix): [Linux] 
Enable email notifications about problems? (Yes/No) [No] 
Enable SNMP traps/notifications about problems? (Yes/No) [No] 
Directory with DDC file and default where to create log files etc. (recommended: same as global setting, if possible)? [/home/oracle/laa_rac] 

Following settings were pre-filled with defaults or your reloaded settings: 
---------------------------------------- 
[MINE_REMOTE_INTERFACE]: Network remote interface: dbvrep812-vip:7901 
[MINE_DATABASE]: Database TNS: DBVLAA 
[TNS_ADMIN]: tnsnames.ora path: /u01/app/oracle/product/11.2.0/db_1/network/admin 
[ORACLE_HOME]: ASM ORACLE_HOME: /u01/app/11.2.0/grid 
[MINE_PLOG]: Filemask for generated plogs: /home/oracle/laa_rac/mine/%S.%E (%S is sequence, %T thread, %F original filename (stripped extension), %P process type, %N process name, %E default extension) 
[LOG_FILE]: General log file: /home/oracle/laa_rac/log/dbvrep_%N_%D.%E 
[LOG_FILE_TRACE]: Error traces: /home/oracle/laa_rac/log/trace/dbvrep_%N_%D_%I_%U.%E 

Checking that these settings are valid... 
Do you want to change any of the settings? [No] 

Following processes are defined: 
1: MINE on DBVLAA 
  Host: dbvrep812-vip, SMTP: No, SNMP: No 
2: APPLY on DBVDEV 
  Not configured. 
Enter number of process to modify it, or "done": [2] 
Fully qualified name of the server for the process (usually co-located with the database, unless mine is offloaded using fetcher): [] dbvrep912-vip -- Target vip
Server type (Windows/Linux/Unix): [Linux] 
Enable email notifications about problems? (Yes/No) [No] 
Enable SNMP traps/notifications about problems? (Yes/No) [No] 
Directory with DDC file and default where to create log files etc. (recommended: same as global setting, if possible)? [/home/oracle/laa_rac] 

Following settings were pre-filled with defaults or your reloaded settings: 
---------------------------------------- 
[APPLY_REMOTE_INTERFACE]: Network remote interface: dbvrep912-vip:7902 
[APPLY_DATABASE]: Database TNS: DBVDEV 
[TNS_ADMIN]: tnsnames.ora path: /u01/app/oracle/product/11.2.0/db_1/network/admin 
[APPLY_STAGING_DIR]: Directory for received plogs: /home/oracle/laa_rac/apply 
[LOG_FILE]: General log file: /home/oracle/laa_rac/log/dbvrep_%N_%D.%E 
[LOG_FILE_TRACE]: Error traces: /home/oracle/laa_rac/log/trace/dbvrep_%N_%D_%I_%U.%E 

Checking that these settings are valid... 
Do you want to change any of the settings? [No] 

Following processes are defined: 
1: MINE on DBVLAA 
  Host: dbvrep812-vip, SMTP: No, SNMP: No 
2: APPLY on DBVDEV 
  Host: dbvrep912-vip, SMTP: No, SNMP: No

oracle@kiwi81[/home/oracle/laa_rac]: ./laa_rac-all.sh 
Setting up Dbvisit Replicate configuration 
Configure database DBVLAA... 

This check fails if the DBID is not the expected one... 

Ok, check passed. 
Configure database DBVDEV... 

This check fails if the DBID is not the expected one... 

Ok, check passed. 
Object grants for database DBVLAA... 

Object grants for database DBVDEV... 

Setting up the configuration 
Initializing......done 
DDC loaded from database (0 variables). 
Dbvisit Replicate version 2.8.04_rc1 
Copyright (C) Dbvisit Software Limited.  All rights reserved. 
DDC file /home/oracle/laa_rac/config/laa_rac-onetime.ddc loaded. 
MINE: Cannot determine Dbvisit Replicate dictionary version. (no 
dictionary exists) 
APPLY: Cannot determine Dbvisit Replicate dictionary version. (no 
dictionary exists) 
dbvrep> #clear the no-DDC-DB-available warning 
dbvrep> process clear previous warnings 
dbvrep> set ON_WARNING SKIP 
Variable ON_WARNING set to SKIP for process *. 
dbvrep> set ON_ERROR EXIT 
Variable ON_ERROR set to EXIT for process *. 
dbvrep> 
dbvrep> # Configuring default processes 
dbvrep> choose process MINE 
Process type MINE set to: MINE. 
dbvrep> choose process APPLY 
Process type APPLY set to: APPLY. 
dbvrep> PROCESS SWITCH_REDOLOG 
Redo log switch requested.dbvrep> PROCESS SETUP MINE DROP DICTIONARY 
0 dictionary objects dropped. 
dbvrep> PROCESS SETUP MINE CREATE DICTIONARY 
dbvrep> PROCESS SETUP MINE LOAD DICTIONARY 
Oldest active transaction SCN: 1179659 (no active transaction) 
Supplemental logging on database set. 
Loading dictionary table DBRSCCOL$ 
Loading dictionary table DBRSCDEF$ 
Loading dictionary table DBRSCOL$ 
Loading dictionary table DBRSIND$ 
Loading dictionary table DBRSOBJ$ 
Loading dictionary table DBRSTAB$ 
Loading dictionary table DBRSUSER$ 
Loading dictionary table DBRSV_$DATABASE 
dbvrep> PROCESS SETUP APPLY DROP DICTIONARY 
0 dictionary objects dropped. 
dbvrep> PROCESS SETUP APPLY CREATE DICTIONARY 
dbvrep> PROCESS SETUP APPLY LOAD DICTIONARY 
dbvrep> PROCESS PREPARE_DP SETUP CLEAR 
dbvrep> PROCESS SETUP PAIR MINE AND APPLY 
Applier SCN set (start=1179673, current=1179673). 
dbvrep> SET APPLY.INSTANTIATE_SCN NOW 
Variable INSTANTIATE_SCN set to NOW for process APPLY. 
dbvrep> SET MINE._PREPARE_SUPLOG_TYPE PK 
Variable _PREPARE_SUPLOG_TYPE set to PK for process MINE. 
dbvrep> EXCLUDE CREATE TABLE %.DBMS_TABCOMP_TEMP_UNCMP #Ignore tables 
created by Compression Advisor 
Exclude rule created. 
dbvrep> EXCLUDE CREATE TABLE %.DBMS_TABCOMP_TEMP_CMP #Ignore tables 
created by Compression Advisor 
Exclude rule created. 
dbvrep> EXCLUDE CREATE TABLE %.SCHEDULER$_% #Ignore tables created by 
Oracle scheduler (also used by schema/full expdp/impdp) 
Exclude rule created. 
dbvrep> EXCLUDE CREATE TABLE %.CMP1$% #Ignore tables created by 
Compression Advisor since 11.2.0.4 
Exclude rule created. 
dbvrep> EXCLUDE CREATE TABLE %.CMP2$% #Ignore tables created by 
Compression Advisor since 11.2.0.4 
Exclude rule created. 
dbvrep> EXCLUDE CREATE TABLE %.CMP3$% #Ignore tables created by 
Compression Advisor since 11.2.0.4 
Exclude rule created. 
dbvrep> EXCLUDE CREATE TABLE %.CMP4$% #Ignore tables created by 
Compression Advisor since 11.2.0.4 
Exclude rule created. 
dbvrep> SET PREPARE_SCHEMA_EXCEPTIONS none 
Variable PREPARE_SCHEMA_EXCEPTIONS set to none for process *. 
dbvrep> PROCESS SUPPLEMENTAL LOGGING SCHEMA "SCOTT" ENABLE PRIMARY KEY 
dbvrep> PROCESS SWITCH_REDOLOG 
Redo log switch requested.dbvrep> PROCESS WAIT_SCN_FLIP 
Waited 1 seconds until scn_to_timestamp changed. 
dbvrep> #single-scn instantiation: lock all tables and schemas 
dbvrep> PROCESS LOCK SCHEMAS "SCOTT" 
Locking all schemas. 
...locked 4 of 4 tables from SCOTT schema. 
Lock done. 
dbvrep> #single-scn instantiation: unlock all tables and schemas, but 
keep the SCN 
dbvrep> PROCESS LOCK RELEASE LOCKS 
dbvrep> 
dbvrep> #prepare the tables (we use OFFLINE as neither MINE nor APPLY 
is running; with OFFLINE we won't wait on network timeout) 
dbvrep> PREPARE OFFLINE SCHEMA "SCOTT" 
Table SCOTT.BONUS instantiated at SCN 1187845 
Table SCOTT.DEPT instantiated at SCN 1187845 
Table SCOTT.EMP instantiated at SCN 1187845 
Table SCOTT.SALGRADE instantiated at SCN 1187845 
dbvrep> 
dbvrep> #single-scn instantiation: unlock all tables and schemas, 
forget the SCN (so it does not affect any further PREPARE statements) 
dbvrep> PROCESS LOCK CLEAR SCN 
dbvrep> PROCESS SWITCH_REDOLOG 
Redo log switch requested.dbvrep> #prepare script for instantiation 
dbvrep> PROCESS PREPARE_DP WRITE DP_NETWORKLINK DIRECTORY DATA_PUMP_DIR 
FILE /home/oracle/laa_rac/APPLY.sh DBLINK DBVLAA USERID 
SYSTEM/kiwi123@DBVDEV 
Created Data Pump script /home/oracle/laa_rac/APPLY.sh, using network 
import. 
dbvrep> create ddcdb from ddcfile 
DDC loaded into database (406 variables). 
dbvrep> set ON_WARNING SKIP 
Variable ON_WARNING set to SKIP for process *. 
dbvrep> set ON_ERROR SKIP 
Variable ON_ERROR set to SKIP for process *. 
OK-0: Completed successfully. 
WARN-1850: No DDC DB available, dictionary table does not exist. 
These steps are required after the laa_rac-all.sh script runs: 

1) Create the necessary directory(ies) on the servers: 
dbvrep812-vip: /home/oracle/laa_rac 
dbvrep912-vip: /home/oracle/laa_rac 

2) Copy the DDC files to the server(s) where the processes will run: 
dbvrep912-vip: /home/oracle/laa_rac/laa_rac-APPLY.ddc 
dbvrep812-vip: /home/oracle/laa_rac/laa_rac-MINE.ddc 

3) Review that path to dbvrep executable is correct in the run scripts: 
/home/oracle/laa_rac/laa_rac-run-dbvrep812-vip.sh 
/home/oracle/laa_rac/laa_rac-run-dbvrep912-vip.sh 

4) Copy the run script to the server(s) where the processes will run: 
dbvrep812-vip: /home/oracle/laa_rac/laa_rac-run-dbvrep812-vip.sh 
dbvrep912-vip: /home/oracle/laa_rac/laa_rac-run-dbvrep912-vip.sh 

5) Ensure firewall is open for listen interfaces 0.0.0.0:7902, 0.0.0.0:7901 used by the processes. 

6) Make sure the data on apply are in sync as of time when setup was run. 
Scripts for Data Pump/export/DDL were created as requested: 

Create referenced database links (if any) before running the scripts. 
/home/oracle/laa_rac/APPLY.sh 

7) Start the replication processes on all servers: 
dbvrep812-vip: /home/oracle/laa_rac/laa_rac-run-dbvrep812-vip.sh 
dbvrep912-vip: /home/oracle/laa_rac/laa_rac-run-dbvrep912-vip.sh 

8) Start the console to monitor the progress: 
/home/oracle/laa_rac/start-console.sh 

The above list is stored in /home/oracle/laa_rac/Nextsteps.txt.  

oracle@kiwi81[/home/oracle/laa_rac]: scp laa_rac-run-dbvrep912-vip.sh oracle@kiwi91:/home/oracle/laa_rac/laa_rac-run-dbvrep912-vip.sh                                                                                                                                            
oracle@kiwi81[/home/oracle/laa_rac]: scp laa_rac-APPLY.ddc oracle@kiwi91:/home/oracle/laa_rac/laa_rac-APPLY.ddc                                                                                                                                    

oracle@kiwi81[/home/oracle]: . oraenv
ORACLE_SID = [LAA1] ? +ASM1
oracle@kiwi81[/home/oracle]: crsctl status resource dbvrep812-vip
NAME=dbvrep812-vip
TYPE=app.appvip_net1.type
TARGET=ONLINE
STATE=ONLINE on kiwi81
oracle@kiwi81[/home/oracle]: crsctl status resource ora.dev.dbvlaa.svc
NAME=ora.dev.dbvlaa.svc
TYPE=ora.service.type
TARGET=ONLINE
STATE=ONLINE on kiwi81
oracle@kiwi81[/home/oracle]: ssh kiwi91
oracle@kiwi91[/home/oracle]: crsctl status resource dbvrep912-vip
NAME=dbvrep912-vip
TYPE=app.appvip_net1.type
TARGET=ONLINE
STATE=ONLINE on kiwi91
oracle@kiwi91[/home/oracle]: crsctl status resource ora.dev.dbvdev.svc
NAME=ora.dev.dbvdev.svc
TYPE=ora.service.type
TARGET=ONLINE
STATE=ONLINE on kiwi91
oracle@kiwi91[/home/oracle]: exit

-- Running on 1 node in each cluster

oracle@kiwi81[/home/oracle]: cd laa_rac
oracle@kiwi81[/home/oracle/laa_rac]: ./laa_rac-run-dbvrep812-vip.sh 
Initializing......done 
DDC loaded from database (406 variables). 
Dbvisit Replicate version 2.8.04_rc1 
Copyright (C) Dbvisit Software Limited.  All rights reserved. 
DDC file /home/oracle/laa_rac/laa_rac-MINE.ddc loaded. 
Starting process MINE...Created directory /home/oracle/laa_rac/ddc_backup 
Created directory /home/oracle/laa_rac/log/trace/ 
Created directory /home/oracle/laa_rac/mine/ 
started 
oracle@kiwi81[/home/oracle/laa_rac]: ssh kiwi91

Start Apply Process Running on Node 1 on Target (kiwi91:DEV) 

oracle@kiwi91[/home/oracle]: cd laa_rac
oracle@kiwi91[/home/oracle/laa_rac]: ./laa_rac-run-dbvrep912-vip.sh 
Initializing......done 
DDC loaded from database (407 variables). 
Dbvisit Replicate version 2.8.04_rc1 
Copyright (C) Dbvisit Software Limited.  All rights reserved. 
DDC file /home/oracle/laa_rac/laa_rac-APPLY.ddc loaded. 
Starting process APPLY...Created directory /home/oracle/laa_rac/ddc_backup 
Created directory /home/oracle/laa_rac/log/ 
Created directory /home/oracle/laa_rac/log/trace/ 
Created directory /home/oracle/laa_rac/apply 
started 

oracle@kiwi81[/home/oracle/laa_rac]: ./start-console.sh 
Initializing......done 
DDC loaded from database (407 variables). 
Dbvisit Replicate version 2.8.04_rc1 
Copyright (C) Dbvisit Software Limited.  All rights reserved. 
/ Dbvisit Replicate 2.8.04_rc1(MAX edition) - Evaluation License expires in 30 days 
MINE is running. Currently at plog 20 (redo sequence 23 [1] 13 [2]) and SCN 1200910 (03/13/2017 15:32:28). 
APPLY is running. Currently at plog 20 and SCN 1200877 (03/13/2017 15:32:24). 
Progress of replication laa_rac:MINE->APPLY: total/this execution 
-------------------------------------------------------------------------------------- 
SCOTT.DEPT:                   100%  Mine:1/1             Unrecov:0/0         Applied:1/1         Conflicts:0/0       Last:13/03/2017 15:30:19/OK 
SCOTT.EMP:                    100%  Mine:1/1             Unrecov:0/0         Applied:1/1         Conflicts:0/0       Last:13/03/2017 15:28:48/OK 
-------------------------------------------------------------------------------------- 
2 tables listed. 
dbvrep> 
 

Step 1 - Shutdown Mine Process
===============================

dbvrep> shutdown mine 
Upon restart MINE will go back to redolog 23 (thread 1), going back no logs. 
Upon restart MINE will go back to redolog 12 (thread 2), going back 1 log. 
Are you sure you want to shutdown? (Yes/No) [No] Yes 
Dbvisit Replicate MINE process shutting down.

Step 2 - Relocate Database Service and vip
=========================================== 

Log in as Root and then relocate VIP

oracle@kiwi82[/home/oracle/laa_rac]: su -
root@kiwi81[/root]: crsctl status resource dbvrep812-vip 
NAME=dbvrep812-vip 
TYPE=app.appvip_net1.type 
TARGET=ONLINE STATE=ONLINE on kiwi81 
root@kiwi81[/root]: crsctl relocate resource dbvrep812-vip 
CRS-2673: Attempting to stop 'dbvrep812-vip' on 'kiwi81'
CRS-2677: Stop of 'dbvrep812-vip' on 'kiwi81' succeeded 
CRS-2672: Attempting to start 'dbvrep812-vip' on 'kiwi82' 
CRS-2676: Start of 'dbvrep812-vip' on 'kiwi82' succeeded
root@kiwi81[/root]: crsctl status resource dbvrep812-vip 
NAME=dbvrep812-vip 
TYPE=app.appvip_net1.type 
TARGET=ONLINE
STATE=ONLINE on kiwi82
root@kiwi81[/root]: exit

## As the oracle user relocate the service DBVLAA to node 2 (kiwi82)

oracle@kiwi81[/home/oracle/laa_rac]: srvctl relocate service -d LAA -s DBVLAA -i LAA1 -t LAA2
oracle@kiwi81[/home/oracle/laa_rac]: srvctl status service -d PROD -s DBVLAA
Service DBVLAA is running on instance(s) LAA2 

Step 3 - Start MINE process from 2nd node
==========================================

oracle@kiwi82[/home/oracle/laa_rac]: ./laa_rac-run-dbvrep812-vip.sh

Initializing......done
DDC loaded from database (406 variables).
Dbvisit Replicate version 2.8.04_rc1
Copyright (C) Dbvisit Software Limited.  All rights reserved.
DDC file /home/oracle/laa_rac/laa_rac-MINE.ddc loaded.
Starting process MINE...started

Step 4 - Confirm replication is running via the Console
========================================================
oracle@kiwi82[/home/oracle/laa_rac]: ./start-console.sh 
DDC loaded from database (407 variables). 
Dbvisit Replicate version 2.8.04_rc1 
Copyright (C) Dbvisit Software Limited.  All rights reserved. 
| Dbvisit Replicate 2.8.04_rc1(MAX edition) - Evaluation License expires in 30 days 
MINE is running. Currently at plog 25 (redo sequence 25 [1] 16 [2]) and SCN 1207240. 
APPLY is running. Currently at plog 25 and SCN 1206804 (03/13/2017 16:39:59). 
Progress of replication laa_rac:MINE->APPLY: total/this execution 
------------------------------------------------------------------------------------------------- 
SCOTT.DEPT:                   150%  Mine:2/0             Unrecov:0/0         Applied:3/3         Conflicts:0/0       Last:13/03/2017 16:34:45/OK 
SCOTT.EMP:                    100%  Mine:1/0             Unrecov:0/0         Applied:1/1         Conflicts:0/0       Last:13/03/2017 15:28:48/OK 
dbvrep>

### REPEAT STEPS on NODES KIWI91/92 for APPLY PROCESS.

KIWI81/82 --  MINE Action Script


#!/bin/bash
#
# Description:
# This is a sample action script for adding the Dbvisit Replicate Mine process
# as a cluster resource
# Note:
# This script needs to be updated to reflect the required environment details
# and should be tested before running on production systems
#

############################
## Echo usage of program
############################
usage ()
{
program=`basename $0`

cat <<EOF
Usage:
 ${program} start|stop|check|clean
EOF
exit 1
}

###########################################
# Function to change database environments
###########################################
set_env ()
{
 export ORAENV_ASK=NO
 export ORACLE_SID=$1
 . oraenv >> /dev/null
}

###########################################
# Function to setup parameter variables
###########################################
setup_parameters ()
{
 DBVREP=/usr/bin/dbvrep
 DDCFILE=/home/oracle/laa_rac/laa_rac-MINE.ddc
 PROCESS=MINE
 db=LAA
 set_env $db
}

###########################################
# Function used to stop replicate process
###########################################
stop_replicate ()
{
 echo "Stopping: $DBVREP --ddcfile $DDCFILE shutdown $PROCESS"
 $DBVREP --ddcfile $DDCFILE shutdown $PROCESS

 for c1 in {1..10}
 do
 # process might take few seconds to shutdown, check every 2 seconds 10 times
 sleep 3
 check=`ps -ef | grep $PROCESS | grep -v grep | wc -l`
 if [ $check -eq 0 ]
 then
 echo "Completed: $DBVREP --ddcfile $DDCFILE shutdown $PROCESS"
 exit 0
 fi
 echo "... sleep for 3 sec and check again"
 done

 # if we get to this stage the above process did not
 # stop as expected, exit with status 1
 echo "Error: $DBVREP --ddcfile $DDCFILE shutdown $PROCESS"
 exit 1
}

###########################################
# Function used to stop replicate process
###########################################
start_replicate ()
{
 echo "Starting: $DBVREP --daemon --ddcfile $DDCFILE start $PROCESS"
 $DBVREP --daemon --ddcfile $DDCFILE start $PROCESS
}

###########################################
# Function used to stop replicate process
###########################################
check_replicate ()
{
 echo "Starting: $DBVREP --ddcfile $DDCFILE healthcheck for $PROCESS"
 $DBVREP --ddcfile $DDCFILE healthcheck for $PROCESS
}

#################
## Main Section
#################

## Basic test number of input values
if test $# -ne 1
 then
 usage
 exit 1
fi

# First call function to set the environment or specific global variables
setup_parameters

#
# This is logged to CRSD agent log file
echo "`date` Action script '$_CRS_ACTION_SCRIPT' for resource [$_CRS_NAME] called for action $1"
#

case "$1" in
 'start') start_replicate
 ;;

 'stop') stop_replicate
 ;;

 'check') check_replicate
 ;;

 'clean')
 exit 0
 ;;

 *) usage
 ;;

esac

oracle@kiwi82[/home/oracle/laa_rac]: chmod u+rwx action_script.scr

-- We can test the action script by using it to stop/start the MINE process on node kiwi82 as dbvrep812-vip
and service DBVLAA are still running from before on this node.

oracle@kiwi82[/home/oracle/laa_rac]: ./action_script.scr
Usage:
 action_script.scr start|stop|check|clean
oracle@kiwi82[/home/oracle/laa_rac]: ./action_script.scr start
Wed Apr 5 15:07:37 NZST 2017 Action script '' for resource [] called for action start
Starting: /usr/bin/dbvrep --daemon --ddcfile /home/oracle/laa_rac/laa_rac-MINE.ddc start MINE
Initializing......done
DDC loaded from database (407 variables).
Dbvisit Replicate version 2.8.04_126_g0810a26
Copyright (C) Dbvisit Software Limited. All rights reserved.
DDC file /home/oracle/laa_rac/laa_rac-MINE.ddc loaded.
Starting process MINE...started
oracle@kiwi82[/home/oracle/laa_rac]: ./action_script.scr check
Wed Apr 5 15:07:59 NZST 2017 Action script '' for resource [] called for action check
Starting: /usr/bin/dbvrep --ddcfile /home/oracle/laa_rac/laa_rac-MINE.ddc healthcheck for MINE
Initializing......done
DDC loaded from database (407 variables).
Dbvisit Replicate version 2.8.04_126_g0810a26
Copyright (C) Dbvisit Software Limited. All rights reserved.
DDC file /home/oracle/laa_rac/laa_rac-MINE.ddc loaded.
oracle@kiwi82[/home/oracle/laa_rac]: ps -ef|grep -i dbv
oracle 23448 1 23 15:07 ? 00:00:10 dbvrep MINE laa --daemon --ddcfile /home/oracle/laa_rac/laa_rac-MINE.ddc start MINE
oracle 23812 3842 0 15:08 pts/0 00:00:00 grep -i dbv 

Repeat on KIWI 91/92 -- APPLY Action Script

#!/bin/bash
#
# Description:
# This is a sample action script for adding the Dbvisit Replicate Mine process
# as a cluster resource
# Note:
# This script needs to be updated to reflect the required environment details
# and should be tested before running on production systems
#

############################
## Echo usage of program
############################
usage ()
{
program=`basename $0`

cat <<EOF
Usage:
 ${program} start|stop|check|clean
EOF
exit 1
}

###########################################
# Function to change database environments
###########################################
set_env ()
{
 export ORAENV_ASK=NO
 export ORACLE_SID=$1
 . oraenv >> /dev/null
}

###########################################
# Function to setup parameter variables
###########################################
setup_parameters ()
{
 DBVREP=/usr/bin/dbvrep
 DDCFILE=/home/oracle/laa_rac/laa_rac-APPLY.ddc
 PROCESS=APPLY
 db=DEV
 set_env $db
}

###########################################
# Function used to stop replicate process
###########################################
stop_replicate ()
{
 echo "Stopping: $DBVREP --ddcfile $DDCFILE shutdown $PROCESS"
 $DBVREP --ddcfile $DDCFILE shutdown $PROCESS

 for c1 in {1..10}
 do
 # process might take few seconds to shutdown, check every 2 seconds 10 times
 sleep 3
 check=`ps -ef | grep $PROCESS | grep -v grep | wc -l`
 if [ $check -eq 0 ]
 then
 echo "Completed: $DBVREP --ddcfile $DDCFILE shutdown $PROCESS"
 exit 0
 fi
 echo "... sleep for 3 sec and check again"
 done

 # if we get to this stage the above process did not
 # stop as expected, exit with status 1
 echo "Error: $DBVREP --ddcfile $DDCFILE shutdown $PROCESS"
 exit 1
}

###########################################
# Function used to stop replicate process
###########################################
start_replicate ()
{
 echo "Starting: $DBVREP --daemon --ddcfile $DDCFILE start $PROCESS"
 $DBVREP --daemon --ddcfile $DDCFILE start $PROCESS
}

###########################################
# Function used to stop replicate process
###########################################
check_replicate ()
{
 echo "Starting: $DBVREP --ddcfile $DDCFILE healthcheck for $PROCESS"
 $DBVREP --ddcfile $DDCFILE healthcheck for $PROCESS
}

#################
## Main Section
#################

## Basic test number of input values
if test $# -ne 1
 then
 usage
 exit 1
fi

# First call function to set the environment or specific global variables
setup_parameters

#
# This is logged to CRSD agent log file
echo "`date` Action script '$_CRS_ACTION_SCRIPT' for resource [$_CRS_NAME] called for action $1"
#

case "$1" in
 'start') start_replicate
 ;;

 'stop') stop_replicate
 ;;

 'check') check_replicate
 ;;

 'clean')
 exit 0
 ;;

 *) usage
 ;;

esac

oracle@kiwi91[/home/oracle/laa_rac]: chmod u+rwx action_script.scr

-- Similarly Testing Stop with action_script on NODE kiwi91

oracle@kiwi91[/home/oracle/laa_rac]: ./action_script.scr stop
Wed Apr 5 15:17:03 NZST 2017 Action script '' for resource [] called for action stop
Stopping: /usr/bin/dbvrep --ddcfile /home/oracle/laa_rac/laa_rac-APPLY.ddc shutdown APPLY
Initializing......done
DDC loaded from database (407 variables).
Dbvisit Replicate version 2.8.04_126_g0810a26
Copyright (C) Dbvisit Software Limited. All rights reserved.
DDC file /home/oracle/laa_rac/laa_rac-APPLY.ddc loaded.
Dbvisit Replicate APPLY process shutting down.
OK-0: Completed successfully.
Completed: /usr/bin/dbvrep --ddcfile /home/oracle/laa_rac/laa_rac-APPLY.ddc shutdown APPLY
oracle@kiwi91[/home/oracle/laa_rac]: ps -ef|grep -i dbv
oracle 6548 5782 0 15:17 pts/0 00:00:00 grep -i dbv

oracle@kiwi82[/home/oracle/laa_rac]: su -
Password:
root@kiwi82[/root]: . oraenv
ORACLE_SID = [root] ? +ASM2
The Oracle base has been set to /u01/app/grid
root@kiwi82[/root]: crsctl add resource dbvMine -type cluster_resource -attr "ACTION_SCRIPT=/home/oracle/laa_rac/action_script.scr, CHECK_INTERVAL=30, START_DEPENDENCIES='hard(dbvrep812-vip,ora.laa.db,ora.asm,ora.laa.dbvlaa.svc) pullup(ora.laa.dbvlaa.svc,dbvrep812-vip)',STOP_DEPENDENCIES='hard(ora.laa.dbvlaa.svc,dbvrep812-vip)' ACL='owner:oracle:rwx,pgrp:oinstall:rwx,other::r--' PLACEMENT='favored' HOSTING_MEMBERS='kiwi81'"
root@kiwi82[/root]: crsctl status resource dbvMine
NAME=dbvMine
TYPE=cluster_resource
TARGET=OFFLINE
STATE=OFFLINE
root@kiwi82[/root]: crsctl start resource dbvMine
CRS-2672: Attempting to start 'dbvMine' on 'kiwi82'
CRS-2676: Start of 'dbvMine' on 'kiwi82' succeeded
root@kiwi82[/root]: ps -ef|grep -i dbv
oracle   32235     1 48 15:32 ?        00:00:16 dbvrep MINE laa --daemon --ddcfile /home/oracle/laa_rac/laa_rac-MINE.ddc start MINE
root     32456 29931  0 15:32 pts/0    00:00:00 grep -i dbv
root@kiwi82[/root]: crsctl stop resource dbvMine
CRS-2673: Attempting to stop 'dbvMine' on 'kiwi82'
CRS-2677: Stop of 'dbvMine' on 'kiwi82' succeeded
root@kiwi82[/root]: ps -ef|grep -i dbv
root     32174 29931  0 15:32 pts/0    00:00:00 grep -i dbv
root@kiwi82[/root]: crsctl start resource dbvMine
CRS-2672: Attempting to start 'dbvMine' on 'kiwi82'
CRS-2676: Start of 'dbvMine' on 'kiwi82' succeeded
root@kiwi82[/root]: crsctl status resource dbvMine 
NAME=dbvMine 
TYPE=cluster_resource 
TARGET=ONLINE 
STATE=ONLINE on kiwi82 

root@kiwi91[/root]: . oraenv
ORACLE_SID = [root] ? +ASM1
The Oracle base has been set to /u01/app/grid
root@kiwi91[/root]: crsctl add resource dbvApply -type cluster_resource -attr "ACTION_SCRIPT=/home/oracle/laa_rac/action_script.scr, CHECK_INTERVAL=30, START_DEPENDENCIES='hard(dbvrep912-vip,ora.dev.db,ora.asm,ora.dev.dbvdev.svc) pullup(ora.dev.dbvdev.svc,dbvrep912-vip)',STOP_DEPENDENCIES='hard(ora.dev.dbvdev.svc,dbvrep912-vip)' ACL='owner:oracle:rwx,pgrp:oinstall:rwx,other::r--' PLACEMENT='favored' HOSTING_MEMBERS='kiwi91'"
root@kiwi91[/root]: crsctl status resource dbvApply
NAME=dbvApply
TYPE=cluster_resource
TARGET=OFFLINE
STATE=OFFLINE
root@kiwi91[/root]: crsctl start resource dbvApply
CRS-2672: Attempting to start 'dbvApply' on 'kiwi91'
CRS-2676: Start of 'dbvApply' on 'kiwi91' succeeded
root@kiwi91[/root]: ps -ef|grep -i dbv
oracle   29109     1 27 17:24 ?        00:00:10 dbvrep APPLY la --daemon --ddcfile /home/oracle/laa_rac/laa_rac-APPLY.ddc start APPLY
root     29368 27631  0 17:24 pts/0    00:00:00 grep -i dbv
root@kiwi91[/root]: crsctl stop resource dbvApply
CRS-2673: Attempting to stop 'dbvApply' on 'kiwi91'
CRS-2677: Stop of 'dbvApply' on 'kiwi91' succeeded
root@kiwi91[/root]: ps -ef|grep -i dbv
root     30218 27631  0 17:26 pts/0    00:00:00 grep -i dbv
root@kiwi91[/root]: crsctl start resource dbvApply
CRS-2672: Attempting to start 'dbvApply' on 'kiwi91'
CRS-2676: Start of 'dbvApply' on 'kiwi91' succeeded
root@kiwi91[/root]: crsctl status resource dbvApply
NAME=dbvApply
TYPE=cluster_resource
TARGET=ONLINE
STATE=ONLINE on kiwi91
root@kiwi91[/root]:

oracle@kiwi82[/home/oracle/laa_rac]: ./start-console.sh
Initializing......done
DDC loaded from database (407 variables).
Dbvisit Replicate version 2.8.04_rc1
Copyright (C) Dbvisit Software Limited.  All rights reserved.
- Dbvisit Replicate 2.8.04_rc1(MAX edition) - Evaluation License expires in 30 days
MINE is running. Currently at plog 89 (redo sequence 59 [1] 50 [2]) and SCN 1371036 (04/05/2017 17:59:41).
APPLY is running. Currently at plog 89 and SCN 1371002 (04/05/2017 17:59:38).
Progress of replication laa_rac:MINE->APPLY: total/this execution
--------------------------------------------------------------------------------------------------------------------------------------------
SCOTT.DEPT:                   100%  Mine:5/0             Unrecov:0/0         Applied:5/0         Conflicts:0/0       Last:--/--
SCOTT.EMP:                    100%  Mine:33/10           Unrecov:0/0         Applied:33/10       Conflicts:0/0       Last:05/04/2017 17:58:45/OK
SCOTT.SALGRADE:               100%  Mine:5/0             Unrecov:0/0         Applied:5/0         Conflicts:0/0       Last:--/--
--------------------------------------------------------------------------------------------------------------------------------------------
3 tables listed.

root@kiwi82[/root]: . oraenv
ORACLE_SID = [root] ? +ASM2
The Oracle base has been set to /u01/app/grid
root@kiwi82[/root]: crsctl relocate resource dbvMine -n kiwi81 -f
RS-2673: Attempting to stop 'dbvMine' on 'kiwi82'
CRS-2677: Stop of 'dbvMine' on 'kiwi82' succeeded
CRS-2673: Attempting to stop 'dbvrep812-vip' on 'kiwi82'
CRS-2673: Attempting to stop 'ora.laa.dbvlaa.svc' on 'kiwi82'
CRS-2677: Stop of 'ora.laa.dbvlaa.svc' on 'kiwi82' succeeded
CRS-2672: Attempting to start 'ora.laa.dbvlaa.svc' on 'kiwi81'
CRS-2676: Start of 'ora.laa.dbvlaa.svc' on 'kiwi81' succeeded
CRS-2677: Stop of 'dbvrep812-vip' on 'kiwi82' succeeded
CRS-2672: Attempting to start 'dbvrep812-vip' on 'kiwi81'
CRS-2676: Start of 'dbvrep812-vip' on 'kiwi81' succeeded
CRS-2672: Attempting to start 'dbvMine' on 'kiwi81'
CRS-2676: Start of 'dbvMine' on 'kiwi81' succeeded
root@kiwi82[/root]: crsctl status resource dbvMine
NAME=dbvMine
TYPE=cluster_resource
TARGET=ONLINE
STATE=ONLINE on kiwi81
oracle@kiwi81[/home/oracle/laa_rac/sql]: ps -ef|grep -i dbv
oracle    2359     1 15 20:30 ?        00:00:13 dbvrep MINE laa --daemon --ddcfile /home/oracle/laa_rac/laa_rac-MINE.ddc start MINE
oracle@kiwi81[/home/oracle/laa_rac/sql]: . oraenv
ORACLE_SID = [+ASM1] ? LAA1
The Oracle base has been changed from /u01/app/grid to /u01/app/oracle
oracle@kiwi81[/home/oracle/laa_rac]: ./start-console.sh
Initializing......done
DDC loaded from database (407 variables).
Dbvisit Replicate version 2.8.04_rc1
Copyright (C) Dbvisit Software Limited.  All rights reserved.
- Dbvisit Replicate 2.8.04_126_rc1(MAX edition) - Evaluation License expires in 30 days
MINE is running. Currently at plog 141 (redo sequence 85 [1] 76 [2]) and SCN 1441295 (04/05/2017 20:35:10).
APPLY is running. Currently at plog 141 and SCN 1441245 (04/05/2017 20:35:02).
Progress of replication laa_rac:MINE->APPLY: total/this execution
--------------------------------------------------------------------------------------------------------------------------------------------
SCOTT.DEPT:                   100%  Mine:6/0             Unrecov:0/0         Applied:6/1         Conflicts:0/0       Last:05/04/2017 18:04:33/OK
SCOTT.EMP:                    100%  Mine:35/2            Unrecov:0/0         Applied:35/12       Conflicts:0/0       Last:05/04/2017 20:35:02/OK
SCOTT.SALGRADE:               100%  Mine:7/0             Unrecov:0/0         Applied:7/2         Conflicts:0/0       Last:05/04/2017 18:03:47/OK
--------------------------------------------------------------------------------------------------------------------------------------------
3 tables listed.

-- Test 1 Crash kiwi92

-- Test 2 Crash kiwi81