We're here to help!

 Dbvnet Connection Refused error in standby version 8

Problem Details

Error connect: Connection refused

Problem:

Running a dbvnet connectivity test between a primary and standyb host reveal the following error:

oracle@dbvlab01[/usr/dbv8/dbvnet]: ./dbvnet -e "uname -n"
Error connect: Connection refused

There are various reasons you may get this error, including firewall blocking ports.  This document explain one scenario and providing you with two options to resolve it.

Description: 

The main problem being the /etc/hosts file have the hostname as part of localhost resolving to the loopback address 127.0.0.1 - which is not recommended. The hostname should always resolve to a IPAddress and not to localhost(127.0.0.1).

The issue will be encountered only during the configuration and testing and not during installation.

We are trying to simulate the issue with two lab servers dbvlab01(Primary) dbvlab03(Standby).

 oracle@dbvlab01[/usr/dbv8/dbvnet]: cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
10.0.2.51 dbvlab01.dbvisit.co.nz dbvlab01
10.0.2.53 dbvlab03.dbvisit.co.nz dbvlab03

The netstat output for the port 7890 clearly shows the host as dbvlab01.dbvisit.co.nz

oracle@dbvlab01[/usr/dbv8/dbvnet]: netstat -a |grep 7890
tcp 0 0 dbvlab01.dbvisit.co.nz:7890 *:* LISTEN

To simulate the error In dbvlab03 i commented the hostname for dbvlab03.dbvisit.co.nz and added it to be part of localhost.

oracle@dbvlab03[/usr/dbv8/dbvnet]: cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 dbvlab03
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
10.0.2.51 dbvlab01.dbvisit.co.nz dbvlab01
#10.0.2.53 dbvlab03.dbvisit.co.nz dbvlab03 -- commented here
10.0.2.41 dbvlab101.dbvisit.co.nz dbvlab101
oracle@dbvlab03[/usr/dbv8/dbvnet]:

The netstat command for the port instead of listening to dbvlab03 it started to listen to
locahost,its imperative that we look into the hostname resolution for the netstat command whenever we are working on a dbvnet issue in V8 standby 

oracle@dbvlab03[/usr/dbv8/dbvnet]: netstat -a |grep 7890
tcp 0 0 localhost:7890 *:* LISTEN

Testing dbvnet from dbvlab01 to see if we are able to connect to the target dbvlab03 we receive the error:

oracle@dbvlab01[/usr/dbv8/dbvnet]: ./dbvnet -e "uname -n"
Error connect: Connection refused

Solution:

Option 1: Fix the /etc/hosts entries

Reverted the changes in the /etc/hosts file to make sure that the hostname resolve the the required IP address and not to the localhost loopback (127.0.0.1) address - see below for the new /etc/hosts file

oracle@dbvlab03[/usr/dbv8/dbvnet]: cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
10.0.2.51 dbvlab01.dbvisit.co.nz dbvlab01
10.0.2.53 dbvlab03.dbvisit.co.nz dbvlab03
10.0.2.41 dbvlab101.dbvisit.co.nz dbvlab101

oracle@dbvlab01[/usr/dbv8/dbvnet]: cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
10.0.2.51 dbvlab01.dbvisit.co.nz dbvlab01
10.0.2.53 dbvlab03.dbvisit.co.nz dbvlab03
10.0.2.41 dbvlab101.dbvisit.co.nz dbvlab101
10.0.2.42 dbvlab102.dbvisit.co.nz dbvlab102

After the updates you must restart the dbvnet process

1. Stop Dbvnet: ./dbvnet -d stop
2. Start Dbvnet: ./dbvnet -d start

Issued a test command from dbvlab01 to test if we are able to contact the remote host and it was successfully able to contact the remote host.

oracle@dbvlab01[/usr/dbv8/dbvnet]: ./dbvnet -e "uname -n"
dbvlab03.dbvisit.co.nz

Use the netstat command to confirm it is listening to the localhost or the hostname 

oracle@dbvlab03[/usr/dbv8/dbvnet]: netstat -a |grep 7890
tcp 0 0 dbvlab03.dbvisit.co.nz:7890 *:* LISTEN

To resolve the issue described above, changes to the /etc/hosts file is required where we have added the hostname as part of localhost to resolve to its specified IP address. If you have some restrictions in making these changes a workaround may be used by adding a alias to the hostname in /etc/hosts file to point to the correct IP address - see option 2 below.

Option 2: Implement a "workaround" 

Below are the steps which must be followed to do it. The steps are quite extensive and needs to be followed exactly.

The files that are modified as part of this is

1. Add hostname alias in /etc/hosts file in both primary and standby servers.

2. dbvnetd.conf file in both primary and standby servers.

3.  dbvhost_<DDC>.sh file in both primary and standby servers.

4. Update the DDC file in primary with SOURCE and DESTINATION parameter.

5. update the dbvagent.conf file in both primary and standby servers.

Step1:

dbvlab01 --> primary

dbvlab03 -->standby

oracle@dbvlab01[/home/oracle]: cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 dbvlab01
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
10.0.2.51 dbvlab01.dbvisit.co.nz dbvlab01 primary
10.0.2.53 dbvlab03.dbvisit.co.nz dbvlab03 standby
10.0.2.41 dbvlab101.dbvisit.co.nz dbvlab101
10.0.2.42 dbvlab102.dbvisit.co.nz dbvlab102

oracle@dbvlab03[/usr/dbv8/standby]: cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 dbvlab03
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
10.0.2.51 dbvlab01.dbvisit.co.nz dbvlab01 primary
10.0.2.53 dbvlab03.dbvisit.co.nz dbvlab03 standby
10.0.2.41 dbvlab101.dbvisit.co.nz dbvlab101

Step2: 

oracle@dbvlab01[/usr/dbv8/dbvnet/conf]: cat dbvnetd.conf
[general]
;logdir=log
passphrase=Test123
listener_address=primary
listener_port=7890
remote_address=standby
remote_port=7890
;rotation_size=100000000
;max_logs_size=1000000000
;cert=internal(conf/server.pem)
;prikey=internal(conf/server.pem)
;ca=internal(conf/ca.pem)
;certkey=internal(test)
;buffer_size=524288
debug=2
compress=1
;session_timeout=600
;workdir=../standby

oracle@dbvlab03[/usr/dbv8/dbvnet/conf]: cat dbvnetd.conf
[general]
;logdir=log
passphrase=Test123
listener_address=standby
listener_port=7890
remote_address=primary
remote_port=7890
;rotation_size=100000000
;max_logs_size=1000000000
;cert=internal(conf/server.pem)
;prikey=internal(conf/server.pem)
;ca=internal(conf/ca.pem)
;certkey=internal(test)
;buffer_size=524288
debug=2
compress=1
;session_timeout=600
;workdir=../standby

Step3:

changed the dbvhost_<ddc>.sh in both primary and standby

oracle@dbvlab01[/usr/dbv8/standby]: cat dbvhost_devdb.sh
echo primary
oracle@dbvlab03[/usr/dbv8/standby]: cat dbvhost_devdb.sh
echo standby

Step4:

oracle@dbvlab01[/usr/dbv8/standby/conf]: cat dbv_devdb.env |grep SOURCE
# SOURCE - Primary server name with primary database
SOURCE = primary
oracle@dbvlab01[/usr/dbv8/standby/conf]: cat dbv_devdb.env |grep DESTINATION
# DESTINATION - Standby server name with standby database.
DESTINATION = standby

Step5:

oracle@dbvlab01[/usr/dbv8/dbvagent/conf]: cat dbvagent.conf
[general]
;logdir=log
;repo=conf/dbvagent.db
passphrase=Test123
listener_address=primary
listener_port=7891
oracle@dbvlab03[/usr/dbv8/dbvagent/conf]: cat dbvagent.conf
[general]
;logdir=log
;repo=conf/dbvagent.db
passphrase=Test123
listener_address=standby
listener_port=7891

Once the above steps are completed, remember to restart dbvnet, dbvagent and dbvserver processes and test sending and applying log to ensure all processes are working as expected.