We're here to help!

Custom mapping/renaming of Oracle table to Kafka topic

Follow

Question:

We are evaluating the Dbvisit Replicate Connector for Kafka, and using the Confluent Platform (open source) version  3.0.0. Our source is an Oracle table. We can successfully send messages to a Kafka topic, whose name is by default:
<TOPIC-PREFIX>-<ORACLE-SCHEMA-NAME>.<ORACLE-TABLE-NAME>


Our goal is to be able to send messages to a specific Kafka topic, so we need to be able to fully customize the Kafka topic name that we send messages to. Looking at the documentation it seems that only the TOPIC-PREFIX is configurable via the topic.prefix configuration parameter in the Dbvisit Replicate Connector for Kafka properties file.

Is it possible to determine the name of the output Kafka topic?

 

Solution:

Table/schema renames are not (currently) supported for Kafka as a target (that is, using the Dbvisit Replicate Connector for Kafka). But this can be achieved with a new feature introduced to Kafka Connect in version 3.2.0, and that is SMT (single message transformations). This can be easily achieved on a per table basis with a minimum amount of required configuration. 


For example, for a table in Oracle called SCOTT.TEST, and assuming also that you were using the default topic prefix of "REP-" (although this can be omitted and/or changed to suit your preference) specified within the Replicate Connector properties file, this can be mapped to a topic called REP-MIKE.NEWTEST as per the following:

transforms=MyRegex
transforms.MyRegex.type=org.apache.kafka.connect.transforms.RegexRouter
transforms.MyRegex.regex=REP-SCOTT.TEST
transforms.MyRegex.replacement=REP-MIKE.NEWTEST

 

Have more questions? Submit a request

Comments