Persist to db and send messages – Fault Tolerant Manner

SO How to reliably/atomically update the database and publish messages/events? 

Link For example, you cannot guarantee that a commit to Cassandra and a message delivery to Kafka would be done atomically or not done at all.

Pattern: Transactional outbox

Let’s imagine that your application is using a relational database. A straightforward way to reliably publish messages is to apply the Transactional outbox pattern. This pattern uses a database table as a temporary message queue. As part of the database transaction that creates, updates, and deletes business objects, the service sends messages by inserting them into the OUTBOX table. Atomicity is guaranteed because this is a local ACID transaction. The OUTBOX table acts a temporary message queue.
The MessageRelay is a component that reads the OUTBOX table and publishes the messages to a message broker

This pattern has the following drawbacks:

• Potentially error prone since the developer might forget to publish the message/event after updating the database.

This pattern also has the following issues:

• The Message Relay might publish a message more than once. It might, for example, crash after publishing a message but before recording the fact that it has done so. When it restarts, it will then publish the message again. As a result, a message consumer must be idempotent, perhaps by tracking the IDs of the messages that it has already processed. Fortunately, since Message Consumers usually need to be idempotent (because a message broker can deliver messages more than once) this is typically not a problem.

There are a couple of different ways to move messages from the database to the message broker. We’ll look at each one.

PUBLISHING EVENTS BY USING THE POLLING PUBLISHER PATTERN Link
If the application uses a relational database, a very simple way to publish the messages inserted into the OUTBOX table is for the MessageRelay to poll the table for unpublished messages.
Polling the database is a simple approach that works reasonably well at low scale. The downside is that frequently polling the database can be expensive.

PUBLISHING EVENTS BY APPLYING THE TRANSACTION LOG TAILING PATTERN  Link
A sophisticated solution is for MessageRelay to tail the database transaction log (also called the commit log). Every committed update made by an application is represented as an entry in the database’s transaction log. A transaction log miner can read the transaction log and publish each change as a message to the message broker. Figure 3.14 shows how this approach works.
There are a few examples of this approach in use: Debezium, LinkedIn Databus, DynamoDB streams

 

Ofcourse, the transaction log tailing pattern has its own set of drawbacks. As mentioned here, some of them are:

• Relatively obscure
• Database specific solutions
• Low level DB changes makes it difficult to determine the business level events
• Tricky to avoid duplicate publishing