Queues : Inbox Outbox design

Posted on by Deep.Kulshreshtha

The Problem ( in simple words )

.

Softwares lose data working with MQ systems.

.

High volume processing requires data to be managed within memory. MQ-based systems like Kafka, Rabbit MQ, and Active MQ are some examples.

.

While these systems manage millions of messages, they are ‘transient in nature’. Meaning – once a message is delivered. It’s done ! No redelivery, no success/ failure feedback, no querying the data back.

.

This means the software ought to work with 100% reliability. Anyone in the real world knows that simply isn’t true !

.

In the real world, software breaks. Code runs into unhandled conditions and error scenarios.

.

.

The Problem ( in nerdy words )

.

From : https://event-driven.io/en/outbox_inbox_patterns_and_delivery_guarantees_explained/

.

The problems of “at least once” and “exactly once” are NOT managed by MQ systems.

.

.

At least once:

.

A guarantee that a message will be delivered at least once. Maybe more.

If we receive a message multiple times – then our system should be able to handle it ( be idempotent )

.

Idempotent:

.

If I send a message once OR multiple times, the system will remain in the same state.

In other words – sending the message multiple times, will NOT mess up the system.

.

Exactly once:

.

The guarantee that a message will be delivered once and only once.

Kafka provides this with the groupId, offset counts for each topic.

.

.

At most once:

.

The message will be delivered a maximum once

.

And it is possible for the message to be NOT delivered even once. Ofcourse, we don’t want this to happen.

.

.

.We want a message to be delivered and processed only once !

.

Since a memory-only system is transient in nature. Persisting message states and guaranteeing “exactly once” delivery is NOT possible within MQ systems....

.

.

The Solution ( in simple words )

.

Persist incoming and outgoing data into local memory. ( Inbox Outbox pattern )

.

A mailing system works as follows:

.

1.Fetch incoming mails from a server to a local folder – Inbox
2.Send outgoing mails from a local folder to the server – Outbox

.

This design decouples the data exchange ( message delivery ), from data processing ( message processing )

.

How does it help ?

.

●In case of a read-time failure. We can re-read the data from Inbox.
●In case of a write-time failure. We can re-send the data from Outbox.

.

.

The Solution ( in Nerdy words )

.

Use the Inbox/ Outbox design pattern.

.

Inbox

.

1.Create a table with the relevant fields.
2.All incoming messages should come into this table.
3.A daemon reads data from the table and sends them for processing

.

.

Outbox

.

1.Create a table with the relevant fields
2.All outgoing messages are first written into the table.
3.A daemon reads data from the table and sends them to MQ.

.

The tables can also be used for audit and reporting purposes.

.

.

.

Problem solved !

2 thoughts on “Queues : Inbox Outbox design”

  1. Seems like we are just discussing decoupling message delivery and receipt from logic on received messages. Yes?

    1. Thats right @Paul.

      You’ve put it as an intuitive design, but most standard libraries do NOT provide this function. And therefore this needs to be custom implemented.

Comments are closed.