Event-Driven Database Architecture with Harper MQTT Topics

Event-driven database architecture with MQTT topics blurs the traditional lines between persistent storage and real-time messaging. By treating database tables as MQTT topic namespaces, this pattern provides a unified system where data persistence and live updates coexist seamlessly. In this model, developers no longer juggle separate systems or write extra synchronization logic because every change in the database can immediately flow to interested clients, and clients can even publish messages that directly update database records.

What Is MQTT?

MQTT (Message Queuing Telemetry Transport) is a lightweight publish/subscribe protocol designed for resource-constrained devices and high-latency or unreliable networks. Clients connect to an MQTT broker, subscribe to topics of interest, and publish messages tagged with topic names. When a message is published, the broker dispatches it to all clients subscribed to that topic. Its simplicity and efficiency make MQTT a natural fit for event-driven architectures.

The Traditional Divide

In conventional systems, real-time features are built by combining:

• A database for durable, persistent storage
• A message broker (e.g., MQTT, Kafka) for live event distribution
  
  

This separation leads to challenges:

• Data Synchronization Complexity: Applications must keep the database and message broker in sync, often with custom polling or change-data-capture logic.
• Dual State Management: State exists in both the database and messaging layers, forcing developers to write extra code to coordinate updates.
• Increased Infrastructure: Running and monitoring two systems increases operational overhead and potential failure points.

How It Works

Instead of treating databases and brokers as distinct, event-driven database architecture uses database tables themselves as MQTT topic namespaces. Defining a table in your schema automatically creates a corresponding topic hierarchy:

type UserProfiles    @table @export
type ChatMessages    @table @export
type IoTSensorData   @table @export

‍

These declarations yield topics like:

• UserProfiles/123
• ChatMessages/room-456/msg-789
• IoTSensorData/device-001/temperature

‍

Each topic maps one-to-one with a record or logical path in your schema.

‍

When a client subscribes to an MQTT topic backed by a table, they effectively open a live view on that record.

// Polling approach (traditional):
setInterval(async () => {
  const user = await database.get('UserProfiles', 123);
  updateUI(user);
}, 1000);

// Database-native MQTT:
mqttClient.subscribe('UserProfiles/123');
mqttClient.on('message', (topic, message) => {
  const userData = JSON.parse(message);
  updateUI(userData);
});

‍

In the traditional approach, clients poll at intervals, risking stale data or unnecessary queries. But with the database-native MQTT, clients will receive updates only when the record changes, as though the table itself is a live data stream.

Harper's MQTT topics align directly with database table paths. A topic like my-resource/some-id corresponds exactly to the record in the my-resource table with ID some-id. This creates seamless interoperability between database operations and real-time messaging:

mqttClient.subscribe('my-resource/123');
mqttClient.on('message', (topic, message) => {
  console.log('Record updated:', JSON.parse(message));
});

‍

Conversely, publishing MQTT messages with the retain flag directly updates the database record:

mqttClient.publish('my-resource/123', JSON.stringify({
  name: "Updated Profile",
  status: "active"
}), { retain: true });

‍

A key feature of MQTT is retained messages, which ensures subscribers immediately receive the current state upon subscribing, no extra fetch required.

// Subscribe and get current state instantly
mqttClient.subscribe('UserProfiles/123');
// First message contains current data; later messages are updates.

‍

This atomic combine-of-fetch-and-listen eliminates race conditions and complex caching layers.

‍

Publishing with the MQTT retain flag set to true updates the database record and notifies all subscribers. Without retention, messages are ephemeral and don’t alter the stored state.

• Retained messages ensure eventual consistency for stateful data (e.g., user profiles, sensor readings).
• Non-retained messages suit ephemeral events (e.g., chat messages, notifications).

Conclusion

Event-driven database architecture with MQTT topics are a paradigm shift in how we think about real-time applications. By treating database tables as natural message topics, this approach eliminates the artificial separation between data persistence and real-time messaging.

This unified model simplifies application development, reduces infrastructure complexity, and provides natural scalability for real-time features. As applications increasingly require immediate responsiveness to data changes, this architecture pattern offers a compelling foundation for building the next generation of real-time, data-driven applications.

The main thing here is recognizing that databases and message brokers aren't inherently separate concerns, they're different aspects of the same fundamental need to manage and distribute state changes in real-time systems. With this unified view, developers can build more robust, scalable, and maintainable real-time applications.