When to use pub/sub, a queue or Kafka: a plain guide for growing businesses

When a new order is placed, five things should happen: the warehouse is told, the invoice is created, the customer gets an email, the finance dashboard updates, and an AI forecast gets another data point. The simplest way to do this is for the order system to call each one directly. That works for a while, and then one of the five is slow, one is down for an hour, and the order system is the one that fails.

When should you use pub/sub, a queue or a direct API call?

Use a direct call when the caller needs the answer to continue. Use a queue when work can wait and each task should be handled by one worker. Use pub/sub when several systems need to react to the same event. Use a Kafka-style log when many readers need the same history and need to replay it.

What is pub/sub?

Publish/subscribe (pub/sub) is a pattern where a sender publishes a message to a named topic, and every system subscribed to that topic receives its own copy. The publisher does not know who is listening. That is the point: adding a sixth system to the order example means adding a subscription, not changing the order system.

AWS describes Amazon SNS as a fully managed pub/sub service that delivers to endpoints such as HTTP endpoints, email, SMS, queues and Lambda functions. The same guide notes that SNS messages are not persisted and are delivered in real time. SNS retries failed deliveries but does not store messages for later, so a subscriber that stays down past the retry policy can miss them unless a durable queue or a dead-letter queue sits behind the subscription. The standard remedy is to subscribe a queue to the topic, so each system gets a durable inbox. AWS calls this fan-out, and its SQS documentation recommends the combination to prevent message loss when subscribers are offline.

What is a queue, and when is it enough?

A queue holds work until a consumer takes it, and each message goes to one consumer. Amazon SQS shows the lifecycle: a producer sends a message, a consumer receives it and has a visibility timeout in which to process it, and then deletes it so it is not handled again. If the consumer fails, the message reappears. SQS keeps messages for 4 days by default and up to 14 days, and offers dead-letter queues for messages that keep failing.

Queues are the right tool when you want to absorb bursts, retry safely and decouple a slow system from a fast one. Calling an ERP that accepts ten requests a second is the classic case. Standard queues deliver at least once, so a message can arrive twice and the handler must tolerate that, a theme we cover for webhooks in webhooks vs polling. Most small integrations never need more than queues and topics.

What is Kafka, and how does it work?

Apache Kafka is a platform that stores streams of events in a durable, ordered log and lets many applications read them. The Kafka documentation describes it as an event streaming platform that combines publishing and subscribing, durable storage, and stream processing.

Five ideas cover how it works:

• Topic. A named stream of events, such as orders. Topics are multi-producer and multi-subscriber.
• Partition. A topic is spread over several partitions on different servers, called brokers. Events with the same key, for example one customer ID, go to the same partition, and consumers read a partition's events in the order they were written.
• Retention. The Kafka documentation states that events are not deleted after consumption. You set per topic how long they are kept.
• Offset. Each consumer tracks its position in each partition, called an offset, and Confluent's consumer design notes that a consumer can specify an offset to reconsume data if needed. This is replay.
• Consumer group. A set of consumers that share a topic's work. Confluent's documentation states that each partition is consumed by exactly one consumer within each consumer group at any given time, while different groups each read the whole stream independently.

The consequence for a business is that the log becomes a shared record of what happened. A new system, such as an AI forecasting service, can start reading from last week's events instead of asking every other team for an export. A bug fix in the invoicing consumer can be followed by reprocessing yesterday's events. A queue cannot do either, because the messages are gone.

The costs are also real. Partitions have to be planned, because they set how many consumers in a group can work in parallel. Order is preserved within a partition only. Someone must monitor consumer lag, the gap between the newest event and the one a consumer has reached.

Where is Kafka used in the real world?

Kafka was built at LinkedIn, and the origin explains what it is for. Jay Kreps, one of its creators, wrote that LinkedIn built Kafka to serve as a central hub of data streams. Before it, the company had built improvised piping between systems on an as-needed basis and ended up with pipelines between all kinds of different systems. Kafka replaced that mesh of point-to-point connections with one log that producers wrote to and consumers read from, each at its own position.

LinkedIn's engineering site describes Kafka as a core part of its infrastructure, powering activity tracking, message exchanges and metric gathering. It states a scale of over 100 clusters, more than 4,000 brokers and over 7 trillion messages processed daily. That is a scale few companies approach, and it is a poor benchmark for a 100-person business.

Other uses are documented first-hand:

• Fleet and shipment tracking, payments and IoT. The Kafka project's own introduction lists processing payments in stock exchanges, banks and insurers, tracking cars, trucks, fleets and shipments, and capturing IoT sensor data.
• Pricing, dispatch and fraud at Uber. Uber's engineers describe streaming applications that calculate up-to-date pricing, enhance driver dispatching and fight fraud, with pipelines that process data from Kafka and write results back to Kafka. They also report that replaying weeks of old data into Kafka for a backfill risked overwhelming the system, so they built a backfiller that reads history from Hive instead, a useful limit on what replay can do.

Those uses share three traits: continuous high-volume events, several independent consumers of the same stream, and a need to reprocess history.

How do the AWS and Azure options compare?

The cloud providers each offer a topic, a queue and a stream, and the names differ. This table maps them, using each provider's own documentation.

Details worth knowing. AWS's documentation says MSK runs open-source versions of Apache Kafka, so existing applications and tools work without code changes, and that you specify the number of broker nodes per Availability Zone, with a minimum of one per zone. Microsoft states that Event Hubs lets you run Kafka workloads without code changes or cluster management, and that Service Bus supports transactions, duplicate detection and dead-lettering while Event Hubs does not offer dead-lettering. Azure's own guidance shows the services side by side in one system: Service Bus to process orders, Event Hubs to capture site telemetry, and Event Grid to respond to events like an item being shipped.

Which should a 20 to 200 person company start with?

Start with a managed queue or topic. On AWS, that means SNS in front of one SQS queue per consuming system. On Azure, it means Service Bus topics and subscriptions. This covers most business integration at this size, it needs almost no operations work, and the services are priced by use.

Illustrative example: Consider a distributor whose order system must notify the warehouse system, create an invoice in the accounting package, send a confirmation email and update a dashboard. The team publishes one order-placed message to an SNS topic and gives each of the four systems its own SQS queue. When the accounting package goes down for two hours, its queue fills and drains afterwards, and nothing else is affected. A year later, an analyst wants to train a demand forecast on the past six months of orders. The queues cannot answer that, so the team loads the history from the order database. To let future analysts replay recent events, it adds a Kinesis stream that also receives order events from then on and keeps them for the retention period. They did not need Kafka on day one, and the early design did not block adding a log later.

Move toward a Kafka-style log when you see these signals:

1. Several teams or systems need the same history. Not only the latest event, but the stream from last Tuesday, read at different speeds.
2. You must replay. Reprocessing after a bug, loading a new system, or rebuilding a search index from events.
3. Volume is sustained and high. A steady high rate rather than occasional bursts. Check the quotas of your queue or topic service against your forecast.
4. You process streams over time. Counting, joining and flagging events within windows, as in fraud signals or fleet tracking.
5. A change feed from your databases feeds a warehouse or AI models. Our guide to connecting AI to ERP, CRM and data warehouse explains why that matters for AI projects.

If none of these apply, a log adds cost and skills you do not need. If one or two apply, choose a managed stream, Kinesis or Event Hubs, before a managed Kafka cluster, and take MSK when you need Kafka's tools or have Kafka skills. Pair the choice with an honest look at volume and failure handling, covered further in high-volume data systems.

How Kastling approaches this

The starting point is the business flow, not the broker: which events matter, who must react to them, what a lost or duplicated event would cost, and who is told when something fails. Kastling would connect the systems a business already runs, begin with the simplest managed queue or topic on AWS or Azure that fits, design for duplicates and dead-lettered messages from the start, and agree how success is measured before building. A move to a streaming log would be proposed only when replay or volume actually calls for it. This work belongs to AI & Cloud Infrastructure, and the team brings in AWS and Azure specialists where a design needs them.