How to build a system that handles hundreds of millions of data points

Counting data points is easy. Building a system that stores them, answers questions about them and survives its own mistakes is the hard part. A fleet of sensors, a satellite image feed or a telemetry stream from a product in the field can all grow from a pilot into hundreds of millions of records a day without anyone planning for it. The design below shows what to decide first and what to postpone.

What does a high volume data pipeline look like?

A high volume data pipeline moves data through five stages, and each stage has one job. Ingestion accepts data from devices, a buffer absorbs bursts, a processor cleans and routes records, object storage keeps the raw copy for the retention period you set, and a query store serves the questions people ask.

The ingestion services are the front door. AWS says its IoT message broker supports MQTT, MQTT over WSS and HTTPS, and Microsoft says IoT Hub scales to millions of simultaneously connected devices and millions of events per second. IoT Hub can also route messages to Storage containers and Event Hubs. Systems that receive data from software, such as application logs, can skip this stage and write straight to the buffer.

How much storage do 5,000 sensors need?

Work the arithmetic before you pick any technology, and do it with a peak, not an average. The figures below are a worked example with stated assumptions, not a measurement.

Illustrative example: Consider a company that operates 5,000 devices, such as pumps, meters or vehicle trackers, and each sends one reading per second. Assume every reading is 100 bytes once it carries a device ID, a timestamp, a few measured values and a quality flag.

Those numbers translate directly into buffer capacity. Amazon states that each Kinesis shard supports up to 1,000 records per second and 1 MB per second for writes. If every reading is its own record, 5,000 records per second needs at least 5 shards, because the record limit binds before the byte limit does. If devices batch 10 readings into one message, the record rate drops to 500 per second and 1 shard covers it. Everything scales in a straight line: 100,000 devices means 8.64 billion records a day, about 315 TB a year and at least 100 shards. Microsoft's guidance for Event Hubs gives about 1 MB per second of ingress per partition on the Standard tier, so 0.5 MB per second fits one partition on bytes alone, though more partitions give consumers room to work in parallel. Treat both as starting points and check current quotas.

The reverse check also helps. At 200 million records a day, the average is about 2,300 per second. That is a modest write rate for many databases, so the peak, the retention window and the query pattern decide the design.

Which store fits which data shape?

The shape of the data picks the store. Forcing all three shapes into one database is a common reason these projects stall.

A time-series database is built for numeric readings queried by time. A columnar store keeps each column together, which makes scans over a few columns across billions of rows fast. Both are query stores, never the only copy. Keep the original files in object storage, and rebuild any query store from them if you must.

How do hot, warm and cold tiers and rollups keep cost down?

Most people query the last few hours in detail and everything older only in summary. Tiering follows that habit. The hot tier holds recent detail in the query store, the warm tier holds older detail as compressed files in object storage, and the cold tier holds archives you rarely open.

Object storage vendors build this in. Amazon S3 offers classes from Standard through Glacier Instant Retrieval, Flexible Retrieval and Deep Archive, with minimum storage durations of 90 days for Glacier Instant and Flexible Retrieval and 180 days for Deep Archive. Azure Blob Storage has hot, cool, cold and archive tiers, with archive data taking up to 15 hours to rehydrate. A lifecycle rule moves objects down the tiers on a schedule, so nobody does it by hand.

Rollups do the same job inside the query store. A rollup is a precomputed summary, such as the minimum, maximum, average and count per device per minute. Timescale's documentation describes this as continuous aggregates, which store results grouped by time bucket and refresh in the background, so a query for hourly averages no longer rescans the whole table. In the sizing example, a one-minute rollup turns 432 million daily rows into 7.2 million, a 60-fold cut, and an hourly rollup cuts it a further 60-fold.

Partition by time first and by device second. Time partitions let you drop or archive a whole day at once and keep queries on recent data small. A device-based key keeps one device's readings together in order. Event Hubs documentation notes that a per-device or per-user identity makes a good partition key, and it warns that changing partition counts later alters which partition each key lands in.

How do you keep a high-volume pipeline reliable?

Four failure modes appear in every stream system. Each has a standard answer.

Backpressure

Backpressure is what happens when data arrives faster than a downstream stage can process it. The buffer is the answer: it holds the backlog while consumers catch up, and you alert on consumer lag instead of discovering it from a customer. Retention sets how long the buffer can hold. Kinesis keeps records for 24 hours by default and up to 365 days, and Event Hubs allows 1 hour by default and up to 7 days on Standard. If a consumer outage could outlast retention, the raw copy in object storage is your safety net.

Duplicate records

Assume everything arrives at least once. AWS states that records may be delivered more than once because of producer retries and consumer retries, and that applications must anticipate this. Make writes idempotent, meaning that applying the same record twice has the same effect as applying it once. The usual method is a unique key per record, such as device ID plus timestamp plus sequence number, with the destination ignoring repeats. AWS suggests embedding a primary key in the record for strict cases.

Late and out-of-order data

Devices lose connectivity and send a batch an hour later. Apache Flink separates event time, when something happened on the device, from processing time, when your system saw it. It uses watermarks to declare how far event time has progressed, and it calls records that arrive after that point late elements. Decide how long you will wait before closing a time window, and send anything later to a correction path that updates the affected rollup.

Replay

Replay means reprocessing history from stored data, for example after you fix a bug or add a new metric. Event Hubs consumers can start from a specific offset, a timestamp or the beginning of the stream, but only within retention. For anything older, replay from object storage. This is the strongest reason to land raw data there first.

What does a real archive at scale look like?

The Sentinel-2 satellite program on the Registry of Open Data on AWS shows the object storage pattern with images. The registry describes a two-satellite constellation that covers the Earth's land surface globally every 5 days. Scenes and metadata sit in Amazon S3 buckets, one set of buckets is Requester Pays so the reader carries the access cost, and the registry lists notification topics for new scenes.

The lesson is structural. Each scene is an object, a catalog of metadata finds it, and an event announces new arrivals. Nobody loads pixels into a database. Companies with inspection photos, scans or drone footage can copy the same layout.

When do you not need any of this?

Skip the full pipeline when the volume is modest. A company with a few hundred devices reporting every minute generates about half a million records a day, which is a size a single relational database can store and index without a stream in front of it. Start there, write raw files to object storage in parallel, and add the buffer when load tests say you need it. Our guide to when to use pub/sub, a queue or Kafka covers that threshold in more detail, and how Meta handles scale shows what the far end looks like and what a smaller company should not copy.

What is the verdict?

Decide the questions first. List the five or ten questions people will actually ask of this data, such as "which devices exceeded a limit this week" or "what was the daily trend per site," and the time range each one needs. Those questions fix what you roll up and how long you keep detail.

Keep raw data cheaply in object storage, partitioned by date, because it lets you change your mind later. Pre-aggregate everything else, and let tiering move old detail to cheaper storage. Add components only when a measured limit forces you to, and size each one from a worked example like the one above.

How Kastling approaches this

A data platform engagement at Kastling would start with the questions and the retention rules, not the tools. The team would map the sources, agree the peak volume and the cost of a gap, and connect to the systems already in place. It would then design the buffer, raw storage and query layers on AWS or Azure, test with synthetic peak load and agree how success is measured. See AI & Cloud Infrastructure for the service, and types of databases for choosing the query store.