This is the engine behind a live outage map that thousands of New Jersey residents check during storms. Every 15 minutes it turns millions of raw records into a simple, up-to-date answer for each town — entirely on its own, with no one touching it. Here's the trip that data takes, from the moment a file shows up to the status on someone's screen. (Generalized — no internal names.)
An upstream system automatically drops a handful of files into cloud storage on its own schedule — a yearly list of towns, a daily list of every customer, and live outage updates every 15 minutes. Nobody uploads anything by hand.
The customer file is enormous — about 41 million lines, far too big to open all at once. So instead of loading the whole thing, the program reads it as a stream, one record at a time, and immediately drops each record into one of 256 buckets. When a bucket fills up (about 1 MB) it's saved off to storage and emptied — then it keeps filling. Nothing ever piles up, so the memory it uses stays flat and tiny no matter how big the file gets.
Every record has a long ID number. Divide it by 256 and keep the leftover — a number from 0 to 255. That leftover picks which of 256 "buckets" (called shards) the record goes in. So 41 million lines get sorted into 256 small, evenly-filled files — and finding one record later means opening one small file, never digging through everything.
Quick to do, always gives the same answer, and spreads records evenly — the same ID always lands in the same bucket.
Because live events and customer records are sorted by the very same rule, a match always sits in the same bucket. To pair them, the system opens one event bucket and the matching record bucket, then lines up the IDs — no giant search, no scanning everything.
All those town totals get written into one final file — townSummary.json. Think of it as the cheat sheet the page reads from.
Open the page and it just grabs that pre-made cheat sheet and shows your town's status. It feels instant — because all the hard work was already done ahead of time. No one waits while 41 million records get crunched.
This pipeline is one project — there's more where it came from.