"We're hiring developers. Find the three words hidden in the code to apply."

An input field. A placeholder: [xxxxx]-[xx]-[xxxx]. No other instructions.

Okay. Say less.

Step 1: The HTML Comment (Word 1: world)

First move: F12 Inspect Element. Always view source first, that's just developer law.

The HTML was clean. Almost suspiciously clean. But right at the top, sandwiched between two normal-looking comments, was this:

<!-- build:a9c3e7f2d8b1 -->
<!-- 1:776f726c64 -->
<!-- deployed:2026-06-10T04:22:18Z -->

The build: and deployed: comments make total sense. But 1:776f726c64? That's giving treasure map energy.

776f726c64 is a hex string. Decoded: Hex To ASCII

77→w  6f→o  72→r  6c→l  64→d

"world", 5 letters. Exactly matches [xxxxx]. And the 1: prefix told me this was clue number one, with more coming.

Step 2: The Config File (Word 2: is)

The JS file (p.js) had a fire-and-forget fetch that looked… unnecessary:

fetch('./static/config.json').catch(function(){});

It fetches the config but does absolutely nothing with the result. No callback. No usage. Just vibes. That's sus.

I opened config.json and found this:

"meta": {
    "sha": "a9c3e7f2d8b1",
    "seq": 2,
    "seed": "aXM=",
    ...
}

"seq": 2 tells you it's clue #2. And "seed": "aXM=" is Base64:

aXM= → 0x69 0x73 → 'i' 's'

"is", 2 letters. Matches [xx]. Respectfully, hiding a 2-letter word in a Base64-encoded JSON field called seed is kind of diabolical (I respect it).

Step 3: The Console Log (Word 3: ours)

Back in p.js, the _boot() function logs a bunch of fake system startup messages. Very aesthetic. All styled the same grey:

var s = 'color:#bbb; font-size:11px;'; // light grey, every log uses this
var h = 'color:#555; font-size:11px;'; // darker, used exactly once

16 console logs. 15 use s. One uses h:

console.log('%c[paasa] ours', h);

Not s. Not a technical message. Just "ours". Styled differently so it stands out in the console if you're paying attention (I almost wasn't).

"ours", 4 letters. Matches [xxxx]. Clue #3, hiding in plain sight.

The Passphrase

Three encoding methods. Three files. Three words.

world-is-ours

Typed it in. Hit enter. It worked.

Is this the hardest CTF I've ever done? No. Was it genuinely clever and fun? Yeah actually. The three layers (hex, Base64, and a visual diff) each required a slightly different way of looking at the code. Not brute force. Just attention.

Which is probably exactly what they're hiring for.

The Assignment

2CentsCapital sent over a backend engineering assignment. The brief was to build a trade clearing and analytics engine. HTTP endpoint for orders, WebSocket feed for streaming, a matching engine for limit and market orders with price-time priority, persistence to PostgreSQL, and load test evidence. The performance target was 2,000 orders per second with sub-100ms median latency. Estimated effort was six to twelve hours.

The six to twelve hour estimate was optimistic. What started as "build a limit order book" turned into a tour through Kafka delivery semantics, JVM heap forensics, and Go goroutine lifecycle management. The assignment asked for 2,000 req/s. The system ended up doing 18,210. The bugs that stood between those two numbers are what this post is about.

The Tech Stack and Why It Looks Like This

The First Version Was Wrong

Most of these decisions were not made upfront. They were made incrementally, each one forced by a problem with the previous approach.

The first version was a single Go process. HTTP handler receives an order, calls the matching engine, writes the result to PostgreSQL, sends a WebSocket event, returns a response. Clean. Simple. Wrong.

The problem became obvious the moment I thought about the load test. A single synchronous request chain means the HTTP handler is blocked waiting for the matching engine, which is blocked waiting for the database write, which means every client is waiting for all of it. At 2,000 concurrent orders per second, that chain becomes a bottleneck at whichever step is slowest. You do not have a trading system. You have a request queue wearing a trading system's clothes.

Why Not Just Use Goroutines?

The obvious Go answer is to fire a goroutine and return immediately.

func (h *Handler) CreateOrder(c *gin.Context) {
    order := parseOrder(c)
    go matchEngine.Process(order)
    c.JSON(202, gin.H{"status": "accepted"})
}

This fixes the blocking problem. For a single-node demo with no persistence requirement, it works. The assignment actually allows it.

The problem is everything else. If the process crashes mid-match, the order is gone. No log, no replay, no recovery. At 2,000 req/s you are spawning goroutines faster than they drain if anything downstream is slow, and goroutines are cheap but unbounded. The matching goroutine still needs to write to PostgreSQL and notify WebSocket clients, so if you call those directly you have just moved the coupling, not removed it.

Goroutines solve the concurrency problem inside a single process. They do not solve durability, backpressure, or cross-service decoupling. You end up using both anyway: goroutines for parallelism inside the matching service, and Kafka for communication between services.

The Decoupling Problem

The fix is to break the chain. The HTTP handler should accept an order and return immediately. The matching engine should process it whenever it gets there. The database writer should persist trades whenever they land. The WebSocket broadcaster should emit events independently. None of them should know whether the others are alive.

Once you decide that, you need something to carry messages between them. That something is Kafka.

Kafka gives you three things that matter here. First, durability: messages are written to disk, so a service restart does not lose anything. Second, replayability: you can rebuild state by replaying the log from any offset. Third, consumer group semantics: multiple independent services can subscribe to the same topic without stepping on each other. The matching engine and the database service both read from the same ORDER_ACCEPTED topic, each getting their own copy, each tracking their own offset.

The Stack

Go is the language because goroutines are cheap enough to run one per trading instrument with no meaningful memory overhead, and channels give you a communication primitive that maps directly to how an order book works. Orders go in. Trades come out. That is a channel.

Apache Kafka carries every message between every service. The sarama library from IBM handles the Go client side. It is the only part of this stack that requires reading actual documentation before using it, which became clear the hard way.

PostgreSQL is the system of record. Every matched trade eventually lands here. Redis does not sit in front of it as a read cache. What it actually does is more specific: the IdempotencyKey middleware checks an Idempotency-Key header on every POST /orders request, looks it up in Redis, and returns a 409 Conflict if the key already exists. A successful order publish sets the key with a one-hour TTL. This prevents duplicate submissions under concurrent retries. The GET endpoints hit PostgreSQL directly. No read cache exists yet.

The four services that came out of this design:

• API Service: accepts HTTP requests, validates orders, publishes to the ORDER_ACCEPTED Kafka topic, returns 202 immediately.

• Matching Service: consumes ORDER_ACCEPTED (group: matching-engine-group), runs the in-memory order book logic, publishes matched trades to TRADE_EXECUTED.

• Database Service: consumes both ORDER_ACCEPTED (to insert each order into PostgreSQL) and TRADE_EXECUTED (to atomically update order quantities and insert the matched trade). Both consumers run in the same process under consumer group persistence-trades.

• Stream Service: consumes TRADE_EXECUTED (group: stream-trades), broadcasts real-time events to connected WebSocket clients at /ws/trades.

All four compile from the same Go codebase into separate binaries. Docker Compose orchestrates all of them along with Kafka, Zookeeper, PostgreSQL, and Redis.

Phase 1: Building the Foundation and Discovering the Wrong Abstraction

How a Matching Engine Actually Works

A limit order book is two sorted lists: bids sorted highest-to-lowest, asks sorted lowest-to-highest. When a new order arrives, if the best bid price is greater than or equal to the best ask price, a trade executes. Otherwise the order waits.

The videos that helped me understand this from first principles:

• How orders affect the order book (Udacity)

• Understand the orderbook like a quant

The natural data structure for "give me the highest bid quickly" is a max-heap. The natural structure for "give me the lowest ask quickly" is a min-heap. Go's container/heap package gives you both with a single interface. That is what OrderBook uses.

The order book lives entirely in memory. Matched trades are safe because Kafka offsets are not committed until the database write succeeds, so they replay on restart. Unmatched limit orders sitting in the heaps are gone.

The Mutex Mistake

The first version of the matching engine was something any Go developer would write naturally, which is how you know it was wrong. I had an OrderBook struct holding two heaps, protected by a sync.Mutex. Every operation locked it. It seemed correct.

type OrderBook struct {
    mu         sync.Mutex
    Instrument string
    BuyOrders  *OrderHeap
    SellOrders *OrderHeap
    TradeOut   chan *trades.Trade
}

func (ob *OrderBook) AddOrder(order *orders.MatchOrder) {
    ob.mu.Lock()
    defer ob.mu.Unlock()
    if order.Side == "BUY" {
        heap.Push(ob.BuyOrders, order)
    } else {
        heap.Push(ob.SellOrders, order)
    }
}

The problem, which became obvious once I stopped congratulating myself on how clean this looked, was that TryMatch did not lock the mutex. I was protecting writes but not reads. Two goroutines could be racing on the same heap and producing garbage fills. Even setting that aside, the mutex sends the wrong message to every future reader. It implies this struct is accessed concurrently and needs careful handling. The struct should never be accessed concurrently in the first place.

The Right Model: One Goroutine Per Instrument

The insight is that BTC-USD and ETH-USD have zero relationship to each other. Their order books are completely independent. If each instrument gets its own goroutine, that goroutine owns its order book exclusively. No goroutine ever touches another instrument's book. There is nothing to lock.

To understand goroutines and channels deeply, this video is worth your time:

• Golang Concurrency - All the Basics you have to know!

The implementation uses a map of channels. When an order arrives for BTC-USD, a dispatcher looks up the BTC-USD channel, creates it if it does not exist, and sends the order down it. A dedicated goroutine on the other end reads from that channel sequentially and runs the matching logic. No two goroutines ever share an order book.

func (engine *Engine) getOrderChannel(instrument string) chan *orders.MatchOrder {
    engine.mutex.RLock()
    ch, exists := engine.OrderChannels[instrument]
    engine.mutex.RUnlock()

    if exists {
        return ch
    }

    engine.mutex.Lock()
    defer engine.mutex.Unlock()

    if ch, exists := engine.OrderChannels[instrument]; exists {
        return ch
    }

    newCh := make(chan *orders.MatchOrder, 100)
    engine.OrderChannels[instrument] = newCh

    go engine.runInstrumentProcessor(instrument, newCh)
    return newCh
}

func (engine *Engine) runInstrumentProcessor(instrument string, orderChan <-chan *orders.MatchOrder) {
    book := orderbook.NewOrderBook(instrument)
    book.TradeOut = engine.TradeChannels

    for order := range orderChan {
        book.TryMatch(order)
        if order.Quantity > 0 && order.Type == "LIMIT" {
            book.AddOrder(order)
        }
    }
}

matching/engine/engine.go

The engine.mutex protecting OrderChannels is the one legitimate lock: it serializes channel creation using a double-check pattern while keeping the hot path read-only. The OrderBook itself has no mutex, by design.

Phase 2: The Matching Logic Had Two Silent Bugs

The Market Order Corruption

The assignment requires market orders. A market order has no price. It matches against whatever is available on the book until it fills or the book runs out of liquidity. The bug was simple and quietly catastrophic. If a market order arrived and could not fully fill, the leftover quantity was being pushed into the limit order book.

for order := range orderChan {
    book.TryMatch(order)
    if order.Quantity > 0 && order.Type == "LIMIT" {
        book.AddOrder(order)
    }
}

matching/engine/engine.go

Without order.Type == "LIMIT", a market order with remaining quantity and a price of 0.0 would sit in a price-sorted heap alongside limit orders at $70,000. The container/heap invariant breaks. Future matches execute at incorrect prices or against incorrect counterparties. In a real exchange, this is the kind of bug that lives quietly in production until someone's trade history looks like abstract expressionism.

The Price-Time Priority Bug

Exchanges make one fairness promise above everything else. If two orders arrive at the same price, the earlier one gets matched first. This is price-time priority. The original OrderHeap.Less only compared prices.

func (h *OrderHeap) Less(i, j int) bool {
    if h.IsBuy {
        return h.data[i].Price > h.data[j].Price
    }
    return h.data[i].Price < h.data[j].Price
}

matching/orderbook/heap.go

When two orders share the same price, Less returns false for both comparisons, making them equal. Go's container/heap is not stable. Equal elements can appear in any order depending on heap state after previous push and pop operations. The trader who submitted first might get matched second. Randomly. With no error, no log, and no way to detect it.

The fix was adding a Timestamp field to MatchOrder, populated with time.Now().UnixNano() at the moment the order hits the API service.

type MatchOrder struct {
    ID         uuid.UUID `json:"id"`
    ClientID   uuid.UUID `json:"client_id"`
    Instrument string    `json:"instrument"`
    Side       string    `json:"side"`
    Type       string    `json:"type"`
    Price      float64   `json:"price"`
    Quantity   float64   `json:"quantity"`
    Timestamp  int64     `json:"timestamp"`
}

internals/orders/models.go

Then update the comparator to use the timestamp as a tiebreaker.

func (h *OrderHeap) Less(i, j int) bool {
    if h.IsBuy {
        if h.data[i].Price != h.data[j].Price {
            return h.data[i].Price > h.data[j].Price
        }
        return h.data[i].Timestamp < h.data[j].Timestamp
    }
    if h.data[i].Price != h.data[j].Price {
        return h.data[i].Price < h.data[j].Price
    }
    return h.data[i].Timestamp < h.data[j].Timestamp
}

matching/orderbook/heap.go

The timestamp is serialized into the Kafka message so arrival order survives the network boundary. For buy orders, highest price wins; ties go to whoever arrived first. For sell orders, lowest price wins with the same tiebreaker. This works correctly on a single node; multi-instance deployments need a logical clock instead.

Phase 3: The Kafka Layer Had a Data Loss Hole and a Duplicate Risk

For Kafka fundamentals, this is the best single video I found:

• Master Apache Kafka in One Shot: LLD + HLD Setup + Partitioning + Pub/Sub

The Silent Trade Vanisher

The original consumer code silently dropped trades whenever the database was unavailable, committed the offset, and moved on. No error surfaced. No alert fired. The trades ceased to exist.

To understand why this matters, you need to know how Kafka consumer offsets work. Every partition in a Kafka topic is an ordered log of messages. Consumers track their position in that log using an offset. When you call MarkMessage, you are telling Kafka that you have successfully processed up to this point. On the next restart, Kafka will start delivering from after that offset. If you mark a message as done before you actually finish processing it, it is gone. Kafka will never redeliver it.

type ConsumerHandler struct {
    Process func(message *sarama.ConsumerMessage)
}

func (h ConsumerHandler) ConsumeClaim(session sarama.ConsumerGroupSession, claim sarama.ConsumerGroupClaim) error {
    for msg := range claim.Messages() {
        if h.Process != nil {
            h.Process(msg)
        }
        session.MarkMessage(msg, "")
    }
    return nil
}

kafka/consumer.go

MarkMessage runs unconditionally. If Process fails halfway through a database write, the trade is committed as processed and disappears from the Kafka log. Matched in the engine. Published to Kafka. Consumed by the database service. Failed to write. Gone. The fix was changing Process to return an error and making MarkMessage conditional on success.

type ConsumerHandler struct {
    Process func(message *sarama.ConsumerMessage) error
}

func (h ConsumerHandler) ConsumeClaim(session sarama.ConsumerGroupSession, claim sarama.ConsumerGroupClaim) error {
    for msg := range claim.Messages() {
        if h.Process != nil {
            if err := h.Process(msg); err != nil {
                log.WithError(err).Errorf(
                    "failed to process message topic=%s partition=%d offset=%d",
                    msg.Topic, msg.Partition, msg.Offset,
                )
                continue
            }
        }
        session.MarkMessage(msg, "")
    }
    return nil
}

kafka/consumer.go

This is at-least-once delivery: a trade might be processed more than once during recovery, but it will never be dropped. A unique constraint on trade_id in PostgreSQL makes duplicate processing a no-op.

func (s *Service) UpdateDatabase(ctx context.Context) {
    handler := kafka.ConsumerHandler{
        Process: func(msg *sarama.ConsumerMessage) error {
            var trade Trade
            if err := json.Unmarshal(msg.Value, &trade); err != nil {
                return err
            }
            if err := s.repo.ApplyTrade(ctx, &trade); err != nil {
                log.WithError(err).Error("failed to apply trade to database")
                return err
            }
            log.Info("Trade applied to database")
            return nil
        },
    }
    consumer := kafka.NewConsumer("TRADE_EXECUTED", "persistence-trades", handler)
    defer consumer.Close()
    consumer.Start()
}

internals/trades/service.go

Idempotent Producer

Without Idempotent = true, a network timeout after Kafka has received but not acknowledged a message causes the producer to retry. Kafka stores the duplicate. The matching engine sends the same order twice. Someone gets matched twice and has a very interesting conversation with their broker.

cfg.Producer.RequiredAcks = sarama.WaitForAll
cfg.Producer.Partitioner = sarama.NewHashPartitioner
cfg.Producer.Return.Successes = true
cfg.Producer.Return.Errors = true
cfg.Producer.Idempotent = true
cfg.Net.MaxOpenRequests = 1

kafka/producer.go

Idempotent = true assigns sequence numbers so Kafka discards duplicates silently. MaxOpenRequests = 1 is required: without it, Kafka cannot correctly order in-flight retries.

One thing the config does not make obvious: sarama.NewAsyncProducer is non-blocking. SendMessage enqueues the message to an internal channel and returns immediately. WaitForAll tells the Kafka broker to acknowledge only after all in-sync replicas have written the message, but that acknowledgment arrives asynchronously in a background goroutine that drains the Successes() channel. The producer never blocks the caller. This is why the graceful shutdown sequence calls producer.Close() last: it blocks until all in-flight messages are flushed before the process exits.

Graceful Shutdown

One more gap. SIGTERM hit the matching service while trades were still in the buffer. The process exited. The trades in the channel were not published. The trades did not exist anywhere. This is the kind of thing that only fails in production, at 2am, after a deployment.

func (te *TradeExecutor) Start() {
    te.wg.Add(1)
    go func() {
        defer te.wg.Done()
        for trade := range te.TradeCh {
            tradeJSON, err := json.Marshal(trade)
            if err != nil {
                log.WithError(err).Error("Failed to marshal trade for publishing")
                continue
            }
            te.Producer.SendMessage("TRADE_EXECUTED", trade.Instrument, tradeJSON)
        }
    }()
}

func (te *TradeExecutor) Stop() {
    te.wg.Wait()
    te.Producer.Close()
}

matching/engine/executor.go

The shutdown sequence goes like this. Stop the consumer first so no new orders arrive. Then drain the per-instrument channels so every already-received order gets processed. Then close the trade channel. Then call Stop() which blocks on wg.Wait() until the executor goroutine has published every trade to Kafka. Finally close the producer. In that exact order. Skipping any step produces data loss that looks like a race condition and smells like a deployment problem.

Phase 4: The Load Test and 18,000 RPS

With the correctness fixes in place, the first load test came in at 12,286 req/s with zero failures. Good, but not done.

Then I noticed the KAFKA_NUM_PARTITIONS setting, which I had not set, which meant it was using the default value of 1. One partition for all ten trading instruments. A single log. One consumer thread. All the per-instrument goroutine parallelism I had carefully built in the matching engine was being funneled through a single Kafka partition before it got anywhere near that parallelism.

To understand why partitions matter: a Kafka partition is the unit of parallelism. Only one consumer in a consumer group can read from a given partition at a time. With one partition, no matter how many consumers you have, only one is active. With ten partitions, you can have ten active consumers reading in parallel.

The producer was already using sarama.NewHashPartitioner with the instrument symbol as the key, so BTC-USD always routes to the same partition and ordering per instrument is preserved. The goroutine-per-instrument design and RoundRobin rebalance strategy were already ready. The only missing piece was telling Kafka to create more than one partition.

environment:
  KAFKA_NUM_PARTITIONS: 10

docker-compose.yml

Also gave Kafka a reasonable amount of memory to work with instead of the default 256MB, which is a perfectly reasonable heap size if you are not planning to use Kafka under any actual load.

environment:
  KAFKA_HEAP_OPTS: "-Xmx1G -Xms1G"
  KAFKA_LOG_RETENTION_HOURS: 1
  KAFKA_LOG_RETENTION_BYTES: 1073741824

docker-compose.yml

A note on the retention config. One hour and 1GB is aggressive. This is tuned for the benchmark environment to prevent the dedicated /data/ partition from filling up during repeated test runs. In a real deployment you would set retention based on how long you need to be able to replay messages for recovery. A reasonable production default is 7 days or the size of your disk, whichever comes first.

The final numbers across two runs:

The assignment target was 2,000 req/s with sub-100ms median latency. The final median is 20ms at 18,210 req/s, which satisfies the requirement in the same way that a firehose satisfies a request for a glass of water.

Where This Goes Next

The application is not the bottleneck anymore. At 18k req/s, the ceiling is the HTTP layer itself. JSON parsing, struct allocation, TCP write overhead per request. Switching to gRPC or a binary protocol over persistent connections would push this considerably higher without touching the matching logic.

Horizontal scaling is already supported by the architecture. Adding a second matching engine instance triggers a Kafka partition rebalance, automatically distributing 5 partitions to each instance. The matching logic stays correct because each partition is always owned by exactly one consumer. No code changes needed. Docker Compose does not scale horizontally, but that is a Docker Compose problem, not an application problem.

The matching logic had correctness bugs. The Kafka consumer had a silent data loss path. The producer could create duplicates under network stress. The default broker configuration was bottlenecking a 20-thread machine to a single I/O thread. Each fix was a separate commit, a separate afternoon of reading documentation, a separate moment of staring at logs until something clicked.

Known Limitations

Timestamp ordering under multiple API instances. time.Now().UnixNano() is stamped at the API service, so in a multi-instance deployment, clock skew can corrupt price-time priority for orders arriving nanoseconds apart. The correct fix is a monotonic sequence number issued by the matching engine per instrument, not a wall clock.

Float64 price representation. All prices and quantities use float64. IEEE 754 floating-point arithmetic does not represent most decimal values exactly. The OrderHeap.Less comparator uses != to compare prices, which can produce surprising results for values that differ only in floating-point rounding error. Production matching engines use fixed-point integers internally, representing prices as integer ticks scaled to the minimum price increment of the instrument, which eliminates rounding errors entirely and makes comparison semantics exact.

In-memory order book state is not durable. A matching service restart empties both heaps. Unmatched limit orders exist in PostgreSQL as PENDING but nothing rebuilds the book from them. A production system snapshots the book periodically and replays from the snapshot plus subsequent Kafka offsets on startup.

Self-trade prevention is absent. A self-trade occurs when a participant's own buy order matches against their own sell order. Exchanges reject these because they are economically meaningless and can be used to manipulate reported volume. This implementation has no check on ClientID in the matching logic. A single client can match against themselves indefinitely.

Log retention values are benchmark-only. One hour and 1GB prevents the test disk from filling during repeated runs. In production these values define your replay window; a service down longer than one hour must rebuild from the database instead. Seven days is a reasonable starting point.

The code is on GitHub. If you find a better approach or a bug I missed, open an issue. The correctness bugs in a matching engine are the kind that only reveal themselves when you look carefully, and there is no particular reason to believe all of them have been found.

The gap between "it works on my machine" and "it works at scale" is where all the interesting engineering lives.

The centerpiece of that feature is a green dot. A tiny circle next to a profile picture that means: this person is here, right now.

I hit offline detection bugs that kept users "online" 40 seconds after they'd closed the app. I watched my server silently stop accepting new connections at 300 users because I'd wired the whole system into one binary. I found an article on systemdesign.one, spent a weekend with ChatGPT breaking it down, rebuilt the whole thing, named a goroutine the Reaper, and eventually stress-tested it to 51,322 concurrent connections on my laptop.

This is the full story. Including the parts that hurt.

Where I Started: gRPC Streaming and the First Problem

The transport choice was gRPC streaming from the start. Clients open a long-lived stream to the server and subscribe to presence updates for a list of user IDs. When anyone in that list changes status, the server pushes an event down the stream. One persistent connection per client.

That part was correct.

The first design question: how does the server know a user is online? Heartbeats. When a client connects, the gateway starts a goroutine that calls RecordHeartbeat on the presence service every 15 seconds with the user's ID and a timestamp. The presence service writes that timestamp to Redis. As long as heartbeats arrive, the user is online.

Online detection worked immediately. Offline detection is where things started breaking.

Problem 1: Nobody Was Going Offline

Attempt 1: Redis Keyspace Notifications for Offline Detection

I googled "redis presence detection" and found keyspace notifications. Set a TTL key per user. When it expires, Redis fires an event. Subscribe to those events. Offline detection, free, no extra goroutine, elegant.

I genuinely felt smart for about four hours.

Then I read that keyspace notifications fire for every key event in your entire Redis instance. Not just yours. Every SET, every DEL, every expiry across all your keys. And Redis doesn't expire keys in real time -- it uses lazy expiration or a periodic sweep. Users were showing offline 30 to 40 seconds after reconnecting. The expired key sat un-swept. The new heartbeat key sat ignored. Two conflicting states, zero resolution, one confused frontend.

Why it failed:

• Key expiration in Redis is not deterministic [1]

• Keyspace notifications fire on every key event cluster-wide -- CPU cost scales with total Redis write volume, not your user count

• You're trusting Redis's garbage collector to be your presence engine. Bold.

Problem 2: One Binary, All the Problems

Attempt 2: One Binary to Rule Them All

Keyspace notifications were out. I moved to a heartbeat-based model -- the client sends a signal every 15 seconds, the server stores it in Redis with a timestamp, and I'd figure out offline detection separately. That was the right call.

What came next was not.

I had gRPC streaming. I had heartbeats. I put everything into one binary.

Every stream connection held its own Redis Pub/Sub subscription. Every heartbeat triggered a direct database call from inside the stream handler. Around 300 concurrent streams, the default Redis client pool (10 connections) was fully saturated. New connections blocked waiting for a free slot. PostgreSQL followed. The server didn't crash dramatically. It just... stopped accepting new connections. Quietly. Like it gave up.

Why it failed:

• One Redis Pub/Sub subscription per client connection

• One database call per heartbeat, per stream goroutine

• Default connection pools hit the ceiling before you hit 400 users

The problem was architectural. Stateful connection management and stateless business logic do not belong in the same binary. Their connection pools compete. The long-lived streams always win.

The Architecture That Survived

At some point during Attempt 2, I stopped writing code and started reading.

I found the Real-Time Presence Platform System Design article on systemdesign.one [4]. Then I spent a weekend with ChatGPT going through it section by section the delayed-trigger pattern for offline detection, the gateway split, the single shared Pub/Sub subscription. Every piece had a reason. I just hadn't understood the reasons yet.

The pattern: separate the stateful connection layer (the gateway, which holds open streams) from the stateless logic layer (the presence service, which processes heartbeats and publishes events). One Redis Pub/Sub subscription in the gateway fans out to every connected client. The presence service never touches a client connection directly.

I rebuilt the whole thing. Here's what actually works.

The flow:

1. Client opens a gRPC stream to the Gateway with a list of user IDs they want to watch

2. Gateway starts a heartbeat loop, calling the API server's internal RecordHeartbeat RPC every 15 seconds

3. Presence Service writes to a Redis Sorted Set (score = timestamp, member = userID)

4. If it's a new connection, it publishes an online event to Redis Pub/Sub

5. The Reaper goroutine sweeps for expired users every 5 seconds, publishes offline events

6. Gateway's EventsListener picks up events from Redis and fans them out to the right connections

Why One Redis Command Replaced Three Data Structures

You could use a Redis Set for who's online. A Hash for last-seen timestamps. A separate TTL key per user for expiration. Three data structures, three commands, three places to get out of sync.

A Sorted Set does all of it in one structure.

ZADD presence:active_users <unix_timestamp> <user_id>

ZADD is idempotent: one command handles both insert and update. No SET+EXPIRE dance, no branching.

To find and clean expired users (offline for more than 30 seconds):

ZRANGEBYSCORE presence:active_users -inf <now - 30>
ZREM presence:active_users <expired_user_ids...>

All O(log N). At 100k users, that's 17 comparisons. Redis barely notices.

The alternative was using key-per-user with TTL expiry and keyspace notifications. We tried that. It's Attempt 2. It's in the graveyard.

How the Heartbeat Works

Every 15 seconds, the gateway lies to the presence service. It tells it the user is still online. The presence service believes it, updates the timestamp, and if it's a new connection, shouts "online" to Redis Pub/Sub. If the lies stop coming, the Reaper sweeps up.

// internals/presence/repository.go

const (
    presenceZSetKey = "presence:active_users"
    ttlSeconds      = 30
)

func (r *Repository) RecordHeartbeat(ctx context.Context, userID string) (bool, error) {
    now := float64(time.Now().Unix())
    cutoff := float64(time.Now().Unix() - ttlSeconds)

    score, err := r.redisClient.ZScore(ctx, presenceZSetKey, userID).Result()
    var isNewConnection bool

    if err == redis.Nil {
        // User not in the set at all. Fresh connection.
        isNewConnection = true
    } else if err == nil && score < cutoff {
        // User exists but their last heartbeat expired. They're back.
        isNewConnection = true
    } else if err != nil {
        return false, err
    }

    // Update (or add) the user's timestamp
    err = r.redisClient.ZAdd(ctx, presenceZSetKey, redis.Z{
        Score:  now,
        Member: userID,
    }).Err()

    return isNewConnection, err
}

Source: internals/presence/repository.go

The isNewConnection flag matters. If a user is already online and sends another heartbeat, we don't publish a redundant online event. Only genuine state transitions get broadcast:

// internals/presence/service.go

func (s *Service) RecordHeartbeat(ctx context.Context, req *pb.HeartBeatRequest) (*emptypb.Empty, error) {
    isNewConnection, err := s.repo.RecordHeartbeat(ctx, req.UserId)
    if err != nil {
        log.Printf("Failed to record heartbeat for user %s: %v", req.UserId, err)
        return &emptypb.Empty{}, err
    }

    if isNewConnection {
        event := gateway.PresenceEvent{
            UserID:   req.UserId,
            Status:   "online",
            LastSeen: time.Now().Unix(),
        }
        payload, err := json.Marshal(event)
        if err == nil {
            s.redisClient.Publish(ctx, "system:presence_events", payload)
        }
    }

    return &emptypb.Empty{}, nil
}

Source: internals/presence/service.go

The gateway side. keepAliveLoop runs as a goroutine for the lifetime of the stream:

// internals/gateway/handler.go

func (s *Server) keepAliveLoop(ctx context.Context, userID string) {
    ticker := time.NewTicker(15 * time.Second)
    defer ticker.Stop()

    s.sendHeartbeat(userID) // fire immediately on connect

    for {
        select {
        case <-ctx.Done():
            return
        case <-ticker.C:
            s.sendHeartbeat(userID)
        }
    }
}

Source: internals/gateway/handler.go

Heartbeat interval is 15 seconds. TTL is 30 seconds. That gives two missed heartbeats before the user is considered offline. One missed heartbeat from a flaky mobile connection doesn't trigger a false offline event.

How Offline Detection Works: The Reaper

After heartbeats were working, I had a new problem. Users who closed the app stayed "online" on everyone else's screen. 30 seconds. 40 seconds. Sometimes longer.

There was no disconnect event. No callback. When the gRPC stream closed, the gateway's heartbeat goroutine exited. That was it. Nobody told the presence service. Nobody published an offline event. The user's timestamp just sat in the sorted set, aging, while their green dot stayed green.

I needed something to notice the silence.

I wrote a goroutine that runs every 5 seconds, queries Redis for users whose last heartbeat is older than 30 seconds, removes them from the sorted set, and publishes an offline event for each one. I called it the Reaper.

// internals/presence/reaper.go

func (r *Reaper) Start(ctx context.Context) {
    ticker := time.NewTicker(5 * time.Second)
    defer ticker.Stop()

    for {
        select {
        case <-ctx.Done():
            return
        case <-ticker.C:
            r.sweep(ctx)
        }
    }
}

func (r *Reaper) sweep(ctx context.Context) {
    sweepCtx, cancel := context.WithTimeout(ctx, 4*time.Second)
    defer cancel()

    expiredUsers, err := r.repo.GetAndRemoveExpiredUsers(sweepCtx)
    if err != nil {
        log.Printf("Reaper failed to fetch expired users: %v", err)
        return
    }

    if len(expiredUsers) > 0 {
        now := time.Now().Unix()
        for _, userID := range expiredUsers {
            event := gateway.PresenceEvent{
                UserID:   userID,
                Status:   "offline",
                LastSeen: now,
            }
            payload, err := json.Marshal(event)
            if err == nil {
                r.redisClient.Publish(sweepCtx, "system:presence_events", payload)
            }
        }
    }
}

Source: internals/presence/reaper.go

The sorted set query and removal:

// internals/presence/repository.go

func (r *Repository) GetAndRemoveExpiredUsers(ctx context.Context) ([]string, error) {
    cutoff := fmt.Sprintf("%f", float64(time.Now().Unix()-ttlSeconds))

    // fetch all users whose last heartbeat score is older than 30s
    expiredUsers, _ := r.redisClient.ZRangeArgs(ctx, redis.ZRangeArgs{
        Key:     presenceZSetKey,
        ByScore: true,
        Start:   "-inf",
        Stop:    cutoff,
    }).Result()

    if len(expiredUsers) > 0 {
        // convert to []interface{} for ZRem
        members := make([]interface{}, len(expiredUsers))
        for i, u := range expiredUsers { members[i] = u }
        r.redisClient.ZRem(ctx, presenceZSetKey, members...)
    }

    return expiredUsers, nil
    // full implementation: https://github.com/dis70rt/bluppi-backend/blob/main/internals/presence/repository.go
}

Source: internals/presence/repository.go

The 4-second context timeout on sweep is intentional. If Redis is unresponsive, the Reaper doesn't hang forever and block subsequent sweeps. It logs the failure and tries again in 5 seconds. LinkedIn's presence platform uses a similar delayed-trigger approach for offline detection [3].

The Gateway: Multi-Device Connections and Targeted Fanout

The ConnectionManager uses a nested map: userId -> connectionId -> Connection. Each device gets a UUID. Disconnecting one device only cleans up that specific connection.

// internals/gateway/manager.go

type Connection struct {
    ID                 string
    UserID             string
    Chan               chan PresenceEvent
    SubscribedToTarget []string // which users this connection is watching
}

type ConnectionManager struct {
    mu sync.RWMutex
    // userId -> connectionId -> Connection
    userConnections map[string]map[string]*Connection
    // targetUserId -> set of connectionIds watching them
    targetWatchers map[string]map[string]struct{}
    // connectionId -> Connection reference (for quick lookup during push)
    connById map[string]*Connection
}

Source: internals/gateway/manager.go

Three maps. Looks like overkill. It's not.

• userConnections: "which connections does this user have?" Needed for cleanup.

• targetWatchers: "who's watching this user?" Needed for fanout. This is the inverted index.

• connById: "give me the connection object by ID." Needed for pushing events.

When a presence event arrives for User B, the PushEvent method looks up User B in targetWatchers, gets the set of connection IDs watching them, resolves each to a Connection via connById, and pushes the event to its channel:

// internals/gateway/manager.go

func (cm *ConnectionManager) PushEvent(targetUserID string, event PresenceEvent) {
    cm.mu.RLock()
    defer cm.mu.RUnlock()

    watchers, exists := cm.targetWatchers[targetUserID]
    if !exists {
        return
    }

    for connID := range watchers {
        if conn, ok := cm.connById[connID]; ok {
            select {
            case conn.Chan <- event:
            default: // Drop event if channel is full to avoid blocking
            }
        }
    }
}

Source: internals/gateway/manager.go

That select { default: } is load-bearing. Without it, a slow consumer blocks the entire fanout. With it, a full channel just drops the event. The user misses one status update and gets the next one 15 seconds later. Better than a deadlock that freezes the entire gateway.

The Redis Pub/Sub listener ties it together. One goroutine, one subscription, fans out to all connections:

// internals/gateway/listener.go

func (el *EventsListener) Start(ctx context.Context) {
    pubsub := el.redisClient.Subscribe(ctx, "system:presence_events")
    defer pubsub.Close()

    for {
        select {
        case <-ctx.Done():
            return
        case msg := <-pubsub.Channel():
            var event PresenceEvent
            if err := json.Unmarshal([]byte(msg.Payload), &event); err != nil {
                log.Printf("Failed to unmarshal presence event: %v", err)
                continue
            }
            el.connManager.PushEvent(event.UserID, event)
        }
    }
}

Source: internals/gateway/listener.go

Goroutines as Actors: How Go Solved the Mutex Problem

Here's the part nobody explains when they show you a sync.RWMutex slapped on a map and call it a day.

The ConnectionManager has a mutex. It protects the three index maps: registration, deregistration, lookups. That's the right scope for a lock.

But event delivery has no lock at all. PushEvent holds only an RLock to read the watcher index, then sends to a channel. It never waits. It never coordinates. Each connection processes its own events in complete isolation.

This is the Actor Model [8], implemented with zero framework, zero library, zero ceremony. In Go, a goroutine plus a channel is an actor. The channel is the mailbox. The goroutine is the event loop. Messages never share memory across actors.

Look at SubscribePresence:

// internals/gateway/handler.go

func (s *Server) SubscribePresence(req *pb.SubscribeRequest, stream pb.PresenceGateway_SubscribePresenceServer) error {
    ctx := stream.Context()

    userID, err := middlewares.GetUserID(ctx)
    if err != nil {
        return err
    }

    conn, connID := s.connManager.AddConnection(userID, req.TargetUserIds)
    defer s.connManager.RemoveConnection(userID, connID)

    go s.keepAliveLoop(stream.Context(), userID)

    for {
        select {
        case <-stream.Context().Done():
            return stream.Context().Err()

        case event, ok := <-conn.Chan: // the actor receiving its mailbox message
            if !ok {
                // channel closed by RemoveConnection: clean shutdown signal,
                // no extra sync primitive needed
                return nil
            }
            grpcMessage := &pb.PresenceUpdate{
                UserId:   event.UserID,
                Status:   event.Status,
                LastSeen: timestamppb.New(time.Unix(event.LastSeen, 0)),
            }
            if err := stream.Send(grpcMessage); err != nil {
                return err
            }
        }
    }
}

Source: internals/gateway/handler.go

Each call to SubscribePresence is its own actor instance. It has one goroutine, one channel (conn.Chan, capacity 20), one job: drain the mailbox and stream events to the client. It shares no state with any other connection's event loop.

The alternative is a shared event queue with a mutex on every dequeue. At 50k concurrent connections, every event delivery serializes through one lock. Under that contention, sync.Mutex triggers O(N) goroutine wake-ups per unlock. The scheduler spends more time arbitrating lock ownership than your code spends doing actual work.

With the actor approach: PushEvent does one non-blocking channel send and moves on. No coordination. No waiting. The worst case is a dropped event when a consumer's buffer is full -- which is a better outcome than a deadlocked gateway serving nobody.

The only mutex in the entire system is on the maps. Not on message delivery. That's the point.

The Two-Binary Split: API Server vs. Gateway Server

The API server (cmd/api/main.go, port 50051) handles business logic: users, music, rooms, parties, and the internal RecordHeartbeat RPC. Stateless. Horizontally scalable. Talks to PostgreSQL, Memgraph, Redis, Solr.

The Gateway server (cmd/gateway/main.go, port 50050) handles long-lived gRPC streams. Stateful by nature (it holds open connections). Talks to Redis Pub/Sub and the API server over internal gRPC. (Note: you can use WebSockets instead of gRPC streams here; the core principles of connection management and fanout remain exactly the same.)

The gateway authenticates clients via Firebase Auth through a gRPC stream interceptor, then calls the API server with the user ID attached as gRPC metadata. The API server never sees raw client connections for presence. It processes heartbeats and that's it.

Load Test: 51,322 Users, Zero Errors

I wrote a custom load test tool in Go (cmd/loadtest/presence/main.go) that spins up concurrent gRPC streams. Each simulated user subscribes to their own events, connects, receives their online event, and holds the stream open.

Important caveat: this is a connection-overhead test, not a fanout test. Each user watches only themselves. In production, one popular user might have thousands of watchers -- that fanout path has different latency characteristics. The numbers below measure how many concurrent streams the gateway holds open without errors, not worst-case fanout latency.

Results from a single laptop (i5-13500HX, 15 GiB RAM):

*Target was 100k. Stopped at 51,322 because the load generator ran out of RAM. The backend was fine. My laptop was not.

Zero errors across all runs. p99 latency at 51k users: 15.28 ms. The backend didn't break. The load generator did.

For context, LinkedIn targets sub-second presence delivery for their 700M+ user base [3]. Our p99 at 51k is 15ms. Different scale, sure, but the architecture patterns are the same.

Known Limitations

I'm still working on some of these. Listing them anyway because honesty is more useful than a blog post that pretends everything is solved.

• Single-node gateway. If the gateway goes down, all streams drop. No cross-node fanout yet. Fixing this requires a dispatcher layer similar to LinkedIn's architecture [3] or consistent hashing across gateway instances.

• Reaper sweep interval. The 5-second tick means offline detection can lag up to 35 seconds (30s TTL + 5s sweep). For a music app's friends activity, this is fine. For an ICU patient monitoring system, this would be a resume-generating event.

• No cross-data center replication. Single region. Adding geo-distribution would need CRDT-based replication on the Redis layer, similar to what the systemdesign.one article describes [4].

• ulimit capped the load test. The default 1024 open file limit on the load generator machine was the actual bottleneck. Raising it and using distributed load generators would push past 100k.

Closing

This took months. For a green dot.

The full source is on GitHub: dis70rt/bluppi-backend. The presence system lives in internals/presence/ and internals/gateway/. Go read it, break it, tell me what's wrong with it.

Two things I wish I knew at the start:

If you're reaching for Redis keyspace notifications for offline detection -- don't. Write a Reaper. It's 66 lines of Go and it does exactly what you need with zero surprises.

If you're about to let every client hold a Redis Pub/Sub subscription from inside your API server -- also don't. Split the gateway out. The connection pool you save might be your own.

And if you're a student building something that "sounds simple" and it starts taking way longer than expected: that's normal. That's just what real systems feel like when you stop using the whiteboard version.

References

1. Redis documentation, "Redis keyspace notifications." redis.io/docs

2. gRPC documentation, "Keepalive." grpc.io/docs

3. Gupta & Lay, "Now You See Me, Now You Don't: LinkedIn's Real-Time Presence Platform" (2018). engineering.linkedin.com

4. Kim, "Real-Time Presence Platform System Design" (2023). systemdesign.one

5. Barber, "Building Real-Time Infrastructure at Facebook" (2017). SREcon / USENIX. youtube.com

6. PubNub, "Is Anyone Home? An Intro to Presence Webhooks" (2020). pubnub.com

7. Redis documentation, "Sorted Sets." redis.io/docs

A complete guide to exposing a Go gRPC backend behind college NAT using Cloudflare Tunnel, Docker Compose, and TLS, including every mistake I made along the way.

Background

I'm a student building Bluppi, a Flutter app with a Go gRPC backend. My server runs on my laptop on college WiFi, behind a NAT I don't control, no port forwarding, no static IP.

My stack:

• Backend: Go gRPC server (bluppi-api:50051) + Go gRPC gateway (bluppi-gateway:50050) + Python FastAPI (bluppi-audio-api:8000)

• Infrastructure: Docker Compose

• Tunnel: Cloudflare Tunnel (cloudflared)

• Client: Flutter (Android/iOS)

• Domain: bluppi.saikat.in

What should have been a simple "expose my API to the internet" turned into a 6-hour debugging session. Here's everything I learned.

Part 1: Setting Up Cloudflare Tunnel

Create the tunnel

# Login to Cloudflare
cloudflared tunnel login

# Create tunnel
cloudflared tunnel create bluppi

# Route your domain to the tunnel
cloudflared tunnel route dns bluppi bluppi.saikat.in

# Copy credentials to project
mkdir -p cloudflared
cp ~/.cloudflared/<tunnel-id>.json cloudflared/
cp ~/.cloudflared/cert.pem cloudflared/

Initial config.yml (what I started with)

tunnel: <tunnel-id>
credentials-file: /etc/cloudflared/<tunnel-id>.json

ingress:
  - hostname: bluppi.saikat.in
    path: /api/v2/.*
    service: grpc://bluppi-gateway:50050

  - hostname: bluppi.saikat.in
    path: /api/rest/.*
    service: http://bluppi-audio-api:8000

  - hostname: bluppi.saikat.in
    service: grpc://bluppi-api:50051

  - service: http_status:404

Initial docker-compose.yml tunnel service

tunnel:
  container_name: bluppi-tunnel
  image: cloudflare/cloudflared:latest
  command: tunnel --config /etc/cloudflared/config.yml run
  restart: unless-stopped
  networks:
    - cloudflared
  depends_on:
    - bluppi-api
    - bluppi-gateway
    - audio-api
  volumes:
    - ./cloudflared:/etc/cloudflared:ro

Part 2: The Mistakes (And How to Fix Them)

Mistake 1: Port 7844 blocked (college network)

Symptom:

dial tcp 198.41.192.107:7844: i/o timeout

cloudflared defaults to connecting Cloudflare's edge on TCP port 7844. College networks often block non-standard ports.

Fix: Force HTTP/2 protocol which falls back to port 443:

# docker-compose.yml
command: tunnel --protocol http2 --config /etc/cloudflared/config.yml run

Or in config.yml:

protocol: http2

Mistake 2: Too many HA connections

Symptom:

already connected to this server, trying another address

cloudflared opens 4 parallel connections by default, but the Delhi edge (del03) only had 1-2 distinct IPs reachable from my network.

Fix: Reduce HA connections:

ha-connections: 1

Mistake 3: grpc:// is not a valid cloudflared service protocol

Symptom:

malformed HTTP response "\x00\x00\x06\x04..."

grpc:// is not a recognised protocol in cloudflared. Those bytes are raw HTTP/2 frames being received by cloudflared which expected HTTP/1.x.

Fix: Initially, I changed grpc:// to h2c:// (HTTP/2 cleartext) for internal gRPC services. However, this is a trap. While h2c:// stops this immediate error, it leads you right into Mistake 4 because of Cloudflare's proxy limitations.

Mistake 4: Cloudflare Free plan blocks gRPC on public hostnames

Symptom:

malformed header: missing HTTP content-type

Even with the gRPC toggle enabled in the Cloudflare Dashboard, the Free plan does not fully proxy gRPC through orange-cloud (proxied) hostnames. Cloudflare strips the content-type: application/grpc header.

References:

• cloudflared issue #491 — gRPC support

• cloudflared issue #1304 — Support h2c origin servers

Fix: This is a fundamental limitation, no simple config change fixes it. You cannot use h2c://. The real solution is adding TLS to your origin server and using https:// (Detailed in Part 3).

Part 3: The Real Fix (TLS on the Origin)

Step 1: Generate self-signed certificates

mkdir -p certs
openssl req -x509 -newkey rsa:4096 \
  -keyout certs/server.key \
  -out certs/server.crt \
  -days 365 -nodes \
  -subj "/CN=localhost"

Note on CN: Since we use noTLSVerify: true in cloudflared config, the CN value doesn't matter. localhost works fine.

Step 2: Configure your gRPC Server

Regardless of whether you are using Go, Python, Node, or Rust, you must configure your gRPC server to use the generated server.crt and server.key. Your specific language's gRPC library will handle the TLS implementation.

Step 3: Verify TLS + ALPN on your server

# Check TLS works
openssl s_client -connect localhost:50051

# Check ALPN h2 is advertised (critical for gRPC)
openssl s_client -connect localhost:50051 -alpn h2

You must see:

ALPN protocol: h2

If you see No ALPN negotiated, gRPC will not work through the tunnel. Check your server's TLS configuration.

Key point: grpc-go automatically advertises ALPN: h2 when you use credentials.NewServerTLSFromFile. This is what enables cloudflared to negotiate HTTP/2.

Part 4: Final Working Configuration

docker-compose.yml

Since cloudflared is running on the host, we just need to make sure our backend services expose their ports to localhost.

  tunnel:
    container_name: bluppi-tunnel
    image: cloudflare/cloudflared:latest
    command: tunnel --no-autoupdate --protocol http2 --ha-connections 1 --config /etc/cloudflared/config.yml run
    restart: unless-stopped
    networks:
      - cloudflared
    depends_on:
      - bluppi-api
      - bluppi-gateway
      - audio-api
    volumes:
      - ./cloudflared:/etc/cloudflared:ro

networks:
  cloudflared:
    name: cloudflared

cloudflared/config.yml

tunnel: <tunnel-token>
credentials-file: /etc/cloudflared/<tunnel-id>.json
ha-connections: 1
protocol: http2

ingress:
  # 1. Dedicated REST API
  - hostname: bluppi.saikat.in
    path: /api/rest/.*
    service: http://bluppi-audio-api:8000

  # 2. gRPC Gateway
  - hostname: bluppi.saikat.in
    path: /api/v2/.*
    service: https://bluppi-gateway:50050
    originRequest:
      noTLSVerify: true
      http2Origin: true

  # 3. Pure gRPC API (Catch-All)
  - hostname: bluppi.saikat.in
    service: https://bluppi-api:50051
    originRequest:
      noTLSVerify: true
      http2Origin: true

  - service: http_status:404

Why https:// not h2c://?
cloudflared does NOT support h2c:// (HTTP/2 cleartext) as an origin protocol — see issue #1304. You must use https:// with a TLS-enabled origin, then add noTLSVerify: true to accept the self-signed cert, and http2Origin: true to force HTTP/2 negotiation.

Part 6: Verification

Test gRPC end-to-end with grpcurl

# Install grpcurl (or you can use Postman)

# Test locally (bypassing tunnel) — baseline
grpcurl -plaintext localhost:50051 list
# Expected: Unauthenticated (server is running, auth is rejecting unauthenticated requests) (as I have setup JWT auth)

# Test through Cloudflare tunnel
grpcurl bluppi.saikat.in:443 list
# Expected: same Unauthenticated error = tunnel is fully transparent ✅

Part 7: Complete Mistake Summary

Key Takeaways

1. grpc:// is not a valid cloudflared service protocol — use h2c:// or https://

2. Cloudflare Free plan cannot proxy gRPC on public hostnames — cloudflare's gRPC toggle requires Pro plan to work properly

3. cloudflared connects to origins using h2 (TLS), not h2c (cleartext) — your gRPC origin MUST serve TLS, even behind a private Docker network

4. http2Origin: true is mandatory — without it, cloudflared uses HTTP/1.1 to the origin, which gRPC servers reject immediately

5. noTLSVerify: true is safe inside Docker — the tunnel itself provides encryption; internal self-signed certs are fine

6. Always test with grpcurl before testing Flutter — if grpcurl bluppi.saikat.in:443 list returns Unauthenticated (same as localhost), your Flutter app will work

7. College/restricted networks block UDP and non-standard ports — always force --protocol http2 to use TCP 443

References

• cloudflared issue #491 — gRPC support request

• cloudflared issue #1304 — Support h2c origin servers

• Cloudflare Docs — Cloudflare Tunnel

Written after 6 hours of debugging. Hopefully this saves you the same pain.

If you're here, you probably already know that Firebase is great for building apps with real-time data. However, many developers run into a big issue—Firebase costs can quickly get out of hand due to too many read operations.

In 2018, a startup in Colombia faced this exact problem when they scaled up to 2 million daily active users (DAUs). A small, but costly mistake in their code ended up leaving them with a $30,356.56 bill from Google Clouds in just 72 hours. Why? Because that tiny error caused 2 million users to each trigger 16,000 document reads, leading to over 40 billion requests to Firestore in less than 48 hours. Read the full article

You can avoid such excessive costs by optimizing your reads. In this blog, we'll dive into specific strategies to stop paying for every single read without sacrificing app performance.

Understanding Firebase Read Costs

Firebase's pricing model is simple. In Firestore, you are charged based on the number of documents you read. Each time a document is read from the database—whether through a query, real-time listener, or a simple fetch—it counts as a read.

Check out the Firestore Pricing here.

Before jumping into solutions, it’s important to understand where things can go wrong.

• Fetching unnecessary data: For instance, if you’re retrieving large collections when you only need a few documents, you’re paying for extra reads.

• Inefficient queries: Poor query structuring can result in pulling unnecessary data or performing repeated reads.

• Overuse of real-time syncing: Real-time listeners are great, but they can result in multiple reads as they sync data even when it’s not needed immediately.

Strategies to Reduce Firebase Read Costs

Optimize Your Data Structure

A well-organized database can greatly cut down the number of reads. If your Firestore database isn't structured efficiently, it can cause you to fetch more data than needed, leading to extra read costs.

Choosing when to use nested data vs flattened data is crucial for reducing unnecessary reads in Firestore.

Example: Instead of storing all user posts in one collection, divide them into smaller subcollections by category or user ID.

Query Optimization

Firestore is a NoSQL database, and it's important to use its indexing features. Writing efficient queries can help lower the number of documents read.

Firestore automatically indexes each document by its document ID. But for complex queries, like filtering on several fields, you might need to create composite indexes.

Example: If you want to query users by both age and city, you'll need a composite index for age and city.

Filtering documents with where() clauses retrieves only the necessary data.

final QuerySnapshot querySnapshot = await FirebaseFirestore.instance
      .collection('users')
      .where('age', isGreaterThanOrEqualTo: 18)
      .where('city', isEqualTo: 'New York')
      .get();

Make sure to combine filters that make sense together to reduce the number of reads.

Implement Pagination Using limit() for Efficient Data Retrieval

final int pageSize = 10;
DocumentSnapshot? lastVisible;

Future<void> getPosts() async {
  final query = FirebaseFirestore.instance
      .collection('posts')
      .orderBy('createdAt')
      .limit(pageSize)
      .startAfterDocument(lastVisible!);
  
  final querySnapshot = await query.get();

  querySnapshot.docs.forEach((doc) => print(doc.data()));
  lastVisible = querySnapshot.docs.isNotEmpty ? querySnapshot.docs.last : null;
}

Limiting documents can significantly reduce costs and improve performance, especially when dealing with large datasets.

Avoid Unnecessary Real-time Listeners

Real-time listeners are powerful but expensive if used without careful consideration. In many cases, you don't need real-time updates for every piece of data.

Manual Refreshing: Instead of relying on real-time updates, users can manually trigger a refresh to load the latest data.

In this blog, we will look at advanced caching methods that can greatly reduce your Firestore costs. These strategies are more than just basic offline caching and can save you a lot of money while improving your app's performance.

Advanced Strategies to Reduce Firebase Costs

You can use SharedPreferences to store small pieces of data that are frequently accessed, like user info or settings. This works well for data that doesn’t change much. However, SharedPreferences is only suitable for small amounts of data. If you store too much, it can slow down your app, use more memory, and hit storage limits.

import 'package:shared_preferences/shared_preferences.dart';

Future<void> saveUserInfo(String name, int age, String email) async {
  final SharedPreferences prefs = await SharedPreferences.getInstance();
  await prefs.setString('userName', name);
  await prefs.setInt('userAge', age);
  await prefs.setString('userEmail', email);
}

For example, you can store user information in SharedPreferences as small chunks of data that are repeatedly used throughout your application.

Use Provider for efficient state management to reduce Firestore reads.

Without proper state management, your app might request the same data from Firestore multiple times, increasing the number of reads unnecessarily. By using Provider, once data is fetched from Firestore, it can be stored and accessed efficiently throughout the app without additional Firestore calls.

import 'package:flutter/material.dart';
import 'package:cloud_firestore/cloud_firestore.dart';

class UserProvider extends ChangeNotifier {
  Map<String, dynamic>? _userData;
  Map<String, dynamic>? get userData => _userData;

  Future<void> fetchUserData(String userId) async {
    if (_userData != null) return; // Skip fetching if data is already loaded

    try {
      final doc = await FirebaseFirestore.instance.collection('users').doc(userId).get();
      _userData = doc.data();
    } catch (e) {
      print('Error fetching user data: $e');
    } 
      
    notifyListeners();
  }
}

Don’t forget to add the Provider class to the main.dart file and then use it in the UI. This ensures the data is fetched only once when the app starts and is shared efficiently across all UI components, screens, and pages.

If you know that the data is not going to change frequently, you can save it on the client side, essentially caching the data. This can be achieved through various methods, such as using Hive, SQLite, or path_provider, each serving a different purpose.

Server-Side Optimization for Cost Reduction

If all the users are requesting the same data from your Firebase database, it can unnecessarily increase your reads. To avoid this, you can use the Firebase Admin SDK to create bundles, request the data only once, store it in those bundles, and send the bundles to the clients. This will reduce Firebase reads and improve your app's performance, as it only needs to request the data once.

from google.cloud import firestore
from google.cloud.firestore_bundle import FirestoreBundle

db = firestore.Client()
bundle = FirestoreBundle("bundle-name")

for user in db.collection("users").stream():
    bundle.add_document(user.get())
    
    for post in db.collection("posts").where("user_id", "==", user.id).stream():
        bundle.add_document(post.get())

bundle_buffer = bundle.build()

with open("cacheBundle.bin", "wb") as file:
    file.wrtie(bundle_buffer)

The above code snippet might not be entirely correct as I forgot the exact syntax, but it was something similar to this. Check Docs

Once you've created the Firestore bundle and serialized it, you can save the bundle as a file (e.g., cacheBundle.bin) and send it to all of your clients via HTTP or upload it to Firebase Cloud Storage. This way, clients can access the cached data from the bundle without repeatedly making requests to Firebase Database.

Conclusion

Optimizing Firebase reads can help you save costs and improve app performance. By structuring your data efficiently, optimizing queries, minimizing real-time listeners, and using caching strategies, you can reduce unnecessary reads without sacrificing performance.

Feel free to share your own methods or any corrections in the comments below. Happy coding, and may your Firebase costs stay low while your app thrives!