The Firehose You Never Subscribed To — Three States of a WebSocket Subscription
Table of Contents
Every job I run broadcasts its status to every connected WebSocket client. Every step update too. The ranch mule worker panes, the workflow UI, my own debugging scripts — they all sit on the same /ws firehose and filter in JavaScript, or don’t filter at all and just burn bandwidth reading updates for jobs they will never render.
When Herd H3.4 asked for per-job subscriptions, the obvious implementation was the boring one: clients subscribe, unsubscribed clients get nothing. The problem with the obvious implementation is the ranch mule worker pane. It connects, it never sends a subscribe op, and it expects to see everything. Make “no subscription” mean “nothing” and every existing client goes dark on deploy day.
So I shipped something slightly weirder, and I think it’s the right shape. Commit 62f409e gives each client three states instead of two.
The state machine nobody asked for
The subscription state lives on the client wrapper, and it has one boolean that looks redundant until you think about it for a second:
type WebSocketClient struct {
conn *websocket.Conn
closeOnce sync.Once
closed chan struct{}
// H3.4 per-job subscription state.
mu sync.RWMutex
subscribed map[string]bool
subscribeAll bool
subscribedOnce bool // has the client ever sent a subscribe op?
}
The routing decision is four lines of logic that encode the whole contract:
func (wsc *WebSocketClient) ShouldReceive(jobID string, jobOK bool) bool {
wsc.mu.RLock()
defer wsc.mu.RUnlock()
if wsc.subscribeAll {
return true
}
if !wsc.subscribedOnce {
// Never subscribed: full firehose (backward compatible).
return true
}
if !jobOK {
return false // job-filtered client: agent events withheld
}
return wsc.subscribed[jobID]
}
Read it as a three-state machine:
- Never subscribed (
subscribedOnce == false): receive everything. This is the pre-H3.4 behavior, frozen in place. Old clients don’t just work; they are indistinguishable from what they were last week. - Subscribed to ≥1 job: receive only
job_updateandjob_step_updateframes whose job id is in your set. Frames with no job id at all — agent events — are withheld, because a client that asked for job scoping shouldn’t silently keep receiving the global stream through the side door. - Ever subscribed, now empty: receive nothing. This is the state that makes the machine three-state instead of two. Unsubscribing from your last job does not fall back to the firehose. It stays silent until you subscribe again or send
subscribe_all.
That third state is the one people get wrong. If you implement filtering as len(subscribed) > 0 ? filter : allow, states 1 and 3 collapse, and unsubscribe-from-everything quietly becomes subscribe-to-everything. The client that tried to go quiet gets a face full of firehose. subscribedOnce exists purely to keep those states apart.
The protocol is three ops and a synchronous ack
Inbound control messages are plain JSON text frames on the same connection, and they are acked synchronously in the read loop:
{"op":"subscribe","job_id":"…"} → {"op":"subscribed","job_id":"…"}
{"op":"unsubscribe","job_id":"…"} → {"op":"unsubscribed","job_id":"…"}
{"op":"subscribe_all"} → {"op":"subscribed_all"}
Synchronous acks matter more than they sound. The server mutates subscription state and then writes the ack, on the client’s own read goroutine. When your UI receives subscribed, the filter is already in effect — there is no window where the client believes it’s subscribed and the server disagrees. For a workflow dashboard that subscribes and then waits for the first step update, that ordering is the difference between “renders instantly” and “sometimes needs a retry.”
Anything that isn’t a valid control op is ignored, not rejected. A firehose client that starts echoing data frames back doesn’t get dropped; it gets treated like the noise it is.
Who owns the writes?
Here’s the part I’d flag to anyone copying this design. The read loop writes acks. The hub’s broadcast loop writes data frames. A third goroutine writes pings every 54 seconds. That’s three goroutines writing to one websocket.Conn.
Gorilla’s contract is blunt about this: applications are responsible for serializing writes to a connection. My serialization story is accidental rather than architectural — the hub holds its RLock across each WriteJSON, the ping and ack writers don’t take that lock. In practice the interleave window is tiny and the failure mode (an interleaved frame) would show up as a client disconnect, not corruption. But it’s real debt, and the right fix is the standard one: give each client a send channel and make the hub, the pinger, and the acker all produce messages onto it, with one writer goroutine draining to the connection. I left it out of scope for H3.4 because the filter works and I didn’t want to smuggle a rewrite of the write path into a feature commit. Deadlines are deadlines; the debt gets its own ticket.
The lock on the subscription state follows the same read-your-writes logic in reverse: writes only ever happen on the client’s read goroutine (AddSubscription, RemoveSubscription, SubscribeAll), but ShouldReceive is called from the hub’s broadcast goroutine. One writer, many readers — a sync.RWMutex per client, held briefly. The hub extracts the job id once per broadcast from the message payload (*job.Job gives ID, *job.JobStep gives JobID, anything else has none), then asks each client whether it wants the frame. Per-client decisions, per-client lock. No global subscription table, no lock convoy when a chatty job floods the channel.
Two more details I’m quietly pleased with:
The poller inherits the filter for free. JobStreamer — the polling tail that emits job status changes — goes through the same BroadcastJobUpdate / BroadcastJobStepUpdate path as everything else. I didn’t write a second filter for it; the filtering lives in one place, at the hub’s fan-out point, so every producer automatically obeys it. If I’d filtered in the producers instead, every future broadcast source would need to remember the rules. Rules at the fan-out beat rules at the source.
Full broadcast channel means skip, not stall. The hub’s broadcast channel is capped at 256 messages and producers select with a default branch that drops the update. A slow fan-out never blocks the workflow engine. Subscribers get backpressure by missing frames, not by slowing jobs down — and since the client side can always re-read job state from the API, a dropped frame is a stale pane for a few seconds, not a lost update.
Tests are the spec
Acceptance lives in internal/api/websocket_test.go: client A subscribed to job 1, client B subscribed to job 2, a subscribe_all client, and a never-subscribed client, all connected at once while both jobs broadcast steps. The assertions are the three-state machine spelled out — A sees only job 1, B only job 2, subscribe_all sees everything, the default client sees everything, and a client that unsubscribed from everything sees nothing, including the agent events.
Four client personas, one test, and the entire backward-compatibility argument is executable. When some future me refactors the hub, the ranch mule worker pane is protected by a table row instead of by my memory.
The best kind of compatibility work is the kind users never notice. Nobody deployed a new worker pane this week. Nobody noticed the firehose got narrower. That’s the whole job.