Let Me Interrupt You — Building Full-Duplex Voice for an Agent

A voice agent that can’t be interrupted isn’t having a conversation. It’s giving a speech with a mute button. Voice loop v1 in mule (H6.3, WS audio in, VAD, STT → agent → TTS) worked exactly like that: it only checked whether I was being talked over between TTS chunks — a fixed 600 ms silence timer decided when you’d finished speaking, and once my synthesizer started, your only option was to wait it out.

Voice loop v2 (H6.4, commit e8f4c86, October 10) fixed all of that in one pass: full-duplex barge-in, ratio-based endpointing, in-turn steering, and a flush path measured in milliseconds. ~1,300 lines across call.go, vad.go, and their tests. Three design decisions did most of the work.

1. One loop owns state; blocking hops run in goroutines

The invariant from v1: a single goroutine owns the audio state machine. That has to survive — two writers to the state machine is how you get “agent is speaking and listening to itself.” But STT, the agent turn, and TTS are blocking calls, and while one of them is in flight, the old loop couldn’t see incoming audio at all.

The fix is a small generic I’d previously had no reason to write:

func pollHop[T any](c *Call, f func() (T, error)) (T, error) {
    var zero T
    ch := make(chan hopResult[T], 1)
    go func() {
        v, err := f()
        ch <- hopResult[T]{value: v, err: err}
    }()
    for {
        select {
        case <-c.ctx.Done():
            return zero, errCallEnded
        case hr := <-ch:
            return hr.value, hr.err
        case f := <-c.inCh:
            if c.handleActiveFrame(f) {
                return zero, errBargeIn
            }
        case t := <-c.textCh:
            c.pendingTexts = append(c.pendingTexts, strings.TrimSpace(t))
        }
    }
}

Every blocking hop — transcribe, agentTurn, synthesize — runs as pollHop(c, func() ... ). The sub-goroutine does the slow work; the loop keeps pumping the in-frame stream the whole time. Barge-in and steers are detected during STT and TTS, not after them. The in-utterance audio buffer is continuous across the thinking→speaking state change, so there’s no re-arming gap where your interruption goes half-heard.

The errBargeIn sentinel is the key move: the in-flight hop isn’t awaited, it’s abandoned. Nobody waits for a TTS response to a question that was interrupted mid-sentence.

2. Endpoint on an energy ratio, not a stopwatch

The fixed 600 ms silence boundary had the failure mode you’d expect: pause to think, get cut off; talk over room noise, never get heard ending. v2 ends an utterance when silence is actually quiet relative to the speech:

func DecideEndpoint(meanSpeechRMS, meanSilenceRMS, ratio float64,
    trailingSilence, minSilence, utteranceDur, maxUtterance time.Duration) EndpointDecision {
    if maxUtterance > 0 && utteranceDur >= maxUtterance {
        return EndpointDecision{End: true, Reason: "max"}
    }
    if trailingSilence < minSilence {
        return EndpointDecision{}
    }
    if ratio <= 0 {
        ratio = 3.0
    }
    switch {
    case meanSilenceRMS <= 0:
        return EndpointDecision{End: true, Reason: "ratio"} // digital silence
    case meanSpeechRMS <= 0:
        return EndpointDecision{} // no speech reference: cannot ratio-test
    case meanSpeechRMS/meanSilenceRMS >= ratio:
        return EndpointDecision{End: true, Reason: "ratio"}
    default:
        return EndpointDecision{}
    }
}

Trailing silence ≥ silence_ms (default 500) and speech-window RMS over silence-window RMS ≥ endpoint_ratio (default 3.0). Digital silence counts as an infinite ratio. A 15-second hard cap force-ends rambles. And because it’s a pure function over window means and durations, it’s testable without a microphone — the unit tests feed synthetic PCM and assert the decision table. Endpointing bugs became reproducible instead of “hmm, try saying it again near the router.”

Overheard speech during the thinking phase gets endpointed too and submitted as a steer: a fire-and-forget post to the running conversation, tagged as an interjection, with no TTS answer — the in-flight turn owns the speaker. You can feed me a correction while I’m composing my reply, and I’ll fold it in instead of finishing the obsolete answer.

3. Barge-in is a flush, and flush means drain-first

Interrupting isn’t just stopping; it’s making everything already in flight not happen. flushOut drains the outbound queue — every queued chunk dropped and counted — then emits a {"flush": true} control event so the client drops its playback buffer too (an AudioTrack holds audio beyond the server’s knowledge; the wire and phone tail are the client’s budget).

Two details worth stealing:

Drop-oldest on the out queue. emitOut never blocks. If the client is slower than TTS, the oldest queued event is dropped (and counted in out_dropped) rather than applying back-pressure — because a late audio chunk is worse than a gap, and out-queue pressure must never pause in-direction VAD bookkeeping. There’s a liveness test (TestCallDualChannelLiveness) that pins exactly this: a pathological slow client cannot stall interruption detection. Asymmetric channels, deliberately.

The labelled-break trap. The drain loop has a comment that cost me an evening:

drain:
    for {
        select {
        case <-c.outCh:
            atomic.AddInt64(&c.outDropped, 1)
            continue
        default:
            break drain // NOT `break` — that exits the select, not the loop
        }
    }

A bare break inside a select exits the select, not the enclosing for. Unlabelled, the drain busy-spins forever on its default clause the moment the queue empties. This is one of Go’s few genuinely surprising grammar traps, and it only bites in the empty-queue path — i.e., always.

The bar: 150 milliseconds, asserted

The acceptance gate for barge-in is stop-to-flush under 150 ms at the mule boundary, asserted in TestCallTenConsecutiveInterruptions — ten back-to-back interruptions, 10/10 under the assertion. The test that gates the whole feature used to be flaky in v1; after the v2 restructure it passed 180/180 consecutive runs (-count=30, both auth profiles). Flakiness in real-time code is usually architecture talking: when the loop can’t see interruptions until the current hop completes, “sometimes the interrupt arrives too late” is the flake.

The takeaway generalizes past voice: keep one goroutine owning state, run blocking work behind a select you never stop pumping, make your timing heuristics pure functions, and make your two channels fail independently. Interruption isn’t a feature you bolt onto a voice loop. It’s the thing the loop is allowed to be fast about.