Streaming

Token-by-token delivery to your end users’ browsers, with exact resume and honest loss signals.

Streaming delivers a run’s output as it is generated — token by token, straight to your end user’s browser over Server-Sent Events. It is built for chat experiences: the user sends a message, and the answer types itself out in real time.

Streaming is a side channel. Runs execute, persist, and complete exactly the same whether anyone is watching or not, and the run record fetched from the API remains the source of truth. If a stream is interrupted, nothing about the run is affected — the protocol below tells the client precisely what it can trust and when to fall back to fetching the result.

How it fits together

1. Your backend mints a stream credential for a thread — one call with any server SDK. The credential grants read access to that thread’s stream and nothing else; your API key never reaches the browser.
2. Your frontend opens an SSE connection with that credential — most easily via @promptjuggler/browser, which handles the whole protocol.
3. Your backend triggers runs on the thread as usual. Everything those runs stream — including every prompt node of a workflow — arrives on the connection, tagged by run and channel.

Getting a credential

POST /api/v1/threads/{thread}/stream-token (createStreamToken in every SDK) returns:

{
  "token": "eyJ…",
  "expiresAt": "2026-07-25T12:00:00+00:00",
  "url": "https://stream.promptjuggler.com/stream/0198…"
}

Hand the response to the browser verbatim — the url is the thread’s SSE endpoint, so the frontend needs no per-environment configuration. Tokens are valid for 24 hours; the browser SDK requests a fresh one on every reconnect, so expiry takes care of itself. The thread does not need to exist yet: mint the token first, connect, then trigger the first run against the same thread ID.

Each plan includes a simultaneous streaming connection allowance (10× your concurrency). Connections beyond it are refused with 429 until one closes.

The event protocol

The stream is an ordered, replayed log. Every event carries an SSE id; a client that reconnects with Last-Event-ID resumes exactly where it left off, losing nothing across network blips and deploys. Events are JSON, keyed by runId and channel:

Event	Meaning
token	One text delta, with segment and seq ordering metadata.
reset	The run is re-generating a segment (a retry): discard its buffered text.
data	An emit tool produced a structured payload: tool and payload.
tool_start	The model used a tool: tool and a ref correlator, at the point it used it.
tool_end	That tool finished: the same ref, plus status (ok or error) — and, for a search, the queries it ran and the citations it found.
done	The run finished. Fetch the run via the API when you need the authoritative result.
failure	The run failed, after all retries. Carries code and message.
stale	The server could not resume without loss — anything buffered may be incomplete.
reconnect	The server is handing the connection off (a deploy): reconnect immediately.

Two rules keep clients honest:

• Connect before you trigger. A fresh subscription starts at the live tip of the thread, not in the past. Open the stream first, then start the run, and you see every event from the beginning.
• stale means fall back. Replay covers reconnects within the retention window (minutes). Beyond it — or if events had to be shed under extreme load — the server says stale instead of leaving the client to guess. Runs that finish under a stale shadow are flagged, and the authoritative text is one API call away.

A tool_end carries no segment or seq. It is a status update rather than a position, found by its ref wherever the chip already sits — which is what lets an async tool (a prompt or workflow call, a knowledge search) resolve seconds after the model moved on. A tool that never ends stays pending, which is exactly what a call still running looks like.

A chip is published where the model used the tool, not where its turn ended — including for the tools the provider runs inside its own turn, web search and remote MCP. So a model that keeps talking after calling something gives you two text blocks with the chip between them, live and on a refetch alike. The one thing a fetched run adds is a text block’s citations, which a provider attaches to the finished message rather than to the deltas it was streamed as; see Transcript.

The browser SDK folds all of this — segments, resets, resume, staleness — into a single maintained text per run with a gapped flag that tells you the one thing you need to know: whether to fetch the run for the full result. Emit payloads ride the same stream as data events and are maintained the same way, as runs[runId].data (see emit tools). Tool events fold into runs[runId].transcript, the same ordered shape a fetched run returns — see Transcript.

Workflows and channels

Every prompt node in a workflow streams. Events are tagged with the node’s channel, so a workflow declares at design time which of its conversations are user-facing: put the user-visible prompt on a support channel, connect with ?channel=support (or channels: ['support'] in the browser SDK), and internal research or synthesis lanes never reach the browser.

Streaming requires API access, so it is available on all paid plans.