Streaming

How a streamed completion arrives on the wire, what the last frame carries, and what happens when a client disconnects.

The transport

Set stream: true and the response is text/event-stream: one JSON object per data: line, each followed by an empty line. Buffering is disabled so a proxy does not hold frames back.

HTTP/1.1 200 OK
content-type: text/event-stream; charset=utf-8
cache-control: no-cache, no-transform
connection: keep-alive

The frames

The first frame carries the assistant role, then one frame per delta, then a frame with finish_reason and the routing block, then exactly one terminator. A client that follows the format reads the terminator and stops; it must not treat a missing terminator as a complete answer.

data: {"id":"chatcmpl-…","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"role":"assistant"},"finish_reason":null}]}

data: {"id":"chatcmpl-…","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"Hallo"},"finish_reason":null}]}

data: {"id":"chatcmpl-…","object":"chat.completion.chunk","choices":[{"index":0,"delta":{},"finish_reason":"stop"}],"llmeu":{…}}

data: [DONE]

The routing block travels with the last frame

The final frame carries the same llmeu object the non-streaming response returns: trace_id, model_used, model_version, provider, endpoint_id, inference_region, sovereignty_class, policy_id, retention and the cost estimate. In streaming mode usage is only known at the end, which is why the block travels with the last frame rather than the first.

Client disconnects

Closing the connection stops the work and writes the trace with what was actually produced. A cancelled request is still a call that happened, so it appears in usage; it is not recorded as a completed completion.

What cannot be streamed

Embeddings and the platform API are single JSON responses. stream: true on a non-streaming route is a validation error, not a silent fallback to one frame.