Streaming

Both the OpenAI (/v1/chat/completions) and Anthropic (/v1/messages) surfaces support streaming over Server-Sent Events. Streaming is the recommended default for anything with a long prompt, long output, or a high token cap — it avoids request timeouts and lets you render tokens as they arrive.

OpenAI-style stream

curl -N https://api.uberllm.ai/v1/chat/completions \
  -H "Authorization: Bearer $UBERLLM_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "deepseek/deepseek-v3.1",
    "stream": true,
    "messages": [{"role": "user", "content": "Count to five, one per line."}]
  }'

The response is a sequence of data: lines, each a chat completion chunk, ending with data: [DONE].

from openai import OpenAI

client = OpenAI(base_url="https://api.uberllm.ai/v1", api_key="ull_...")

with client.chat.completions.create(
    model="deepseek/deepseek-v3.1",
    stream=True,
    messages=[{"role": "user", "content": "Write a short poem."}],
) as stream:
    for chunk in stream:
        delta = chunk.choices[0].delta.content
        if delta:
            print(delta, end="", flush=True)

Usage accounting

UberLLM injects stream_options: {"include_usage": true} for upstreams that support it, so the final chunk carries a usage object with prompt and completion token counts. Your request is settled against those numbers; when a provider omits usage, the gateway falls back to a tokenizer estimate and flags the request as estimated in your activity log.

Aborts

If you disconnect mid-stream, the gateway cancels the upstream request and settles for the tokens that were actually streamed. You are never charged for a full response you did not receive, and no reserved balance is left stuck.

Keep-alives

The gateway sends an immediate : ping comment on connect and periodic keep-alives, so long-running streams survive proxy idle timeouts.