AIForum.

How to Implement Streaming AI API Responses with Server-Sent Events (SSE)

On this page

THE SHORT ANSWER

Streaming an AI response over server-sent events means sending model output incrementally instead of waiting for the full completion. Enabling it is usually a single request flag; the real work is parsing the event stream, defeating output buffering, and deciding what happens when a stream breaks mid-generation.
Quick facts

Streaming an AI API response over server-sent events (SSE) means delivering the model’s output to the caller incrementally instead of waiting for the complete completion. In practice you enable it with a request flag — the OpenAI Responses API uses stream: true — and then consume a sequence of events rather than a single JSON body. The difficult part is rarely the flag. It is buffering, event parsing, and deciding what the application should do when a stream stops halfway through.

This guide is written for developers and technically curious practitioners who already have a working non-streaming API call and want to convert it. It covers the wire format, minimal implementations in Python, JavaScript and Go, a raw curl check you can run before touching application code, and the failure modes that most often make a stream arrive all at once at the very end.

What SSE actually sends on the wire

Server-sent events are a text protocol carried over a long-lived HTTP response. The response uses the content type text/event-stream and stays open while the server writes messages into it. Each message is a small block of text whose payload lines begin with data:.

The Chrome for Developers guide on how LLMs stream responses shows this concretely. A curl call to the Gemini API’s streamGenerateContent endpoint with alt=sse and --no-buffer logs a sequence of data: lines. An early line carries a short fragment such as “A T-Rex”; the next carries a longer continuation. Each payload is a JSON object containing a candidates array, and later messages carry finishReason and usageMetadata with token counts. The exact JSON differs between providers, but the shape does not: many small payloads, each prefixed by data:, delivered as the model generates them.

Two details matter when you write a parser. First, the data: prefix is framing, not payload — strip it, along with the single following space, before parsing JSON. Second, an event’s payload can be spread across multiple data: lines, so a robust parser accumulates lines until a blank line terminates the event. Some providers also send a terminal sentinel such as [DONE] instead of a JSON object. Treat a payload that is not valid JSON as a possible end marker rather than letting it crash the reader.

Chunks are not tokens

A frequent confusion is treating a “chunk” as one token or one word. A chunk is a transport-level delivery unit chosen by the provider. The Gemini example above shows a first chunk of two words and a second containing a full clause. Never build logic that assumes chunk boundaries align with sentences, words, or token counts. If you need exact token accounting, read the usage metadata the provider sends at the end of the stream, as the Chrome example shows in its usageMetadata field.

Prerequisites

  • A working non-streaming request to your provider, with the API key stored in an environment variable rather than in source code.
  • An HTTP client that can read a response body incrementally: an official SDK, or curl --no-buffer for manual checks.
  • If a reverse proxy, CDN, or web server sits between your client and the provider, the ability to change its buffering configuration.

Enabling streaming with an SDK

The OpenAI streaming guide shows the same pattern across languages: set the stream flag, then iterate over events as they arrive.

Python (Responses API)

from openai import OpenAI

client = OpenAI()

stream = client.responses.create(
    model="your-model",
    input=[
        {"role": "user", "content": "Say 'double bubble bath' ten times fast."},
    ],
    stream=True,
)

for event in stream:
    print(event)

The expected result is that events print with visible gaps between them, not in a single burst after a pause.

JavaScript

import { OpenAI } from "openai";

const client = new OpenAI();

const stream = await client.responses.create({
  model: "your-model",
  input: [
    { role: "user", content: "Say 'double bubble bath' ten times fast." },
  ],
  stream: true,
});

for await (const event of stream) {
  console.log(event);
}

Go

The Go SDK exposes explicit iteration rather than a callback. You call client.Responses.NewStreaming(...), loop with for stream.Next(), read the current event through stream.Current().Type, and finish by checking stream.Err(). That final error check is the Go equivalent of an exception handler: a stream that ends because of a network fault and a stream that ends because the model finished can otherwise look identical.

Verify the transport before writing application code

Before wiring SSE into an application, confirm that the endpoint genuinely streams and that nothing between you and the provider is buffering. A raw request is the fastest way to find out. The Chrome for Developers guide uses this form against the Gemini API:

curl "https://generativelanguage.googleapis.com/v1beta/models/gemini-1.5-flash:streamGenerateContent?alt=sse&key=$GOOGLE_API_KEY" 
  -H 'Content-Type: application/json' 
  --no-buffer 
  -d '{"contents":[{"parts":[{"text": "Tell me a long T-rex joke, please."}]}]}'

Keep the key out of shell history and logs, and mask it in anything you paste into a forum. The observable check is timing: text should appear progressively. If everything appears at once after a pause, buffering is happening somewhere — most often --no-buffer was omitted, or an intermediary is holding the response.

Forwarding a stream to a browser

If your frontend talks directly to the provider, remember that the browser’s EventSource object cannot set custom request headers, so it cannot carry an Authorization header. Two patterns work in practice.

  1. Server proxy. Your backend holds the API key, opens the provider stream, and re-emits data: lines to the browser. This also gives you a place to log, meter, and redact.
  2. fetch plus a readable stream. The browser sends a normal POST with headers and reads the response body incrementally, parsing SSE frames in your own code.

EventSource brings automatic reconnection, which is a benefit for notification feeds and a liability for generation. A reconnect mid-generation typically restarts the model call and can duplicate tokens already displayed. For chat-style output, prefer fetch-based streaming and treat reconnection as an explicit application decision.

Streaming through PHP or WordPress

WordPress’s REST API and most PHP setups buffer output by default, because the HTTP API is designed around complete responses rather than incremental ones. An illustrative design — a pattern to evaluate, not a configuration tested for this article — is a dedicated endpoint that disables output buffering, sets Content-Type: text/event-stream, writes each data: line, and calls flush() after every write, with the web server configured not to buffer that specific route. Use the same curl check against your own endpoint: if the browser shows nothing until the end, buffering is still active somewhere in the chain.

Failure modes and how to diagnose them

SymptomLikely causeDiagnosticFix
Everything arrives at once at the endOutput buffering in the proxy, framework, or clientRun curl --no-buffer directly against the providerDisable buffering at the layer that is holding the response
Stream stops after roughly a minuteIdle read timeout on a proxy or load balancerCompare timings against the gateway’s timeout settingsRaise the read timeout and emit periodic keepalive comment lines
JSON parse errors partway throughParsing the framing, or hitting a non-JSON sentinelLog the raw line before parsingStrip the data: prefix and handle [DONE] explicitly
Text appears twice after a network blipAutomatic reconnection replaying the requestInspect your client library’s retry behaviorDisable automatic retries for generation streams
Token usage is unclearUsage metadata only arrives at the end of the streamInspect the final event’s usage fieldsPersist usage from the terminal event; check your provider’s policy for interrupted generations

A verification checklist

  1. Raw curl against the provider shows progressive output.
  2. The same curl against your own endpoint also shows progressive output.
  3. Your parser strips the data: prefix and tolerates multi-line payloads and a non-JSON sentinel.
  4. Errors raised mid-stream surface to the user instead of silently truncating the answer.
  5. Usage or completion metadata from the final event is captured for accounting.
  6. Reconnection behavior is deliberate, not inherited from a library default.

What to do next

Start with the raw curl check, because it separates “the provider is not streaming” from “my stack is buffering” in a single step. Once that passes, move the same check one layer inward — your proxy, then your endpoint, then the browser — until the buffering layer reveals itself. If you are running local models, the same framing applies to any OpenAI-compatible server that supports stream: true, and the same buffering questions apply to whatever sits in front of it.

If you hit a case this guide does not cover — an unusual proxy, a framework that buffers in a surprising place, or a provider whose event format differs — share your environment, the raw output you see, and the error text in the community so others can reproduce it.

FAQ

Do I need EventSource to consume an AI stream in the browser?

No. EventSource cannot set custom headers such as Authorization, so it is a poor fit for direct provider calls. A fetch request with a readable response body works and lets you parse SSE frames yourself. EventSource's automatic reconnection is also undesirable for generation streams, because a reconnect typically restarts the model call.

Why does my streamed response arrive all at once?

Almost always because something is buffering the response rather than because streaming is unsupported. Test with a raw curl request using --no-buffer directly against the provider. If that streams correctly, the buffering is in your proxy, framework, or application layer.

Is each streamed chunk one token?

No. A chunk is a delivery unit chosen by the provider and can contain a few words or a longer clause. For accurate token counts, read the usage metadata that providers send in the final event of the stream.

How should I handle a stream that fails midway?

Detect the failure through your client's error path rather than treating a short response as a complete one, surface the truncation to the user, and disable automatic retries that would replay the generation. Whether an interrupted generation is still billed depends on the provider, so check its policy rather than assuming.

References

  1. OpenAI, Streaming API responses — https://developers.openai.com/api/docs/guides/streaming-responses
  2. Chrome for Developers, How LLMs stream responses — https://developer.chrome.com/docs/ai/streaming

Related terms

Server-sent events (SSE)
A text-based protocol for pushing messages from a server to a client over a single long-lived HTTP response, using the text/event-stream content type.
Chunk
A transport-level unit of streamed output chosen by the provider. It does not correspond to one token or one word, so parsing logic should not depend on chunk boundaries.
Event stream framing
The data: prefix and blank-line separators that wrap each payload. The prefix must be stripped before JSON parsing, and a payload may span multiple data: lines.
Output buffering
The accumulation of response bytes by a client, framework, or proxy before they are forwarded. It is the most common reason a working stream appears all at once.
Usage metadata
Token counts and completion information that providers typically send in a final stream event rather than in each chunk.

Leave a Reply

Your email address will not be published. Required fields are marked *