Have you ever watched an AI generate text, one word at a time? That moment you see the first few words appear, you’re not just reading a response; you’re watching a thought form. This immediate, flowing interaction is what makes modern AI tools feel alive. It’s also why I believe any developer building with large language models today needs to understand streaming. Waiting 30 seconds for a complete block of text feels outdated. Users expect to see progress, to feel a conversation happening in real time. That’s the shift from static to dynamic, from waiting to participating. Let’s build applications that meet this expectation.
So, what’s the core idea? Instead of asking an AI for a complete answer and waiting, we ask it to send us pieces of the answer as it creates them. Each piece, often a word or a short phrase, is called a token. We receive these tokens over a continuous connection and show them to the user immediately. This method changes everything about how an application feels.
Why does this approach work so much better? First, it tackles the biggest problem in user experience: perceived latency. A user might tolerate a two-second wait for a full page to load, but staring at a blank box for ten seconds while an essay is written? That feels broken. By showing the first token in under a second, the application signals that work has begun. The user’s mind engages with the partial content, making the total wait feel shorter. Second, it respects resources. If a user reads the first sentence and realizes it’s off-topic, they can stop the generation. You’ve saved processing power and cost on tokens the user didn’t want.
How do we make this happen technically? We need a two-part system: a backend that can handle a long-lived connection and a frontend that can listen to a steady stream of data. The magic link between them is often a technology called Server-Sent Events, or SSE. Think of SSE as a one-way radio broadcast from your server to the client. The server opens a connection and just keeps sending messages down the line. It’s simpler and more efficient for this task than other methods like WebSockets when you only need server-to-client communication.
Let’s start with the backend, the heart of the operation. We’ll use a Python framework like FastAPI because it handles asynchronous code beautifully, which is essential for managing multiple streams. Our job is to create an endpoint that does three things: connects to an LLM provider, asks for a streamed response, and then forwards each token to the client. Here is a foundational piece of that code.
from fastapi import FastAPI
from sse_starlette.sse import EventSourceResponse
import asyncio
import openai
app = FastAPI()
async def generate_ai_stream(prompt: str):
"""Connects to OpenAI and streams tokens."""
client = openai.AsyncOpenAI()
stream = await client.chat.completions.create(
model="gpt-4",
messages=[{"role": "user", "content": prompt}],
stream=True
)
async for chunk in stream:
if token := chunk.choices[0].delta.content:
yield f"data: {token}\n\n"
await asyncio.sleep(0.001) # Small delay to prevent overwhelming the client
@app.get("/stream")
async def stream_response(prompt: str):
"""The endpoint that clients connect to."""
return EventSourceResponse(generate_ai_stream(prompt))See how the generate_ai_stream function is an async generator? It yields each token as it arrives, formatted specifically for SSE. The frontend receives these as distinct events.
But what if you’re not using OpenAI? The principle is the same. Whether you’re calling Anthropic’s Claude, a model running locally via Ollama, or another service, you look for their streaming option. The key is to set up a provider-agnostic layer in your code. This way, switching models doesn’t mean rewriting your entire streaming logic. You just create a new adapter that knows how to handle that provider’s specific streaming response format.
Now, sending raw tokens directly to the browser is a good start, but it’s not efficient. Sending a network packet for every single word creates a lot of overhead. This is where a token buffer becomes useful. Instead of firing off each token immediately, we collect a few of them—maybe a short phrase—and send them as one batch. This reduces the number of tiny network calls without noticeably harming the real-time feel for the user. You can tune this based on your needs; a creative writing tool might use smaller batches for a more delicate flow, while a code generator might use larger ones.
class TokenBuffer:
def __init__(self, batch_size=3):
self.batch = []
self.batch_size = batch_size
def add(self, token):
"""Adds a token and returns a batch if ready."""
self.batch.append(token)
if len(self.batch) >= self.batch_size:
ready_batch = ''.join(self.batch)
self.batch = [] # Clear the buffer
return ready_batch
return NoneWhat happens when things go wrong? Network connections drop. The AI service might have a hiccup. A robust streaming system needs to handle these gracefully. This involves setting timeouts on connections, catching exceptions cleanly, and sending a clear closing message to the client. You should also implement a way for the client to cancel the stream, which immediately stops the backend from requesting more tokens, saving cost.
One of the most powerful aspects of receiving tokens one by one is the ability to inspect and modify them on the fly. This is token-level processing. Imagine you’re building a customer service bot. You could run each token through a content filter the moment it’s generated. If the filter detects inappropriate language, you can stop the stream before the offending word is ever sent to the user and redirect the AI to apologize and start over. This real-time moderation is far more effective than checking a completed block of text after the damage is done.
# A simple safety filter within the stream
async def monitored_stream(prompt):
safety_blocklist = ["bad_word", "another_bad_term"]
buffer = TokenBuffer()
async for token in get_ai_token_stream(prompt):
# Check each token against a blocklist
if any(bad in token.lower() for bad in safety_blocklist):
yield "data: [Content filtered.]\n\n"
break # Stop the stream
if batched := buffer.add(token):
yield f"data: {batched}\n\n"
# Send any remaining tokens in the buffer
if buffer.batch:
yield f"data: {''.join(buffer.batch)}\n\n"The frontend’s job is to listen. In JavaScript, you use the EventSource API to connect to our /stream endpoint. It listens for “message” events, takes the token data, and appends it to the webpage. This creates the characteristic typing effect. But have you considered what makes a frontend feel truly responsive? It’s not just displaying text. It’s providing a UI that reacts. You might show a blinking cursor while waiting for the first token, then replace it with the text. You should also provide a prominent “Stop” button that disconnects the EventSource, which signals your backend to cancel the AI request.
// Frontend code to consume the stream
function startStream(prompt) {
const eventSource = new EventSource(`/stream?prompt=${encodeURIComponent(prompt)}`);
const outputDiv = document.getElementById('ai-output');
outputDiv.innerHTML = ''; // Clear previous content
eventSource.onmessage = function(event) {
// Append the new token batch to the page
outputDiv.innerHTML += event.data;
// Optional: auto-scroll to the bottom
outputDiv.scrollTop = outputDiv.scrollHeight;
};
eventSource.onerror = function() {
eventSource.close();
console.log("Stream ended.");
};
// Function to stop the stream manually
window.stopStream = () => { eventSource.close(); };
}When you put all these pieces together—a robust async backend, efficient token batching, real-time processing, and a reactive frontend—you create an experience that feels instantaneous and interactive. You’re not just building a feature; you’re building user trust. The application feels intelligent and responsive because it communicates its process.
The move from batch to streaming is not just a technical upgrade; it’s a fundamental change in how we conceive of human-AI interaction. We’re moving from a request-reply model to a continuous dialogue. This opens doors for applications we haven’t even imagined yet. What kind of real-time, collaborative tool could you build with this foundation? The next step is to try it. Start small, get a simple stream working from a local model, and feel the difference. Then, layer in the complexity.
Building this well means your applications will stand out. They’ll feel faster, more engaging, and more reliable. I encourage you to take these concepts, experiment with the code, and see what you can create. If you found this walkthrough helpful, please share it with a colleague who might be building the next great AI interface. I’d love to hear about your projects and any clever twists you’ve added—leave a comment below and let’s discuss.
As a best-selling author, I invite you to explore my books on Amazon. Don’t forget to follow me on Medium and show your support. Thank you! Your support means the world!
101 Books
101 Books is an AI-driven publishing company co-founded by author Aarav Joshi. By leveraging advanced AI technology, we keep our publishing costs incredibly low—some books are priced as low as $4—making quality knowledge accessible to everyone.
Check out our book Golang Clean Code available on Amazon.
Stay tuned for updates and exciting news. When shopping for books, search for Aarav Joshi to find more of our titles. Use the provided link to enjoy special discounts!
📘 Checkout my latest ebook for free on my channel!
Be sure to like, share, comment, and subscribe to the channel!
Our Creations
Be sure to check out our creations:
Investor Central | Investor Central Spanish | Investor Central German | Smart Living | Epochs & Echoes | Puzzling Mysteries | Hindutva | Elite Dev | JS Schools
We are on Medium
Tech Koala Insights | Epochs & Echoes World | Investor Central Medium | Puzzling Mysteries Medium | Science & Epochs Medium | Modern Hindutva
