How to Handle LLM Token Limits for Long Documents with Claude, LangChain, and Python

I first noticed the problem while building a simple Q&A bot for a legal contract review. The contract was fifty pages long. My initial approach was straightforward: dump the entire document into the prompt and ask questions. But every answer came back either incomplete or hallucinated—the model simply forgot the middle of the contract. That’s when I realised that no matter how powerful the LLM, its context window is a hard boundary. You cannot push an elephant through a keyhole. The solution wasn’t a better model but a smarter pipeline: one that counts tokens, streams output, prunes history on the fly, and compresses old memory into summaries. This article walks through exactly how I built that system in Python, using Anthropic Claude and LangChain. By the end, you’ll have a production‑ready architecture for handling arbitrarily long inputs without losing context.

---

Token counting is not optional. Have you ever sent a long prompt and received a response that ignores half your instructions? That’s silent truncation. Most APIs simply clip your input when you exceed the limit, leaving you wondering why the model forgot. The first line of defence is an accurate token counter. I built one that supports multiple model families—tiktoken for OpenAI, character‑based estimation for Claude with a calibration factor. Here is the core logic:

from dataclasses import dataclass
import tiktoken

@dataclass
class TokenBudget:
    model: str
    max_context_tokens: int
    system_prompt_tokens: int
    conversation_tokens: int
    document_tokens: int
    reserved_output_tokens: int

    @property
    def available_tokens(self):
        return self.max_context_tokens - (self.system_prompt_tokens + self.conversation_tokens + self.document_tokens + self.reserved_output_tokens)

def count_tokens(text: str, model: str = "claude-3-5") -> int:
    if "claude" in model:
        return int(len(text) / 3.8)  # conservative ratio
    enc = tiktoken.encoding_for_model(model)
    return len(enc.encode(text))

The reserved_output_tokens is critical—you must leave room for the model’s reply. Without it, you’ll get a truncated stream. I set it at 4,000 tokens for most tasks. That little buffer saved me from dozens of silent failures.

---

Streaming makes long outputs bearable. When you ask a model to summarise a hundred‑page document, waiting for the full response to arrive before showing anything feels like watching paint dry. Streaming sends tokens as soon as they are generated, so the user sees the answer building in real time. Here is how I wired it with Anthropic’s API:

from anthropic import Anthropic
import json

client = Anthropic(api_key="your-key")

def stream_answer(system_prompt: str, messages: list):
    with client.messages.stream(
        model="claude-3-5-sonnet-20241022",
        max_tokens=4096,
        system=system_prompt,
        messages=messages,
    ) as stream:
        for text in stream.text_stream:
            yield text  # each chunk is a string fragment

Notice the yield—this turns your function into a generator. In a FastAPI endpoint you can use StreamingResponse to push tokens over HTTP. The user never waits more than a few milliseconds for the first word. That difference in perceived speed changes everything.

---

Dynamic context windows prevent the model from forgetting. The real challenge is what to keep and what to drop when the conversation history grows too long. I built a WindowManager that prunes old messages, summarises redundant chunks, and ensures the critical context always fits within the token budget.

from typing import List, Dict

class WindowManager:
    def __init__(self, max_tokens: int = 180000, compression_ratio: float = 0.85):
        self.max_tokens = max_tokens
        self.threshold = int(max_tokens * compression_ratio)

    def trim(self, messages: List[Dict], system_tokens: int, doc_tokens: int) -> List[Dict]:
        # Simulate token counting
        total = system_tokens + doc_tokens + sum(len(m["content"])*4 for m in messages)
        while total > self.threshold and messages:
            removed = messages.pop(0)
            total -= len(removed["content"]) * 4
        return messages

This is a simplistic version—real‑world systems prioritise messages by recency and importance. I later replaced the blind pop(0) with a scoring function that keeps assistant‑user exchanges that contain explicit instructions or questions. When the window is tight, I summarise the oldest group of messages into a single condensed turn.

How do you decide which messages are expendable? For me, any message that begins with “Okay, let me check” or “Thank you” can be summarised in one line.

---

Chunked memory compression turns documents into tiny summaries. Instead of stuffing an entire book into the prompt, you break it into sections, summarise each, and store the summaries. When a user asks a question, you retrieve only the relevant chunks and their summaries. Here is a memory store I use:

from langchain.text_splitter import RecursiveCharacterTextSplitter

class ChunkedMemory:
    def __init__(self, chunk_size: int = 4000, overlap: int = 200):
        self.splitter = RecursiveCharacterTextSplitter(
            chunk_size=chunk_size,
            chunk_overlap=overlap
        )
        self.chunks = []

    def ingest(self, doc: str):
        chunks = self.splitter.split_text(doc)
        for i, chunk in enumerate(chunks):
            summary = self._quick_summarize(chunk)
            self.chunks.append({
                "index": i,
                "text": chunk,
                "summary": summary
            })

    def _quick_summarize(self, text: str) -> str:
        # uses a cheap model (e.g., GPT-3.5-turbo) to create a one‑sentence summary
        return "placeholder summary for demo"

The summaries are stored alongside the original text. When a question comes in, I do a semantic search over the summaries first, then include only the top‑two matching full chunks in the prompt. This keeps the token usage low while retaining high‑fidelity detail where it matters.

---

Sliding windows and hierarchical summarisation extend the approach further. For extremely long documents (book‑length), a single level of chunking isn’t enough. I built a two‑layer compression: first summarise every 4,000‑token chunk, then summarise groups of those summaries into a top‑level abstract. The system retrieves the most relevant bottom‑level chunks and presents the top‑level abstract for context. This pattern is sometimes called “multi‑level summarisation.” It allows me to answer questions about a 500‑page book using only 8,000 tokens of prompt—while still being accurate.

---

Putting it all together: a long‑document Q&A system with streaming. I combined all the pieces into a single class that accepts a document, a question, and returns a streaming answer that respects token limits. The flow is simple:

1. Ingest the document into ChunkedMemory (split and summarise).
2. Accept the question, retrieve relevant chunks.
3. Build the system prompt + retrieved chunks + conversation history.
4. Use TokenBudget to verify everything fits; if not, trim conversation history.
5. Stream the answer using stream_answer.
6. Save the new exchange into conversation history for future trimming.

I exposed this as a FastAPI endpoint that returns StreamingResponse. That let me integrate it with a web front‑end. The latency reduction was dramatic—first token appeared in under 200 ms for most queries.

---

The most common pitfall I see is ignoring the output reservation. Developers calculate input tokens but forget that the model needs space to generate. The result is a half‑finished answer that silently truncates. Always reserve at least twice the expected output length. I also learned to use rich for logging token budgets in development. Here is an example log:

from rich import print

budget = TokenBudget(model="claude-3-5-sonnet", ...)
print(budget)
# TokenBudget(model=claude-3-5-sonnet, used=145,000/180,000 [80.6%], available=30,000)

That one line saved me hours of debugging.

---

If you are building anything that deals with long documents or extended conversations, start with accurate token counting and a robust trimming strategy. The rest—streaming, chunked memory, sliding windows—are easier to add once you have the foundation. I built this system while staring at a fifty‑page contract, but it applies equally to codebases, technical manuals, or a year’s worth of chat logs.

Now I want to hear from you. Have you hit the wall with token limits? What creative workarounds have you tried? Drop a comment below, share this article with a teammate who’s fighting the same problem, and hit the like button if it saved you an hour of frustration. The hardest part is realising the model isn’t broken—your context window is. Fix that, and you unlock everything else.

---

As a best-selling author, I invite you to explore my books on Amazon. Don’t forget to follow me on Medium and show your support. Thank you! Your support means the world!

---

101 Books

101 Books is an AI-driven publishing company co-founded by author Aarav Joshi. By leveraging advanced AI technology, we keep our publishing costs incredibly low—some books are priced as low as $4—making quality knowledge accessible to everyone.

Check out our book Golang Clean Code available on Amazon.

Stay tuned for updates and exciting news. When shopping for books, search for Aarav Joshi to find more of our titles. Use the provided link to enjoy special discounts!

---

📘 Checkout my latest ebook for free on my channel!
Be sure to like, share, comment, and subscribe to the channel!

---

Our Creations

Be sure to check out our creations:

Investor Central | Investor Central Spanish | Investor Central German | Smart Living | Epochs & Echoes | Puzzling Mysteries | Hindutva | Elite Dev | JS Schools

---

We are on Medium

Tech Koala Insights | Epochs & Echoes World | Investor Central Medium | Puzzling Mysteries Medium | Science & Epochs Medium | Modern Hindutva