RAG Ingestion Pipeline Guide: Better Chunking, Embeddings, and Retrieval Accuracy

You know that frustrating moment when you ask a question to an AI assistant and it gives you an answer that’s confidently wrong, or pulls from the wrong document? I’ve been there. In my work building these systems, I realized the problem is almost never the language model itself. It’s what happens before the question is even asked—how we prepare the documents. Get this part wrong, and even the most powerful AI will struggle. I wrote this to save you the months of trial and error I went through. If you’re building something that needs to find and use information accurately, this is where you should focus.

Why does this happen? Think about how we usually feed documents to an AI. A common first try is to simply cut a long text into 500-character pieces. The code looks clean and simple.

# This is the common, problematic starting point
from llama_index.core.node_parser import SimpleNodeParser

basic_parser = SimpleNodeParser.from_defaults(chunk_size=500, chunk_overlap=50)

But here’s the issue: text doesn’t care about character counts. A sentence explaining a crucial legal clause or a complex formula can be split right down the middle. When you ask a question later, the system looks at these fragments. The piece containing the beginning of the answer and the piece containing the end might be stored separately. The system retrieves one incomplete chunk and the AI is left to guess the rest, often leading to what we call a “hallucination.” The answer was in the data, but our preparation made it invisible. Have you ever searched for a file on your computer only to find it was saved in the wrong folder with a meaningless name? That’s what naive chunking does to your AI’s knowledge.

So, what’s the better way? We need strategies that respect the natural boundaries in the text. Let’s move beyond the simple split and look at four practical methods, each with its own strength.

The first upgrade is the Recursive Splitter. Instead of counting characters, it tries to split at natural breaks like paragraphs, then sentences, until the chunks are roughly the right size. It keeps related ideas together more often.

from llama_index.core.node_parser import SentenceSplitter

recursive_parser = SentenceSplitter(chunk_size=512, chunk_overlap=64, paragraph_separator="\n\n")

For general articles or documentation, this is a strong, straightforward choice. But what if the ideas span multiple sentences or change topic mid-paragraph?

This brings us to a more advanced technique: Semantic Chunking. This method uses an embedding model itself to decide where to split. It reads the text and finds the spots where the topic or meaning subtly shifts, creating chunks that are cohesive in their ideas, not just their length. Imagine a textbook chapter moving from explaining a theory to showing an example—a semantic splitter would likely place a chunk boundary there.

from llama_index.core.node_parser import SemanticSplitterNodeParser
from llama_index.embeddings.openai import OpenAIEmbedding

embed_model = OpenAIEmbedding()
semantic_parser = SemanticSplitterNodeParser(buffer_size=1, breakpoint_percentile_threshold=95, embed_model=embed_model)

The result is chunks that feel more complete. However, sometimes an answer needs the full context of a few surrounding sentences. What if the key detail is in sentence five, but sentences one and ten provide necessary framing?

The Sentence Window strategy is designed for this. It creates chunks centered on a single sentence, but also stores the sentences immediately before and after it as “context.” During retrieval, it fetches the core sentence and its surrounding window. This gives the language model the focal point plus the environment it existed in, which is incredibly powerful for precision.

from llama_index.core.node_parser import SentenceWindowNodeParser

window_parser = SentenceWindowNodeParser.from_defaults(window_size=3, window_metadata_key="context", original_text_metadata_key="original_sentence")

Choosing the right strategy depends on your documents. Legal contracts? Semantic chunking might preserve clauses. A Q&A manual? Sentence windows could pinpoint answers. Technical code with comments? A recursive split on newlines might work best. The choice isn’t theoretical; it directly impacts performance.

Now, we have well-prepared text chunks. The next critical step is turning them into something the computer can compare: vectors, or embeddings. This is where we translate meaning into math. The choice of embedding model is as important as the chunking. A model trained on scientific papers will represent technical terms brilliantly but might falter with slang or customer support tickets.

You can easily test different models. For example, a popular open-source option like BAAI/bge-large-en-v1.5 offers a great balance of speed and accuracy for general English.

from llama_index.embeddings.huggingface import HuggingFaceEmbedding

hf_embed_model = HuggingFaceEmbedding(model_name="BAAI/bge-large-en-v1.5")

Once you’ve chosen a model, you build your index—the searchable library of your content. Using a vector store like ChromaDB with LlamaIndex makes this process clean.

from llama_index.core import VectorStoreIndex, StorageContext
from llama_index.vector_stores.chroma import ChromaVectorStore
import chromadb

# Connect to a persistent database
chroma_client = chromadb.PersistentClient(path="./chroma_db")
chroma_collection = chroma_client.get_or_create_collection("knowledge_base")
vector_store = ChromaVectorStore(chroma_collection=chroma_collection)
storage_context = StorageContext.from_defaults(vector_store=vector_store)

# Create the index with your documents and chosen embedder
index = VectorStoreIndex.from_documents(your_documents, storage_context=storage_context, embed_model=hf_embed_model)

This creates your ready-to-query knowledge base. But a production system needs more. How do you handle updates without re-processing everything? You use incremental ingestion, adding only new or changed documents. How do you ensure the retrieved chunks are diverse and not repetitive? You implement algorithms like MMR (Maximal Marginal Relevance) at query time, which prioritizes unique information.

Finally, you must measure. Tools like RAGAS can score your system on metrics like whether the retrieved context actually contains the answer (faithfulness) and whether that answer is relevant to the question. This turns guesswork into engineering.

Building this pipeline is the unsung hero of a reliable AI application. It’s less about flashy AI and more about thoughtful data craftsmanship. By investing time here—choosing the right way to split your text, selecting a suitable embedding model, and structuring your index for scale—you build a foundation that makes the AI look smart. The language model gets the glory, but the ingestion pipeline does the essential work.

Was there a particular chunking strategy you’re curious to try with your own documents? Share your thoughts or questions below—I’d love to hear what you’re building. If you found this walkthrough helpful, please like or share it with someone else who might be piecing together their own RAG system.

---

As a best-selling author, I invite you to explore my books on Amazon. Don’t forget to follow me on Medium and show your support. Thank you! Your support means the world!

---

101 Books

101 Books is an AI-driven publishing company co-founded by author Aarav Joshi. By leveraging advanced AI technology, we keep our publishing costs incredibly low—some books are priced as low as $4—making quality knowledge accessible to everyone.

Check out our book Golang Clean Code available on Amazon.

Stay tuned for updates and exciting news. When shopping for books, search for Aarav Joshi to find more of our titles. Use the provided link to enjoy special discounts!

---

📘 Checkout my latest ebook for free on my channel!
Be sure to like, share, comment, and subscribe to the channel!

---

Our Creations

Be sure to check out our creations:

Investor Central | Investor Central Spanish | Investor Central German | Smart Living | Epochs & Echoes | Puzzling Mysteries | Hindutva | Elite Dev | JS Schools

---

We are on Medium

Tech Koala Insights | Epochs & Echoes World | Investor Central Medium | Puzzling Mysteries Medium | Science & Epochs Medium | Modern Hindutva