llama-index
LlamaIndex is a Python framework for building LLM applications over your own data, with retrieval-augmented generation as its core competency. It provides data loaders, chunking, embeddings, vector indexes, retrievers, query and chat engines, and agent workflows behind a few high-level classes; the five-line SimpleDirectoryReader-to-VectorStoreIndex path is the canonical demo. The llama-index package on PyPI is a starter bundle: llama-index-core plus OpenAI LLM and embedding defaults. Everything else (other LLMs, embedding models, vector stores) is a separate llama-index-* integration package from a catalog of 300+.
The most focused RAG framework in Python: if the problem is answering questions over your documents, its abstractions earn their keep. Pin versions and budget for upgrade churn until it reaches 1.0.
Use it if
- Your app is fundamentally retrieval over documents: internal search, chat over PDFs, knowledge assistants
- You want to swap LLMs, embedding models, and vector stores behind one interface instead of committing to a single vendor's SDK
- You need ready-made ingestion machinery: readers for many formats, sentence-aware chunking, and metadata handling
- You are prototyping RAG and want the five-line index-and-query path working before you customize retrievers and chunking
- You are just calling one LLM API with prompts; the provider SDK plus a few functions is less machinery than adopting a framework
- You need API stability: the package is still 0.x, breaking changes land between minor releases, and 2024-era tutorial code frequently no longer imports
- Dependency weight bothers you: the starter bundle pulls nltk and OpenAI integrations whether you use them or not, and a real app accumulates a dozen llama-index-* packages that must stay version-compatible with core
- Your workload is agent orchestration more than retrieval; LangGraph or a hand-rolled state machine can fit better than retrofitting query engines
Setup reality
pip install llama-index gets you the starter bundle wired to OpenAI, so the quickstart only works after export OPENAI_API_KEY. Going beyond the defaults means finding the right integration on LlamaHub (llama-index-llms-ollama, llama-index-vector-stores-chroma, and so on) and pip-installing each one; import paths mirror package names, which helps. Expect churn: pre-1.0 versioning means you pin versions and read changelogs on every upgrade, and the README itself warns that the docs are more current than the README.
Patterns
Index a folder and ask a questionbuild-index-and-query
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader
documents = SimpleDirectoryReader("data").load_data()
index = VectorStoreIndex.from_documents(documents)
query_engine = index.as_query_engine()
print(query_engine.query("What does the contract say about termination?"))Defaults to OpenAI for both LLM and embeddings; fails immediately without OPENAI_API_KEY set.
Use a local or non-OpenAI LLMswap-llm
from llama_index.core import Settings
from llama_index.llms.ollama import Ollama
Settings.llm = Ollama(model="llama3.1", request_timeout=360.0)Requires pip install llama-index-llms-ollama; Settings is a global that all components created afterwards read.
Use local embeddingsswap-embeddings
from llama_index.core import Settings
from llama_index.embeddings.huggingface import HuggingFaceEmbedding
Settings.embed_model = HuggingFaceEmbedding(
model_name="BAAI/bge-small-en-v1.5"
)Requires pip install llama-index-embeddings-huggingface; changing the embed model invalidates existing indexes, so re-embed everything.
Persist an index to disk and reload itpersist-and-reload
index.storage_context.persist(persist_dir="./storage")
from llama_index.core import StorageContext, load_index_from_storage
storage_context = StorageContext.from_defaults(persist_dir="./storage")
index = load_index_from_storage(storage_context)Default storage is in-memory and dies with the process; disk persistence writes JSON, fine for small corpora only.
Back the index with a real vector storeexternal-vector-store
import chromadb
from llama_index.core import VectorStoreIndex, StorageContext
from llama_index.vector_stores.chroma import ChromaVectorStore
client = chromadb.PersistentClient(path="./chroma_db")
collection = client.get_or_create_collection("docs")
vector_store = ChromaVectorStore(chroma_collection=collection)
storage_context = StorageContext.from_defaults(vector_store=vector_store)
index = VectorStoreIndex.from_documents(documents, storage_context=storage_context)Needs pip install llama-index-vector-stores-chroma chromadb; the same StorageContext pattern applies to every vector store integration.
Control chunk size and overlapcontrol-chunking
from llama_index.core.node_parser import SentenceSplitter
splitter = SentenceSplitter(chunk_size=512, chunk_overlap=50)
index = VectorStoreIndex.from_documents(documents, transformations=[splitter])Retrieval quality is usually more sensitive to chunking than to which LLM you picked; tune this first.
Retrieve more context per querytune-retrieval
query_engine = index.as_query_engine(
similarity_top_k=8,
response_mode="compact",
)The default top_k of 2 is often too few for real corpora; compact mode packs retrieved chunks into fewer LLM calls.
Chat engine with conversation historychat-with-memory
chat_engine = index.as_chat_engine(chat_mode="condense_plus_context")
response = chat_engine.chat("What is the notice period?")
follow_up = chat_engine.chat("And who does it apply to?")as_query_engine is stateless; chat engines keep history and rewrite follow-ups into standalone retrieval queries.
Stream the answer token by tokenstream-response
query_engine = index.as_query_engine(streaming=True)
response = query_engine.query("Summarize the report")
for token in response.response_gen:
print(token, end="")Streaming covers the synthesis step only; retrieval still completes before the first token arrives.
See which chunks produced the answerinspect-sources
response = query_engine.query("What changed in Q3?")
for node in response.source_nodes:
print(node.score, node.metadata.get("file_name"))
print(node.text[:200])Check source_nodes whenever answers look wrong; bad answers are usually bad retrieval, not a bad LLM.
Agent that calls Python functions as toolsfunction-agent
import asyncio
from llama_index.core.agent.workflow import FunctionAgent
from llama_index.llms.openai import OpenAI
def multiply(a: float, b: float) -> float:
"""Multiply two numbers."""
return a * b
agent = FunctionAgent(tools=[multiply], llm=OpenAI(model="gpt-4o-mini"))
result = asyncio.run(agent.run("What is 7 times 8?"))Agent runs are async, so wrap them in asyncio.run outside notebooks; function docstrings become the tool descriptions.
Install only what you usecore-only-install
pip install llama-index-core \
llama-index-llms-openai \
llama-index-embeddings-huggingfaceSkip the llama-index starter bundle in production so you control exactly which integrations and transitive deps ship.
Alternatives
| Package | Registry | Pick it when |
|---|---|---|
| langchain | PyPI | You want the larger ecosystem and LangGraph-style agent orchestration around your retrieval |
| haystack-ai | PyPI | You want explicit, typed pipeline graphs for production RAG and search |
| txtai | PyPI | You want a lighter all-in-one embeddings database with optional LLM plumbing |