Agentic Memory from Llama Index

How to Structure Memory

As I argued in my earlier post on AI Agents, memory is the difference between a clever one‑off response and a system that actually learns you: your preferences, your past asks, which tools worked (and which didn't). For the high-level theory, see Duncan Calvert's "AI Agent Memory". Here, we'll stay pragmatic: short‑term vs. long‑term memory and how LlamaIndex operationalizes both with a clean, composable API.

Short‑term memory (STM) is the rolling context that keeps the model coherent right now. Long‑term memory (LTM) is slimmer, curated, and retrieved only when relevant. Something must decide what to keep, how to compress it, and when to bring it back. LlamaIndex's Memory object plus specialized memory blocks give you exactly that.

Memory

In LlamaIndex, Memory is the orchestrator. It owns the plain‑text rolling chat buffer and automatically flushes overflowed tokens into long‑term memory blocks once you exceed a global token_limit. If you don't attach any blocks, those tokens are simply dropped. You tune the short‑term buffer's protection with chat_history_token_ratio (e.g., 0.7 to reserve 70% of the context for the live chat) and set token_flush_size to control how many tokens get evicted at a time. Conversations are isolated under a session_id, and you can persist memory by pointing async_database_uri to a real database (instead of the default in‑memory SQLite). At inference time, Memory reassembles the prompt by merging the short‑term buffer with whatever long‑term content still fits under token_limit, injected via your chosen insert_method (commonly "system"). Conceptually: memory.put() logs new events; memory.get() returns the token‑bounded context the model will actually see.

StaticMemoryBlock

StaticMemoryBlock is your non‑negotiable, always‑on context: agent identity, safety policies, core constraints, or stable user profile facts. It never retrieves, ranks, or summarizes; it always gets appended to the LLM context on every call. You typically set priority=0, which effectively says: "don't ever truncate this." It's cheap, deterministic, and perfect for what you'd otherwise hard‑code into a system prompt.

FactExtractionMemoryBlock

FactExtractionMemoryBlock converts long chats into durable, compact facts. When short‑term memory overflows, this block asks an LLM (llm) to extract salient information: preferences, decisions, constraints, key entities, up to max_facts. If it overgrows, it summarizes itself to stay tight. The result is a lightweight, always‑useful layer you can inject almost every turn. The cost: extra LLM calls and lossy compression. In practice, give it priority=1, so it nearly always survives truncation right after your static policy block.

VectorMemoryBlock

VectorMemoryBlock is your semantic recall layer. Flushed messages are embedded (embed_model) and written to a vector store (vector_store). On each step, it can pull back the most relevant past snippets (similarity_top_k, retrieval_context_window) and optionally post‑process them. This is ideal when you need to reach way back ("remember that Athena query we debugged last month?") or when you want verbatim, auditable recovery. Because retrieved spans can be token‑heavy, assign it a lower priority (e.g., priority=2+) so it's trimmed first when the context gets tight.

Priority

The priority you assign to each block defines your degradation path under strict token budgets. A production‑ready default:

1. StaticMemoryBlock (priority=0): identity/rules must never drop.
2. FactExtractionMemoryBlock (priority=1): distilled, low‑cost facts that almost always help.
3. VectorMemoryBlock (priority=2+): deep semantic recall included only when there's space or clear relevance.

This mirrors the theory: guarantee invariant context, keep a compact evergreen factual spine, and fall back to heavy semantic retrieval when necessary. Need something bespoke (scores, counters, domain state)? Subclass BaseMemoryBlock and Memory will still flush, retrieve, and truncate it under the same rules.

Best Practices

Adopt the three‑layer stack with explicit priorities. Set a strict token_limit, keep chat_history_token_ratio high enough for live turns, and flush in coherent chunks (token_flush_size) so long‑term blocks receive meaningful spans. Bound max_facts so facts stay cheap to inject; tighten similarity_top_k / retrieval_context_window to keep vector recalls lean. Use separate session_ids and a persistent async_database_uri when you need continuity beyond process restarts. Prefer insert_method="system" for clean, deterministic prompt injection.

from llama_index.core.memory import (
    Memory,
    StaticMemoryBlock,
    FactExtractionMemoryBlock,
    VectorMemoryBlock,
)

blocks = [
    StaticMemoryBlock(
        name="core_identity",
        static_content="You are Atlas, a finance agent. Always be precise, cite data, no hallucinations.",
        priority=0,
    ),
    FactExtractionMemoryBlock(
        name="facts",
        llm=llm,
        max_facts=100,
        priority=1,
    ),
    VectorMemoryBlock(
        name="ltm_vec",
        vector_store=vector_store,
        embed_model=embed_model,
        priority=2,
        similarity_top_k=3,
        retrieval_context_window=5,
    ),
]

memory = Memory.from_defaults(
    session_id="user-123",
    token_limit=32_000,
    chat_history_token_ratio=0.7,
    token_flush_size=2_000,
    memory_blocks=blocks,
    insert_method="system",
)

response = await agent.run(user_msg, memory=memory)

Conclusion

Memory is what turns a clever LLM into a situated agent. LlamaIndex's Memory orchestrator plus a layered stack of StaticMemoryBlock (0) for invariants, FactExtractionMemoryBlock (1) for compact durable facts, and VectorMemoryBlock (2) for deep semantic recall maps cleanly onto short‑ vs. long‑term memory theory while staying within strict token limits. Priorities give you a predictable degradation path; knobs like token_limit, chat_history_token_ratio, and token_flush_size keep prompts lean. Start with the three‑block baseline, measure token pressure and retrieval usefulness, and only customize where your workload proves it's needed. The payoff is an agent that remembers what matters, forgets what doesn't, and scales its recall gracefully as conversations (and stakes) grow.