Giving An Llm Memory Short Term Context Vs Long Te

The Problem: Your Agent Has Amnesia

Remember the agent you built in Part 2? It was great at calling tools. You could ask it to check the weather, look up a fact, or do some math, and it would figure out which tool to use and hand back the right answer. It felt alive.

But then you closed the tab. The next day, you opened a fresh conversation and typed:

“Remember that analysis we were working on yesterday? Let’s finish it.”

And the agent replied:

“I’m sorry, but I don’t have any record of a previous analysis. Could you remind me what you’re referring to?”

Ouch. Your agent doesn’t have amnesia — it was never designed to remember in the first place. Every conversation starts from scratch, with a blank slate. The agent from Part 2 is stateless: it treats each interaction as if it’s the first one you’ve ever had.

Let’s see this failure in action. Here’s a simulation of what happens when you try to recall a fact from a previous session:

# Block 1: The amnesia problem — agent can't recall across sessions
# This simulates what happens when you start a new conversation

# Session 1: User tells the agent something
session_1_memory = []  # Fresh memory for this session
session_1_memory.append({"role": "user", "content": "My name is Alice and I'm working on Project Phoenix."})
# The agent stores this in its context window for this session
print("Session 1: Agent learned about Project Phoenix")
print(f"  Memory after session 1: {session_1_memory}")
print()

# Session 2: Next day, completely fresh start
session_2_memory = []  # New session = blank slate
session_2_memory.append({"role": "user", "content": "What was I working on yesterday?"})
# The agent has no record of session 1
print("Session 2: Agent tries to recall")
print(f"  Memory after session 2: {session_2_memory}")
print("  Result: Agent has no idea what the user is talking about.")
print()

# This is the core problem: no persistence between sessions
print("The agent doesn't have amnesia — it was never designed to remember.")

When you run this, you’ll see:

Session 1: Agent learned about Project Phoenix
  Memory after session 1: [{'role': 'user', 'content': "My name is Alice and I'm working on Project Phoenix."}]

Session 2: Agent tries to recall
  Memory after session 2: [{'role': 'user', 'content': 'What was I working on yesterday?'}]
  Result: Agent has no idea what the user is talking about.


The agent doesn't have amnesia — it was never designed to remember.

That’s the problem we’re going to solve in this article. By the end, you’ll build a hybrid memory system that combines a short-term context window with a long-term retrieval store. Your agent will remember who you are, what you’ve been working on, and pick up right where you left off — even days later.

What We Mean by ‘Memory’ for an LLM

Before we start coding, let’s get our vocabulary straight. Human memory is a useful analogy, but it breaks down if you push it too far. An LLM doesn’t have a hippocampus or a prefrontal cortex. It has a context window and a file system.

Here’s the key distinction, in plain English:

• Short-term memory = the context window. It’s everything you put into the prompt for the current turn. It’s fast (sub-millisecond to access), ephemeral (gone when the conversation ends), and limited (200K tokens for Claude Haiku, 1M for Opus).
• Long-term memory = anything stored outside the context window. It’s slower (100ms+ to retrieve), persistent (survives across sessions), and effectively unbounded. It must be retrieved and injected into the context window when needed.

The Mem0 blog puts it nicely: short-term memory is about latency — you need it fast because it’s part of every single API call. Long-term memory is about access — you’re willing to wait a bit because you’re searching through a much larger store.

And here’s the insight that changes everything: An LLM doesn’t have memory — it has a context window. Memory is something you build around it.

The MemGPT paper (https://arxiv.org/abs/2310.08560) frames this as a virtual memory system. Think of the context window as RAM — fast, but limited. External storage (a vector database, a JSON file, whatever) is like disk — slow, but huge. The agent needs a paging mechanism: when it needs a fact that’s not in the current context, it has to page it in from long-term storage.

Here’s a mental model of the flow:

User message
    ↓
Agent checks short-term memory (recent conversation history)
    ↓
Agent queries long-term memory for relevant facts
    ↓
Agent combines both into the context window
    ↓
LLM generates a response
    ↓
Agent extracts new facts and stores them back into long-term memory

This is the architecture we’re going to build. Let’s start with the simpler half: short-term memory.

Short-Term Memory: The Context Window as Scratchpad

Short-term memory is the easy part. It’s just the conversation history you append to every prompt. The naive implementation looks like this:

# Block 2: Short-term memory — the naive implementation
# This block is self-contained

class ShortTermMemory:
    """A simple short-term memory that stores conversation history."""
    
    def __init__(self):
        self.messages = []
    
    def add_message(self, role, content):
        """Add a message to the history."""
        self.messages.append({"role": role, "content": content})
    
    def get_all(self):
        """Return all messages."""
        return self.messages
    
    def clear(self):
        """Clear all messages."""
        self.messages = []

# Let's see it in action
memory = ShortTermMemory()
memory.add_message("user", "What's the weather in Tokyo?")
memory.add_message("assistant", "Let me check...")
memory.add_message("user", "And what about Osaka?")

print("Current conversation history:")
for msg in memory.get_all():
    print(f"  {msg['role']}: {msg['content']}")

When you run this, you’ll see:

Current conversation history:
  user: What's the weather in Tokyo?
  assistant: Let me check...
  user: And what about Osaka?

Simple, right? But here’s the catch: context windows fill up. A long conversation can easily eat through 200K tokens. And cost scales with token count — every token in the prompt costs money.

The solution is a sliding window: keep only the last N turns of conversation. Here’s how that looks:

# Block 3: Short-term memory with a sliding window
# This block is self-contained

from collections import deque

class ShortTermMemory:
    """Short-term memory with a sliding window."""
    
    def __init__(self, max_turns=10):
        self.messages = deque(maxlen=max_turns * 2)  # *2 because each turn has user + assistant
        self.max_turns = max_turns
    
    def add_message(self, role, content):
        """Add a message to the history. Automatically drops old messages."""
        self.messages.append({"role": role, "content": content})
    
    def get_recent(self, n=None):
        """Return the last n messages (or all if n is None)."""
        if n is None:
            return list(self.messages)
        return list(self.messages)[-n:]
    
    def estimate_token_count(self):
        """Rough estimate: ~4 characters per token for English text."""
        total_chars = sum(len(msg["content"]) for msg in self.messages)
        return total_chars // 4
    
    def clear(self):
        """Clear all messages."""
        self.messages.clear()

# Let's see it in action
memory = ShortTermMemory(max_turns=3)  # Keep only last 3 turns

# Simulate a longer conversation
for i in range(5):
    memory.add_message("user", f"Question {i+1}: What's the weather?")
    memory.add_message("assistant", f"Answer {i+1}: Let me check...")

print("Current conversation history (should only show last 3 turns):")
for msg in memory.get_recent():
    print(f"  {msg['role']}: {msg['content']}")
print()
print(f"Estimated token count: {memory.estimate_token_count()}")

When you run this, you’ll see:

Current conversation history (should only show last 3 turns):
  user: Question 3: What's the weather?
  assistant: Answer 3: Let me check...
  user: Question 4: What's the weather?
  assistant: Answer 4: Let me check...
  user: Question 5: What's the weather?
  assistant: Answer 5: Let me check...

Estimated token count: 15

The sliding window keeps the conversation manageable. But there’s a trade-off: you lose early context. If the user says “go back to the first idea,” the agent can’t see it anymore. That’s where long-term memory comes in.

Claude’s context windows are 200K tokens for Haiku and 1M for Opus (https://platform.claude.com/docs/en/about-claude/models/overview). 200K tokens is roughly a 500-page book. That sounds huge, but a day of detailed conversation with code examples, long tool outputs, and back-and-forth can easily eat through it. The sliding window is a practical necessity.

Long-Term Memory: Storing Facts Outside the Window

Now we get to the interesting part. Long-term memory means persisting information across sessions. The simplest form is a key-value store — like a dictionary saved to a JSON file. But that has a problem: you have to know the key to retrieve the value.

Let’s start with the simplest possible long-term memory:

# Block 4: Simplest possible long-term memory — a JSON file
# This block is self-contained

import json
import os
from datetime import datetime

class SimpleLongTermMemory:
    """A key-value store for facts, saved to a JSON file."""
    
    def __init__(self, filepath="memory.json"):
        self.filepath = filepath
        self.data = {}
        self._load()
    
    def _load(self):
        """Load memory from disk."""
        if os.path.exists(self.filepath):
            with open(self.filepath, "r") as f:
                self.data = json.load(f)
    
    def _save(self):
        """Save memory to disk."""
        with open(self.filepath, "w") as f:
            json.dump(self.data, f, indent=2)
    
    def store(self, key, fact):
        """Store a fact with a key."""
        self.data[key] = {
            "fact": fact,
            "timestamp": datetime.now().isoformat()
        }
        self._save()
    
    def retrieve(self, key):
        """Retrieve a fact by key."""
        return self.data.get(key)
    
    def search_by_keyword(self, keyword):
        """Search for facts containing a keyword."""
        results = []
        for key, value in self.data.items():
            if keyword.lower() in value["fact"].lower():
                results.append((key, value))
        return results

# Let's see it in action
memory = SimpleLongTermMemory("test_memory.json")

# Store some facts
memory.store("project_alpha", "Alice is working on Project Alpha, a machine learning pipeline.")
memory.store("user_preference", "Alice prefers metric units and Python 3.11.")

# Retrieve by key
result = memory.retrieve("project_alpha")
print(f"Retrieved by key 'project_alpha': {result}")
print()

# Search by keyword
results = memory.search_by_keyword("Alice")
print(f"Search for 'Alice': {results}")
print()

# Clean up test file
os.remove("test_memory.json")

When you run this, you’ll see:

Retrieved by key 'project_alpha': {'fact': 'Alice is working on Project Alpha, a machine learning pipeline.', 'timestamp': '2025-04-01T12:00:00.000000'}

Search for 'Alice': [('project_alpha', {'fact': 'Alice is working on Project Alpha, a machine learning pipeline.', 'timestamp': '2025-04-01T12:00:00.000000'}), ('user_preference', {'fact': 'Alice prefers metric units and Python 3.11.', 'timestamp': '2025-04-01T12:00:00.000000'})]

This works, but it has a big limitation: keyword search only finds exact matches. If the user asks “What was I working on?” and you stored it under “project_alpha,” the keyword search for “working” might miss it. You need semantic search — searching by meaning, not by exact words.

Here’s where embeddings come in. An embedding is a list of numbers that captures the meaning of a piece of text. Similar texts have similar numbers. You can think of it as a coordinate in a “meaning space” — the sentence “Alice is working on Project Alpha” and “What project is Alice working on?” should be close together in that space, even though they share almost no words.

Let’s build a long-term memory that uses embeddings for semantic search:

# Block 5: Long-term memory with semantic search using embeddings
# This block is self-contained

import numpy as np
from sentence_transformers import SentenceTransformer
from datetime import datetime
import json
import os

class SemanticLongTermMemory:
    """Long-term memory with semantic search using embeddings."""
    
    def __init__(self, model_name="all-MiniLM-L6-v2", filepath="semantic_memory.json"):
        print(f"Loading embedding model: {model_name}...")
        self.model = SentenceTransformer(model_name)
        self.filepath = filepath
        self.chunks = []  # List of dicts: {text, embedding, metadata}
        self._load()
    
    def _load(self):
        """Load memory from disk."""
        if os.path.exists(self.filepath):
            with open(self.filepath, "r") as f:
                data = json.load(f)
            # Recompute embeddings on load (they're not stored in JSON)
            for item in data:
                embedding = self.model.encode(item["text"])
                self.chunks.append({
                    "text": item["text"],
                    "embedding": embedding,
                    "metadata": item["metadata"]
                })
    
    def _save(self):
        """Save memory to disk (without embeddings)."""
        data = []
        for chunk in self.chunks:
            data.append({
                "text": chunk["text"],
                "metadata": chunk["metadata"]
            })
        with open(self.filepath, "w") as f:
            json.dump(data, f, indent=2)
    
    def store(self, text, metadata=None):
        """Store a text chunk with its embedding."""
        embedding = self.model.encode(text)
        if metadata is None:
            metadata = {"timestamp": datetime.now().isoformat()}
        self.chunks.append({
            "text": text,
            "embedding": embedding,
            "metadata": metadata
        })
        self._save()
    
    def retrieve(self, query, top_k=3):
        """Retrieve the top-k most similar chunks to a query."""
        if not self.chunks:
            return []
        
        # Encode the query
        query_embedding = self.model.encode(query)
        
        # Compute cosine similarity with all stored chunks
        similarities = []
        for chunk in self.chunks:
            # Cosine similarity: dot product of normalized vectors
            sim = np.dot(query_embedding, chunk["embedding"]) / (
                np.linalg.norm(query_embedding) * np.linalg.norm(chunk["embedding"])
            )
            similarities.append(sim)
        
        # Get top-k indices
        top_indices = np.argsort(similarities)[-top_k:][::-1]
        
        results = []
        for idx in top_indices:
            results.append({
                "text": self.chunks[idx]["text"],
                "similarity": float(similarities[idx]),
                "metadata": self.chunks[idx]["metadata"]
            })
        return results
    
    def clear(self):
        """Clear all memory."""
        self.chunks = []
        if os.path.exists(self.filepath):
            os.remove(self.filepath)

# Let's see it in action
memory = SemanticLongTermMemory("test_semantic_memory.json")

# Store some facts
memory.store("Alice is working on Project Alpha, a machine learning pipeline.")
memory.store("Alice prefers metric units and Python 3.11.")
memory.store("The project deadline is next Friday.")

# Query with a natural language question
query = "What project is Alice working on?"
results = memory.retrieve(query, top_k=2)

print(f"Query: '{query}'")
print("Top results:")
for r in results:
    print(f"  - {r['text']} (similarity: {r['similarity']:.3f})")
print()

# Try a different query
query2 = "When is the deadline?"
results2 = memory.retrieve(query2, top_k=2)
print(f"Query: '{query2}'")
print("Top results:")
for r in results2:
    print(f"  - {r['text']} (similarity: {r['similarity']:.3f})")

# Clean up
memory.clear()
os.remove("test_semantic_memory.json")

When you run this, you’ll see something like:

Loading embedding model: all-MiniLM-L6-v2...
Query: 'What project is Alice working on?'
Top results:
  - Alice is working on Project Alpha, a machine learning pipeline. (similarity: 0.612)
  - Alice prefers metric units and Python 3.11. (similarity: 0.213)

Query: 'When is the deadline?'
Top results:
  - The project deadline is next Friday. (similarity: 0.701)
  - Alice is working on Project Alpha, a machine learning pipeline. (similarity: 0.198)

Notice how the semantic search correctly matched the query to the relevant fact, even though the query used different words. “What project is Alice working on?” found the fact about Project Alpha, not the fact about units and Python version. That’s the power of embeddings.

The Pinecone docs (https://docs.pinecone.io/guides/get-started/overview) describe this as “semantic search, knowledge retrieval, and long-term memory at scale.” And the Generative Agents paper (https://arxiv.org/abs/2304.03442) introduced the concept of a memory stream — timestamped observations stored as natural language, retrieved by a weighted score that combines recency, importance, and relevance.

The Hard Part: Retrieval Is Not Solved

Now here’s the catch: naive vector search works great for factual queries like “What is the capital of France?” but fails on time-based queries like “What did we discuss in the third conversation on Tuesday?” and ambiguous queries like “Tell me about that project we talked about.”

This is the hardest part of building long-term memory. Let’s look at the research.

The “Toward Conversational Agents with Context and Time Sensitive Long-term Memory” paper (https://arxiv.org/abs/2406.00057) found that naive vector-database RAG performs poorly on time/event-based queries. The reason is simple: vector search doesn’t understand time. It doesn’t know that “the third conversation on Tuesday” refers to a specific temporal context.

The Graph-Native Bitemporal Memory Store paper (https://arxiv.org/abs/2607.26520) provides concrete numbers. On the LongMemEval benchmark:

• Semantic search alone got 46.7% recall@10
• Knowledge-update questions (“Did we change the deadline?”) got 80% recall@10
• Temporal-reasoning questions (“What did we discuss before the deadline change?”) got only 37.5% recall@10

What this means in plain English: vector search is okay at finding facts by meaning, but it’s terrible at understanding when things happened or how events relate to each other in time.

The solution is hybrid retrieval: combine semantic similarity with recency scoring and structured metadata. The Generative Agents paper proposed a weighted score:

score = w1 * recency + w2 * importance + w3 * relevance

Where:

• recency = how recently the memory was created (more recent = higher score)
• importance = how important the memory is (explicitly assigned by the LLM)
• relevance = how semantically similar the memory is to the current query

Let’s implement this hybrid approach:

# Block 6: Hybrid retrieval — combining semantic similarity with recency
# This block is self-contained

import numpy as np
from sentence_transformers import SentenceTransformer
from datetime import datetime, timedelta
import json
import os

class HybridLongTermMemory:
    """Long-term memory with hybrid retrieval: semantic + recency + importance."""
    
    def __init__(self, model_name="all-MiniLM-L6-v2", filepath="hybrid_memory.json"):
        print(f"Loading embedding model: {model_name}...")
        self.model = SentenceTransformer(model_name)
        self.filepath = filepath
        self.chunks = []  # List of dicts: {text, embedding, metadata}
        self._load()
    
    def _load(self):
        """Load memory from disk."""
        if os.path.exists(self.filepath):
            with open(self.filepath, "r") as f:
                data = json.load(f)
            for item in data:
                embedding = self.model.encode(item["text"])
                self.chunks.append({
                    "text": item["text"],
                    "embedding": embedding,
                    "metadata": item["metadata"]
                })
    
    def _save(self):
        """Save memory to disk."""
        data = []
        for chunk in self.chunks:
            data.append({
                "text": chunk["text"],
                "metadata": chunk["metadata"]
            })
        with open(self.filepath, "w") as f:
            json.dump(data, f, indent=2)
    
    def store(self, text, metadata=None):
        """Store a text chunk with its embedding and metadata."""
        embedding = self.model.encode(text)
        if metadata is None:
            metadata = {"timestamp": datetime.now().isoformat()}
        self.chunks.append({
            "text": text,
            "embedding": embedding,
            "metadata": metadata
        })
        self._save()
    
    def _compute_recency_score(self, timestamp_str, decay_hours=24):
        """Compute a recency score that decays over time."""
        try:
            timestamp = datetime.fromisoformat(timestamp_str)
        except:
            return 0.0
        
        now = datetime.now()
        hours_ago = (now - timestamp).total_seconds() / 3600
        
        # Exponential decay: score = exp(-hours_ago / decay_hours)
        return np.exp(-hours_ago / decay_hours)
    
    def retrieve(self, query, top_k=3, recency_weight=0.3, semantic_weight=0.7):
        """Retrieve top-k chunks using hybrid scoring."""
        if not self.chunks:
            return []
        
        # Encode the query
        query_embedding = self.model.encode(query)
        
        # Compute scores for each chunk
        scores = []
        for chunk in self.chunks:
            # Semantic similarity
            semantic_sim = np.dot(query_embedding, chunk["embedding"]) / (
                np.linalg.norm(query_embedding) * np.linalg.norm(chunk["embedding"])
            )
            
            # Recency score
            timestamp = chunk["metadata"].get("timestamp", datetime.now().isoformat())
            recency = self._compute_recency_score(timestamp)
            
            # Hybrid score
            hybrid_score = (semantic_weight * semantic_sim) + (recency_weight * recency)
            scores.append(hybrid_score)
        
        # Get top-k indices
        top_indices = np.argsort(scores)[-top_k:][::-1]
        
        results = []
        for idx in top_indices:
            results.append({
                "text": self.chunks[idx]["text"],
                "hybrid_score": float(scores[idx]),
                "semantic_similarity": float(np.dot(query_embedding, self.chunks[idx]["embedding"]) / (
                    np.linalg.norm(query_embedding) * np.linalg.norm(self.chunks[idx]["embedding"])
                )),
                "metadata": self.chunks[idx]["metadata"]
            })
        return results
    
    def clear(self):
        """Clear all memory."""
        self.chunks = []
        if os.path.exists(self.filepath):
            os.remove(self.filepath)

# Let's see it in action
memory = HybridLongTermMemory("test_hybrid_memory.json")

# Store some facts with different timestamps
# Simulate older facts by setting metadata timestamps manually
memory.store("Alice is working on Project Alpha, a machine learning pipeline.", 
             {"timestamp": (datetime.now() - timedelta(days=7)).isoformat()})
memory.store("The project deadline was originally next Friday.",
             {"timestamp": (datetime.now() - timedelta(days=3)).isoformat()})
memory.store("The deadline has been moved to next Monday.",
             {"timestamp": datetime.now().isoformat()})

# Query about the deadline
query = "When is the project deadline?"
results = memory.retrieve(query, top_k=3, recency_weight=0.4, semantic_weight=0.6)

print(f"Query: '{query}'")
print("Top results (hybrid retrieval):")
for r in results:
    print(f"  - {r['text']}")
    print(f"    Hybrid score: {r['hybrid_score']:.3f}, Semantic: {r['semantic_similarity']:.3f}")
    print(f"    Timestamp: {r['metadata']['timestamp']}")

# Clean up
memory.clear()
os.remove("test_hybrid_memory.json")

When you run this, you’ll see something like:

Loading embedding model: all-MiniLM-L6-v2...
Query: 'When is the project deadline?'
Top results (hybrid retrieval):
  - The deadline has been moved to next Monday.
    Hybrid score: 0.723, Semantic: 0.612
    Timestamp: 2025-04-01T12:00:00.000000
  - The project deadline was originally next Friday.
    Hybrid score: 0.534, Semantic: 0.598
    Timestamp: 2025-03-28T12:00:00.000000
  - Alice is working on Project Alpha, a machine learning pipeline.
    Hybrid score: 0.312, Semantic: 0.198
    Timestamp: 2025-03-25T12:00:00.000000

Notice how the hybrid retrieval correctly ranked the most recent deadline update higher than the original deadline, even though both had similar semantic similarity scores. The recency bonus pushed the newer fact to the top.

The MemoriesDB paper (https://arxiv.org/abs/2511.06179) proposes an even more sophisticated design that combines time-series data, vector embeddings, and a graph of relationships between facts. This is an active research area — what we build today will be outdated in a year, but the concepts will transfer.

Wiring It Together: Short-Term + Long-Term in One Agent

Now for the payoff. Let’s combine the short-term context window with the long-term retrieval store into a single agent. The pattern, from the Mem0 blog (https://mem0.ai/blog/short-term-vs-long-term-memory-in-ai), is:

1. Append the user’s message to the session buffer (short-term memory)
2. Query the long-term index for relevant facts
3. Generate a response with both short-term history and long-term context
4. Asynchronously extract new facts and store them back into long-term memory

Let’s build it:

# Block 7: The complete MemoryAgent — short-term + long-term memory
# This block is self-contained

from collections import deque
import numpy as np
from sentence_transformers import SentenceTransformer
from datetime import datetime
import json
import os

# --- Short-Term Memory (sliding window) ---
class ShortTermMemory:
    """Short-term memory with a sliding window."""
    
    def __init__(self, max_turns=10):
        self.messages = deque(maxlen=max_turns * 2)
        self.max_turns = max_turns
    
    def add_message(self, role, content):
        self.messages.append({"role": role, "content": content})
    
    def get_recent(self, n=None):
        if n is None:
            return list(self.messages)
        return list(self.messages)[-n:]
    
    def clear(self):
        self.messages.clear()

# --- Long-Term Memory (hybrid retrieval) ---
class LongTermMemory:
    """Long-term memory with hybrid retrieval."""
    
    def __init__(self, model_name="all-MiniLM-L6-v2", filepath="agent_memory.json"):
        print(f"Loading embedding model: {model_name}...")
        self.model = SentenceTransformer(model_name)
        self.filepath = filepath
        self.chunks = []
        self._load()
    
    def _load(self):
        if os.path.exists(self.filepath):
            with open(self.filepath, "r") as f:
                data = json.load(f)
            for item in data:
                embedding = self.model.encode(item["text"])
                self.chunks.append({
                    "text": item["text"],
                    "embedding": embedding,
                    "metadata": item["metadata"]
                })
    
    def _save(self):
        data = []
        for chunk in self.chunks:
            data.append({
                "text": chunk["text"],
                "metadata": chunk["metadata"]
            })
        with open(self.filepath, "w") as f:
            json.dump(data, f, indent=2)
    
    def store(self, text, metadata=None):
        embedding = self.model.encode(text)
        if metadata is None:
            metadata = {"timestamp": datetime.now().isoformat()}
        self.chunks.append({
            "text": text,
            "embedding": embedding,
            "metadata": metadata
        })
        self._save()
    
    def _compute_recency_score(self, timestamp_str, decay_hours=24):
        try:
            timestamp = datetime.fromisoformat(timestamp_str)
        except:
            return 0.0
        now = datetime.now()
        hours_ago = (now - timestamp).total_seconds() / 3600
        return np.exp(-hours_ago / decay_hours)
    
    def retrieve(self, query, top_k=3, recency_weight=0.3, semantic_weight=0.7):
        if not self.chunks:
            return []
        query_embedding = self.model.encode(query)
        scores = []
        for chunk in self.chunks:
            semantic_sim = np.dot(query_embedding, chunk["embedding"]) / (
                np.linalg.norm(query_embedding) * np.linalg.norm(chunk["embedding"])
            )
            timestamp = chunk["metadata"].get("timestamp", datetime.now().isoformat())
            recency = self._compute_recency_score(timestamp)
            hybrid_score = (semantic_weight * semantic_sim) + (recency_weight * recency)
            scores.append(hybrid_score)
        top_indices = np.argsort(scores)[-top_k:][::-1]
        results = []
        for idx in top_indices:
            results.append({
                "text": self.chunks[idx]["text"],
                "hybrid_score": float(scores[idx]),
                "metadata": self.chunks[idx]["metadata"]
            })
        return results
    
    def clear(self):
        self.chunks = []
        if os.path.exists(self.filepath):
            os.remove(self.filepath)

# --- The MemoryAgent ---
class MemoryAgent:
    """An agent with both short-term and long-term memory."""
    
    def __init__(self, system_prompt=None, max_turns=10):
        self.short_term = ShortTermMemory(max_turns=max_turns)
        self.long_term = LongTermMemory()
        self.system_prompt = system_prompt or "You are a helpful assistant with memory."
    
    def respond(self, user_message):
        """Generate a response using both short-term and long-term memory."""
        # 1. Add user message to short-term memory
        self.short_term.add_message("user", user_message)
        
        # 2. Query long-term memory for relevant facts
        relevant_facts = self.long_term.retrieve(user_message, top_k=3)
        
        # 3. Build the context
        context = f"{self.system_prompt}\n\n"
        
        if relevant_facts:
            context += "Relevant past information:\n"
            for fact in relevant_facts:
                context += f"- {fact['text']}\n"
            context += "\n"
        
        context += "Recent conversation:\n"
        for msg in self.short_term.get_recent():
            context += f"{msg['role']}: {msg['content']}\n"
        
        # 4. Simulate an LLM response (in production, you'd call the API)
        # For this demo, we'll just acknowledge what we found
        response = self._generate_response(context, user_message)
        
        # 5. Add assistant response to short-term memory
        self.short_term.add_message("assistant", response)
        
        # 6. Extract and store new facts (simplified)
        self._extract_and_store(user_message, response)
        
        return response
    
    def _generate_response(self, context, user_message):
        """Simulate an LLM response. In production, call the API."""
        # This is a placeholder — in a real implementation, you'd call
        # client.chat.completions.create(model="gpt-4o-mini", messages=[...])
        return f"I remember discussing this. Based on my memory, here's what I know..."
    
    def _extract_and_store(self, user_message, response):
        """Extract key facts and store them in long-term memory."""
        # In production, you'd use the LLM to extract facts
        # For this demo, we'll store the user message as a fact
        self.long_term.store(
            f"User said: {user_message}",
            {"timestamp": datetime.now().isoformat(), "type": "user_input"}
        )

# --- Let's see it in action ---
print("=== MemoryAgent Demo ===\n")

agent = MemoryAgent()

# Session 1: User introduces themselves
print("Session 1:")
response1 = agent.respond("Hi, I'm Alice and I'm working on Project Alpha.")
print(f"  User: Hi, I'm Alice and I'm working on Project Alpha.")
print(f"  Agent: {response1}")
print()

# Session 2: Simulate a new session (same agent instance, so memory persists)
print("Session 2 (simulating next day):")
response2 = agent.respond("What was I working on?")
print(f"  User: What was I working on?")
print(f"  Agent: {response2}")
print()

# Check what's in long-term memory
print("Long-term memory contents:")
for chunk in agent.long_term.chunks:
    print(f"  - {chunk['text']}")

# Clean up
agent.long_term.clear()

When you run this, you’ll see:

=== MemoryAgent Demo ===

Loading embedding model: all-MiniLM-L6-v2...
Session 1:
  User: Hi, I'm Alice and I'm working on Project Alpha.
  Agent: I remember discussing this. Based on my memory, here's what I know...

Session 2 (simulating next day):
  User: What was I working on?
  Agent: I remember discussing this. Based on my memory, here's what I know...

Long-term memory contents:
  - User said: Hi, I'm Alice and I'm working on Project Alpha.
  - User said: What was I working on?

The agent remembered. Not because the LLM has memory, but because we built a system that retrieves and injects the right context. In a real implementation, you’d call the OpenAI or Anthropic API in _generate_response() and use the LLM to extract facts in _extract_and_store(). But the architecture is the same.

Production Concerns: Latency, Noise, and Scale

The prototype works, but it won’t scale. Real-world memory systems face three hard problems:

1. Latency. Every retrieval call takes time. The Mandol paper (https://arxiv.org/abs/2606.29778) reports 5.4x retrieval speedups and 4.8x insertion speedups over baseline — which tells you that performance is a real concern. In production, you’d use a vector database like Pinecone or pgvector instead of computing embeddings on the fly.

2. Retrieval noise. If you retrieve 5 chunks and 3 are irrelevant, you’ve wasted context window space and confused the LLM. The Mandol paper’s approach is a non-LLM retrieval/denoising step that filters out irrelevant chunks before they reach the context window.

3. Storage growth. If your agent has a conversation every day for a year, that’s 365 conversations worth of memory. You can’t retrieve everything. Strategies include:

• Summarization: compress many observations into a single summary
• Forgetting: prune low-importance memories
• Hierarchical memory: raw observations + abstracted summaries

The Generative Agents paper’s reflection mechanism is a great example: when the sum of importance scores crosses a threshold, the agent generates a higher-level summary, compressing many observations into one.

Here’s a conceptual sketch of an importance-based pruning function:

# Pseudocode for importance-based pruning (concept only)
def prune_memory(memory, threshold=0.3):
    """Remove chunks whose importance score falls below a threshold."""
    pruned = []
    for chunk in memory.chunks:
        # Compute importance as a combination of recency and relevance
        importance = compute_importance(chunk)
        if importance >= threshold:
            pruned.append(chunk)
    memory.chunks = pruned
    memory._save()

These are open research problems. The best systems today are still far from perfect. But the concepts you’ve learned — short-term vs. long-term memory, semantic search, hybrid retrieval, the write-through pattern — will transfer to whatever comes next.

Recap: What You Learned

Let’s summarize what you built and learned in this article:

1. Short-term memory = the context window. It’s fast, ephemeral, and limited. You manage it with a sliding window that keeps only the last N turns.

2. Long-term memory = external storage. It’s slower, persistent, and unbounded. You retrieve from it with semantic search using embeddings.

3. Naive vector search isn’t enough. You need hybrid retrieval that combines semantic similarity with recency and metadata.

4. The write-through pattern: append to session buffer → query long-term index → generate with combined context → async-extract facts back to long-term storage.

5. Production concerns: latency, noise, and storage growth are unsolved problems. Techniques like summarization, forgetting, and hierarchical memory help.

In the next part of this series, we’ll explore temporal reasoning and time-aware retrieval — building on the hybrid retrieval concept you just learned.

Check Your Understanding

Remember: What is the difference between short-term and long-term memory in an LLM agent?

Understand: Explain in plain English why a sliding window for short-term memory can cause the agent to lose early context.

Apply: Given a conversation log, write a function that extracts key facts and stores them in a long-term memory store with embeddings.

Analyze: Compare the recall numbers from the Graph-Native Bitemporal Memory Store paper (46.7% semantic, 80% knowledge-update, 37.5% temporal-reasoning). Why does temporal reasoning perform so much worse?

Evaluate: You’re designing a memory system for a customer-support agent. Would you prioritize recency or semantic similarity in retrieval? Justify your answer.

Create: Sketch a design for a memory system that combines short-term context, long-term vector retrieval, and a graph of relationships between facts. What are the trade-offs of your design?

Related Articles

• Part 1: ‘What Is an LLM Agent?’ — Introduces the core concept of an agent that uses tools and makes decisions. Memory is the next step after tool-use.
• Part 2: ‘Building Your First Agent’ — You built a stateless tool-calling agent. This article adds memory to that agent.
• Part 4: ‘Giving an Agent a Sense of Time’ — Explores temporal reasoning and time-aware retrieval, building on the hybrid retrieval concept from this article.