Context Engineering: Designing Tiered Memory Architectures for Long-Horizon AI Agents

When foundational models first introduced one-million and two-million token context windows, a dangerous myth took hold in enterprise engineering: RAG and memory systems are dead; just dump everything into the prompt.

In production, this brute-force approach fails rapidly. Stuffing hundreds of thousands of raw operational tokens into every turn of an autonomous loop triggers three severe problems:

1. Context Rot (“Lost in the Middle”): As distraction tokens accumulate (such as verbose tool outputs, stack traces, and failed retry loops), the model’s attention mechanism degrades, leading to subtle reasoning failures.
2. Latency Explosion: Time-to-First-Token (TTFT) scales with prompt length, turning a snappy agent into a sluggish bottleneck.
3. Runaway Token Economics: Paying to re-process the entire history of a 40-step workflow on every single tool call destroys unit economics.

Enter Context Engineering: the discipline of dynamically curating, compressing, and structuring the exact state an AI agent sees at each step of execution. Just as modern computers rely on a memory hierarchy (CPU registers, L1/L2 cache, RAM, and SSD), long-horizon enterprise agents require a Tiered Cognitive Memory Architecture.

The Three Tiers of Agentic Memory

To build an agent that can execute complex workflows over days or weeks without losing the plot, we divide its state into three distinct tiers:

1. Tier 1: Working Memory (The Scratchpad)

Working memory is what resides directly inside the active LLM context window. Instead of an append-only chat log, enterprise working memory should be treated as a mutable state buffer:

• System Core & Active Goal: Fixed instructions and the current task rubric (pinned to take advantage of provider prompt caching).
• Rolling Window of Recent Turns: Only the last $N$ tool interactions in full fidelity.
• Structured Scratchpad: A compact JSON state object where the agent records verified facts, active hypotheses, and pending subtasks.

2. Tier 2: Episodic Memory (Experiential Recall)

Episodic memory answers the question: “What happened the last time I tried a similar task?” Whenever an agent completes a workflow (or recovers from an error), a background synthesis step distils the trajectory into a compact Episode Record that captures the initial trigger, the tools invoked, the mistakes made, and the final resolution. These episodes are embedded and indexed with temporal decay weights so recent operational lessons rank higher than stale ones.

3. Tier 3: Semantic Memory (Consolidated Knowledge)

Semantic memory stores timeless domain truths, user preferences, and entity relationships extracted across hundreds of episodes. Rather than keeping raw transcripts, a consolidation worker periodically extracts durable triples (e.g., Service_Payments -> depends_on -> DB_Ledger_Prod) into a structured store or property graph.

Out-of-the-Box Implementation: A Tiered Memory Manager in Python

Below is a production-ready pattern showing how to assemble a ContextAssembler that keeps the active prompt lean by pruning stale tool outputs, maintaining a structured scratchpad, and injecting relevant past episodes on demand.

from dataclasses import dataclass, field
from datetime import datetime, timezone
from typing import List, Dict, Any
import json
import math


@dataclass
class Episode:
    episode_id: str
    task_summary: str
    resolution: str
    pitfalls_avoided: List[str]
    created_at: datetime
    similarity_score: float = 0.0

    def recency_weighted_score(self, half_life_days: float = 14.0) -> float:
        """Applies exponential temporal decay so fresh operational episodes win."""
        age_days = (datetime.now(timezone.utc) - self.created_at).total_seconds() / 86400.0
        decay = math.exp(-math.log(2) * (age_days / half_life_days))
        return (0.7 * self.similarity_score) + (0.3 * decay)


@dataclass
class WorkingScratchpad:
    objective: str
    verified_facts: Dict[str, Any] = field(default_factory=dict)
    completed_steps: List[str] = field(default_factory=list)
    open_blockers: List[str] = field(default_factory=list)

    def to_prompt_block(self) -> str:
        return json.dumps(
            {
                "objective": self.objective,
                "verified_facts": self.verified_facts,
                "completed_steps": self.completed_steps[-5:],
                "open_blockers": self.open_blockers,
            },
            indent=2,
        )


class ContextEngineeringHarness:
    def __init__(self, max_recent_turns: int = 6):
        self.max_recent_turns = max_recent_turns
        self.turn_history: List[Dict[str, str]] = []

    def record_turn(self, role: str, content: str, is_verbose_tool_output: bool = False):
        """Compresses bulky raw tool payloads once they fall out of the immediate step."""
        if is_verbose_tool_output and len(content) > 1200:
            preview = content[:600] + "\n...[TRUNCATED BY CONTEXT HARNESS]...\n" + content[-300:]
            content = preview
        self.turn_history.append({"role": role, "content": content})

    def build_context_window(
        self,
        scratchpad: WorkingScratchpad,
        candidate_episodes: List[Episode],
        semantic_facts: List[str],
    ) -> List[Dict[str, str]]:
        # 1. Rank and select top 2 episodic memories using recency + similarity
        ranked_episodes = sorted(
            candidate_episodes,
            key=lambda ep: ep.recency_weighted_score(),
            reverse=True,
        )[:2]

        episodic_block = "\n".join(
            f"- [{ep.episode_id}] Task: {ep.task_summary} | Resolution: {ep.resolution} "
            f"| Watch out for: {', '.join(ep.pitfalls_avoided)}"
            for ep in ranked_episodes
        )

        # 2. Construct the pinned state header
        state_header = f"""### ACTIVE SCRATCHPAD (TIER 1)
{scratchpad.to_prompt_block()}

### RELEVANT PAST EPISODES (TIER 2)
{episodic_block or "No prior episodes matched."}

### DOMAIN INVARIANTS (TIER 3)
{chr(10).join(f"- {fact}" for fact in semantic_facts)}
"""
        # 3. Combine pinned state with a sliding window of recent turns
        recent_window = self.turn_history[-self.max_recent_turns :]
        return [{"role": "user", "content": state_header}] + recent_window

Key Engineering Takeaways

1. Never Let Raw Tool Payloads Accumulate: If an agent queries a log store and receives a 15,000-token stack trace, extract the relevant error signature into the WorkingScratchpad and evict the raw payload before the next reasoning hop.
2. Separate Read and Write Paths for Memory: Do not ask the primary reasoning agent to manage its own vector database writes inline. Offload episode reflection and semantic extraction to an asynchronous background worker after the task completes.
3. Design for Prompt Caching: Keep static system instructions and Tier 3 domain invariants at the very top of the prompt prefix so model providers can serve 80%+ of your input tokens from cache at a fraction of the cost.

Why Alps Agility?

At Alps Agility, we combine deep AI expertise with advanced engineering to help you implement autonomous agents that cut costs and improve operational efficiency.

Contact us today to start transforming your enterprise with Agentic AI.

Struggling to move AI from prototype to production? We help enterprises build robust, scalable AI architectures. Book a Generative AI Readiness Assessment.