Key Takeaway
Keep a small, editable text memory and update it with feedback at each step so a frozen language model can correct past mistakes and adapt over long runs without changing its weights.
ON THIS PAGE
Key Findings
Keeping a short, human-readable memory that gets revised after each prediction beats simply appending history or sending the entire past to the model. The update loop (predict → judge → rewrite memory) improves accuracy on long-running tasks, especially where past beliefs become wrong over time, like clinical records and market forecasting. Smaller models benefit disproportionately from this approach, sometimes outperforming much larger models that use full history. Weather forecasting showed smaller gains, suggesting the method helps most when semantic errors or contradicting evidence need to be corrected. Chain of Thought Pattern
Not sure where to start?Get personalized recommendations
Data Highlights
112.6 percentage-point absolute gain in top-1 clinical diagnosis accuracy on MIMIC-IV versus the best step-wise summary baseline (0.6434 vs. 0.5175).
2About a 6.6% reduction in mean squared error on S&P 500 forecasting (3.821 vs. 4.090).
3A 3-billion-parameter model using the memory loop reached Acc@1 = 0.4545, outperforming a 70-billion-parameter model using full-history (Acc@1 = 0.4126).
What This Means
Engineers building long-running AI assistants or monitoring systems will get more robust, auditable behavior without retraining models. Product and ML leads in healthcare and finance can use this to let deployed models adapt to new evidence while keeping a transparent record of what changed. Research teams studying online adaptation or continual evaluation can use the method as a lightweight alternative to retraining. LLM-as-Judge
Key Figures
Fig 1: Figure 1: Illustrative comparison. (a) Simple LLM lacks memory. (b) LLM with Context suffers from growing input size. (c) LLM-as-RNN uses an iterative memory state to summarize historical information from evaluating outputs.
Fig 2: Figure 2: Overview of LLM-as-RNN framework. At each time step, the system fuses the previous memory state with new input to generate a response, evaluates that response to create a feedback signal, and then updates the natural language memory state to guide future interactions.
Fig 3: Figure 3: Temporal dynamics across iterative timesteps ( t = 1 … 5 t{=}1\ldots 5 ) for three datasets. As feedback-driven memory updates accumulate, the performance increase.
Fig 4: Figure 4: Scaling across backbone families. Performance vs. model size (B params, log-scale) for Zero-shot, FHC, MemPrompt, and LLM-as-RNN. Rows: Llama/Gemma/GPT backbones; columns: MIMIC Acc@1, Weather Align, S&P 500 MSE. LLM-as-RNN yields consistent gains across sizes and families.
Ready to evaluate your AI agents?
Learn how ReputAgent helps teams build trustworthy AI through systematic evaluation.
Learn MoreYes, But...
The approach increases per-step compute because each timestep typically requires multiple model calls (predict, critique, update). It depends on reliable feedback—delayed labels or noisy self-critique can cause memory drift or reinforce errors. The human-readable memory helps audits but can also create false confidence; privacy and adversarial input must be managed with redaction, access controls, and rollback options. Guardrails Pattern
Full Analysis
The method frames inference as a recurrent loop where the model keeps a compact, editable text memory (bounded by a fixed token budget) instead of appending everything to the context. At each timestep the system: (1) predicts the next output using the current memory and new input, (2) evaluates that output using ground truth or a model-based critic, and (3) rewrites the memory to reflect corrected beliefs or new strategies. The memory is always human-readable, so the model’s evolving beliefs are transparent and auditable. Memory Poisoning Across three benchmarks—clinical trajectories (MIMIC-IV), continuous weather streams, and S&P 500 forecasting—the memory-rewrite loop consistently outperformed zero-shot, full-history concatenation, and step-wise summary baselines. Gains were largest on tasks where earlier outputs can mislead future decisions (clinical and financial data). The approach is model-agnostic and especially helps smaller models approximate longer-horizon tracking that otherwise requires far larger models. Trade-offs include higher inference cost and reliance on feedback quality, but the readable state and improved long-run performance make it a practical option for systems that need online adaptation without retraining. Tree of Thoughts Pattern
Explore evaluation patternsSee how to apply these findings
Credibility Assessment:
Includes May D. Wang, a highly cited and established researcher (high h-index), boosting credibility.