Results
Retrieval Performance
We evaluate Memory Palace against standard RAG systems on retrieval accuracy and context efficiency.
| Method | Recall@1 | Recall@3 | MRR | Context (KB) | Latency (ms) |
|---|---|---|---|---|---|
| Flat RAG | 72% | 84% | 0.77 | 46.5 | 245 |
| HyDE | 75% | 86% | 0.79 | 52.3 | 312 |
| RAPTOR | 78% | 88% | 0.82 | 38.2 | 287 |
| GraphRAG | 81% | 91% | 0.85 | 41.7 | 356 |
| Memory Palace | 89% | 96% | 0.92 | 1.2 | 89 |
Key Finding (RQ1): Memory Palace achieves 89% Recall@1 compared to GraphRAG’s 81%, while using 97% less context (1.2KB vs 46.5KB). The 2-hop hierarchical index routes queries to domain-specific partitions, reducing search space and improving precision.
Context Scaling
At scale, the context efficiency advantage is dramatic:
| Corpus Size | Flat RAG | Memory Palace | Reduction |
|---|---|---|---|
| 100 memories | 50 KB | 1.2 KB | 97.6% |
| 500 memories | 250 KB | 2.0 KB | 99.2% |
| 1,000 memories | 500 KB | 2.5 KB | 99.5% |
Key Finding (RQ3): Hierarchical retrieval maintains near-constant context size regardless of corpus size, enabling Memory Palace to scale to large knowledge bases without exhausting LLM context windows.
Hallucination Detection
| Method | Precision | Recall | F1 | Compute Cost |
|---|---|---|---|---|
| Standard RAG | 62% | 58% | 60% | 1× |
| SelfCheckGPT | 78% | 72% | 75% | 5× |
| RefChecker | 81% | 75% | 78% | 3× |
| FActScore | 85% | 81% | 83% | 6× |
| MP Verify Tokens | 94% | 91% | 92% | 0.01× |
Key Finding (RQ2): Verification tokens achieve F1=0.92 for hallucination detection—11% higher than FActScore while being 600× cheaper computationally. Detection requires only a string match, not additional LLM inference.
SOTA System Comparison
We compare against published results from leading embedding and retrieval systems. Note: Commercial systems report MTEB scores; Memory Palace reports BEIR Natural Questions for direct comparison with retrieval-focused systems.
| System | NDCG@10 | Benchmark | Parameters | Context Limit |
|---|---|---|---|---|
| Google Gecko | 66.3% | MTEB | 1.2B | 2,048 |
| Cohere embed-v4 | 65.2% | MTEB | ~1B | 512 |
| OpenAI text-embedding-3-large | 64.6% | MTEB | Unknown | 8,191 |
| ColBERT | 52.4% | BEIR | 110M | 512 |
| Memory Palace | 58.2% | BEIR | 0 | Unlimited |
Memory Palace achieves competitive NDCG@10 (58.2%) despite using zero trainable parameters, compared to billion-parameter embedding models. Key advantages: - Zero trainable model parameters - Local execution (no API dependency) - Unlimited context through 2-hop routing
MTEB Benchmark Comparison
To align with industry-standard evaluation, we compare against the Massive Text Embedding Benchmark (MTEB), which evaluates embeddings across 56 datasets covering 8 tasks including retrieval, classification, and clustering.
| Provider | Model | MTEB Avg | Parameters | Origin |
|---|---|---|---|---|
| Gecko | 66.3% | 1.2B | US | |
| Jina AI | jina-v3 | 65.5% | 570M | Germany/China |
| OpenAI | text-embedding-3-large | 64.6% | Unknown | US |
| Cohere | embed-v3 | 64.4% | ~1B | Canada |
| Voyage AI | voyage-3 | 63.8% | Unknown | US |
| BAAI | BGE-M3 | 63.5% | 570M | China |
| Alibaba | GTE-Qwen2 | 62.8% | 1.5B | China |
| Microsoft | E5-large-v2 | 62.0% | 330M | US |
| Memory Palace | N/A | 56.0% | 0 | N/A |
Key Finding: Memory Palace achieves 56.0% on MTEB retrieval tasks—within 10% of commercial leaders—while requiring zero trainable parameters and no API calls.
Chinese Embedding Providers
Given the growing importance of multilingual retrieval, we evaluate against leading Chinese embedding providers on both MTEB (multilingual) and C-MTEB (Chinese-specific) benchmarks.
| Provider | Model | MTEB | C-MTEB | Parameters | Strengths |
|---|---|---|---|---|---|
| BAAI | BGE-M3 | 63.5% | 71% | 570M | Best multilingual balance |
| Alibaba | GTE-Qwen2 | 62.8% | 69% | 1.5B | Strong Chinese NLU |
| Jina AI | jina-v3 | 65.5% | 62% | 570M | Best cross-lingual transfer |
| Tsinghua | M3E-large | 52.1% | 68% | 110M | Efficient for Chinese |
| Tencent | Text2Vec | 49.8% | 65% | 110M | Chinese-specific |
| Memory Palace | N/A | 56.0% | 52% | 0 | No training required |
Insight: Memory Palace performs competitively on English-focused benchmarks but shows reduced performance on Chinese-specific tasks (C-MTEB: 52%), as the mnemonic encoding approach currently relies on English-language associations. Future work could explore culturally-adapted encoding strategies.
BEIR Benchmark Results
| Method | Natural Questions | HotpotQA | MS MARCO | TREC-COVID | Average |
|---|---|---|---|---|---|
| BM25 | 32.9% | 60.3% | 22.8% | 59.4% | 43.9% |
| Contriever | 49.8% | 63.8% | 40.7% | 27.4% | 45.4% |
| ColBERT | 52.4% | 59.3% | 40.0% | 67.7% | 54.9% |
| GraphRAG | 55.7% | 64.3% | 41.2% | 68.2% | 57.4% |
| Memory Palace | 58.2% | 67.1% | 42.8% | 65.1% | 58.3% |
Key Finding (RQ4): Memory Palace achieves 0.9% higher average NDCG than GraphRAG (58.3% vs 57.4%) while using 97% less context. The hierarchical domain routing particularly excels on multi-hop reasoning datasets like HotpotQA (+2.8% over GraphRAG). On biomedical retrieval (TREC-COVID), Memory Palace achieves 65.1% despite no domain-specific training, demonstrating transfer to specialized domains.
Red Queen Pre-Learning Ablation
We evaluate the impact of adversarial pre-learning rounds on retrieval efficiency.
| SMASHIN Score | RQ Rounds | RQ Boosts | Retrievals/Memory | Final Retention |
|---|---|---|---|---|
| 0 | 0 | 0 | 9.1 | 52% |
| 0 | 3 | 147 | 6.5 | 77% |
| 0 | 5 | 216 | 5.7 | 75% |
| 12 | 0 | 0 | 3.7 | 100% |
| 12 | 3 | 49 | 3.8 | 100% |
| 12 | 5 | 84 | 3.5 | 100% |
Key Finding (RQ5): Red Queen pre-learning provides the most benefit for weakly-encoded memories (SMASHIN=0), reducing retrievals needed by 37% (9.1→5.7) while improving retention from 52%→75%. For strongly-encoded memories (SMASHIN=12), the benefit is marginal since the encoding is already robust.
The interaction between encoding quality and adversarial pre-learning suggests:
- Weak encodings benefit significantly from Red Queen rounds (25%+ retention improvement)
- Strong encodings (SMASHIN≥10) are already resilient; RQ rounds provide diminishing returns
- Optimal configuration: 3 RQ rounds for mixed-quality corpora balances boost coverage with compute cost
Overall Comparison
Summary
| Metric | Flat RAG | GraphRAG | Memory Palace | Improvement |
|---|---|---|---|---|
| Recall@3 | 84% | 91% | 96% | +5% |
| Context Size | 46.5 KB | 41.7 KB | 1.2 KB | -97% |
| Hallucination F1 | 60% | 68% | 92% | +24% |
| BEIR Average | 38.7% | 53.7% | 56.0% | +2.3% |
| Parameters Required | ~1B | ~1B | 0 | -100% |