The problem with agent memory
Verimem is a persistent memory layer whose add() routes every write through an anti-confabulation admission gate — does the source actually entail the fact? — and whose search() returns provenance on every read. A hippocampus with a notary at the door.
✓ 100% free & open source — AGPL-3.0 license. Self-hosted, local-first: no account, no API key, no billing.
Other memory layers store whatever their extractor emits. Verimem doesn't.
On write, a candidate fact is admitted, downgraded, or refused — decided by whether its cited source actually entails it. A cheap, no-LLM lexical screen first downgrades unsupported "it works / verified / done" claims; then, on by default since v0.5.0, a source⊢fact entailment check refuses any claim the cited source doesn't actually support. Measured on SNLI it reaches AUROC 0.971, and that number is judge-independent.
A structured review of mem0, Zep, Letta, Cognee and MemOS found that none of them ship a write-admission gate — and an independent 2026 survey of always-on agents (arXiv 2606.30306) finds the field concentrates far more on accumulating and retrieving agent state than on governing it — the write-admission gap, exactly. That gate — plus the provenance every read carries back — is the whole point.
Illustrative — the gate in three verdicts
Reads don't just return text — they return each fact's status and write-time grounding_score, so your code can trust-condition instead of trusting blindly. And update() never destroys the old fact: it supersedes it, leaving an auditable history() trail. explain() returns a full TrustReport — provenance, checks, and any conflict, stated or abstained-on with the reason.
memory.py · the write goes through the gate, the read returns provenance
from verimem import Memory
mem = Memory() # local SQLite, offline
mem.add("The deployment uses PostgreSQL 16.") # write goes THROUGH the gate
for hit in mem.search("which database?"): # read returns provenance
print(hit["text"], hit["status"], hit["grounding_score"])
# -> The deployment uses PostgreSQL 16. ADMITTED 0.97| Metric | Result | On the record |
|---|---|---|
| Write-gate entailment | AUROC 0.971 | Source⊢fact, judge-independent (SNLI). The write-path moat. |
| End-to-end QA · HaluMem, full pipeline | 0.66–0.68 | Our own extraction → gated store → answer, seven consecutive full runs cluster at 0.66–0.68 (mean 0.667, n=3 clean) vs MemOS self-reported 0.672 — parity, not a win, and we say it that way. Judge is Claude, not GPT-4; raw result files in the repo. |
| Cross-user generalization | 0.716 | The same recipe on a never-seen user (n=169): 0.716. The recipe isn’t tuned to one user’s data. |
| Memory-boundary abstention | 1.000 × 7 runs | Refuses to invent when the store can’t support an answer — 1.000 across seven consecutive end-to-end runs. This is the axis competitors don’t measure. |
| Multilingual gate hole — closed | 8/10 → 0/10 | Reproduced from live testing: the same unsupported hype claim passed the gate clean in 8 of 10 languages (the lexical screen was EN/IT-only). Fixed with a semantic dual-check that uses the multilingual embedder itself as the detector, then hardened by an adversarial review that falsified the first "0 FP" claim on held-out sentences: questions, negations and reported speech are now excluded by deterministic guards, by design. Post-fix on the real write path: hype flagged 10/10 languages, held-out false positives 0/14 — calibration script, review findings and raw files in the repo. |
| Memory-conflict resolution | 0.15 → 0.82 | A contradicting update wins 5.5× more often after the fix; wrong retractions cut 99 → 7 with the precision floor (0 cross-attribute). Adversarial critic verdict: claim_holds. |
| Hallucination rate (a dial) | 0.233 → 0.111 | −52% as a switch (ENGRAM_GROUNDING_GATE), while keeping the correct answers. You choose recall-first or trust-first — on QA and on the write path. |
| Retrieval recall@5 · LongMemEval-s | 0.8745 | Full 500, judge-free, same e5 embedder, zero external APIs. Fusion ON vs 0.8525 OFF (+2.2 pp). This is recall@k, not end-to-end QA accuracy. |
| Latency · concurrency | 38ms / 166–237ms | Write 38ms p50 with the full gate; read 166–237ms with history + TrustReport at 5k facts — warm, single process. Shared-service mode (the recommended deployment): ~258ms write / ~102ms read at 14.3 ops/s, zero errors. First call in a throwaway process pays ~26s of model load — an anti-pattern; use the service. Concurrent-save SLO: 40/40 process-level writers, zero lost writes. |
| Test suite | 7,761 passing | Self-run, reproducible from the repo. No third-party leaderboard. One live-wire LLM smoke test is environment-bound (needs a hosted provider) and excluded from the count. |
| TrustMem-Bench · trust axes | Verimem 60/60 | Our own deterministic trust benchmark — no LLM, no network: six axes, one command. Verimem clears 60/60; mem0 OSS covers 40/60 (0/10 on absence, forget, provenance). A raw-store baseline by design — competitors are invited to run it. |
| VeriBench · the trust standard | deterministic · self-run | The open benchmark for trusted memory — it scores what recall@k can't see, because a symmetric recall score cannot tell a confident wrong answer from an honest "I don't know". On the causal axis, a trust-only store that corroborated a spurious correlation nets negative even at λ=1 (provenance ≠ causality). On the adversarial axis — collusion plus a trusted sleeper — only a two-channel policy (independent corroboration and outcome) stays net-positive; each single channel fails one attack. Model-free, one command, spec in the repo — competitors invited to run it. Full results, protocol and head-to-head → |
| Source trust on a REAL corpus | cartel 0.90 → 0.20 · 3/3 seeds | Held-out HaluEval, criteria pre-registered before the first run: a 4-identity cartel that self-confirms to 0.90 under naive ≥2-source counting is demolished to 0.20 by independence + audit-deconfounding; honest sources restored to 0.95, the cartel's hallucinated answers drop out of recall (→ 0.0). Robustness curve (18 points): wrong answers written by deceivers = 0/18 at every noise level; the residue is honest slips — a per-claim disease, declared, not hidden. |
| SLA knob · operates at declared risk | TCE ≤ 0.011 · risk 1.1% @ 73% coverage | A strong AUROC says scores discriminate; it does not say the λ knob operates at its declared risk (Oxford 2603.21172). Measured held-out with calibration fit on dev only: raw scores rank near-oracle (E-AURC 0.0008) but promise a different risk than delivered; after a pure isotonic calibration every declared λ target is met — TCE ≤ 0.011 across λ ∈ {0.5–9}, observed risk 1.1% at 73% coverage. Declared trade-off: calibration flattens fine ranking — raw for ranking, calibrated for operating. |
Every figure here is self-run and reproducible from the repository — it is not a third-party leaderboard placement. Retrieval numbers are recall@k, not the end-to-end QA accuracy that Mem0 and Zep headline, so they are not directly comparable.
Admit, downgrade or refuse each write by source entailment. No competitor here ships one — and since July it is multilingual: the same semantic screen in 10 languages, calibrated at 0 false positives on the bench. A delivery-path defect found by our own exam (Aug 28) kept that screen silent on one write path; the fix is in review and tracked in the public exam.
Every fact returns status + grounding_score; update() supersedes, history() stays auditable; explain() gives a full "how do you know?" dossier.
Two clocks (when learned vs when true), answer-with-history ("changed from X to Y on Z"), as_of time-travel, and deep recall of dormant facts — not just valid_until.
verimem gateway serve: multi-tenant HTTP API on your own box — hashed keys, per-tenant isolated stores, rate limiting, hot backups, remote provisioning. Your data never leaves your infrastructure.
Persistent counters of what the gate actually DID — writes admitted, quarantined, rejected, honest abstentions — per store and per tenant, with a dependency-free /dashboard page. Observable actions, not marketing claims.
Index PDF / DOCX / HTML / EPUB; passages come back with file, version and character offsets, and can be promoted to memory through the gate.
Bootstrap from your ChatGPT / Claude export: conversations are listed first, filterable by title, date or project — nothing is ingested without an explicit selection.
delete(purge_history=True) collapses the whole chain — erased data does not resurface from history, as_of, or deep recall.
Memory tools at session start in Claude Code, Cursor, Cline, Zed — plus a typed, zero-dependency TypeScript client for the gateway, contract-tested against the live server.
Hosted MCP mode: no API key, no per-token billing — the host's LLM does the work. Runs air-gapped (verimem airgap verifies zero egress).
The composition ring derives NEW facts from verified ones and pushes them through the SAME admission gate as every writer — admitted survivors are signed (engine writes never testify for themselves), traced (retractable if a parent falls) and labeled with the exact check that passed: proven / unbeaten(bound) / refuted(counterexample). "Held to 10^6" and "proven" never get conflated.
A read doesn't only abstain — when the store holds a better-guaranteed truth about the same subject, it CORRECTS, citing both facts (a refuted fact is never served). And the store probes ITSELF: it builds the query that would falsify a stored fact — independent counter-evidence proposes a refuted label, surviving grows its unbeaten bound. Falsification you don't have to wait for.
An unforgeable HMAC of WHO is speaking rides inside each write's provenance ref, complementing the entailment gate's WHAT deserves admission. Content authenticity and channel authenticity — the two halves no deterministic content filter alone can certify against an adaptive adversary. Opt-in; audit reports coverage and names offenders.
| System | Write gate | Provenance on read | Approach | Maturity |
|---|---|---|---|---|
| Mem0 | ✗ | ✗ | Flat vector + LLM summary | Established, widely adopted |
| Zep / Graphiti | ✗ | partial (temporal) | Temporal knowledge graph | Commercial, mature |
| HippoRAG | ✗ | ✗ | OpenIE + PageRank | Research |
| Verimem | ✓ | ✓ | Fusion recall + gate + sleep consolidation | Brand-new · 0 adoption yet |
Competitor scores and adoption drift by source and over time; the two columns we stand on — a write gate and provenance on read — come from a structured competitor review, not a marketing table.
A brand-new release. Nobody depends on it yet — including us, beyond the maintainer's daily driver.
pip install verimem "mcp<2" — v0.7.0 live on PyPI (AGPL-3.0); the public repo is on GitHub. Known issue, stated: 0.7.0's MCP entrypoint crashes with mcp>=2 — the pin above is the workaround until the fixed release ships. Still brand-new, though.
Provenance-independence and audit-deconfounding now hold on a real held-out corpus (HaluEval, pre-registered criteria, 3/3 seeds): a manufactured-consensus cartel that self-confirms to 0.90 under naive counting is demolished to 0.20, honest sources restored to 0.95, and its hallucinated answers drop out of recall entirely. Honest limit, measured: under heavy honest noise (trusted sources slipping ~15%+) separation degrades — the residue is 100% honest slips, a per-claim disease, not an attack. Flags stay default OFF: the evidence informs the flip, the flip is a product decision.
Every TrustReport now declares it — Verimem certifies who asserted a fact, how independently it was corroborated, and how fresh it is, not that it is causally true. A do(X) question needs an interventional fact; a corroborated observational one cannot answer it.
Every number is reproducible from the repo, but none is third-party audited. The QA vs MemOS figure uses a Claude judge (not GPT-4) over n=2 users. Treat them as reproducible, not certified.
The full pipeline (our own extraction → QA) clusters at 0.66–0.68 across seven consecutive runs (mean 0.667 on n=3 clean stores) vs MemOS's own 0.672, and holds 0.716 on a never-seen user. The caveat stays: judge is Claude not GPT-4, no third-party audit. We say parity, not "beats".
The multi-tenant gateway (isolated per-tenant stores, hashed keys, backups, metering) is real and tested — on your infrastructure. A managed cloud service does not exist yet.
Conversational entity extraction (typed tier) is shipped but off by default; multi-hop reasoning over the graph is the weakest measured axis (0.39–0.44) and the current workstream.
① Python — from source
pip install verimemProvides the verimem command and the from verimem import Memory SDK — add() / search() / history() / explain(). Local SQLite, offline, gate on by default.
github.com/aureliocpr-ctrl/verimem ↗ · AGPL-3.0 · README · BENCHMARKS.md
② As an MCP server — Claude Code, Cursor, Cline, Zed
{
"mcpServers": {
"verimem": {
"command": "verimem",
"args": ["mcp"],
"env": { "ENGRAM_HOSTED": "1" }
}
}
}Restart your host; the memory tools become callable with zero API key — the host's LLM does the work.
③ Self-host team gateway
verimem gateway keys create --tenant acme
verimem gateway serve # 127.0.0.1:8377 · /uiverimem gateway keys create --tenant acme then verimem gateway serve — a multi-tenant HTTP API with per-tenant isolated stores, rate limiting and hot backups. Open /ui for the trust odometer; a typed TypeScript client ships in the repo.
④ See it working — the trust console
verimem console # your store · odometer · graph · liveverimem console opens your own local store in the browser — trust odometer, knowledge graph where every conclusion carries its chain of custody, blocked-claims log, live over SSE. The team gateway serves the same console at /ui; agents read the whole state from /v1/snapshot.