Giving Our Hermes Agent a Real Memory: Why We Chose Hindsight (and How to Set It Up)

If you run Hermes Agent — the open-source, self-hosted AI agent from Nous Research — you've probably hit the same wall everyone does eventually: out of the box, it forgets.

Not entirely. But close enough to be annoying. So we added an external memory provider called Hindsight, and the agent went from "competent goldfish" to something that actually remembers our projects across sessions, platforms, and restarts. This post explains what Hindsight is, why we picked it over the eight other options Hermes ships with, and exactly how to set it up — including which mode to choose if, like us, you're running on a tiny free-tier VPS with no GPU.

First, what Hermes remembers by default

This part trips people up, so let's be precise.

Hermes already has a built-in memory system. It lives in two plain text files — MEMORY.md and USER.md — stored under ~/.hermes/memories/. The agent manages these itself: it can add, replace, or remove entries, and at the start of every session it loads them into its context as a frozen snapshot.

The catch is that this memory is bounded and curated on purpose. There are character limits. It's meant to hold a tight summary of who you are, what you're working on, and a handful of hard-won facts — not the full history of everything you've ever discussed. Think of MEMORY.md as a single index card the agent keeps rewriting to stay current.

That's genuinely useful, but it's not a filing cabinet. The index card can't hold three weeks of a research project, every decision you made along the way, and the side-notes that never made it into a document. For that, you need something bigger sitting alongside it.

That's what external memory providers are for. They never replace the built-in memory — they run on top of it. The index card stays; you're adding the cabinet.

The nine options Hermes gives you

Hermes ships with a roster of pluggable memory providers, and only one can be active at a time. The docs describe eight provider plugins — Honcho, OpenViking, Mem0, Hindsight, Holographic, RetainDB, ByteRover, and Supermemory — with a ninth, Memori, listed in the comparison table.

They split into rough camps:

• Cloud-leaning, hands-off — Honcho, Supermemory, RetainDB ($20/mo), Memori. These do clever user-modeling, though most are paid and your memories live on their servers. (Honcho is the exception that also offers a free self-hosted mode if you want to run it yourself.)
• Self-hosted, free, more setup — OpenViking (needs a running server), Mem0 in its open-source mode, Holographic (pure local SQLite, zero dependencies), ByteRover (local-first CLI).
• Flexible, either way — Hindsight, which runs as a paid cloud service or as a free local instance, your choice.

If you want the absolute simplest free option with no moving parts at all, Holographic is worth a look — it's just a local SQLite database and needs nothing installed. We tried it. It's fine. But it retrieves; it doesn't reason.

Why we landed on Hindsight

Three things tipped it.

1. It's a knowledge graph, not just a search box. Most providers store facts and pull back the ones that look similar to your question (semantic search). Hindsight does that too, but it also extracts the entities — the people, projects, and decisions — out of each conversation and links them together. When you later ask about "the affiliate outreach," it doesn't just find the one message where you said those words; it knows which creators, which emails, and which decisions are connected to it.

2. The reflect tool — the thing no other provider has. Hindsight ships three tools: hindsight_retain (store), hindsight_recall (search), and hindsight_reflect (synthesize). That third one is the differentiator. Reflect performs cross-memory synthesis — it combines multiple separate memories into a new insight rather than just handing back the closest match. The official docs are blunt about this being something "no other provider offers." For an agent that's supposed to get smarter the longer it runs, that's the whole ballgame.

3. Free locally, and it captures everything automatically. With auto_retain on (the default), Hindsight quietly stores full conversation turns — including the agent's tool calls — without you lifting a finger. And in local mode it costs nothing but the LLM API key it uses for extraction. For a solo operator that's the right price.

The honest trade-off: Hindsight exposes only three tools versus five on some providers, and the cross-session magic depends on that extraction LLM doing good work. But for "remember my projects and connect the dots," it was the clear pick.

How memory actually flows, once it's on

When a provider is active, Hermes does six things automatically every session, and it helps to picture them:

1. Injects what the provider already knows into the system prompt.
2. Prefetches relevant memories before each of your turns — in the background, so it doesn't slow the reply.
3. Syncs each conversation turn to the provider after the response.
4. Extracts durable memories when the session ends.
5. Mirrors any built-in memory writes out to the provider too.
6. Adds the provider's tools so the agent can deliberately search and store.

So you get two layers working together: the small, always-loaded index card (built-in), and the large, queried-on-demand cabinet (Hindsight).

Picking your mode: this is the part that depends on your hardware

Hindsight runs three ways, and choosing wrong is the most common setup mistake.

• Cloud — You get an API key from ui.hindsight.vectorize.io, paste it in, done. Zero ops, but it's a paid service and your memories live in their cloud.
• Local Embedded — Hermes spins up its own local Hindsight daemon with a built-in PostgreSQL database. It starts automatically on first use and shuts itself down after five minutes idle. You supply one LLM API key (OpenAI, Groq, OpenRouter, Anthropic, Ollama, etc.) for the extraction and synthesis work. Free, and your data never leaves your machine.
• Local External — You run a separate Hindsight instance (Docker or self-hosted) and point Hermes at it over HTTP. For people who want full control of the database.

If you're on a small free-tier VPS like ours (Oracle ARM always-free, no GPU): the choice comes down to where the embedding work happens. Local Embedded runs embeddings and reranking locally and only calls out to an LLM for extraction — which is light on a small box because the daemon hibernates after five minutes of inactivity and only wakes on demand. That intermittent footprint is exactly what you want on a memory-constrained instance. If even that feels tight, Cloud mode offloads everything and asks nothing of your CPU at all. The one thing you don't want on a tiny ARM box is trying to run heavy local embedding models full-time.

We went with Local Embedded pointed at a cheap hosted LLM for extraction — free storage, data stays put, near-zero idle cost.

Setting it up

The setup wizard does the heavy lifting and installs only what your chosen mode needs.

The simple path:

hermes memory setup      # interactive picker — choose "hindsight", then your mode
hermes memory status     # confirm it's active

If you'd rather configure it by hand, the provider line is the same either way — only the key differs by mode.

For cloud mode, the key is your Hindsight Cloud API key:

hermes config set memory.provider hindsight
echo "HINDSIGHT_API_KEY=your-key" >> ~/.hermes/.env

For local mode, you don't use a Hindsight Cloud key at all — you provide an LLM extraction key, stored under a different variable:

hermes config set memory.provider hindsight
echo "HINDSIGHT_LLM_API_KEY=your-llm-key" >> ~/.hermes/.env

That HINDSIGHT_LLM_API_KEY is your OpenAI / Groq / OpenRouter / etc. key — Hindsight uses it for extraction and synthesis only; the daemon and its PostgreSQL come up on their own the first time they're needed. Getting these two variables confused is the most common local-setup snag, so it's worth pasting the right one the first time.

On the desktop app, it's now even simpler: Settings → Memory & Context → pick Hindsight, choose a mode, paste a key, save. No files, no .env editing — Hindsight is currently the one provider with a full in-app config panel.

A couple of practical notes from our setup:

• The wizard auto-installs dependencies (hindsight-client for cloud, hindsight-all for local) and auto-upgrades them on session start if they're outdated, so you don't have to babysit versions.
• Config lives at $HERMES_HOME/hindsight/config.json. The settings worth knowing early: recall_budget (low/mid/high — how hard it searches), memory_mode (hybrid is the default and gives you both auto-injection and the manual tools), and auto_retain / auto_recall (both on by default — leave them).
• To eyeball what it's storing in local mode, there's a UI: hindsight-embed -p hermes ui start.
• Everything is isolated per Hermes profile, so a separate profile gets its own memory bank automatically — no cross-contamination between, say, a work profile and a personal one.

Was it worth it?

For us, unambiguously yes. The built-in MEMORY.md is great at "remember who I am." Hindsight is what makes the agent stop asking us to re-explain a project we set up three weeks ago — and occasionally surface a connection between two things we'd forgotten were related. That second part, the reflect-driven synthesis, is the bit that feels less like a database and more like the agent actually knowing something.

If you're running Hermes and you've been re-pasting context every session, give it the cabinet. Start with hermes memory setup, pick Hindsight, and pick the mode that fits your hardware.

Setup commands and provider details in this post are drawn from the official Hermes Agent documentation. Versions and defaults move quickly on a young project — check the current docs before you copy-paste.