Local-First Memory Stacks: What Self-Hosted pgvector and MCP Servers Actually Buy You for Privacy

The privacy question in AI memory has shifted. It used to be about whether your assistant remembered things. Now it's about where those memories physically live, who can read them, and which jurisdiction's regulators can compel access. A wave of 2026 comparison frameworks is formally classifying AI memory architectures by data sovereignty for the first time, from single-vendor conversational memory baked into a chat product, through SaaS memory layers like Mem0 and Zep, all the way to fully self-managed open stacks built on Postgres with pgvector and an MCP server you control. The tradeoffs are finally being mapped. Some of what you'll find is surprising.

Key Takeaways

• Using MCP does not make a memory layer sovereign. If the Postgres instance and embedding computation run on someone else's infrastructure, the data-sovereignty risk profile is identical to fully proprietary SaaS memory, regardless of protocol.
• The performance excuse for avoiding self-hosted vector search quietly expired in 2026. Postgres with pgvectorscale now benchmarks at 471 QPS versus ~41 QPS for a dedicated vector database at 50 million vectors on identical hardware.
• Regulatory enforcement is no longer a "someday" concern. The EU AI Act's context-layer governance requirements for high-risk systems bind from August 2026, and the NSA's own MCP guidance explicitly recommends local MCP server instances for sensitive data.
• MCP itself is undergoing its largest revision since launch, with a stateless core, OAuth/OIDC-aligned auth, and an enterprise-managed authorization extension now at stable status. The protocol is maturing fast, but security researchers filed over 30 CVEs against MCP components in the first two months of 2026 alone.
• For most teams, the honest answer is a hybrid: self-host the memory layer for sovereignty, accept that inference still touches a frontier provider, and be precise about what is and isn't protected at each hop.

What Does "Local-First Memory" Actually Mean in 2026?

It means your embeddings, your vector indices, and your retrieval logic run on infrastructure you own or exclusively control. The term gets abused. A hosted MCP-compatible memory service is not local-first just because it speaks the Model Context Protocol. The 2026 AI Second Brain Comparison framework makes this distinction cleanly: the dividing line is where the Postgres instance and the embedding computation physically run, not which protocol sits on top.

A genuine local-first memory stack in 2026 typically looks like this: Postgres with the pgvector (or pgvectorscale) extension running on your own server or VPC, an MCP server process co-located with it, and an MCP-compatible client that your agents or assistants connect through. No third-party SaaS touches the stored embeddings or the raw context. The MCP protocol handles the interop layer, so you're not locked into a single client.

This is meaningfully different from what most vendors offer. When a SaaS memory layer says it "supports MCP," what they typically mean is that their hosted service exposes an MCP-compatible API. Your data still lives on their infrastructure. Their operators can, in principle, access it. Their jurisdiction's law enforcement can compel disclosure. The protocol standardization is real and useful, but it doesn't change the residency story at all.

Why Is Protocol-Level Interoperability Not the Same as Sovereignty?

Because interoperability governs how systems talk to each other. Sovereignty governs who can access what's stored. These are orthogonal properties, and conflating them is the most common category error in current memory-architecture marketing.

Consider: you could run a fully MCP-compliant memory server on AWS us-east-1 using a vendor's managed service. Your agents connect cleanly. The protocol works. But your embeddings are on that vendor's infrastructure, subject to their data-processing agreement, accessible to their operations team under their internal policies, and stored in a jurisdiction you may not have chosen deliberately. The secondbrain.us.com comparison classifies these MCP-branded hosted services in the same risk tier as fully proprietary memory, and it's correct to do so.

The industry is currently letting vendors claim a sovereignty halo just because they speak MCP. We see this repeatedly when evaluating tools for our own stack. The real question is blunt: can you pg_dump your memory store tonight and move it to a different provider by morning? If the answer involves a support ticket, a data-export request, or a format conversion that only works with their tooling, you don't have sovereignty. You have a nicer API on top of the same dependency.

Has pgvector Actually Closed the Performance Gap?

Yes, and the margin is not close. The old assumption, that self-hosting Postgres for vector search meant accepting a significant performance penalty compared to purpose-built vector databases, was reasonable through 2024. It is no longer defensible.

Recent benchmarking by Tural Allahverdiyev tested Postgres with the pgvectorscale extension against Qdrant on identical hardware at scale: 50 million vectors, 1536 dimensions, targeting 99% recall. Postgres with pgvectorscale processed 471 queries per second. Qdrant processed 41.47 QPS. On the same machine. That's roughly an 11x throughput advantage for the Postgres-based stack.

The mechanism behind this is Timescale's StreamingDiskANN index, which doesn't require the entire dataset to fit in memory. This matters for self-hosting economics: you can run meaningful vector workloads on hardware that would have been undersized for a dedicated vector database's memory requirements. The practical result is that a team running Postgres on a single decent server can handle vector retrieval workloads that previously required either a managed vector SaaS or a cluster.

We should be honest about the limits. These benchmarks test specific query patterns and recall targets. Your workload might differ. Operational concerns (backup, monitoring, index maintenance) don't disappear just because the raw QPS number is good. But the core argument, that self-hosting vector search means accepting materially worse performance, is no longer supported by the data. The sovereignty choice no longer comes with a technical-inferiority tax.

What Does the Regulatory Landscape Actually Require Right Now?

The EU AI Act's requirements for context-layer governance in high-risk systems take effect from August 2026. This is not a proposed regulation or a draft guideline. It is a binding enforcement date.

The shift that matters here is subtle but consequential. Regulators are moving from asking "where is data stored?" to asking "where is it processed?" A legal expert at DLA Piper, serving as Global Co-Chair of Data Protection, has framed it this way: when an AI agent pulls context from EU data sources, it's performing processing operations that must be mapped and governed under GDPR regardless of where the model itself runs. Storage residency is necessary but no longer sufficient.

This creates a specific problem for SaaS memory layers. Even if the SaaS provider stores your embeddings in an EU data center, the retrieval and augmentation pipeline might route context through inference infrastructure in a different jurisdiction. If you don't control the full path from storage through retrieval to inference, you can't fully map your processing operations, and you can't fully comply.

A local-first pgvector stack gives you control over the storage and retrieval segments of that path. The inference hop still goes to a frontier provider (unless you're running local models, which carries its own capability tradeoffs). But you've reduced the surface area of jurisdictional ambiguity from three hops to one, and that one hop is governed by a direct API agreement rather than a chain of subprocessor relationships.

What Does the NSA's MCP Guidance Actually Say?

The NSA's May 2026 cybersecurity advisory on MCP security explicitly recommends that organizations prefer a local instance of an MCP server when processing private data. This is not ambiguous. A national security agency looked at the MCP deployment landscape and concluded that the risk reduction from local deployment is significant enough to warrant a direct recommendation.

The context here is important: the advisory isn't anti-MCP. It treats MCP as a viable and increasingly standard protocol. The guidance is about deployment topology, not protocol choice. Run MCP, but run it locally when the data matters.

This aligns with what we've observed operationally. The protocol itself is a transport and capability-negotiation layer. Its security properties depend heavily on how and where it's deployed. A local MCP server connected to a local Postgres instance over a Unix socket has a fundamentally different threat surface than a cloud-hosted MCP server accessible over the public internet, even if they run identical code.

How Mature Is MCP as a Foundation for Memory Architecture?

Maturing rapidly, but with real gaps that you need to account for. The 2026-07-28 MCP specification release candidate is the largest revision of the protocol since launch. It delivers a stateless core that scales on ordinary HTTP infrastructure, formal extensions for capabilities like authorization, and auth hardening aligned with OAuth and OpenID Connect. The Python and TypeScript SDKs see roughly 97 million monthly downloads. The protocol was donated to the Agentic AI Foundation under the Linux Foundation in late 2025, making it vendor-neutral and community-governed.

The Enterprise-Managed Authorization extension has been promoted to stable status, with Asana, Atlassian, Canva, Figma, Granola, Linear, and Supabase listed as supporters. This is the centralized-auth story that enterprise IT teams need before they'll approve MCP server deployments at scale.

The counterweight: between January and February 2026 alone, security researchers filed over 30 CVEs targeting MCP servers, clients, and infrastructure components, with the highest-severity finding carrying a CVSS score of 9.6. The protocol is attracting serious security research attention, which is both a sign of adoption maturity and a reminder that the attack surface is being actively probed. If you're deploying an MCP server, local or otherwise, you need to be tracking the CVE stream and patching aggressively.

What Happens When SaaS Memory Providers Change Direction?

You find out how sovereign your data actually was. At least one practitioner account from 2026 documents organizations that needed to self-host after Zep deprecated its open-source edition. This is the lock-in risk in concrete form: a product decision by a third party forces a migration on your timeline, not yours.

With a self-hosted pgvector stack, your memory store is a Postgres database. You can back it up with pg_dump. You can replicate it with standard streaming replication. You can move it between cloud providers, or to bare metal, or to a different continent, using tools that have existed for decades. The operational burden is yours, but so is the control.

Mem0's 2026 state-of-memory report frames this well: privacy and consent architecture, including who can inspect stored memories, how long they're retained, and how a user deletes them, remain application-layer decisions today. Regulatory expectations will get more specific. If those decisions are delegated to a SaaS provider, you're hoping their implementation matches the regulatory interpretation that applies to your use case. If you control the application layer, you can implement exactly the retention and deletion semantics your compliance posture requires.

What Does a Production-Grade Self-Hosted Memory Stack Look Like?

At minimum: Postgres 16+ with pgvector (or pgvectorscale for larger workloads), an MCP server process that handles embedding generation and retrieval, and a client integration layer. Several open-source implementations exist. mcp-memory provides a PostgreSQL-backed MCP server with pgvector for long-term memory. Memory Vault adds hybrid search and knowledge-graph capabilities. local-memory-mcp focuses specifically on having LLMs save memories locally on your own hardware.

The embedding generation step is where the architecture gets honest. Unless you're running a local embedding model (which limits you to smaller, less capable models for now), embedding generation requires a call to an external model API. That call transmits the raw text to be embedded. This is a processing operation that crosses a trust boundary, and it needs to be accounted for in your sovereignty analysis. You can mitigate it (batch processing, contractual guarantees with the embedding provider, or local embedding models if accuracy tradeoffs are acceptable), but you can't pretend it doesn't exist.

We run a stack of frontier models, routed per task. Our experience: embedding behavior generalizes well across frontier models. The vector representations are stable enough that you can switch embedding providers without re-indexing your entire memory store, provided you standardize on a dimension size. Retrieval behavior, meaning how well a model uses retrieved context, varies more. Some models handle long retrieved contexts gracefully. Others degrade. This is the kind of thing you only discover by running the same retrieval workload across multiple models, and it's why model-agnostic architecture matters. If your memory layer is coupled to a single model's context-handling assumptions, you'll feel it when you need to switch.

What Are the Honest Limits of Local-First Memory?

Operational burden is the big one. You're responsible for backups, monitoring, index maintenance, security patching, and capacity planning. Postgres is battle-tested infrastructure, but it still needs care. If your team doesn't have someone comfortable with database operations, a managed Postgres service (where you control the instance but delegate operational tasks) is a reasonable middle ground. You lose some sovereignty compared to bare-metal self-hosting, but you retain control of the data and the schema.

The inference hop is the other honest limit. Even with a fully local memory stack, the moment you send retrieved context to a frontier model for reasoning, that context leaves your infrastructure. This is true for every architecture that uses external model APIs, including ours. Selina's memory is encrypted at rest, but memory is NOT end-to-end encrypted, because a slice of each request reaches a frontier provider at inference. We state this plainly because the alternative, implying a level of encryption we don't provide, would be dishonest. Files and transfers via SelinaSEND are zero-knowledge encrypted. Memory is not. The distinction matters and we maintain it.

Non-content operational metadata has a short retention window, not zero retention. The account is protected. Content is encrypted. These are specific, bounded claims and we don't extend them beyond what we can demonstrate.

Scale ceiling is a consideration, though a less pressing one than it was. pgvectorscale's benchmarks at 50 million vectors are impressive, but if your workload involves billions of vectors with sub-millisecond latency requirements, you're in territory where dedicated vector infrastructure (self-hosted or managed) might still make sense. For most AI memory use cases, where you're storing thousands to low millions of contextual embeddings per user or workspace, Postgres is more than sufficient.

How Should You Think About the Tier System?

The 2026 comparison frameworks generally recognize three tiers:

1. Single-vendor conversational memory. Built into the chat product. Convenient. Zero portability. The vendor controls retention, format, and access. You're a tenant.
2. SaaS memory layers. Dedicated memory services, sometimes MCP-compatible. Better separation of concerns. Still hosted. The data-sovereignty risk is identical to tier one unless you self-host. This is the tier where protocol-level interoperability gets confused with sovereignty.
3. Self-hosted open stacks. Postgres + pgvector + local MCP server. Full control over storage, retrieval, retention, and deletion. Operational burden is yours. The inference hop remains an external trust boundary unless you run local models.

The honest answer for most teams: tier three for the memory layer, with a clear-eyed understanding that the inference layer (tier one or two, depending on how you count it) still involves a frontier provider. The goal is to minimize the surface area of third-party data exposure, not to pretend you can eliminate it entirely while using external models.

An academic perspective from a January 2026 paper on zero-trust AI memory architectures frames the core tension precisely: centralized memory systems enable efficient cross-agent collaboration but expose users' sensitive data to cloud provider risks, while private deployments provide security but limit collaboration. The researchers propose hardware-based trusted execution environments as one resolution. That's promising but not production-ready for most teams. In the meantime, local-first Postgres with a well-scoped MCP interface is the most practical answer available.

What Should You Do This Quarter?

If you're building agents or assistants that handle sensitive context, and you haven't mapped your memory architecture against the three-tier framework, do it now. The August 2026 EU AI Act enforcement date is not far. The work is not primarily technical. It's a data-flow mapping exercise: for each piece of context your system stores and retrieves, where does it physically reside, where is it processed, and under which jurisdiction's authority?

If that mapping reveals dependencies you're not comfortable with, a local pgvector + MCP stack is no longer a compromise solution. The performance is there. The protocol maturity is getting there (track the CVEs, patch promptly, don't assume stable means secure). The tooling exists. The remaining gap is operational willingness, and the regulatory calendar is about to close that gap for you.

If you'd rather not run the memory infrastructure yourself but still want strong data protections at the application layer, start a free 7-day trial of Selina, no card required.

Michael C.

Founder & Principal Engineer, Selina Labs

Michael builds Selina, a privacy-first AI that remembers you across conversations. He ships security-sensitive AI in production — real attacks, real fixes, measured in minutes and dollars — and writes about privacy, security, and LLMs from that seat. Top Rated Plus and expert-verified on Upwork.

• www.upwork.com
• x.com
• medium.com
• selinalabs.io