You followed a guide. Ollama is running, a plugin is indexing your notes, and technically, you have local RAG. Then you ask it a question and the answer is wrong, or shallow, or just misses a connection that would have been obvious to anyone who actually read your notes.
The instinct at this point is to blame the tool. Swap plugins, try a bigger model, throw more VRAM at it if you’ve got it to throw. None of that is the actual problem, and none of it will fix what’s actually broken.
The problem is almost never the tool. It’s that nobody explained what the tool is doing to your data before you trusted it with an answer.
Chunking Is Fragmentation. Embedding Is Abstraction. Same Rules Apply.
Strip away the branding and local RAG is doing two things to your notes before it ever answers a question.
First, it’s breaking your notes into pieces. Small enough for a model to work with, because you can’t hand an LLM your entire vault and expect it to read all of it for every query. That’s fragmentation. It’s the same principle covered in Fragmentation Is How AI Systems Actually Work: systems that fragment well stay resilient, and systems that fragment badly lose the thing that made the whole worth more than its parts. A chunker doesn’t know what matters in your note. It knows a character count, or a token count, or maybe a paragraph boundary if you’re lucky. Cut in the wrong place and you’ve split an idea in half without knowing you did it.
Second, it’s compressing meaning down into a vector. That’s abstraction, reducing something specific into a representation that’s supposed to preserve enough of the original to still be useful. QAJourney’s Abstraction Is Not the Enemy, Losing Track of It Is makes the same point from the QA side: abstraction isn’t the failure, forgetting what got left out of the abstraction is. An embedding model compresses your note down to a few hundred numbers. Whatever nuance didn’t survive that compression is gone, and it’s gone silently. You won’t see an error. You’ll just get a search that doesn’t find the thing you know is in there.
Every local RAG plugin makes both of these decisions for you the moment you install it. Chunk size, chunk overlap, which embedding model, how retrieval scores relevance. You never see the decisions get made. You just see the results, good or bad, with no way to tell which of the two failure points caused it.
That’s the fundamental most tutorials skip. Not “here’s how to install this,” but “here’s what’s happening to your data every time you ask it something.”
Local vs Cloud Isn’t a Tool Choice, It’s a Design Choice
Once you see RAG as fragmentation plus abstraction, “local vs cloud” stops being a brand preference and becomes a question of where those two steps physically happen.
Run it locally and both the fragmentation and the abstraction happen on your hardware, under your control, and under your hardware’s limits. Run it through something like Gemini Notebook (the product formerly known as NotebookLM) and Google is making both decisions for you, on infrastructure that doesn’t share your VRAM ceiling or your privacy concerns.
Neither is wrong. Gemini Notebook is a genuinely good product for understanding a pile of source material you didn’t write. That’s a different job than building a personal knowledge base out of years of your own notes, where you actually want a say in how the fragmentation and abstraction get done, because you’re the one who has to trust the answers.
The point isn’t picking a side. It’s knowing that “local” buys you visibility and control over both steps, and “cloud” trades that away for convenience and someone else’s infrastructure. That’s the actual tradeoff, not a feature comparison.
I Tested This on a GTX 1660. Here’s Where It Broke.
Most local RAG content gets tested on hardware most people don’t own, and written as if installing the plugin is the hard part. Neither was true here. Everything below happened on a desktop with an i7-8700, 16GB of RAM, and a GTX 1660 with 6GB of VRAM, using Obsidian as the vault and my own published QAJourney and EngineeredAI posts as the source content, with Ollama running nomic-embed-text for embedding and qwen2.5:7b-instruct-q4_K_M for generation, both fully local and confirmed working outside of any plugin.
The plan was to test four specific failure modes, cross-note reasoning, a concept split across a chunk boundary, embedding degradation at scale, and naive versus semantic chunking, across two Obsidian plugins. That plan assumed getting to a working local RAG setup was the easy part. It wasn’t. Three separate plugins got tried, and none of them reached the point where those four tests could even run.
Smart Connections, free tier. This is the plugin most guides point to first. It indexed the vault cleanly with its built-in embedding model, no Ollama needed. But there’s no chat or Q&A feature in the free version at all anymore. Smart Chat used to be bundled in, it’s since been split into its own separate plugin, and connecting it to any model, local or cloud, is gated behind Smart Chat Pro. The free tier shows you a ranked list of related notes. It cannot answer a question about them.
Copilot for Obsidian, connected to Ollama. This one is actually built to do what I needed, chat grounded in vault content, with Ollama as a supported provider. Getting there wasn’t a dropdown selection though. Ollama isn’t in Copilot’s default provider list, so both the chat model and the embedding model had to be added as custom entries by hand, provider, model name, base URL, and a CORS toggle to get the local connection to respond at all. Once configured, it failed to initialize its local database. A full restart didn’t fix it. Force-reindexing the vault came back with “semantic search index rebuilt with 0 documents.” Ollama itself stayed responsive and error-free through all of this, confirmed by querying it directly outside the plugin. The failure was entirely in Copilot’s own indexing layer.
Local GPT, via the AI Providers plugin. This looked like the cleanest local-first option, Ollama is a native provider choice here, no manual custom-model wrangling. Configuring it went smoothly, both the embedding and chat models connected without a fight. But once set up, the plugin’s available commands turned out to be per-note actions, summarize, continue writing, fix grammar, find action items. There’s no vault-wide chat or question-answering command anywhere in it. It’s a writing assistant with local model support, not a RAG tool. The embedding provider setting exists for something the plugin calls “Enhanced Actions,” not for retrieval across notes.
Three plugins, three different failure modes: a feature that got paywalled out of the free tier, a working integration that broke at the database layer, and a tool that was never built for the job it looked like it was built for. Ollama and the models themselves worked correctly and quickly through every single attempt, confirmed by direct terminal queries. The constrained hardware was never the bottleneck. The plugin layer was.
That’s not a string of bad luck. That’s the actual current state of free local RAG for a personal knowledge base in Obsidian, tested honestly on hardware most people actually have.
Why “Just Use a Different Plugin” Doesn’t Fix It
Three plugins into this, the tempting read is that Copilot or Local GPT are just bad software, and the fix is finding the right one. Maybe. But notice what actually varied across all three: the failure mode changed every time, database corruption, a paywalled feature, a tool that was never built for the job, while the one constant underneath all of it, Ollama serving the models correctly, never wavered.
That’s the lock-in point. Every plugin author makes their own decisions about what gets exposed, what’s free versus paid, what the plugin is actually designed to do versus what it looks like it should do from the description. You don’t see any of those decisions until you’ve already sunk time into finding out the hard way. Swapping to a fourth plugin doesn’t remove that risk, it just resets the clock on which specific decision trips you up next.
The fix isn’t a better plugin. It’s understanding what a working local RAG setup actually requires closely enough that you can tell, fast, whether a given tool is fighting you because of a bug, a paywall, or a scope mismatch, instead of burning an evening finding out by trial and error.
Once You Know the Pattern, You Can Design Your Own
After three plugins failed to deliver a working setup, this stopped being the optional advanced-user section and became the actual answer. If the tools built specifically for this job can’t reliably do it on a real machine, understanding the pattern well enough to build your own stops being a nice-to-have.
Here’s the thing about vibe-coding a RAG setup without understanding fragmentation and abstraction first: it’s not a shortcut, it’s effort applied to nothing. You can prompt an AI coding assistant to generate an entire RAG pipeline in an afternoon, and it will run, and it still won’t tell you why your retrieval is bad, because you never gave it, or yourself, the actual requirements. Effort multiplied by zero understanding is still zero. You’ll have shipped something, and you still won’t know if it works, and you’ll have no better odds than the plugins that already failed.
Once you understand what you’re actually asking for, chunk size and overlap that match how your notes are structured, an embedding model suited to your hardware, retrieval logic that fits how you actually ask questions, you can direct that same AI assistant with intent instead of hope. That’s covered in more depth in the EAI coding workflow breakdown, but the short version applies here directly: the model isn’t the bottleneck. Knowing what to ask it for is.
A dedicated tool built outside the Obsidian plugin ecosystem, something like AnythingLLM, is also worth trying before going the full custom route, since the plugin layer specifically was where every failure happened here, not the local-AI concept itself. But whichever direction you go, plugin, dedicated app, or your own script, the difference between another failed attempt and a working one is whether you understand what’s supposed to be happening under the hood.
The Fix Was Never a Newer Tool
None of this is new information. Fragmentation and abstraction aren’t RAG-specific concepts, they’re the same fundamentals that show up anywhere systems break things down to work with them. I still keep old manuals and reread old computer books for the same reason: the tools change every year, the fundamentals underneath them mostly don’t.
Local RAG isn’t broken because the tooling is bad. It’s inconsistent because most of us installed a plugin and skipped the part where we learned what it was actually doing. Once you see it as fragmentation and abstraction instead of magic, the failures stop being mysterious, and the fix stops depending on whichever tool ships next.
Related reading: Fragmentation Is How AI Systems Actually Work · LLM Quantization Explained · Q4 vs Q5 vs Q8 Quantization · The EAI Coding Workflow
