I dabbled with local AI assistants and this is what happened?

I spent a few weeks running AI assistants on my own machine instead of in a browser tab. Not because I'm a privacy absolutist. Because I wanted to know what changes when the model lives on your hardware, reads your files, and doesn't ask permission from someone else's server first.

What happened was not what I expected. The tools were better than I assumed. And I got worse.

Let me explain both halves.

Why I went local in the first place

The thing that pushed me over the edge was a small, dumb moment. I wanted an assistant to read a folder of client notes and draft a summary, and I realized I was about to upload a quarter's worth of confidential work to a vendor I had no contract with, no DPA with, and honestly no relationship with beyond a credit card.

That instinct isn't unique to me. In Stack Overflow's 2025 Developer Survey, 87% of developers said they were concerned about the accuracy of information coming from AI agents, and 81% flagged the privacy and security of their data when using them. Meanwhile Gartner projects that 40% of enterprise applications will ship with task-specific AI agents by the end of 2026, up from under 5% a year prior. The agents are arriving whether or not anyone has figured out the data governance.

Running things locally is the most direct answer available to a normal person. Your files stay on your disk. Your keys stay in your keychain. The blast radius of a bad decision is one laptop.

"Local AI assistant" is actually two different things

This is the part every roundup glosses over, and it cost me a weekend.

There are local model runners — apps that download an open-weight model and give you a chat box. Fully offline, genuinely private, and about as useful as a very smart person locked in a room with no phone and no hands.

Then there are local agents — things with tools, memory, file access, and a permission model. These can actually do work: run code, read a directory, hit an API, edit a file. This is what most people picture when they say "AI assistant."

If you install the first kind expecting the second, you will be underwhelmed and blame the model. I did exactly that.

One genuinely useful development: MCP (Model Context Protocol) has become the connective tissue. Anthropic donated it to the Linux Foundation's Agentic AI Foundation, alongside Block's Goose and OpenAI's AGENTS.md as founding projects. Practically, a local agent that speaks MCP inherits a whole ecosystem of integrations instead of waiting on one vendor's roadmap.

My top 5

I judged these on one question: would I still be using this in a month? Not benchmark scores. Not feature matrices. Actual survival in a real workflow.

1. 1 Goose

   Best all-around local agent · Apache-2.0 · Free

   Open-source desktop agent and CLI from the Agentic AI Foundation. Native app for macOS, Linux, and Windows, works with 15+ LLM providers including Ollama for fully local inference, and connects to 70+ MCP extensions.

   Why it winsIt's the shortest distance between "I have a local model" and "something useful just happened." Point it at any provider, plug in what you need, get on with your day.

   The catchConfiguration is on you, and it doesn't remember you between sessions. It's a capable agent runtime, not a companion.
2. 2 Jan

   Best starting point if you've never done this · Apache-2.0 · Free

   An open-source ChatGPT replacement that runs 100% offline. Downloads Llama, Gemma, Qwen, and GPT-oss models directly, exposes an OpenAI-compatible server at localhost:1337, and now supports MCP.

   Why it's hereThe least intimidating on-ramp in the category. Install, pick a model, chat. No Docker, no Python, no YAML.

   The catchHardware is real — budget roughly 16GB of RAM for a 7B model and 32GB for a 13B. And it's closer to a chat client than an assistant.
3. 3 Open Interpreter

   Best if you live in a terminal · AGPL-3.0 · Free

   A natural-language interface to your computer. It writes code and runs it locally with no file-size or runtime limits, and asks for approval before each command.

   Why it's hereThis produced my biggest genuine time savings. "Rename every file in this folder to match this convention" is a five-minute job by hand and a five-second job here.

   The catchA power-user CLI with no memory across sessions and no polish. Local-only mode caps the context window to keep RAM sane, which limits long jobs.
4. 4 AnythingLLM

   Best private document workspace · MIT · Free to self-host

   Desktop app for Mac, Windows, and Linux, with Docker for multi-user. Supports nearly every LLM and embedder, including local ones via Ollama, LM Studio, and LocalAI. No-code agent builder and MCP support.

   Why it's hereIf your job involves a pile of documents you can't upload anywhere, this is the fastest path to "chat with my stuff" without it becoming a research project.

   The catchTelemetry is on by default — turn it off explicitly. And acting across email, calendar, or messaging isn't what it's for.
5. 5 Khoj

   Best personal knowledge base · AGPL-3.0 · Paid cloud tier

   A self-hostable second brain over your own documents, with full local LLM support through llama.cpp or Ollama, reachable from browser, Obsidian, Emacs, desktop, phone, and WhatsApp.

   Why it's hereThe multi-surface reach is unusual for a self-hosted tool. Asking your own notes a question from your phone is a genuinely different experience than searching them.

   The catchSelf-hosting setup is non-trivial, and it's built around retrieval more than action.

Honorable mentions: Hermes Agent (self-improving, server-resident, six execution backends — excellent if you want an agent living on a $5 VPS), and Leon (principled, MIT-licensed, still early on the maturity curve).

And now the part nobody puts in the roundup

Here is what actually happened after three weeks.

I stopped thinking first.

When AI is a browser tab, there's friction. You have to switch context, open a window, decide the question is worth asking. That friction is small, but it's a checkpoint — a moment where you unconsciously ask is this actually worth outsourcing? And a surprising number of times the answer is no, I know this, I'll just do it.

Remove the tab and the friction goes with it. The assistant is in my terminal. It's in my file manager. It's a keystroke away with my whole working directory already in context. So I asked it things I already knew the answer to. I had it draft things I could have written faster myself. I let it summarize documents I should have read, and then I made decisions based on the summary.

That last one is the dangerous one. I caught myself — twice — confidently repeating a conclusion in a conversation that I had never independently verified. It came from a summary of a document sitting eighteen inches from my hand. I could have checked in ninety seconds. I didn't, because the answer arrived already-formed and felt like knowing.

Nobody warns you about this because it doesn't look like a failure. Nothing crashes. There's no error message for "you have stopped forming your own opinions." The output stays good. It's the operator who degrades, slowly, in a way that only shows up when the tool isn't there.

So: a double-edged sword

The upside is real and I'm not giving it up. My data stays where it belongs. Costs are near zero once the hardware is paid for. Nothing breaks when a vendor changes their pricing or their terms. And the good tools genuinely compound — the more of my actual work they can see, the more useful they get.

But convenience is not free, it's just deferred. Every layer of friction you remove between yourself and an answer is a layer of judgment you removed too. Cloud AI at least makes you reach for it. Local AI is already in your hands, always on, always willing, and never once tells you that you didn't need to ask.

Three rules I've adopted since:

1. Answer first, then ask. If I can form my own take in under two minutes, I do, and only then compare notes with the assistant. It goes from oracle to reviewer.
2. Never act on a summary of a document I could have read. Summaries are for triage, not for decisions.
3. One day a week without it. Not for discipline points — as a diagnostic. If the work feels dramatically harder, that's information about how much I've delegated.

The tools aren't the risk. Your relationship with them is. Pick one from the list, run it locally, and pay as much attention to what it's doing to you as to what it's doing for you.