PrivateGPT Explained: Run AI on Your Own Computer

If the idea of typing sensitive documents into ChatGPT makes you uneasy, you're not alone. That discomfort is exactly what pushed a small open-source project called PrivateGPT from a weekend proof-of-concept into one of the most-watched AI repositories on GitHub. This is PrivateGPT explained from the ground up: what it actually is, how it's changed since it first went viral, and what it takes to run AI on your own computer today, in 2026, without your data ever touching someone else's server.

This isn't a theoretical privacy exercise either. It's a real, working setup that developers and privacy-conscious teams are running right now, and by the end of this guide you'll know exactly what pieces you need and how they fit together.

What Is PrivateGPT, Really?

PrivateGPT started in 2023 as a simple script: a way to chat with your own documents fully offline, with nothing leaving your machine. It struck a nerve. The project crossed 50,000 GitHub stars and became one of the most closely followed AI repos of that year, largely because it answered a question a lot of people were quietly asking — can I get the "chat with my files" experience without handing my files to a cloud company?

The project has evolved a lot since then. What's now called PrivateGPT 1.0 is a full rebuild, and it's important to understand what it is today rather than what it was in 2023. Modern PrivateGPT is an open-source application API layer for local AI — not a chatbot app itself, and not something that runs models on its own. It sits on top of whatever local model server you're already running and gives that setup the higher-level features a real application needs: message handling, document ingestion, retrieval with citations, tool use, database querying, tabular data analysis, web search, MCP support, and more.

In plain terms: PrivateGPT is the layer that turns "I have a model running locally" into "I have an actual private AI application I can build on top of."

Why PrivateGPT Exists Separately From Ollama or LM Studio

This is the part that trips people up, so it's worth spelling out clearly. Tools like Ollama, LM Studio, vLLM, and llama.cpp solve one specific problem: how do you actually run a language model on your hardware? They handle loading the model into memory, using your GPU or CPU efficiently, and exposing a basic API to talk to it.

PrivateGPT answers a different question: once a model is running, how do you build something useful on top of it? That's where document ingestion, citation-backed retrieval, custom tools, and structured workflows come in — the same kind of building blocks you'd get from a hosted API like Anthropic's, except running entirely on infrastructure you control.

The relationship looks like this:

Your app / agent / workflow / UI
            ↓
      PrivateGPT API
            ↓
OpenAI-compatible inference server (Ollama, llama.cpp, vLLM, etc.)

PrivateGPT doesn't replace your local inference server — it needs one to function. It connects to it over the standard /v1/chat/completions and /v1/models endpoints, meaning practically any OpenAI-compatible backend works, whether that's Ollama running on your laptop or a beefier vLLM instance on a home server.

How PrivateGPT Compares to Similar Local AI Tools

A lot of local AI tools get lumped together, but they're solving different layers of the same stack. Here's how PrivateGPT actually differs from the tools people usually compare it to.

If Ollama is the engine, and Open WebUI is a car built around a similar engine, PrivateGPT is the chassis and electronics you'd use to build your own car — it's API-first, not a polished end-user product out of the box, though it does ship a built-in workbench UI at /ui for testing.

Setting Up PrivateGPT: What You Actually Need

Getting PrivateGPT running involves two separate pieces: the inference server that runs the model, and PrivateGPT itself, which sits on top of it.

Step 1: Install a Local Inference Server

Ollama is the easiest starting point for most people, since it handles GPU detection and model management automatically. A typical setup looks like this:

ollama pull qwen3.5:35b        # the LLM itself (~24 GB)
ollama pull mxbai-embed-large  # the embeddings model (~670 MB)
ollama serve

The embeddings model matters if you plan to search or query your own documents — it's what converts your text into the vector representations PrivateGPT uses for retrieval.

Step 2: Install PrivateGPT

On Linux or macOS:

curl -LsSf https://astral.sh/uv/install.sh | sh
uv tool install --python 3.11 \
  --find-links https://wheels.privategpt.dev/packages/ \
  "private-gpt[core]"

On Windows:

powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"
uv tool install --python 3.11 `
  --find-links https://wheels.privategpt.dev/packages/ `
  "private-gpt[core]"

Step 3: Point PrivateGPT at Your Model

Once both are installed, PrivateGPT connects to your running inference server through an environment variable (OPENAI_API_BASE) pointing at wherever Ollama or your chosen server is listening. From there, you can open the built-in UI, start uploading documents, and begin querying them — or start building against the API directly if you're integrating this into your own application.

Hardware: What You Actually Need to Run This Comfortably

This is where a lot of beginners get discouraged unnecessarily. You don't need a data center, but the hardware you pick genuinely determines which models are realistic to run.

• RTX 4090 (24GB VRAM) — handles 32B-parameter models comfortably and is the practical ceiling for a single consumer GPU.
• RTX 3090 (24GB VRAM, used market) — same VRAM as the 4090 at a noticeably lower price; a strong value pick if you don't need the extra speed.
• Apple Silicon Macs with 36GB+ unified memory — the unified memory architecture removes the hard VRAM wall you hit on PC GPUs. A Mac with 64–96GB of unified memory can run 70B-class models, which is where local setups start closing the gap with cloud AI.
• Storage — plan for at least 100GB of free NVMe SSD space if you want to keep a few models around. A 70B model at Q4 quantization alone takes roughly 38–42GB. NVMe matters here too: loading a 20GB model from NVMe takes seconds, while a spinning hard drive can stretch that to nearly a minute.

If you're on a modest laptop with 16GB of RAM, you're not locked out — smaller quantized models still run reasonably well — but the smoother, closer-to-cloud-quality experience really opens up once you're running 32B+ models on dedicated VRAM or generous unified memory.

What You Can Actually Build With PrivateGPT

Because PrivateGPT is API-first, the range of things people build with it looks a lot like what people build on hosted AI APIs — just running locally:

• Private document Q&A — upload contracts, research papers, or internal wikis and ask questions with cited sources, without any of it leaving your machine.
• Internal knowledge bases — point it at a company's internal documentation and get grounded answers instead of hallucinated ones.
• Secure help desks — customer or internal support tools that need to stay off shared cloud infrastructure for compliance reasons.
• Tabular and CSV analysis — query spreadsheets and structured data directly through natural language.
• Custom agents and tools — PrivateGPT supports MCP, skills, and custom tool integration, so you can wire it into scripts, internal databases, or automation pipelines the same way you would with a cloud API.

The One Privacy Nuance People Miss

Here's something worth knowing before you assume "local" automatically means "100% private" in every configuration: PrivateGPT's document pipeline runs locally, but the model answering your questions is whatever backend you've pointed it at. If that backend is a local Ollama instance, your data genuinely never leaves your machine. But PrivateGPT can also connect to remote OpenAI-compatible APIs through its openailike provider mode — and if you do that, your document chunks are being sent wherever that backend lives.

For people who want frontier-level model quality without fully trusting a standard cloud API, there are confidential-computing backends built specifically to bridge this gap — encrypting document chunks before they leave your machine and using remote attestation so you can cryptographically verify the backend hasn't been tampered with. It's a middle ground worth knowing about if pure local inference feels too limited for your use case, but you still don't want to hand raw data to a conventional cloud provider.

PrivateGPT vs. Zylon: Which One Do You Actually Need?

PrivateGPT is maintained by the team behind Zylon, and it's worth understanding the difference before you go looking for enterprise features that don't exist in the open-source version.

• PrivateGPT is the open-source application API layer — ingestion, retrieval, citations, tools, MCP, and the core building blocks.
• Zylon is the full commercial platform built around that layer, adding things like Kubernetes deployment, an integrated inference server, an API gateway, LDAP integration, SIEM audit logging, and connectors for SharePoint, Confluence, and FTP.

If you're an individual developer or a small team experimenting with local AI, PrivateGPT alone is the right starting point. If you're a regulated organization needing a packaged, supportable platform with enterprise connectors and compliance logging, that's what Zylon exists for.

Pros and Cons of Running AI This Way

Pros

• Your documents and prompts never leave your machine when paired with a local inference backend
• No per-token costs or subscription fees once your hardware is in place — just electricity
• Full control over which models you run and how they're configured
• Works with almost any OpenAI-compatible inference server, so you're not locked into one ecosystem
• Supports the same kind of building blocks — tools, citations, structured retrieval — that hosted APIs offer

Cons

• Requires real setup effort: installing Python, an inference server, and PrivateGPT itself is more involved than signing up for a cloud API key
• Model quality still lags behind the largest frontier cloud models, especially on the hardest reasoning tasks
• Decent performance requires a real hardware investment — a capable GPU or a high-memory Mac isn't optional if you want to run larger models
• The built-in UI is meant for testing, not a polished end-user product out of the box
• If you point it at a remote backend instead of a local one, the privacy guarantee depends entirely on that backend, not on PrivateGPT itself

Getting Started Without Overcomplicating It

If you're new to this, resist the urge to jump straight to a 70B model on day one. Start with Ollama and a mid-sized model your hardware can comfortably handle, get PrivateGPT talking to it, and upload a single test document. Once you've seen a real answer come back with a citation pointing at your own file, the rest of the setup — bigger models, custom tools, MCP integrations — is just building outward from something that already works.

Final Thoughts

Hopefully this leaves you with PrivateGPT explained clearly enough to actually try it, rather than just another entry on a list of privacy tools. What started as a script for chatting with documents offline has grown into a genuine application layer for local AI — one that lets you run AI on your own computer with the same kind of building blocks you'd get from a cloud API, minus the cloud. Pair it with a solid local inference server and hardware that matches your ambitions, and you end up with something that's not just private by default, but genuinely capable of doing real work.