Where Does RAG Actually Cost You Money? I Decided to Stop Guessing.

RAG has become my current obsession. Every day I dig one layer deeper into my own pipeline, and almost every day something I "knew for sure" turns out to be wrong.

Here's how this started.

I used to read business and finance content on the side — nothing serious, just enough to pick up how companies actually survive. One thing stuck with me: sustainable businesses don't just chase revenue, they obsess over knowing exactly where their cost goes. Burn less fuel, go further. That's the whole game.

I decided to point that same habit at my own RAG pipeline. Stop assuming. Actually dig into every stage and see where the money really goes.

What I found surprised me.

The belief I never questioned

"Embedding is expensive — avoid re-embedding at all costs."

I didn't just hear this once. I heard it everywhere. Every dev I talked to repeated some version of it. So when I was building a semantic cache, I designed around that fear — I assumed a Redis-based semantic search layer would cost more than it saved, purely because "embedding = expensive" was sitting in the back of my head as gospel. I ended up leaning heavily on a plain key-value cache instead, because it felt like the "safe" choice.

I later realized I had optimized for the wrong bottleneck. The savings from avoiding embeddings were tiny compared to the recurring costs sitting elsewhere in the pipeline.

So I priced every stage, step by step, on a real document going through my pipeline.

Turns out embedding isn't the expensive part at all. It's one of the cheapest stages in the entire pipeline. Using OpenAI's current embedding pricing, a typical 1,000-page document in my tests came out to roughly $0.18 to embed — one time.

Once I realized embeddings weren't the bottleneck, I started following the pipeline stage by stage to see where the costs were actually hiding.

Walking the pipeline, looking for where the money really goes

  Upload PDF
      │
      ▼
  Extract + clean text        ── billing quietly starts here
      │
      ▼
  Is this doc already known?  ── the check that saves you real money
      │
   ┌──┴──┐
  same   changed
   │        │
  skip    continue
   │        │
   ▼        ▼
  Chunk only what changed
      │
      ▼
  Attach metadata
      │
      ▼
  Embed (cheap — ~18¢ per 1,000 pages)
      │
      ▼
  Store in vector DB          ── millions of chunks, running 24×7
      │
      ▼
  Index + retrieve            ── costs more than embedding, at scale
      │
      ▼
  LLM reads chunks, answers   ── the real money and latency eater

The dedup check matters more than people give it credit for. Most knowledge bases don't change completely from one day to the next — they evolve slowly. In my case, roughly 80% of documents stay the same and only about 20% actually change. That's close to a classic 80/20 split.

That changes how I think about ingestion. If 80% of the knowledge base hasn't changed, why should 100% of it be processed again?

Here's the mistake I almost made: when a document updates, the lazy approach is to treat it as a brand-new file and re-run the entire thing — full history included. That's a completely different (and much more expensive) problem than it needs to be. The right approach is to diff against what you already have and only touch what actually changed.

Chunking follows the same logic. If only a section of a document changed, there's no reason to re-chunk or re-embed the whole document. Re-process just the chunks that changed. This is where careful metadata (page, section, version) earns its keep — it's what makes "find only the changed part" possible in the first place.

The vector store is a different cost game entirely. You're not paying for the vectors themselves. You're paying for the infrastructure that keeps millions of them searchable in milliseconds — running whether or not anyone's asking a question right now. At small scale this is cheap. At production scale, it's a real recurring line item — and indexing/retrieval costs add up faster than embedding ever did. Unlike embeddings, which are generated once, the vector database has to stay online, maintain indexes, and answer similarity searches for every single query.

And then there's the LLM. This is the actual heavyweight. It's not a one-time cost like embedding — it runs on every single question, from every user, every day. It's also where most of the latency lives. This is the piece I want to spend the most time on going forward, because it's where both the money and the user experience are won or lost.

The cost profile looks nothing like I expected

After walking through the entire pipeline, here's what actually became clear:

• Embeddings are mostly a one-time ingestion cost.
• Vector databases introduce recurring infrastructure costs.
• The LLM becomes a recurring cost on every single user request.

Those are three completely different cost models. Optimizing one — like avoiding re-embedding — doesn't do anything for the other two. That's the trap I fell into with the semantic cache. I was optimizing the cheapest, one-time stage while the recurring stages kept running in the background regardless.

Different costs need different optimizations. One-time ingestion costs benefit from deduplication and incremental updates. Recurring infrastructure costs benefit from efficient indexing and storage. Recurring inference costs benefit from caching, better retrieval, and smaller models where they fit. Once I saw these as three separate problems instead of one, the optimization choices became a lot clearer.

Where I'm at right now

I don't have all the answers yet. I'm learning this by taking my own production system apart, piece by piece, and I'll probably get some things wrong along the way. If you've built RAG systems at scale and I'm off on something, I'd genuinely rather know than keep repeating a wrong assumption — the same way I was repeating "embedding is expensive" for months without ever checking it myself.

Next thing I want to dig into: why vector database infrastructure becomes the next major cost after embeddings, and how I'm thinking about reducing it.

Less noise, more action. Let's dig.