top of page

RAG Is Not Dead. You Just Don't Understand What It Became.

A viral thread is making the rounds this week claiming RAG is dead.


That long-context windows killed it. That vector databases were a two-year detour. That your retrieval pipeline is technical debt.


I read every claim. I ran the math. I checked the research.


Here is what is actually true - and what could cost you real money if you believe it.



The Thread Gets Three Things Right


1. Context windows did get massive.


Claude Opus 4.7 shipped on April 16 with a 1M token context window. Gemini hit 1M a year earlier. This is real progress.


2. Prompt caching changes the economics.


Anthropic's cache reads cost 10% of standard input pricing. That is a genuine 90% discount on repeated queries over the same corpus. For small, static workloads, this makes long-context cheaper than maintaining a full retrieval stack.


3. Naive RAG was always fragile.


Chunking a PDF into 500-token pieces, embedding each piece, and hoping cosine similarity finds the right answer? That was always a compromise. The thread is right that this specific pattern is losing relevance.


If your entire use case is "I have one PDF and I ask it questions ten times a day," then yes - dump it in context, enable caching, and move on.


But that is not the use case the thread is arguing against. And this is where the argument falls apart.



What the Thread Gets Wrong


The "Lost in the Middle" Problem Is Real and Unsolved


The article claims 99%+ recall at 1M tokens, citing needle-in-a-haystack benchmarks.


Here is what those benchmarks actually test: the model finds a single planted sentence ("The special magic number is 7392") buried in pages of unrelated text.


That is not retrieval. That is a parlor trick.


Real-world queries require multi-hop reasoning. "What was our Q3 cost variance compared to the budgeted rate from the February board deck?" That answer lives across multiple sections, in different documents, with context that depends on tables, footnotes, and prior definitions.


Research published through early 2026 consistently shows that LLMs deprioritize information in the middle of long contexts. This is not a bug being patched. It is a structural characteristic of transformer attention mechanisms. Bigger windows make the middle bigger - and the problem worse.


The needle benchmark says "I can find a sentence." Enterprise workloads say "I need to reason across 47 pages of interconnected data." These are fundamentally different tasks.


The Cost Math Is Cherry-Picked


The thread models 100 queries per day over a 500K-token corpus.


Let me model what production actually looks like.


Scenario: SaaS platform, 200 tenants, 50 queries per tenant per day.


That is 10,000 queries per day.


With long-context + caching, you are sending 500K tokens of context per tenant session. Even at cached rates ($0.50/M tokens for reads on Opus 4.7), you are paying for 500K tokens on every cache miss (tenant switch, TTL expiry, corpus update).


With RAG, you retrieve 5-10 relevant chunks (roughly 2K-5K tokens) per query. The rest of your corpus never touches the model.


At 10,000 queries/day:

  • Long-context: 500K tokens x 10,000 = 5B input tokens/day. Even 90% cached, the 10% cache misses alone run $2,500/day.

  • RAG: 5K tokens x 10,000 = 50M input tokens/day = $250/day at standard rates. No cache dependency.


RAG is 10x cheaper at this scale. At 100,000 queries/day, it is 100x cheaper.


The thread's math works for a solo developer querying their codebase. It collapses the moment you build a product other people use.


Data Freshness Is Not an Edge Case


The article says RAG still wins for "corpora that change hourly."


Name a production system where the data does not change.


CRM records update every call. Support tickets arrive every minute. Financial data refreshes daily. Compliance documents get amended quarterly. Employee records change with every hire, promotion, and departure.


With long-context, every data change means re-ingesting the entire corpus into a new prompt. Cache invalidation is not free - it is the hardest problem in computer science for a reason.


With RAG, you update one document, re-embed one chunk, and the index reflects the change in seconds. No full re-read. No cache bust.


If your data never changes, you do not need AI. You need a filing cabinet.


Multi-Tenancy Is Not a Footnote


The thread dismisses multi-tenant data isolation as an edge case.


In SaaS, multi-tenancy is the entire business model.


When you stuff all tenant data into a single model context, you need absolute guarantees that Tenant A's confidential financial data never leaks into Tenant B's response. Prompt injection, context bleed, and attention leakage are all active research problems with no production-grade solutions at 1M token scale.


RAG solves this architecturally. Each tenant's data lives in isolated index partitions. Query-time filtering ensures one tenant's embeddings never enter another tenant's retrieval set. The isolation happens before the model sees anything.


This is not an edge case. This is how you avoid getting sued.


Latency Matters


Processing 500K tokens of context takes seconds - sometimes 10+ seconds for complex reasoning on Opus 4.7.


A well-optimized RAG pipeline retrieves in under 100ms and delivers a response in 1-2 seconds total.


For interactive applications, chatbots, and real-time decision support, that difference is the gap between a product people use and a product people abandon.



What RAG Actually Became


The thread argues against 2023 RAG. Fair enough. 2023 RAG deserved criticism.


But RAG in 2026 is not "chunk, embed, top-k, pray."


Modern retrieval architectures look like this:


Hybrid search: combining dense vector similarity with sparse BM25 keyword matching. Each method catches what the other misses.


Cross-encoder reranking: a dedicated model re-scores retrieved documents for actual relevance, not just semantic proximity. This eliminates the "chunk 11 problem" the thread describes.


Contextual chunking: each chunk carries metadata about its position, its parent document, and its relationship to adjacent content. The chunking-destroys-context critique applied to naive fixed-window chunking. Modern systems preserve structure.


Agentic retrieval: the model decides what to retrieve, evaluates whether the retrieved context is sufficient, and iterates if needed. Retrieval is a tool the model uses, not a fixed pipeline imposed on it.


Graph-augmented retrieval: knowledge graphs connect entities across documents, enabling the multi-hop reasoning that long-context promises but underdelivers on at scale.


The thread describes replacing a complex stack with 30 lines of Python.


Thirty lines of Python is a demo. Production is observability, fallback strategies, access control, audit logging, cache management, and graceful degradation. Whether you build with RAG or long-context, you need all of these. The infrastructure does not disappear. It just moves.



The Actual Lesson from the Bitter Lesson


The thread cites Rich Sutton's 2019 essay to argue that long-context models will eat RAG the way they ate other scaffolding.


Sutton's actual point is subtler: general methods that leverage computation outperform hand-crafted heuristics over time.


RAG is not a hand-crafted heuristic. RAG is a general method. It is search - one of the two scalable approaches Sutton explicitly endorses. Retrieval is how you give a model access to arbitrary-scale knowledge without requiring all of it to fit in working memory at once.


The Bitter Lesson does not predict that models will internalize all information. It predicts that models will get better at using search. That is exactly what agentic retrieval delivers.


If anything, the Bitter Lesson argues for RAG, not against it.



What You Should Actually Build


The real answer is not RAG or long-context. It is knowing when to use each.


Use long-context when:

  • Your corpus fits in the window (under 1M tokens)

  • Your data is static or changes infrequently

  • You have a single tenant or a small user base

  • Query volume is low (under 500/day)

  • You need deep reasoning across an entire document


Use RAG when:

  • Your corpus exceeds the context window

  • Data changes frequently

  • You serve multiple tenants with data isolation requirements

  • Query volume is high

  • Latency matters

  • You need auditability and source attribution

  • Compliance requires you to control exactly what data reaches the model


Use both when:

  • RAG retrieves the most relevant context, then feeds it into a long-context model for deep reasoning

  • This is where the industry is actually heading


The middle of the stack is not dead. It is evolving. The companies that abandon retrieval infrastructure because a LinkedIn thread told them to will spend the next 18 months rebuilding it.



The Bottom Line


"RAG is dead" is engagement bait wrapped in half-truths.


Long-context windows are a genuine breakthrough. Prompt caching is a genuine cost innovation. For simple, single-user, static-corpus use cases, you genuinely do not need a retrieval pipeline anymore.


For everything else - which is most of production AI - RAG is not dead.


It just grew up.



Hazwan is COO of Neuramerge Sdn Bhd, a Malaysia-based AI training and digital tools company. He builds multi-tenant AI systems for enterprise clients and has opinions about people who declare entire architectural patterns dead based on benchmark scores.


 
 
 

Comments

bottom of page