Over the last few weeks I've been redesigning the retrieval pipeline for an AI knowledge system.

Initially, the architecture was fairly typical:

User Question │ ▼ Vector Search │ ▼ Top K Chunks │ ▼ LLM

It worked well while the knowledge base was small.

As more documents were added, I started seeing a few recurring problems:

More irrelevant chunks being retrieved.

Larger prompts and increasing token costs.

Multiple documents discussing the same topic competing with each other.

Vector search returning semantically similar chunks from documents that weren't actually the best source of truth.

I realized the problem wasn't vector search itself.

It was deciding what should be searched before semantic retrieval even began.

Instead of treating every document equally, I separated the system into two independent stages.

Ingestion

During document upload, every document is processed once.

The pipeline extracts structured metadata including:

document type

business role

departments

topics

planner summary

retrieval keywords

authority score

importance score

answerable questions

That information is stored in a SQL registry, while document chunks and embeddings are stored separately in a vector database.

Document │ ▼ Metadata Extraction │ ├────────► SQL Registry │ └────────► Chunking + Embeddings │ ▼ Vector Store

Query Time

Instead of querying the vector database immediately, the retrieval flow became:

User Question │ ▼ Intent Analysis │ ▼ Registry Ranking │ ▼ Retrieval Planner │ ▼ Selected Documents │ ▼ Vector Search │ ▼ Context Assembly │ ▼ LLM

The registry acts as a lightweight ranking layer.

Rather than searching every document, it produces a ranked candidate set based on signals such as:

authority

importance

approval state

departments

document role

planner summary

retrieval keywords

topic overlap

The planner then decides which documents should actually participate in vector retrieval.

The vector database never searches the entire workspace anymore.

Only the planner-selected documents.

A few other changes made a noticeable difference:

similarity thresholding before accepting chunks

duplicate chunk removal

token budgeting before generation

dynamic chunk limits based on query type

ranking retrieved chunks before assembling context

One interesting observation was that improving retrieval often had a larger impact on answer quality than changing the generation model.

I'm curious whether others have moved beyond "vector search first" architectures.

If you've experimented with retrieval planning, metadata-driven routing, or hybrid retrieval systems, I'd be interested in hearing what worked and what didn't.

Vector search isn't the hard part. Deciding what should be searched is