AI‑Powered Document Search That Delivers Answers Across Your Enterprise

The Challenge

A data‑intensive enterprise had amassed hundreds of quarterly reports, financial filings and internal memos. Analysts needed to answer questions such as “What was our Q2 2025 revenue?” or “What factors drove the revenue increase last year?” Gathering and synthesising these insights across disparate PDFs was slow and error‑prone. Traditional RAG systems retrieved isolated paragraphs but struggled to connect the dots between documents, leading to incomplete or misleading answers.

Our Approach

We set out to build a modern document search system that could answer both simple lookups and complex causal questions. The pipeline begins with Azure Document Intelligence: this service extracts text, tables and layout from PDFs, enabling us to index content and metadata. Document Intelligence is designed to analyze images and PDFs and detect content, layout and semantic elements【887842370382046†L360-L377】; it also automates data extraction from complex business documents such as forms and invoices【311226031213347†L18-L19】【311226031213347†L146-L147】.

Once the content is extracted, we organize it as a knowledge graph. Unlike naive RAG that stores independent text chunks, GraphRAG extracts entities and relationships, builds nodes and edges, and performs graph‑aware retrieval. This allows the system to follow multi‑hop paths and retrieve context that explains how facts relate【761941700469273†L116-L132】. By traversing connections between entities, GraphRAG excels at complex queries and reduces hallucinations【761941700469273†L151-L162】.

Key Innovations

• Question classification: incoming questions are classified into three categories — direct lookups (Level 1), multi‑hop reasoning (Level 2) and strategic/causal reasoning (Level 3). This determines whether we perform a simple vector search, traverse the knowledge graph or synthesise evidence across multiple reports.
• Hybrid retrieval with weighted RRF: we fuse several retrieval methods — lexical BM25, semantic vector search and curated “community insights” — using weighted reciprocal rank fusion. The algorithm scores each document based on its rank in each list and adds the weighted contributions, rewarding documents that appear near the top across multiple lists【379698102144572†L249-L267】【983907171868975†L51-L77】. We adjust per‑retriever weights to emphasise semantic or keyword signals depending on the question type【983907171868975†L74-L87】.
• Semantic reranking: after fusion, a cross‑encoder reranker scores each candidate by jointly encoding the query and document. Cross‑encoders provide a deep relevance score and reorder the results【110680732246318†L642-L657】, dramatically improving precision for complex questions.
• Two‑stage retrieval pipeline: our system first gathers a broad candidate pool via hybrid retrieval and weighted RRF, then applies the semantic reranker for final ranking. This combination delivers both recall (finding all relevant passages) and precision (surfacing the best ones). It outperforms naive RAG and single‑method retrieval by a wide margin.

Evaluation & Benchmarks

We created a repeatable offline evaluation to measure retrieval quality independently of the generation model. Domain experts wrote 50 questions at all three complexity levels and identified the correct answer passages. We also used LLM prompts to generate additional question–answer pairs and assigned graded relevance labels (0 = irrelevant, 1 = relevant, 2 = highly relevant). To evaluate ranking, we used nDCG@10 — Normalized Discounted Cumulative Gain. NDCG sums the relevance of retrieved items while applying a logarithmic penalty for lower ranks, ensuring that highly relevant results at the top count more【785398683972972†L161-L173】. It normalizes DCG by the ideal DCG, yielding values from 0 to 1, where 1 indicates a perfect ranking【785398683972972†L193-L195】.

The table below summarises the improvement across four configurations: baseline RAG, classification‑aware RAG, GraphRAG with weighted RRF, and GraphRAG with weighted RRF plus semantic reranking.

By combining weighted reciprocal rank fusion and semantic reranking, we achieved a 50% improvement in ranking quality over the baseline and reduced analysts’ research time by more than 60% — delivering concise, trustworthy answers in seconds.

Business Impact

• Faster insights: The system surfaces revenue figures, causal factors and strategic trends instantly, cutting research time by 60% and freeing analysts for higher‑value work.
• Improved accuracy: Graph‑aware retrieval and reranking reduce hallucinations and ensure answers are grounded in relevant evidence.
• Knowledge reuse: Summarised community reports and cross‑document links turn previously siloed PDFs into a searchable knowledge base.
• Scalable platform: Built on Postgres, the architecture scales to thousands of documents and supports future self‑service bots and agents.

Ready to Transform Your Document Search?

If you’re drowning in reports and need rapid, accurate answers, we can help. Our experts will assess your data landscape and design a custom GraphRAG solution tailored to your business.

• Get a free search audit — we’ll analyse your document archive and outline a roadmap to smarter retrieval.
• See a live demo — experience how GraphRAG and weighted RRF deliver superior results.
• Book a strategy call — discuss your unique challenges with our AI consultants.