Taming Hallucinations in Bengali Legal LLMs: RAG vs. Knowledge Graphs

Independent Research · A Multi-Domain Assessment of Hallucinations in Bengali LLMs: Covering Legal Inquiries of Bangla Gazettes Using RAG and KG (Knowledge Graph)

The Problem: Bengali Legal Text Breaks LLMs

Bengali is spoken by more than 270 million people, yet it remains a low-resource language for NLP — modern LLMs are trained overwhelmingly on English, and their Bengali coverage stays thin. On technical, legal, or numerical Bengali text, they hallucinate freely: a recent benchmark on the Bengali legal domain found that even commercial LLMs, when pressed, will fabricate act names and clause references that read plausibly to real Bangladeshi citizens.

This project tested that failure mode directly against the Bangladesh Public Procurement Rules 2025 (PPR 2025) — a 196-page legal instrument published in the Bangladesh Gazette on 28 September 2025 that governs every public-sector procurement in the country: who can approve what, which document templates apply, which procurement method triggers at which financial threshold, and what deadlines apply at each stage. The PDF itself is typeset in legacy, non-Unicode Bengali fonts (SutonnyMJ, ShonarBangla, NikoshBAN, and others), so even reading the text cleanly was step one of the problem.

Two Competing Architectures, One Benchmark

Rather than picking a side up front, the study built full, working pipelines for both dominant retrieval-grounded paradigms and ran them against the exact same 500-pair benchmark — split across standard, reference-free, and counterfactual question categories, so the systems were also tested on how well they resisted answering questions with false premises.

The Headline Result: A 9× Hallucination Reduction

Across every architecture tested, the single largest effect in the entire study was on hallucination rate. The RAG baseline (BM25 + e5 retrieval, evaluated within the KG framework's 500-question set for a fair comparison) hallucinated on 22.0% of answers. All four knowledge-graph variants landed between 2.4% and 3.2%:

Statistically, that gap is as robust as gaps get in applied NLP: bootstrap resampling put the half-width below 0.022 on hallucination for every system, and the effect size between RAG and the best KG was very large (Cohen's d ≈ 1.6), against only a small effect on answer-quality metrics like cosine similarity. In plain terms: KG calibration is a feature, not a bug — when a query maps cleanly onto the locked ontology, the graph either answers correctly or (via NLI faithfulness gating) declines to answer, rather than confidently inventing a clause number.

Best-vs-Best: The Two Paradigms Aren't Really Competing

The best RAG generator, Llama-3B fine-tuned, reached a BERTScore F1 of 82.45 and a hallucination rate of 17.0%. The best KG, Gemma-3-12B, reached a cosine similarity of 0.817 (the highest of any system, RAG included) and a hallucination rate of just 3.2%. On the one metric that's cleanly comparable across both frameworks — semantic similarity — the two systems are within 0.4 points of each other, but the KG hallucinates roughly 7× less on that same head-to-head pairing.

The two systems are not really competitors — they are complementary. KG gives precise, low-hallucination answers when the question maps cleanly onto the schema; RAG always produces something, and is the safer default when graph coverage is partial.

What Actually Broke: The Engineering Reality

The technical write-up is unusually candid about what didn't work, which is arguably the most useful part for anyone building a similar pipeline:

• Bengali OCR was the real bottleneck. Naive PDF text extraction on the legacy fonts returned garbage bytes disguised as ASCII. Tesseract was tried and rejected; the fix was rasterizing every page at 220 DPI and running EasyOCR, reaching a 98.78% clean per-page Unicode ratio.
• QLoRA fine-tuning went through five broken iterations before converging — NaN gradients from a bf16 dtype choice, a gradient-checkpointing bug, a chat-template auto-detector silently picking the wrong template for Gemma-3, a model too large to fit an A100 under NF4+LoRA, and a pad-token collision that collapsed the loss on Llama-3.1-8B. The fix was float32 compute dtype, disabled gradient checkpointing, and per-family pad-token rebinding.
• Compute had to scale up mid-project: a T4 handled the RAG pipeline fine, but the Qwen2.5-14B KG teacher needed an A100, and the Gemma-3-12B student fine-tune forced a further upgrade to a 102 GB Pro+ instance.
• LLM-generated Cypher mostly failed: of 164 LLM-fallback attempts at generating a Cypher query, only 1 was syntactically valid against Kùzu's OpenCypher restrictions — deterministic query templates carried nearly all of the actual answering.

Limitations, Reported Honestly

Conclusion & Future Work

The clearest finding survives every caveat above: for high-stakes Bengali legal question answering, knowledge graphs deliver dramatically lower hallucination than retrieval-augmented generation, at a small and often favorable cost to answer quality. The natural next step, flagged directly in the study, is a hybrid router — using the KG when entity-linking confidence and Cypher results are both high, and falling back to RAG otherwise — alongside testing the pipeline across multiple gazettes (income-tax rules, customs rules) to see how far it generalizes, and eventually packaging it as a production Gradio chatbot that returns an answer alongside its supporting clause or triple.