CSE 533 — Machine Learning · North South University
Co-authored with Zannatul Islam Proma, supervised by Dr. Mohammad Ashrafuzzaman Khan.
The Problem: Chatbots Are Becoming a Primary Information Source
ChatGPT, Gemini, Grok, Claude, and DeepSeek are increasingly where people turn first for information — to study, to make decisions, to answer everyday questions. But how reliable is that information, really? Chatbots are known to occasionally "hallucinate" false or misleading answers, their training data has a knowledge cutoff that can leave them behind a fast-changing world, and it's often unclear to an end user how a given chatbot was trained or when its knowledge was last updated. This project set out to build a systematic way to actually measure that reliability, rather than relying on anecdote.
Why This Matters
- Trust and accountability: as these systems become more embedded in daily life, verifying the reliability of what they tell people is an ethical AI-development concern, not just a technical one.
- Comparative analysis: knowing how different chatbots stack up on information quality helps users make better choices and gives developers a concrete target to improve against.
- Novelty: a systematic method for evaluating accuracy, authenticity, and up-to-dateness together — rather than accuracy alone — is a genuinely underexplored angle.
- Practical impact: the resulting framework could inform user-facing guidelines, evaluation tooling, or awareness campaigns about how to critically read chatbot answers.
A Proposed Evaluation Framework
The project's core contribution is a four-stage methodology for scoring chatbot answers along three dimensions — accuracy (does the answer factually match the ground truth?), authenticity (does it look grounded in real sources, or fabricated?), and up-to-dateness (does it reflect current information?):
Ground Truth Sources
Because "accuracy" is only meaningful relative to something, the methodology anchors every query to a domain-appropriate authoritative source:
| Query Domain | Ground Truth Source |
|---|---|
| General knowledge | Britannica, Wikipedia |
| Current affairs / news | The New York Times, Associated Press, Reuters, BBC |
| Technical / scientific | IEEE Xplore, arXiv, PubMed |
| Policy / governance | Government websites and official statistics |
| Contested / viral claims | PolitiFact, Snopes |
The Harder Problem: Making "Reliability" Measurable
Defining the methodology surfaced the project's real difficulty — accuracy, authenticity, and up-to-dateness are inherently qualitative judgments, and turning them into something a script can score consistently is genuinely hard:
- Subjectivity and measurability: precise, operational definitions had to be written for each dimension before anything could be scored.
- Ground truth availability: authoritative, current sources aren't equally accessible across every domain a chatbot might be asked about.
- A moving target: both "current events" and the chatbots themselves change constantly, so the evaluation had to account for versioning and reproducibility.
- Scale: manually checking every response against ground truth doesn't scale, which is why the methodology leans on NLP (NLTK, spaCy, Transformer-based similarity models) for semi-automated extraction and comparison — while accepting that authenticity judgments in ambiguous cases still need a human in the loop.
The interesting research problem here wasn't picking a winner among five chatbots — it was designing a scoring methodology precise enough that a "winner" would even mean something.
Scope of this work: this project produced a fully specified evaluation methodology — query design, ground-truth sourcing, NLP-based comparison, and a scoring rubric — rather than a completed head-to-head benchmark. Running that framework at scale across all five chatbots and publishing the resulting scores is the natural next phase.
Ethical Considerations
The evaluation design was kept low-risk by working only with publicly available chatbot responses, with explicit attention to fairness — ensuring the query set and scoring criteria didn't systematically favor any one chatbot's response style over another's substance.
Conclusion
Reliable information from AI chatbots isn't something users can currently take for granted, and this project's contribution is a concrete, reproducible way to start measuring that reliability rather than debating it anecdotally. The next step is execution: running the framework across a real query set, publishing comparative scores across Claude, ChatGPT, Gemini, Grok, and DeepSeek, and using the results to inform how these tools should be used — and improved.
Researching LLM Evaluation or AI Reliability?
I'd welcome a conversation about chatbot evaluation methodology, hallucination detection, or applied NLP more broadly.
Contact Me