CANDI: Contextual Alignment for Niche Domains Question Answering
Quick Answer
CANDI-QA introduces a novel dataset for evaluating large language models (LLMs) in niche domains like medical and financial sectors, focusing on context-sensitive and user-aligned answers.
Quick Take
It categorizes questions into Information Assistance and Applied Inference, assessing over ten , including MTSS-Net, a neuro-symbolic framework. Findings reveal significant challenges in achieving contextual alignment, highlighting the need for improved integration in LLMs for high-stakes applications.
Key Points
- CANDI-QA evaluates LLMs on context-sensitive answers in specialized domains.
- Questions are categorized into Information Assistance and Applied Inference.
- Over ten language models, including MTSS-Net, were assessed for performance.
- Findings indicate significant limitations in current LLMs for niche applications.
- CANDI-QA serves as a benchmark for advancing context-aware language models.
Paper Resources
📖 Reader Mode
~2 min readAbstract:The deployment of large language models (LLMs) in specialized domains like medical diagnostics and financial advisory necessitates evaluating capabilities beyond general knowledge. Traditional question-answering benchmarks often fail to capture the nuanced contextual grounding, user awareness, and domain understanding these fields require. To address this, we introduce CANDI-QA (Contextual Alignment for Niche Domains Question Answering), a novel dataset evaluating LLMs on delivering accurate, context-sensitive, and user-aligned answers in specialized settings. CANDI-QA features expert-curated question-answer pairs structured into two categories: (1) Information Assistance Questions, which are direct, factual queries requiring precise extraction, and (2) Applied Inference Questions, which are multi-hop reasoning tasks needing situational inference to generate actionable insights. We evaluate over ten diverse language models, from compact open-source to state-of-the-art proprietary systems. As a robust baseline, we present MTSS-Net, a lightweight neuro-symbolic framework combining neural retrieval with rule-based reasoning. Our findings highlight the profound challenges of achieving contextual alignment in niche domains, revealing the limitations of current LLMs without enhanced contextual or symbolic integration. Ultimately, CANDI-QA serves as a critical benchmark for advancing research in context-aware language models, stimulating the development of robust, trustworthy AI for high-stakes domains.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2607.11891 [cs.CL] |
| (or arXiv:2607.11891v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2607.11891 arXiv-issued DOI via DataCite |
Submission history
From: Madhur Thareja [view email]
[v1]
Wed, 6 May 2026 06:40:13 UTC (1,858 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.