Deployment-Centered Evaluation: Predicting Query-Level Rejection Risk in a Clinical LLM System
Quick Answer
This study introduces a deployment-centered evaluation method for predicting user rejection of LLM responses in clinical settings, achieving an AUROC of 0.719.
Quick Take
This study introduces a deployment-centered evaluation method for predicting user rejection of LLM responses in clinical settings, achieving an AUROC of 0.719. By incorporating deployment-specific context, such as provider type and department, the model enhances rejection risk prediction, demonstrating its utility in real-world applications like guardrail triggering.
Key Points
- Developed a pre-response classifier for predicting user rejection risk in clinical LLM systems.
- Achieved an AUROC score of 0.719 over 4.5 months of user feedback.
- Utilized deployment-specific context to improve prediction accuracy.
- Demonstrated potential applications in guardrail triggering and user abstention.
- Addresses limitations of static benchmarks in evaluating clinical LLM systems.
Paper Resources
Article Content
From source RSS / original summaryarXiv:2606. 12702v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly integrated into clinical systems, making it essential to evaluate the real-world utility of these systems. However, static benchmarks tend to measure correctness rather than user acceptance, aggregate performance across queries, and require densely annotated datasets -- leading to major blind spots for evaluating clinical systems.
In this work, we perform a deployment-centered evaluation of an LLM system embedded within electronic health records at an academic medical center, where user feedback is sparse but closely reflects the deployment conditions. Specifically, we train a pre-response classifier that estimates the risk that a future interaction will result in the user rejecting the LLM response, based on query content and deployment-specific context available before generation. We conduct a prospective analysis of our model over 4.
5 months of user feedback, finding that our prediction model achieves an AUROC of 0. 719. Further, we estimate the benefit of such predictions in two downstream use cases (guardrail triggering and abstention). Our key conceptual insight is that making use of deployment-specific context (i. e. , the provider type, department name, language model used for response), as opposed to only query content, improves the ability to predict whether the user will reject the system output.
Altogether, our empirical case study demonstrates the feasibility of predicting user rejection using deployment-specific context, opening the door to targeted guardrails.
Reader Mode unavailable (could not extract clean content).
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →Arbor: Tree Search as a Cognition Layer for Autonomous Agents
Arbor introduces a multi-agent framework utilizing structured tree search for optimizing LLM inference, achieving up to 193% throughput-latency improvement compared to vendor-optimized systems. It employs an Orchestrator and Critic agent for stability and coordination, demonstrating hardware-agnostic performance with minimal variance.