LLM-FACETS: A Privacy-Preserving Framework for Evaluating LLM Transparency and Accountability
Quick Answer
LLM-FACETS is an open-source framework designed to evaluate Large Language Models (LLMs) for transparency and accountability, accessible via a browser interface.
Quick Take
It supports three user profiles—technical experts, domain experts, and compliance officers—while ensuring data privacy through self-hosted metrics and explicit API interactions. The framework enhances reproducibility and accountability in AI by allowing integration of new metrics without altering the evaluation pipeline.
Key Points
- Framework designed for non-technical practitioners to audit outputs effectively.
- Utilizes deterministic metrics like BLEU and ROUGE without external data transmission.
- Incorporates token-level visualizations for epistemic uncertainty and multi-judge consensus.
- Open-source implementation allows integration of new metrics seamlessly.
- Cross-validation of 18 metrics ensures reproducibility against reference libraries.
Paper Resources
Article Content
From source RSS / original summaryarXiv:2605. 31167v1 Announce Type: new Abstract: Assessing whether outputs are factually grounded, epistemically calibrated, and methodologically reproducible is a prerequisite for responsible AI deployment.
Yet auditing LLMs remains inaccessible to non-technical practitioners: existing tools require programming expertise and non-trivial environment setup, and cloud-hosted platforms transmit evaluation data to external services, creating barriers for domain experts and compliance officers legally responsible for AI oversight.
We introduce LLM-FACETS (LLM FActuality Cross-EvaluaTion System): an open-source framework with a browser-accessible interface and a plugin architecture, structured around three practitioner profiles (technical experts, domain experts, compliance officers) that mirror the stakeholder categories identified in the EU AI Act and the NIST AI Risk Management Framework.
The architecture makes data flows explicit: deterministic metrics (BLEU, ROUGE, BERTScore) run entirely within the self-hosted server with no outbound transmission; LLM-judge metrics contact external APIs explicitly, with users retaining full credential control.
The framework operationalizes transparency through three mechanisms: token-level log-probability visualization for epistemic uncertainty, multi-judge consensus to mitigate judge bias, and Triad metrics (Faithfulness, Answer Relevance, Context Relevance) to detect and localize hallucinations. A plugin architecture allows any new metric or dataset to be integrated without modifying the evaluation pipeline.
The open-source implementation enables cross-checking across multiple metrics targeting the same property, ensuring reproducibility and decoupling AI accountability from the teams building the systems assessed. We verify the framework through cross-validation of 18 metric implementations against canonical reference libraries.
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →HOBA: Hierarchical On-Policy Bidding Agents for Adaptive Online Advertising
HOBA (Hierarchical On-policy Bidding Agents) is a novel hierarchical reinforcement learning framework that enhances online advertising bidding systems by improving adaptability and reducing hyperparameter tuning costs. It utilizes a for hyperparameter inference, a SARSA agent for expert model selection, and a dynamic expert pool for bid execution, achieving a +3.6% increase in target cost during large-scale deployment and outperforming state-of-the-art baselines on AuctionNet.