GENIE: A Fine-Grained Measure for Novelty
Quick Answer
The paper introduces GENIE, a fine-grained metric for assessing the novelty of model-generated content, addressing the shortcomings of holistic metrics.
Quick Take
It demonstrates that GENIE effectively captures task-specific features of novelty, providing insights into model creativity and the impact of mitigation methods.
Key Points
- GENIE measures novelty in model outputs with task-specific features.
- Holistic metrics fail to capture the complexity of novelty effectively.
- The study evaluates the effectiveness of methods aimed at enhancing creativity.
- GENIE provides insights into which properties contribute to novelty.
Paper Resources
Article Excerpt
From source RSS / original summaryarXiv:2606. 12790v1 Announce Type: new Abstract: have consistently demonstrated a lack of creativity and diversity across tasks. Prior work has focused on addressing whether models are capable of generating creative outputs. Here, we aim to consider novelty and investigate what makes model-generated content novel or not novel in a task-specific manner.
We propose a fine-grained evaluation metric GENIE to measure the novelty of responses along task-specific features with respect to a population of responses. We show that unlike GENIE, holistic metrics struggle to capture the high-dimensionality of novelty and do not provide insight on which properties they target. Finally, we use GENIE to measure the effectiveness of mitigation methods that address creativity to better understand where these methods can improve novelty.
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.