RuleChef: Grounding LLM Task Knowledge in Human-Editable Rules
Quick Answer
RuleChef is an open-source framework that leverages large language models to generate and iteratively improve executable rules for NLP tasks like text classification and Named Entity Recognition.
Quick Take
RuleChef is an open-source framework that leverages large language models to generate and iteratively improve executable rules for NLP tasks like text classification and Named Entity Recognition. By synthesizing rules from task descriptions and labeled examples, it creates a fast and inspectable rule system, enhancing performance through human feedback and additional examples.
Key Points
- RuleChef generates executable rules for NLP tasks using large language models.
- The framework improves rules based on human feedback and additional examples.
- It can bootstrap rules from observed input-output pairs of existing models.
- Preliminary evaluations show effectiveness in classification and NER tasks.
- RuleChef is available as open-source software under Apache 2.0.
Paper Resources
Article Excerpt
From source RSS / original summaryarXiv:2607. 01293v1 Announce Type: new Abstract: We present RuleChef, a framework that uses large language models (LLMs) to generate executable rules for NLP tasks such as text classification, Named Entity Recognition (NER), or relation extraction. Rules are generated based on a task description and a set of labeled examples, then they are iteratively improved based both on additional examples and on human feedback overexisting rules.
RuleChef can also be used to bootstrap rules using the observed input-output pairs from any existing model for a given task. LLMs are used only at learning time, synthesizing rules and iteratively patching them based on failures measured on a held-out split. The result of this process is a fast, deterministic, and inspectable rule system. Preliminary evaluation is performed on both classification and NER tasks. We release RuleChef as open-source software under an Apache 2. 0
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →Quantifying Prior Dominance in Systems
The study introduces the Normalized Context Utilization (NCU) metric to evaluate Retrieval-Augmented Generation (RAG) systems, revealing that Small Language Models (SLMs) outperform larger models in factual extraction. The findings indicate that traditional scaling laws yield diminishing returns, with a commercial API frequently failing against adversarial evidence due to systemic confidence collapse.