JUMP: Single-Pass Membership Inference on Fine-Tuned Diffusion Language Models
Quick Answer
This paper shows that The JUMP attack enhances membership inference on fine-tuned diffusion language models (dLLMs) like LLaDA-8B-Base, achieving a mean ROC-AUC of 0.90, up from 0.82 with the previous SAMA method.
Quick Take
By utilizing joint uncertainty-guided mask probing, JUMP requires only one pass through the model, significantly improving low-FPR detection across six MIMIR domains.
Key Points
- JUMP improves mean ROC-AUC from 0.82 to 0.90 on LLaDA-8B-Base.
- Utilizes joint uncertainty-guided mask probing for efficient scoring.
- Requires only one selector pass and one scoring pass per model.
- Substantially enhances low-FPR detection in membership inference attacks.
- Targets fine-tuned discrete diffusion language models (dLLMs).
DeepSignal Analysis
What happened
The JUMP attack improves membership inference on fine-tuned diffusion language models, achieving a mean ROC-AUC of 0.90, surpassing the previous SAMA method's 0.82. This method utilizes joint uncertainty-guided mask probing, allowing for a single pass through the model, which enhances low false positive rate detection across six MIMIR domains.
Key evidence
- JUMP achieves a mean ROC-AUC of 0.90 on fine-tuned LLaDA-8B-Base, an improvement from SAMA's 0.82.
- The JUMP method employs joint uncertainty-guided mask probing, requiring only one pass through the model for scoring.
- JUMP significantly enhances low false positive rate detection across six MIMIR domains, indicating its effectiveness in membership inference.
Why it matters
The advancement of JUMP in membership inference attacks highlights the vulnerabilities of fine-tuned diffusion language models. As these models become more prevalent in applications, understanding their susceptibility to such attacks is crucial for ensuring data privacy and security. The ability to conduct effective membership inference with fewer model passes could lead to more efficient attacks, raising concerns for organizations relying on these models.
Paper Resources
Source Excerpt
Membership inference attacks (MIAs) test whether a candidate example appeared in a model's training data. We study MIAs for fine-tuned discrete diffusion language models (dLLMs), where membership means inclusion in the target model's fine-tuning set. Unlike autoregressive language models, dLLMs allow an attacker to choose arbitrary mask sets and obtain token distributions for all masked positions in parallel. The prior dLLM attack, SAMA, follows a natural loss-mimicking strategy by averaging rec
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →RAIL Guard: Closing the Evaluation-to-Remediation Gap in Responsible AI for Agents
RAIL Guard introduces a closed-loop AI pipeline for large language models (LLMs) that evaluates outputs across eight dimensions and iteratively remediates failures, achieving 96.9% convergence compared to 49.1% for traditional block-and-retry methods. The system reduces unsafe agent executions by 33% without impacting task completion and is available as open-source SDKs.