Beyond Safe Data: Pretraining-Stage Alignment with Regular Safety Reflection
Quick Answer
The study introduces Safety Reflection Pretraining, enhancing safety alignment in LLMs by integrating self-monitoring during pretraining.
Quick Take
Experiments with a 1.7B model show improved safety classification accuracy and reduced attack success rates compared to traditional data filtering methods.
Key Points
- Safety Reflection Pretraining integrates short safety reflections into pretraining corpora.
- 1.7B models pretrained on FineWeb-Edu showed improved safety classification accuracy.
- The method significantly reduced success rates of inference-stage and finetuning attacks.
- A synthetic environment, MedSafetyWorld, demonstrated the method's effectiveness against unsafe behaviors.
- Findings suggest pretraining should shape behaviors from safe data, not just filter it.
Paper Resources
Source Excerpt
To achieve deeper safety alignment for (LLMs), recent efforts have studied how to push safety interventions earlier into the pretraining stage, primarily by filtering unsafe data or rewriting it into safer forms. We argue that pretraining-stage alignment should go beyond making the data safe: LLMs may compose seemingly benign knowledge and capabilities into unsafe behaviors. To this end, we propose Safety Reflection Pretraining, a pretraining-stage alignment method which re
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →HOBA: Hierarchical On-Policy Bidding Agents for Adaptive Online Advertising
HOBA (Hierarchical On-policy Bidding Agents) is a novel hierarchical reinforcement learning framework that enhances online advertising bidding systems by improving adaptability and reducing hyperparameter tuning costs. It utilizes a for hyperparameter inference, a SARSA agent for expert model selection, and a dynamic expert pool for bid execution, achieving a +3.6% increase in target cost during large-scale deployment and outperforming state-of-the-art baselines on AuctionNet.