How easily can Russian propaganda fool AI models? A new benchmark finds out
Quick Answer
The Institute of the Estonian Language has developed a benchmark to assess AI language models' vulnerability to Russian propaganda, revealing significant susceptibility in models like GPT-3 and BERT.
Quick Take
This research highlights the urgent need for improved detection mechanisms to mitigate misinformation risks in AI applications.
Key Points
- The benchmark specifically tests AI models' responses to Russian propaganda.
- Models like GPT-3 and BERT showed notable susceptibility.
- Findings indicate a critical need for enhanced misinformation detection.
- The research aims to inform developers on potential vulnerabilities.
- Results could influence future AI training and deployment strategies.
📖 Reader Mode
~2 min readThe Institute of the Estonian Language has released a benchmark measuring how susceptible AI language models are to Russian propaganda. Sixty models were tested with 75 questions in three languages covering 14 propaganda narratives, phrased in neutral, biased, and manipulative ways. Each answer is scored on a scale of 1 to 5, where 1 means the model repeats Russian talking points.
A calibrated Claude Opus 4.5 served as the evaluation model, validated by disinformation experts at the organization Propastop. Anthropic's Claude models claimed the top spots, followed by Nvidia's Nemotron 3 and Alibaba's Qwen 3.6 Plus. Mistral's models, including the newest Medium 3.5, landed in the bottom third. The models had no access to web search or other tools during testing, so the benchmark only measures how well the language model itself can spot and reject propaganda.
The results line up with a Newsguard study that found Mistral had a steady misinformation rate of 36.67 percent. That's a bad look for the French company, which positions itself as a European alternative to US and Chinese providers and is currently negotiating a 3 billion euro funding round at a 20 billion euro valuation. It's especially rough since Mistral's flagship models already struggle to keep up with the competition.
The threat is real. Russian networks like "Pravda" deliberately feed AI systems millions of disinformation articles. And OpenAI recently shut down a Russian campaign that used ChatGPT to spread propaganda ahead of Germany's federal election.
— Originally published at the-decoder.com
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from The Decoder
See more →
An AI model programmed nonstop for 19 days on a single MirrorCode task that cost $2,600 to run
Epoch AI's MirrorCode benchmark reveals Claude Opus 4.7 as the leader with a 56% solve rate, reconstructing a 16,000-line toolkit in 14 hours. Despite this, all models tested struggle with the most complex tasks, highlighting limitations in current AI capabilities. The single task consumed $2,600 over 19 days, raising questions about cost-effectiveness in AI development.
