Evaluating OpenAI's Privacy Filter: Cross-Lingual, Cross-Domain PII Detection Across 42 Benchmarks
Quick Answer
This paper shows that OpenAI's Privacy Filter (OPF), a 1.5B-parameter PII detector, was evaluated across 42 benchmarks, achieving an F1 score of 0.855 on AI4Privacy.
Quick Take
While OPF outperformed Presidio and XLM-RoBERTa on PII-annotated benchmarks, it struggled with narrative prose, especially in non-Latin scripts, highlighting its limitations in culturally variable PII types.
Key Points
- OPF achieved F1=0.855 on AI4Privacy and outperformed competitors like Presidio.
- Performance drops significantly on narrative prose, with F1 scores as low as 0.04.
- Strongest results on structured PII types like email (0.78) and phone (0.76).
- Recall bias observed in customer support and medical/legal PII detection.
- Global precision varies widely, ranging from 0.31 to 0.86 across domains.
DeepSignal Analysis
What happened
OpenAI's Privacy Filter (OPF), a 1.5 billion parameter PII detection model, was evaluated across 42 benchmarks in 22 languages. It achieved an F1 score of 0.855 on AI4Privacy but struggled with narrative prose, particularly in non-Latin scripts. OPF outperformed competitors like Presidio and XLM-RoBERTa on certain benchmarks but showed weaknesses in culturally variable PII types.
Key evidence
- OPF achieved an F1 score of 0.855 on the AI4Privacy benchmark and 0.464 on SPY medical, outperforming Presidio and XLM-RoBERTa.
- On multilingual named entity recognition (NER) tasks, XLM-RoBERTa outperformed OPF across all 13 Indic and non-Latin languages.
- OPF's performance degraded significantly in narrative prose, with F1 scores ranging from 0.04 to 0.57 on NER benchmarks for non-Latin scripts.
Why it matters
The evaluation of OPF highlights both its strengths and limitations in PII detection across diverse languages and contexts. While it shows promise in structured data scenarios, its struggles with narrative prose and culturally variable PII types raise concerns about its applicability in real-world settings. This analysis is crucial for organizations considering the deployment of PII detection systems, as it underscores the need for careful evaluation of model performance across various use cases.
Paper Resources
📖 Reader Mode
~2 min readAbstract:We present the first independent, systematic evaluation of OpenAI's Privacy Filter (OPF), a 1.5B-parameter bidirectional PII detector, across 42 synthetic benchmarks spanning 22 languages and 5 domains. Zero-shot, OPF achieves F1=0.855 on AI4Privacy and 0.464 on SPY medical, outperforming Presidio (0.431, 0.273) and XLM-RoBERTa (0.269, 0.111) on PII-annotated benchmarks; on multilingual NER, XLM-RoBERTa leads OPF on all 13 Indic and non-Latin languages. GPT-4o leads on medical, legal, and financial PII (SPY: 0.643 avg, Gretel: 0.527), while OPF leads on structured synthetic PII (0.71 avg) and customer support (0.60). OPF degrades sharply when PII is embedded in narrative prose: F1=0.04--0.57 on NER benchmarks and collapse for non-Latin scripts (Arabic: 0.04, Cyrillic: 0.03). Error analysis shows OPF is strongest on structurally regular PII types (email: 0.78, phone: 0.76) and weakest on culturally variable ones (person: 0.40, address: 0.49), and is recall-biased on customer-support and medical/legal PII (P=0.31--0.54, R=0.70--0.85); global precision spans 0.31--0.86 across all domains.
| Comments: | 11 pages, 5 tables; evaluation of a production PII detection system |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2608.02616 [cs.CL] |
| (or arXiv:2608.02616v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2608.02616 arXiv-issued DOI via DataCite |
Submission history
From: Rohith Uppala [view email]
[v1]
Sat, 23 May 2026 16:59:01 UTC (30 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.