Robust, Scalable Detection of Text Containment in Large Web-Crawled Corpora
Quick Answer
FindMyText is an open-source Python tool that efficiently detects text containment in large corpora, outperforming existing methods on ArXiv, Wikipedia, and web content datasets.
Quick Take
Utilizing a novel fingerprinting mechanism, it enhances the identification of near-verbatim copies, making it ideal for copyright verification. The system's distributed indexing framework allows it to scale effectively for extensive web-crawled datasets.
Key Points
- FindMyText uses a novel mechanism for matching text fingerprints.
- The tool is particularly effective for verifying copyrighted material.
- It scales to large datasets using a distributed, disk-based indexing framework.
- Benchmark tests show superior performance against three datasets.
- Open-source and designed for efficient text containment assessment.
Paper Resources
📖 Reader Mode
~2 min readAbstract:We present FindMyText, an open-source Python package designed to efficiently assess whether a given text appears, in part or in full, within a text corpus. The tool builds on prior techniques for document fingerprinting, but extends them with a novel mechanism to explicitly capture sequences of matching fingerprints. By identifying such chains, the tool can more reliably detect near-verbatim copies of a given text rather than mere textual similarities. This makes FindMyText particularly suited for verifying the presence of copyrighted material in a corpus. Leveraging a distributed, disk-based indexing framework, the system scales to large web-crawled datasets. Using a new benchmark for evaluating text containment methods, we show that FindMyText outperforms alternative approaches across three datasets (ArXiv papers, Wikipedia, and generic web content).
| Comments: | 6 pages + references and appendices |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2607.10020 [cs.CL] |
| (or arXiv:2607.10020v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2607.10020 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Pierre Lison [view email]
[v1]
Fri, 10 Jul 2026 22:46:11 UTC (1,047 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.