Robust, Scalable Detection of Text Containment in Large Web-Crawled Corpora
Quick Answer
FindMyText is an open-source Python tool that efficiently detects text containment in large corpora, outperforming existing methods on ArXiv, Wikipedia, and web content datasets.
Quick Take
FindMyText is an open-source Python tool that efficiently detects text containment in large corpora, outperforming existing methods on ArXiv, Wikipedia, and web content datasets. Utilizing a novel fingerprinting mechanism, it enhances the identification of near-verbatim copies, making it ideal for copyright verification. The system's distributed indexing framework allows it to scale effectively for extensive web-crawled datasets.
Key Points
- FindMyText uses a novel mechanism for matching text fingerprints.
- The tool is particularly effective for verifying copyrighted material.
- It scales to large datasets using a distributed, disk-based indexing framework.
- Benchmark tests show superior performance against three datasets.
- Open-source and designed for efficient text containment assessment.
Paper Resources
📖 Reader Mode
~2 min readAbstract:We present FindMyText, an open-source Python package designed to efficiently assess whether a given text appears, in part or in full, within a text corpus. The tool builds on prior techniques for document fingerprinting, but extends them with a novel mechanism to explicitly capture sequences of matching fingerprints. By identifying such chains, the tool can more reliably detect near-verbatim copies of a given text rather than mere textual similarities. This makes FindMyText particularly suited for verifying the presence of copyrighted material in a corpus. Leveraging a distributed, disk-based indexing framework, the system scales to large web-crawled datasets. Using a new benchmark for evaluating text containment methods, we show that FindMyText outperforms alternative approaches across three datasets (ArXiv papers, Wikipedia, and generic web content).
| Comments: | 6 pages + references and appendices |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2607.10020 [cs.CL] |
| (or arXiv:2607.10020v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2607.10020 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Pierre Lison [view email]
[v1]
Fri, 10 Jul 2026 22:46:11 UTC (1,047 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →Letting the Data Speak: Extracting Keywords from Crowdsourced Collections with AI
The study evaluates three NLP approaches—Named Entity Recognition, Keyword Extraction, and Topic Modelling—using the Their Finest Hour Online Archive to automate keyword extraction from crowdsourced WWII collections. Findings suggest that while NLP methods show promise, no single approach is sufficient, and ethical considerations in automated keyword extraction are crucial for responsible stewardship.