Does My Embedding Reflect That $A = B$? Evaluating Mathematical Equivalence in Embedding Models
Quick Answer
This paper evaluates whether embedding models effectively capture mathematical equivalence, introducing the MELD dataset to highlight issues with terminology-based grouping.
Quick Take
This paper evaluates whether embedding models effectively capture mathematical equivalence, introducing the MELD dataset to highlight issues with terminology-based grouping. A proposed contrastive learning approach improves retrieval tasks, demonstrating enhanced performance on informal-formal mappings and MELD, which consists solely of natural language statements.
Key Points
- Current embedding models group statements by terminology rather than mathematical equivalence.
- The MELD dataset contains mathematically equivalent statements in different languages.
- A contrastive learning approach aligns informal and formal mathematical statements.
- Improvements were observed in informal-formal retrieval tasks and MELD performance.
Paper Resources
📖 Reader Mode
~2 min readAuthors:Jiaying Ye, Samarth Rao, Leo Carlin, Kedar Chintalapati, Saharsh Bhargava, Rachit Jaiswal, Michael Zhou, Jared Darlington, Jarod Alper, Vasily Ilin, Henry Kvinge
Abstract:Because mathematics is highly abstract, a single statement can take very different forms depending on what subfield it is framed in. There are many examples where breakthroughs occurred after researchers discovered that a question had already been answered in a different field. At the same time, the growth of new resources related to formalization has increased the need for tools that enable efficient and reliable navigation between mathematical 'languages' (e.g., from Lean to natural language). In this paper, we investigate whether current embedding models capture mathematical equivalence. To do this, we introduce the Mathematically Equivalent but Lexically Different Pairs (MELD) Dataset, a collection of mathematically equivalent statements that are expressed in very different language. We show that current state-of-the-art embedding models tend to group statements by the terminology used to make them instead of the underlying math. Motivated by this, we propose a contrastive approach to learning embeddings of mathematical text that focuses on aligning informal statements with different formalizations. Our experiments demonstrate that this leads to improvements not only on informal-formal retrieval tasks but also on MELD, which only contains natural language statements.
| Comments: | 18 pages, comments welcome |
| Subjects: | Computation and Language (cs.CL); Machine Learning (cs.LG) |
| Cite as: | arXiv:2606.23959 [cs.CL] |
| (or arXiv:2606.23959v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2606.23959 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Henry Kvinge [view email]
[v1]
Mon, 22 Jun 2026 21:37:58 UTC (272 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →Quantifying Prior Dominance in Systems
The study introduces the Normalized Context Utilization (NCU) metric to evaluate Retrieval-Augmented Generation (RAG) systems, revealing that Small Language Models (SLMs) outperform larger models in factual extraction. The findings indicate that traditional scaling laws yield diminishing returns, with a commercial API frequently failing against adversarial evidence due to systemic confidence collapse.