Automated Data Readiness for Scientific AI
Quick Answer
This paper shows that The REDI framework automates data readiness for scientific AI, transforming datasets into AI-ready formats through a five-stage pipeline.
Quick Take
The REDI framework automates data readiness for scientific AI, transforming datasets into AI-ready formats through a five-stage pipeline. Evaluated across various domains, it achieves near-ideal parallel scaling on 100 nodes, addressing data preparation bottlenecks and enhancing reproducibility.
Key Points
- REDI features a five-stage pipeline: ingest, preprocess, transform, structure, and output.
- The framework automates FAIR compliance and catalog publication via the SetGo tool.
- Preliminary results show near-ideal scaling on 100 nodes for climate data processing.
- File I/O is identified as the dominant cost in the data transformation pipeline.
- REDI transforms raw datasets into reusable community assets across various scientific domains.
Paper Resources
📖 Reader Mode
~2 min readAuthors:Sean R. Wilkinson, Valentine G. Anantharaj, Jong Youl Choi, Ketan Maheshwari, Marshall McDonnell, Massimiliano Lupo Pasini, Polina Shpilker, Renan Souza, Patrick Widener, Sarp Oral, Wesley Brewer
Abstract:Leadership computing facilities steward large-scale scientific datasets that routinely require substantial transformation before serving as AI training data. However, no existing framework fully unifies automated transformation, readiness assessment, provenance tracking, and agent-native deployment. We present REDI, an open-source framework that addresses this gap through a unified five-stage pipeline (ingest, preprocess, transform, structure, and output) with per-stage instrumentation for reproducibility and deployment as an agent-callable skill; companion tool SetGo automates FAIR compliance and catalog publication. Evaluated across climate, proteomics, materials science, and nuclear fusion, REDI transforms all datasets from raw to AI-ready, with outputs validated against domain-expert references, and preliminary results show near-ideal parallel scaling to 100 nodes on Frontier for the climate case. Provenance-instrumented profiling reveals file I/O as the dominant pipeline cost, with format selection a first-order optimization lever. These results establish REDI as a cross-domain platform providing automated data readiness for scientific AI, transforming data preparation bottlenecks into reproducible, reusable community assets.
| Subjects: | Artificial Intelligence (cs.AI); Computational Engineering, Finance, and Science (cs.CE) |
| Cite as: | arXiv:2607.02771 [cs.AI] |
| (or arXiv:2607.02771v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2607.02771 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Sean Wilkinson [view email]
[v1]
Thu, 2 Jul 2026 21:09:13 UTC (196 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →Procedural Memory Distillation: Online Reflection for Self-Improving Language Models
Procedural Memory Distillation (PMD) enhances reinforcement learning by converting cross-episode signals into reusable memory, improving Qwen3-8B and OLMo3-Instruct-7B models by 3.8-5.5% on SCIKNOWEVAL and 7.9-13.6% on . The co-evolution of policy and memory allows for more effective self-supervision, demonstrating significant performance gains when both components are active.