Fine-tuning language encoding models on slow fMRI improves prediction for fast ECoG
Quick Take
Fine-tuning fMRI models enhances ECoG prediction despite fMRI's lower temporal resolution.
Key Points
- Non-invasive fMRI data improves ECoG model performance.
- Models retain predictive ability even with downsampled fMRI data.
- ECoG performance scales with the amount of fMRI tuning data.
📖 Reader Mode
~2 min readAbstract:Neuroscientists have recently turned to intracranial brain recording methods, like electrocorticography (ECoG), for human experiments because of the fine spatial and temporal resolution that they afford. Models trained on this data, however, are fundamentally restricted by the patient populations that can receive the implants necessary for recording. We propose using non-invasive fMRI to bridge the gap in training data. Using spoken language representations fine-tuned on fMRI, we build encoding models of ECoG. These representations showed improved prediction performance in ECoG, even though the temporal resolution of fMRI is two orders of magnitude worse. Prediction improved in frequency bands well beyond what is directly measured in fMRI. Next, to test the procedure's generalization ability, we fine-tuned models on fMRI responses that were temporally downsampled by a factor of 2. Despite the loss in resolution, these models were able to predict fMRI and ECoG responses at levels comparable to the original fMRI-tuned models. Finally, we showed that ECoG performance steadily scales with the amount of fMRI-tuning data. Our results show that "slow" data like fMRI can be a valuable resource for building better models of "fast" brain data like ECoG. In the future, integrating across multiple recording methods may further improve performance in other applications, like decoding.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2605.19224 [cs.CL] |
| (or arXiv:2605.19224v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2605.19224 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Aditya Vaidya [view email]
[v1]
Tue, 19 May 2026 00:49:51 UTC (11,940 KB)
— Originally published at arxiv.org
More from arXiv cs.CL
See more →Time to REFLECT: Can We Trust LLM Judges for Evidence-based Research Agents?
The reliability of LLM judges for evaluating deep research agents is critically assessed using the REFLECT benchmark.