Fine-tuning language encoding models on slow fMRI improves prediction for fast ECoG
Quick Answer
This study demonstrates that fine-tuning language encoding models on non-invasive fMRI data can enhance ECoG prediction accuracy, despite fMRI's lower temporal resolution.
Quick Take
The results indicate that integrating fMRI data can significantly improve ECoG model performance, suggesting a promising avenue for future brain-computer interface applications.
Key Points
- Fine-tuning on fMRI improved ECoG predictions despite fMRI's lower temporal resolution.
- Models showed enhanced performance across frequency bands beyond fMRI measurements.
- Downsampled fMRI data still yielded comparable predictions to original models.
- ECoG performance scales positively with the amount of fMRI tuning data.
- Integration of multiple recording methods may enhance future decoding applications.
Paper Resources
📖 Reader Mode
~2 min readAbstract:Neuroscientists have recently turned to intracranial brain recording methods, like electrocorticography (ECoG), for human experiments because of the fine spatial and temporal resolution that they afford. Models trained on this data, however, are fundamentally restricted by the patient populations that can receive the implants necessary for recording. We propose using non-invasive fMRI to bridge the gap in training data. Using spoken language representations fine-tuned on fMRI, we build encoding models of ECoG. These representations showed improved prediction performance in ECoG, even though the temporal resolution of fMRI is two orders of magnitude worse. Prediction improved in frequency bands well beyond what is directly measured in fMRI. Next, to test the procedure's generalization ability, we fine-tuned models on fMRI responses that were temporally downsampled by a factor of 2. Despite the loss in resolution, these models were able to predict fMRI and ECoG responses at levels comparable to the original fMRI-tuned models. Finally, we showed that ECoG performance steadily scales with the amount of fMRI-tuning data. Our results show that "slow" data like fMRI can be a valuable resource for building better models of "fast" brain data like ECoG. In the future, integrating across multiple recording methods may further improve performance in other applications, like decoding.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2605.19224 [cs.CL] |
| (or arXiv:2605.19224v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2605.19224 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Aditya Vaidya [view email]
[v1]
Tue, 19 May 2026 00:49:51 UTC (11,940 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.