Device-First Feedback: Toward Mobile-Native LLM-Driven Neural Architecture Search
Quick Answer
The study presents an automated mobile deployment pipeline for LLM-generated CNNs, achieving a 25.6x improvement in mobile deployment score on CIFAR-10.
Quick Take
However, while GPU accuracy improved in later cycles, it did not translate to better mobile performance, particularly on CIFAR-100, highlighting the need for multi-dataset on-device testing.
Key Points
- Automated pipeline integrates QLoRA fine-tuning, GPU evaluation, and on-device benchmarking.
- CIFAR-10 shows a 25.6x improvement in mobile deployment score with 46.9% mean quantized accuracy.
- Later cycles improved GPU accuracy but failed to enhance mobile performance.
- CIFAR-100's best mobile score was retained by the pre-QLoRA baseline.
- Study emphasizes the importance of multi-dataset testing for deployment objectives.
DeepSignal Analysis
What happened
The study introduces an automated pipeline for deploying CNNs generated by large language models on mobile devices. It reports a significant improvement in mobile deployment scores on the CIFAR-10 dataset, but later cycles did not yield better mobile performance on CIFAR-100 despite improved GPU accuracy.
Key evidence
- The automated pipeline integrates QLoRA fine-tuning, GPU evaluation, INT8 export, and physical-device benchmarking for mobile deployment.
- On CIFAR-10, the first cycle achieved a 25.6x improvement in mobile deployment score, with a mean quantized accuracy of 46.9%.
- For CIFAR-100, the pre-QLoRA baseline maintained the best mobile score, while later cycles improved GPU accuracy but did not enhance on-device performance.
Why it matters
This research highlights the complexities of deploying AI models on mobile devices, emphasizing that GPU performance does not guarantee effective mobile deployment. The findings suggest a need for rigorous on-device testing across multiple datasets to ensure models perform well in real-world scenarios, particularly for challenging classification tasks.
What to watch
Paper Resources
Source Excerpt
Deploying convolutional neural networks generated by (LLMs) on real mobile hardware requires more than GPU validation accuracy: INT8 TensorFlow Lite export, delegate selection, and on-device latency jointly determine whether a model is usable. We present an automated mobile deployment pipeline that closes the loop from QLoRA fine-tuning of an architecture-generating LLM through GPU evaluation, INT8 export, and physical-device benchmarking to gated augmentation of the traini
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CV
See more →ProMoE-FL: Prototype-conditioned Mixture of Experts for Multimodal Federated Learning with Missing Modalities
ProMoE-FL introduces a Prototype-conditioned Mixture-of-Experts framework for multimodal federated learning, effectively addressing missing modalities. It outperforms existing methods on four chest X-ray datasets, demonstrating superior feature synthesis capabilities in both homogeneous and heterogeneous settings.
