A $1,500 foundation model that rivals larger LLMs - Venturebeat
Quick Answer
Researchers at Sapient developed HRM-Text, a 1B-parameter foundation model trained from scratch for $1,500, achieving competitive performance against larger LLMs on industry benchmarks.
Quick Take
This model utilizes a Hierarchical Recurrent Model architecture, focusing on instruction-response pairs instead of traditional autoregressive methods, significantly reducing training costs and data requirements.
Key Points
- HRM-Text trained for $1,500, a fraction of typical costs.
- Utilizes a Hierarchical Recurrent Model for enhanced sample efficiency.
- Achieved competitive performance against larger open models.
- Focuses on instruction-response pairs rather than raw text.
- Reduces the need for extensive internet-scale data.
Source Excerpt
# Researchers say they trained a foundation model from scratch for about $1,500. Training a foundation from scratch costs millions and requires internet-scale data — which is why most enterprises don't bother. To overcome this brute-force scaling dogma, researchers at Sapient developed HRM-Text, which replaces standard Transformers with a highly sample-efficient Hierarchical Recurrent Model (HRM), an architecture they first introduced last year.
Instead of brute-force autoregressive prediction on raw text, HRM-Text trains exclusively on instruction-response pairs. The researchers were able to train a 1B-parameter HRM-Text from scratch at a fraction of the cost and tokens of normal LLMs. …
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from WebSearch (Tavily)
See more →It's true that Anthropic is the fastest growing company ...
Anthropic is rapidly emerging as a leader in AI with its Claude model, which outperforms competitors in various benchmarks. Their coding agents are also recognized as the best in the industry, contributing to their unprecedented revenue growth and solidifying their position in the enterprise market.
WSJ: OpenAI is considering deep price reductions as competition ...
OpenAI is contemplating significant price cuts in response to competitive pressure from Anthropic, particularly due to the success of Claude Code in developer and coding workflows. This shift could affect pricing strategies in the AI market as companies vie for dominance in coding solutions.