A Comparative Evaluation of Structural Topic Models and BERTopic for Short, Open-Ended Survey Responses
Quick Take
This study compares Structural Topic Models (STM) and BERTopic for analyzing short, open-ended survey responses, finding BERTopic consistently yields higher topic coherence and more interpretable results. While STM supports inferential covariate analysis better, BERTopic excels in descriptive comparisons, suggesting complementary strengths in topic modeling approaches for social science research.
Key Points
- BERTopic outperformed STM in topic coherence across all evaluated conditions.
- Contextual augmentation significantly enhanced BERTopic's performance.
- STM supports stronger inferential covariate analysis compared to BERTopic.
- Qualitative evaluations showed BERTopic produced more interpretable topics.
- The study provides practical guidance for selecting topic modeling methods.
Article Content
From source RSS / original summaryarXiv:2605. 23093v1 Announce Type: new Abstract: Topic modeling in applied psychology increasingly spans two methodological traditions: probabilistic bag-of-words models and newer embedding-based approaches. Yet many evaluations of these methods rely on longer and cleaner benchmark corpora, leaving less guidance for short, open-ended survey responses. This paper compares Structural Topic Models (STM), a probabilistic topic model, and BERTopic, an embedding-based model, for analyzing open-ended survey responses.
We evaluated three STM conditions and five BERTopic conditions, varying typographical correction, stemming, embedding choice, and contextual augmentation, a strategy we introduced to provide additional semantic context for very short responses. Results indicate that BERTopic consistently produced higher topic coherence than STM, with contextual augmentation yielding the strongest performance gains. In contrast, higher-dimensional embeddings alone did not improve coherence and were associated with greater data loss.
Qualitative evaluation showed that BERTopic generated more interpretable and stable topics, while STM topics were often broader and more mixed. However, STM provides stronger support for inferential covariate analysis, whereas BERTopic covariate comparisons are primarily descriptive. These findings suggest that STM and BERTopic offer complementary strengths. We conclude with practical guidance for selecting and combining topic modeling approaches in applied social science research.
Reader Mode unavailable (could not extract clean content).
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →Time to REFLECT: Can We Trust LLM Judges for Evidence-based Research Agents?
The reliability of LLM judges for evaluating deep research agents is critically assessed using the REFLECT benchmark.