Adopt $\neq$ Adapt: Longitudinal Analyses of LLM Conversations in the Wild
Quick Answer
This study analyzes the conversational patterns of approximately 12,000 Microsoft Bing Copilot users, revealing that individual user behaviors are largely stable over time.
Quick Take
This study analyzes the conversational patterns of approximately 12,000 Microsoft Bing Copilot users, revealing that individual user behaviors are largely stable over time. While active users engage in more complex tasks and have more successful interactions, the dataset from WildChat-4.8M skews towards highly proficient users, indicating that typical user-AI interactions are not well represented.
Key Points
- Analyzed conversational trajectories of ~12,000 Bing Copilot users.
- Found that user habits are overwhelmingly sticky over time.
- Active users engage in more complex, professional tasks.
- WildChat-4.8M dataset skews towards highly proficient users.
- Results highlight the difficulty in changing existing user behavior.
Paper Resources
Article Content
From source RSS / original summaryarXiv:2605. 29018v1 Announce Type: new Abstract: Although a growing body of research has begun to describe user--LLM interactions, the picture it paints is largely static; little is known about how individual users change their behavior over time. To address this gap, we analyze the conversational trajectories of $\sim$12,000 randomly sampled Microsoft Bing Copilot users and compare these with data from WildChat-4. 8M.
While the Copilot data contains significant population-level trends, we find that trends in individual user trajectories are much weaker; user habits prove to be overwhelmingly sticky. We also find stark differences between users of different activity levels: more active users have more successful conversations and use the LLM for more complex and professionally oriented tasks. Some user trends also appear in WildChat-4.
8M, but we find evidence that this dataset is significantly skewed towards highly proficient "power" users. Ultimately, our results suggest that existing user behavior is difficult to change and demonstrate the extent of user heterogeneity. Our comparison between datasets highlights that WildChat does not represent typical user-AI interactions, an important caveat for downstream uses of the data.
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →Adversarial Social Epistemology for Assemblies of Humans and Large Language Models
The paper introduces Adversarial Social Epistemology (ASE) to analyze how agents manipulate trust in public communications, highlighting mechanisms that undermine the reliability of testimony and inference. It critiques existing frameworks like epistemic bubbles and misinformation diffusion, proposing a new language for understanding trust breaches and auditing inferential chains in densely interactive environments involving humans and large language models.