Adopt $\neq$ Adapt: Longitudinal Analyses of LLM Conversations in the Wild
Quick Answer
This study analyzes the conversational patterns of approximately 12,000 Microsoft Bing Copilot users, revealing that individual user behaviors are largely stable over time.
Quick Take
While active users engage in more complex tasks and have more successful interactions, the dataset from WildChat-4.8M skews towards highly proficient users, indicating that typical user-AI interactions are not well represented.
Key Points
- Analyzed conversational trajectories of ~12,000 Bing Copilot users.
- Found that user habits are overwhelmingly sticky over time.
- Active users engage in more complex, professional tasks.
- WildChat-4.8M dataset skews towards highly proficient users.
- Results highlight the difficulty in changing existing user behavior.
Paper Resources
Article Content
From source RSS / original summaryarXiv:2605. 29018v1 Announce Type: new Abstract: Although a growing body of research has begun to describe user-- interactions, the picture it paints is largely static; little is known about how individual users change their behavior over time. To address this gap, we analyze the conversational trajectories of $\sim$12,000 randomly sampled Microsoft Bing Copilot users and compare these with data from WildChat-4. 8M.
While the Copilot data contains significant population-level trends, we find that trends in individual user trajectories are much weaker; user habits prove to be overwhelmingly sticky. We also find stark differences between users of different activity levels: more active users have more successful conversations and use the LLM for more complex and professionally oriented tasks. Some user trends also appear in WildChat-4.
8M, but we find evidence that this dataset is significantly skewed towards highly proficient "power" users. Ultimately, our results suggest that existing user behavior is difficult to change and demonstrate the extent of user heterogeneity. Our comparison between datasets highlights that WildChat does not represent typical user-AI interactions, an important caveat for downstream uses of the data.
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →HOBA: Hierarchical On-Policy Bidding Agents for Adaptive Online Advertising
HOBA (Hierarchical On-policy Bidding Agents) is a novel hierarchical reinforcement learning framework that enhances online advertising bidding systems by improving adaptability and reducing hyperparameter tuning costs. It utilizes a for hyperparameter inference, a SARSA agent for expert model selection, and a dynamic expert pool for bid execution, achieving a +3.6% increase in target cost during large-scale deployment and outperforming state-of-the-art baselines on AuctionNet.