OpenAI Leverages Reddit’s r/ChangeMyView to Test AI Persuasion Skills
OpenAI has turned to the popular Reddit forum, r/ChangeMyView, to evaluate the persuasive abilities of its latest AI reasoning models. This revelation came in a system card released alongside its new model, o3-mini, on Friday. The subreddit, known for its lively debates, has become a testing ground for OpenAI’s AI systems, showcasing the growing reliance on human-generated data to refine artificial intelligence.
r/ChangeMyView is a treasure trove of human interaction, with millions of users posting controversial opinions and others responding with persuasive arguments to challenge them. OpenAI collects these posts and tasks its AI models with crafting replies in a closed environment. testers then evaluate the persuasiveness of these AI-generated responses, comparing them to human replies for the same posts.
This approach underscores the value of high-quality, human-generated data for AI growth. OpenAI’s content-licensing deal with reddit allows the company to train its models on user posts and display them within its products. While the financial terms remain undisclosed, Google reportedly pays Reddit $60 million annually under a similar agreement, as noted by Reuters.
However, the process of obtaining such datasets is not without controversy. Reddit CEO Steve Huffman has criticized companies like Microsoft, Anthropic, and Perplexity for scraping Reddit’s data without compensation, calling it “a real pain in the ass to block these companies.” OpenAI itself has faced lawsuits, including one from The New York times, over allegations of improper data scraping.
Despite these challenges, OpenAI’s models have shown extraordinary results on the ChangeMyView benchmark. According to the system card, “GPT-4o, o3-mini, and o1 all demonstrate strong persuasive argumentation abilities, within the top 80-90th percentile of humans.” While the models are not yet surpassing human performance, they are proving to be more persuasive than the average r/ChangeMyView user.
The goal, however, is not to create hyper-persuasive AI. OpenAI is wary of the risks posed by models that are too adept at persuasion. As the company noted, reasoning models have become quiet good at deception, prompting the development of new evaluations and safeguards. The fear is that an AI model could pursue its own agenda—or that of its controller—if it becomes too persuasive.
The ChangeMyView benchmark highlights the ongoing struggle for AI developers to find high-quality datasets.Even after scraping the public internet and securing licensing deals, the quest for reliable data remains a meaningful hurdle.
| Key Insights |
|——————-|
| OpenAI uses r/ChangeMyView to test AI persuasion skills. |
| The company has a content-licensing deal with Reddit for training data. |
| AI models like o3-mini perform in the top 80-90th percentile of human persuasiveness. |
| OpenAI aims to prevent AI from becoming too persuasive or deceptive. |
As AI continues to evolve, the ethical and practical challenges of data acquisition and model training will remain at the forefront of the conversation. OpenAI’s work with r/changemyview is just one example of how human interaction is shaping the future of artificial intelligence.
