Table of Contents
Few months back we reported that Google and Open AI had partnered with Reddit to use its content for training their AI models. Now Reddit has decided to block Microsoft and other search engines, as well as AI models, from crawling its content using robots.txt unless they agree to a content licensing deal.
This has been confirmed by the Reddit CEO Steve Huffman in a recent interview with The Verge. Where he said that the company has taken this step to ensure they have control over their data and its usage.
“Without these agreements, we don’t have any say or knowledge of how our data is displayed and what it’s used for, which has put us in a position now of blocking folks who haven’t been willing to come to terms with how we’d like our data to be used or not used,” Huffman said in an interview this week. He specifically named Microsoft, Anthropic, and Perplexity for refusing to negotiate, saying it has been “a real pain in the ass to block these companies.”
Microsoft’s Response
In response to Reddit’s decision, a Microsoft spokesperson told Search Engine Land:
“Microsoft respects the robots.txt standard and we honour the directions provided by websites that do not want content on their pages to be used with our generative AI models. Bing stopped crawling Reddit after they implemented their updated robots.txt file on July 1, which prohibits all crawling of their site.”
Even Microsoft AI CEO Mustafa Suleyman described web content as “freeware,” asserting that it is available for anyone to use:
“…With respect to content that is already on the open web, the social contract of that content since the ’90s has been that it is fair use. Anyone can copy it, recreate with it, and reproduce with it. That has been freeware, if you like. That’s the understanding.”
Google and Open AI Has Not Been Blocked
Reddit has not blocked Google and Open AI due to their exclusive content licensing deal with both the companies. Where Google is paying Reddit $60 million annually. This agreement was announced in February.
