Llms.txt and the Rise of Agentic SEO: Why Half the Data Says It Doesn't Work, and Smart Teams Are Shipping It Anyway
In June 2026, Cloudflare CEO Matthew Prince dropped a statistic that went largely unnoticed: bots and AI agents now account for over half of all web traffic. For the first time, human visitors are the minority. The internet shifted under our feet, turning machines into the primary audience consuming our content. That shift explains the sudden rise of llms.txt and the broader, messier debate around "agentic SEO." Depending on who you ask, this simple text file is either the most sensible five-minute dev task you can do today, or a complete waste of time solving a problem that hasn't arrived yet. How llms.txt Works Under the Hood Jeremy Howard proposed the format at Answer.AI in late 2024 with a straightforward pitch. Standard webpages are designed for human eyes and browsers - cluttered with navigation menus, heavy JavaScript, ad scripts, and tracking tags that Large Language Models (LLMs) must sift through to find relevant information. Llms.txt bypasses that bloat. Sit...