Rendered at 18:35:06 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
naw103 4 hours ago [-]
One trap when you name AI bots specifically is that is that robots.txt groups don’t merge. Crawlers will only follow the most specific group so if GPT bot has its own group named nothing under User-agent: * applies to it anymore so you need to repeat every disallowed line under each group.
Hissigh 13 hours ago [-]
Are llms.txt actually read by bots now? I found one experiment that showed that no bot has accessed the file in 3 months.
https://www.rebootonline.com/geo/llms-txt-experiment/