da157.id / website
this repo has no description
0
fork

Configure Feed

Select the types of activity you want to include in your feed.

website / templates / robots.txt
at main 46 lines 1.1 kB view raw
awwpotato init
b72ec1f4
1User-agent: * 2Allow: / 3 4User-agent: AI2Bot 5User-agent: Ai2Bot-Dolma 6User-agent: Amazonbot 7User-agent: anthropic-ai 8User-agent: Applebot-Extended 9User-agent: Bytespider 10User-agent: CCBot 11User-agent: ChatGPT-User 12User-agent: Claude-Web 13User-agent: ClaudeBot 14User-agent: cohere-ai 15User-agent: Diffbot 16User-agent: DuckAssistBot 17User-agent: FacebookBot 18User-agent: FriendlyCrawler 19User-agent: Google-Extended 20User-agent: GoogleOther 21User-agent: GoogleOther-Image 22User-agent: GoogleOther-Video 23User-agent: GPTBot 24User-agent: iaskspider/2.0 25User-agent: ICC-Crawler 26User-agent: ImagesiftBot 27User-agent: img2dataset 28User-agent: ISSCyberRiskCrawler 29User-agent: Kangaroo Bot 30User-agent: Meta-ExternalAgent 31User-agent: Meta-ExternalFetcher 32User-agent: OAI-SearchBot 33User-agent: omgili 34User-agent: omgilibot 35User-agent: PanguBot 36User-agent: PerplexityBot 37User-agent: PetalBot 38User-agent: Scrapy 39User-agent: Sidetrade indexer bot 40User-agent: Timpibot 41User-agent: VelenPublicWebCrawler 42User-agent: Webzio-Extended 43User-agent: YouBot 44User-agent: turnitinbot 45User-agent: Turnitin 46Disallow: /