blog/static/robots.txt

# https://platform.openai.com/docs/gptbot
User-agent: GPTBot
Disallow: /

User-agent: *
Disallow: /search/
Disallow: /page/
Disallow: /*/page/
Disallow: /tags/
Disallow: /api/

Sitemap: https://kremalicious.com/sitemap.xml
block GPTBot 2023-08-07 14:12:09 +02:00			`# https://platform.openai.com/docs/gptbot`
			`User-agent: GPTBot`
			`Disallow: /`

robots.txt, sitemap plugin 2013-11-21 00:44:52 +01:00			`User-agent: *`
			`Disallow: /search/`
			`Disallow: /page/`
also ignore category pagination 2013-12-11 22:55:39 +01:00			`Disallow: /*/page/`
build & lint updates 2018-09-30 03:11:08 +02:00			`Disallow: /tags/`
kremalicious as a service - output all posts with all metadata as json 2016-05-02 01:05:53 +02:00			`Disallow: /api/`
robots.txt, sitemap plugin 2013-11-21 00:44:52 +01:00
basic setup for blog posts & media 2018-07-17 23:33:55 +02:00			`Sitemap: https://kremalicious.com/sitemap.xml`