blog/public/robots.txt

User-agent: CCBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Omgilibot
Disallow: /

User-agent: Omgili
Disallow: /

User-agent: FacebookBot
Disallow: /

User-agent: *
Disallow: /search/
Disallow: /page/
Disallow: /*/page/
Disallow: /tags/
Disallow: /api/

Sitemap: https://kremalicious.com/sitemap-index.xml
block all the ai bots 2023-10-06 12:59:26 +02:00			`User-agent: CCBot`
			`Disallow: /`

			`User-agent: ChatGPT-User`
			`Disallow: /`

block GPTBot 2023-08-07 14:12:09 +02:00			`User-agent: GPTBot`
			`Disallow: /`

block all the ai bots 2023-10-06 12:59:26 +02:00			`User-agent: Google-Extended`
			`Disallow: /`

			`User-agent: Omgilibot`
			`Disallow: /`

			`User-agent: Omgili`
			`Disallow: /`

			`User-agent: FacebookBot`
			`Disallow: /`

robots.txt, sitemap plugin 2013-11-21 00:44:52 +01:00			`User-agent: *`
			`Disallow: /search/`
			`Disallow: /page/`
also ignore category pagination 2013-12-11 22:55:39 +01:00			`Disallow: /*/page/`
build & lint updates 2018-09-30 03:11:08 +02:00			`Disallow: /tags/`
kremalicious as a service - output all posts with all metadata as json 2016-05-02 01:05:53 +02:00			`Disallow: /api/`
robots.txt, sitemap plugin 2013-11-21 00:44:52 +01:00
Gatsby → Astro (#829) * basic astro setup, kick out all gatsby configs * move content folder * src/pages setup * more file reorg * more config updates * more reorgs * refactor * refactor * bump astro * refactor * svg icon build system, theme switch * remark plugin for lead paragraph, more refactor * make images work * post meta * custom Picture component * Pagination, More component, 404 fixes * linking fixes * add table of contents * post actions fixes * tag fixes * content changes * content changes: move media files to their posts * more content moving, remove media folder * refactor remark-lead-paragraph * link css file when defined in post frontmatter * move content up again * kbd post update * allow js * downloads solution * add astro check * redirect_from solution * githubLink solution * reorg * exif solution as prebuild step * exif solution on each post during build * isolate lead paragraph extraction to articles * restore Exif components * deploy script update * fix redirects * xml & json feeds * build fix * fix exif readout in production * head and seo tweaks, add feeds * tweak image display * archive pages with single layout * restore tags archive * sitemap setup * restore thanks page functionality * reorg * cleanup * parallel scripts during prebuild * restore jest setup * remove jest, switch to vitest as test runner * adapt CI * test refactor * typescript tweaks * avatar fixes * typings * restore search functionality * theme switch with nanostores * fixes * test fixes * restore changelog functionality * umami script * border color tweak * related posts with fuse.js * plawright e2e testing setup * search tweaks * simplify typekit loading * photo fix * e2e tests * related posts fix * fix tags archive * tweaks * tweaks * linux snapshots * fix header test * new symlink tactic * fix dev server in codespaces * fix yaml * ci fixes * changelog loading tweaks * e2e against dev server on ci * changelog tweaks * ci tweaks * ci tweaks * ci tweaks * docs updates * ci tweaks * refactor photo creation script * package updates * refactor search * ci tweak * ci tweaks * test tweaks, more unit tests * more unit tests * post creation script tweaks * refactor new scripts, test them for real life * more tests * refactor * codeclimate-action update * uses update * limit ci runs * fix theme toggle, test it * more tests * favicon files cleanup * icon components location change * type checking through ci * command fixes * ci fix * search tweaks * ci tweaks * revised favicons, write post draft about it * drafts filtering fix * lint fix, favicon fixes * copy changes * fix related search images * content updates * new codeblock styles, copy tweaks, fixes * package updates * typing fixes * lint fix * content updates * restore link posts * faster theme switching * split up astro utils * related posts fixes * fix * refactor * fixes * copy tweaks * fixes * picture tweaks * image fixes * feed fixes, adapt for json feed v1.1 * e2e test updates * layout tweaks * update snaphots * migrate to createMarkdownProcessor * ci cache tweaks * activate more browsers for e2e testing * switch to macos-13 images * build caching tweaks * markdown fix * set image quality * remove avif generation * picture tweaks * head fixes * add og:image:alt * create-icons test * new post: Favicon Generation with Astro 2023-09-18 03:16:53 +02:00			`Sitemap: https://kremalicious.com/sitemap-index.xml`