mirror of https://github.com/kremalicious/blog.git
15 lines
999 B
Markdown
15 lines
999 B
Markdown
|
---
|
|||
|
|
date: 2023-10-06T10:22:16.581Z
|
|||
|
|
|
|||
|
|
title: Block the Bots that Feed “AI” Models by Scraping Your Website
|
|||
|
|
linkurl: https://neil-clarke.com/block-the-bots-that-feed-ai-models-by-scraping-your-website/
|
|||
|
|
|
|||
|
|
tags:
|
|||
|
|
- development
|
|||
|
|
- ai
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
Neil Clarke with an [excellent overview](https://neil-clarke.com/block-the-bots-that-feed-ai-models-by-scraping-your-website/) about current techniques to block all the "AI" web scraping bots from your content, e.g. via `robots.txt`. The reasons for doing so are numerous:
|
|||
|
|
|
|||
|
|
> “AI” companies think that we should have to opt-out of data-scraping bots that take our work to train their products. [...] These companies should be prevented from using data that they haven’t been given explicit consent for. Opt-out is problematic as it counts on concerned parties hearing about new or modified bots BEFORE their sites are targeted by them. That is simply not practical.[...] The online community is under no responsibility to help them create their products.
|