GentleBot

A small, slow bot that saves pages for personal archiving and research

This is my little bot, run by a human for their own use. It isn't a search engine, and the human doesn't sell or publish anything the GentleBot finds.

Recognise the GentleBot by its user agent:

GentleBot/0.1 (+https://gentlebot.eu; contact: bot@gentlebot.eu)

GentleBot's gentle behaviour

• One page at a time fetching (never in parallel).
• Random wait 5-15s between pages. If your robots.txt specifies a longer delay, the GentleBot honours that instead.
• Fetches at most 50 pages per visit (or 150 for very large sites).
• Obeys robots.txt rules, including Disallow.
• Doesn't retry failed requests.
• If rate limited (429), refused (403), shown a captcha, or blocked in any other way, it stops and does not come back to your site until a human has looked at what happened.
• It only scrapes publicly available pages, never pages behind a login.

How to block it

Add this to your robots.txt to give GentleBot the hose again, and it will stay away from your whole site:

User-agent: GentleBot
Disallow: /

You can also choose to disallow only some paths. Blocking it by user agent or IP address at your server works too; it will treat that as a block and stop.

Contact

If GentleBot is causing you any trouble, or you would just like it to stop, write to bot@gentlebot.eu. A human reads that mail and will reply within a few days. Please include your site's address, and a log line with the time, if you have one.

Security reports: see security.txt