To help Blackhole Pro trap only bad bots, it’s important to add a few rules to your site’s robots.txt file. This post explains the details.

Blackhole robots rules

In order for the Blackhole trap to work properly, it is important to add the provided robots rules to your site’s robots.txt file. The rules are provided on the plugin settings screen in the “Robots Rules” section. They will look something like this:

User-agent: *
Disallow: /*blackhole
Disallow: /?blackhole

That’s just an example, make sure to visit the plugin settings page to get the latest/most current rules.

How to add the rules

For all websites, the robots.txt file is located in the web-accessible root directory of the website. So for example, if your site is example.com, the robots file will be located at the following URL:

https://example.com/robots.txt

If your site has a physical file on the server named “robots.txt”, then you can add the blackhole rules via copy/paste. Or, if your site does not have a physical file named “robots.txt”, then it means that WordPress is generating a robots file for you, automatically and dynamically.

Tip: You can find a link to your site’s robots.txt file on the Blackhole settings page.

So with that in mind, there are two possible scenarios for adding the blackhole rules:

If your site is using the virtual WP-generated robots.txt file, then the required robots rules are added for you automatically. You do not need to add anything or make any changes to your robots.txt file in this case.

*OR* if your site is using a physical robots.txt on the server, then you need to add the robots rules manually. The next section below explains how to do this. Additional information may be found in the plugin documentation and Help tab (on the plugin settings page).

Related: This virtual robots tutorial provides some useful related information about robots.txt.

For physical robots.txt files..

To add the rules to a physical robots.txt file:

  1. Copy and paste the blackhole rules into your robots file, after any/all other existing rules.
  2. Save changes and upload to your server.

Done. Visit the “Useful Resources” section below for resources to check and validate your robots.txt file. For more complex/advanced robots configurations, consult your web developer.

Important: Proper robots.txt syntax is critical for good SEO. So make sure to validate your robots rules after making any changes. For more information check out my post, better robots rules for WordPress.

Why is this necessary?

Why are the robots.txt rules necessary? Because you only want to trap “bad” bots, not good bots. The Blackhole robots rules explicitly instruct all bots to NOT crawl the blackhole link. So any bots that follow the link are disobeying robots.txt and will be banned from further site access.

Useful Resources

Blackhole plugin resources:

Robots.txt resources:

Tip: Google also provides a robust robots.txt checker inside of your Google account (Google accounts are free). Worth checking out.

Lots more great resources on the web to learn about and validate your robots.txt file. Read up, it’s important for SEO.