To help Blackhole Pro trap only bad bots, it’s important to add a few rules to your site’s
robots.txt file. This post explains the details.
Blackhole robots rules
In order for the Blackhole trap to work properly, it is important to add the provided robots rules to your site’s
robots.txt file. The rules are provided on the plugin settings screen in the “Robots Rules” section. They will look something like this:
User-agent: * Disallow: /*blackhole Disallow: /?blackhole
That’s just an example, make sure to visit the plugin settings page to get the latest/most current rules.
How to add the rules
For all websites, the
robots.txt file is located in the web-accessible root directory of the website. So for example, if your site is
example.com, the robots file will be located at the following URL:
If your site has a physical file on the server named “robots.txt”, then you can add the blackhole rules via copy/paste. Or, if your site does not have a physical file named “robots.txt”, then it means that WordPress is generating a robots file for you, automatically and dynamically.
robots.txtfile on the Blackhole settings page.
So with that in mind, there are two possible scenarios for adding the blackhole rules:
If your site is using the virtual WP-generated
robots.txt file, then the required robots rules are added for you automatically. You do not need to add anything or make any changes to your robots.txt file in this case.
*OR* if your site is using a physical
robots.txt on the server, then you need to add the robots rules manually. The next section below explains how to do this. Additional information may be found in the plugin documentation and Help tab (on the plugin settings page).
For physical robots.txt files..
To add the rules to a physical
- Copy and paste the blackhole rules into your robots file, after any/all other existing rules.
- Save changes and upload to your server.
Done. Visit the “Useful Resources” section below for resources to check and validate your
robots.txt file. For more complex/advanced robots configurations, consult your web developer.
robots.txtsyntax is critical for good SEO. So make sure to validate your robots rules after making any changes. For more information check out my post, better robots rules for WordPress.
Why is this necessary?
Why are the
robots.txt rules necessary? Because you only want to trap “bad” bots, not good bots. The Blackhole robots rules explicitly instruct all bots to NOT crawl the blackhole link. So any bots that follow the link are disobeying
robots.txt and will be banned from further site access.
Blackhole plugin resources:
- Learn more about robots.txt
- Validate your robots.txt file
- Check your robots.txt rules
- Google’s robots.txt specifications
- How to create a robots.txt file
Lots more great resources on the web to learn about and validate your
robots.txt file. Read up, it’s important for SEO.