spider trouble - block the tosser: UbiCrawler

Post your favourite links, jokes and anything you like, so long as its not work/coding related ;-)
KEEP it clean (well cleanish) thanks.
davidgwalker
MODS thats what I do that is! But definately NO mod music! It has to be ROCK!
Posts: 1807
Joined: Mon Oct 25, 2004 8:20 pm
Location: Huddersfield, UK
Contact:

spider trouble - block the tosser: UbiCrawler

Postby davidgwalker » Tue Feb 06, 2007 9:21 am

I noticed I had over 50,000 hits from UbiCrawler/v0.5beta (http://law.dsi.unimi.it/) within a few days of this month.

This was starting to drain my site... using my bandwidth for what! In the name of research - piss off, at my cost not!

So here is how to stop the bloody nosey little bastard:

Create a robots.txt file (if you do not already have one) and place it in your root folder.

Within that file add the following:

Code: Select all

User-agent: UbiCrawler
Disallow: /

You will find more details on robots.txt here:
http://www.robotstxt.org/wc/faq.html

Hope it helps you :)

Return to “General talk”

Who is online

Users browsing this forum: No registered users and 2 guests