Stricter rules compliance

[Andrew Flegg <andrew@xxxxxxxx>]

Hi,

There seem to be a large number of grabbers running very aggressively against 
the data with no user agent set at all. Therefore, various IP addresses
have been banned, and some simple checks are done against the user-agent
to ensure it is compliant.

Blocked XML fetches will get the following data returned:

    http://www.bleb.org/tv/data/banned.xml

Other fetches (including bulk downloads) will get the following HTML
file:

    http://www.bleb.org/tv/data/banned.html

If anyone's application is behaving correctly, but is now triggering
these let me know (it should be possible, however).

Cheers,

Andrew

-- 
Andrew Flegg -- mailto:andrew@xxxxxxxx  |  http://www.bleb.org/