[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Stricter rules compliance
There seem to be a large number of grabbers running very aggressively against
the data with no user agent set at all. Therefore, various IP addresses
have been banned, and some simple checks are done against the user-agent
to ensure it is compliant.
Blocked XML fetches will get the following data returned:
Other fetches (including bulk downloads) will get the following HTML
If anyone's application is behaving correctly, but is now triggering
these let me know (it should be possible, however).
Andrew Flegg -- mailto:andrew@xxxxxxxx | http://www.bleb.org/