I should mention RE the below post about a web crawler. I have been writing and running them for a few years now with each one getting more and more stable and learning more and more about each one. I feel pretty confident that when I fire one up now that it is unlikely to crash someones site or suck down everything on the web.

Published on Wednesday 17th February 2010

Comments (0)