This project is read-only.


Abot C# Web Crawler

Abot is an open source C# web crawler built for speed and flexibility. It takes care of the low level plumbing (multithreading, http requests, scheduling, link parsing, etc..). You just register for events to process the page data. You can also plugin your own implementations of core interfaces to take complete control over the crawl process.

Documentation: Docs live on Google Code

Source Code: Code lives on Github 

Last edited Dec 25, 2014 at 7:37 PM by sjdirect, version 6