Project Description

SharkCrawler is a simple web page crawler written in C# for demonstration of how regular crawler is working.

Features

  1. Crawl from a specified starting page
  2. Pause and Resume crawling
  3. Specify how many threads are used to crawl
  4. Use local file system to store crawled pages
  5. Track requests and errors

Last edited Oct 21, 2012 at 7:39 AM by aaroncheung, version 2