A C# Web Crawler

Here is a C# Web Crawler that could help you download web pages and attachments from one web-site. Such kind of crawler used vastly among some online stores during their beginning stage. They called them network spider, robot, crawler or radar.  They were trained well to retrieve only some kind of information that not like Google, Baidu does. They service their own small and special group of customizers.

The logic behind the web crawler is very easy: given by one url, the crawler will download this page, and parse and find all Urls. It repeat the whole process again for all those new found urls until all Urls were visited.  The full source code could be download from here.

 

Something need to fix
 
This crawler could not work well when come across some web sites. I could access those web sites from the FireFox or other web browser. But it seems this crawler could not access those web server, error ‘<404> Not Found 'always report. If any one have some well work solutions, please let me know, thanks!  Maybe I could use external tools to download the web content instead of using native C# web API, may be I could use ‘wget’or something similar.

Reference
http://www.digitalcoding.com/Code-Snippets/C-Sharp/C-Code-Snippet-Download-HTML-Web-Page.html