1 Crawl a website with scrapy
scrapy is a crawler framework writtten in Python, Crawl a website with scrapy described how to use it.
I follow the guide and wrote a crawler, quite simple and easy to write.
I have written a crawler using BeatifulSoup, urllib2, pyquery, eventlet, MongoDB before, using Scrapy can simplify our work.
Have fun to use it！
2 Is there a better crawler than Scrapy?
Someone asked, "Is there a better crawler than Scrapy?" in quora, and in stackflow about crawler,
There also an introduction about open-source crawlers in wikipedia,
You can choose one which best suits your case, or you can write one yourself.
