Web scraper open source
By On September 10, 2012 · Add Comment
What we know about open soucre web scrapping software?
There are many open source scrapers out there. They’re free, but they do require a good deal of time to setup.
At the very basic level, you can use wget which can easily be installed in almost any machine. It’s relatively trival to install for the mac or a linux based system. The great thing about wget is you can also ask it to follow links, so you can effectively “crawl” without having to enter in each URL of a given website manually.
Many of the popular programming languages have their own open source crawlers. Here’s a short list of some of the more stable ones I know of:
| Programming language | Features | ||
|---|---|---|---|
| Java | |||
| Nutch | Apache Nutch is an open source web-search software project. Stemming from Apache Lucene, it now builds on Apache Solr adding web-specifics, such as a crawler, a link-graph database and parsing support handled by Apache Tika for HTML and and array other document formats. | ||
| Heritrix | Heritrix is the Internet Archive’s open-source, extensible, web-scale, archival-quality web crawler project. Heritrix is designed to respect the robots.txt exclusion directives and META robots tags, and collect material at a measured, adaptive pace unlikely to disrupt normal website activity/ | ||
| WebSPHINX | WebSPHINX ( Website-Specific Processors for HTML INformation eXtraction) is a Java class library and interactive development environment for web crawlers. A web crawler (also called a robot or spider) is a program that browses and processes Web pages automatically. | ||
| Python | |||
| Scrapy | Scrapy is a fast high-level screen scraping and web crawling framework, used to crawl websites and extract structured data from their pages. It can be used for a wide range of purposes, from data mining to monitoring and automated testing. | ||
| Scrape.py | scrape.py is a Python module for scraping content from webpages. Using it, you can easily fetch pages, follow links, and submit forms. Cookies, redirections, and SSL are handled automatically. (For SSL, you either need a version of Python with the socket.ssl function, or the curl command-line utility. | ||
| HarvestMan | HarvestMan is a web crawler application written in the Python programming language. HarvestMan can be used to download files from websites, according to a number of user-specified rules. The latest version of HarvestMan supports as much as 60 plus customization options. HarvestMan is a console (command-line) application. HarvestMan is the only open source, multithreaded web-crawler program written in the Python language. HarvestMan is released under the GNU General Public License. | ||
| Mechanized (ported from the perl version) | Stateful programmatic web browsing in Python, after Andy Lester’s Perl module WWW:Mechanize. mechanize.Browser and mechanize.UserAgentBase implement the interface of urllib2.OpenerDirector, so: any URL can be opened, not just http. | ||
| Ruby | |||
| Anemone | Anemone is a Ruby library that makes it quick and painless to write programs that spider a website. It provides a simple DSL for performing actions on every page of a site, skipping certain URLs, and calculating the shortest path to a given page on a site. | ||
| Ruby: Not Really Crawlers but can be used like one | |||
| hpricot | Hpricot is a fast, flexible HTML parser written in C. It’s designed to be very accommodating (like Tanaka Akira’s HTree) and to have a very helpful library (like some JavaScript libs – JQuery, Prototype – give you.) The XPath and CSS parser, in fact, is based on John Resig’s JQuery.Also, Hpricot can be handy for reading broken XML files, since many of the same techniques can be used. If a quote is missing, Hpricot tries to figure it out. If tags overlap, Hpricot works on sorting them out. | ||
| Mechanize | The Mechanize library is used for automating interaction with websites. Mechanize automatically stores and sends cookies, follows redirects, and can follow links and submit forms. Form fields can be populated and submitted. Mechanize also keeps track of the sites that you have visited as a history. | ||
| Nokogiri | Nokogiri is an HTML, XML, SAX, and Reader parser. Among Nokogiri’s many features is the ability to search documents via XPath or CSS3 selectors. XML is like violence – if it doesn’t solve your problems, you are not using enough of it.FEATURES:XPath support for document searchingCSS3 selector support for document searchingXML/HTML builderNokogiri parses and searches XML/HTML very quickly, and also has correctly implemented CSS3 selector support as well as XPath support. | ||
| PHP | |||
| Snoopy | Snoopy is a PHP class that simulates a web browser. It automates the task of retrieving web page content and posting forms, for example. | ||
| PHPCrawl | PHPCrawl is a framework for crawling/spidering websites written in the programming language PHP, so just call it a webcrawler-library or crawler-engine for PHP. It provides several options to specify the behaviour of the crawler like URL- and Content-Type-filters, cookie-handling, robots.txt-handling, limiting options, multiprocessing and much more. | ||
| Erlang | |||
| eBot | Ebot, an Opensource Web Crawler built on top of a nosql database (apache couchdb, riak), AMQP database (rabbitmq), webmachine and mochiweb. Ebot is written in Erlang and it is a very scalable, distribuited and highly configurable web cawler. | ||
| Bixo | Bixo, an open source web mining toolkit that runs as a series of Cascading pipes on top of Hadoop. | ||
| DEiXTo | DEiXTo, a powerful tool for creating “extraction rules” (wrappers) that describe what pieces of data to scrape from a web page; consists of GUI and a stand-alone extraction rule executor. | ||
| GNU Wget | GNU Wget, command line tool for retrieving files using HTTP, HTTPS and FTP. | ||
| Pattern | Pattern, a web mining module for Python; bundles tools for data retrieval (Google + Twitter + Wikipedia API, web spider), text analysis (rule-based shallow parser, WordNet interface, tf-idf, …) and data visualization (graph networks). | ||
| ScraperWiki | ScraperWiki, a collaborative platform for web-scraping and screen-scraping code and views. | ||
| Scrapy | Scrapy, a fast high-level screen scraping and web crawling framework in Python. | ||
| Trapit | Trapit, system for personalizing content based on keywords, URLs and reading habits. | ||
| Web Mining Services | Web Mining Services, provides free, customized web extracts to meet your needs. | ||
| Web Mining Services | WebSundew, a powerful web scraping and web data extraction tool that extracts data from the web pages with high productivity and speed. |
Some more open Source solutions:
| WebHarvest | Written in JavaLeverages XSLT, Xquery and Regex to performs its scraping voodoo | ||
| BeautifulSoap | Written in PythonLeverages libraries like lxml and html5lib.I must mention that their client list includes notables like MovableType and Reddit, so I guess they have their game sorted out. | ||
| Solvent + Piggy Bank | These are firefox extensions written in Javascript, authored at MIT.Piggy Bank is actually a mashup module to aggregate and integrate info from various sites. Solvent is another add-on that works with Piggy Bank to develop screen scrapers.They have some nice screencasts to show you how their tool can scrape sites like craigslist and Starbucks coffee shops.Basic knowledge of Javascript is necessary |
Visual Software
If you’re in the market looking for something a bit less technically demanding, here are some offerings:
| IRobotSoft | This is a desktop app that will allow you to configure scraper flows and the data fields you wish to capture.It leverages something called HTQL (Hyper-TExt Query Language) to extract its web data.Price: Free | ||
| NeedleBase | A visual tool allowing you to easily create scrapers + gives you cool features like duplicate culling/merging data sets and all.Its pretty easy to use but I’m not sure how it performs when things get a wee bit complicated (e.g. with AJAX and all) Price: Free for low volume scrapes (login with your Google account)(I think for higher volumes you need to pay up) | ||
| 80 legs | 80 legs service lets you set up completely customized web crawlers that let you extract the data you need from websites. Here are just a few of the settings you can customize:Select which websites to crawl by entering URLs or uploading a seed listSpecify what data to extract by using a pre-built extractor or creating your ownRun a directed or general web crawlerSelect how many web pages you want to crawlChoose specific file types to analyze | ||
| Extractiv | Extractiv lets you transform unstructured web content into highly-structured semantic data.With our powerful web crawling and text extraction tools, you can:Crawl millions of domains every hourExtract tons of semantic information from the content on those domainsDo it all at highly-affordable prices | ||
| Arachnode.net | .NET architecture arachnode.net is the most comprehensive open source C#/.NET web crawler available. Use arachnode.net from any .NET language. | ||
| WebSundew | WebSundew is a powerful web scraping tool that extracts data from the web pages with high productivity and speed. WebSundew enables users to automate the whole process of extracting and storing information from the web sites. You can capture large quantities of bad-structured data in minutes at any time in any place and save results in any format. Our customers use WebSundew to collect and analyze the wide range of data that exists on the Internet related to their industry. | ||
| Scraperwiki.com | Scraperwiki.com have a web-based platform – or data hub – where programmers write scripts to get, clean and analyse data sets. They can simply schedule code to run automatically, and they can reuse the structured data with our flexible API. |
