Scraping multiple sites in parallel and memory consumption - Google Groups

Scraping multiple sites in parallel and memory consumption - Google Groups

Ross Chapman

Post reply       May 4 (2 days ago)

Other recipients:  

 

Depending on how thorough you're scraping, you can disable image loading, running javascript, and loading plugins (flash, silverlight, etc...) when constructing casper in the options argument.  This will definitely save memory.

Eg:

var casper = require('casper').create({

    loadImages:  false,

    javascriptEnabled: false,

    loadPlugins: false

});