[Node.js] Stream all things!

Node.js come alone with many Stream API. Stream is useful when handling large trunck of data.

For example, we have a big file to read from file system:

// create-big-file.js

const fs = require('fs')
const file = fs.createWriteStream('./big.file')

for (let i = 0; i <= 1e6; i++) {
    file.write('awefawgrga afewfa')
}

file.end();

 

If we are reading it using normal API:

const fs = require('fs')
const server = require('http').createServer();
server.on('request', (req, res) => {
    fs.readFile('./big.file', (err, data) => {
        if (err) throw err;

        res.end(data);
    })
});
server.listen(8000);

 

When running the code, we found that the normal memory cost is 8MB, when reading the large file, memory cost is 400+MB, basiclly it buffer all the file content into the memory, this is not the way we want.

 

Using Stream:

const fs = require('fs')
const server = require('http').createServer();
server.on('request', (req, res) => {
    /*fs.readFile('./big.file', (err, data) => {
        if (err) throw err;

        res.end(data);
    })*/
    const src = fs.createReadStream('./big.file')
    src.pipe(res)
});

server.listen(8000);

 

After updating the code, the memory usage of stream version stay around 30-40MB.