The Ultimate Guide to Working with Files in Node

Working with files in Node.js is easy because we can make use of the fs built-in module for interacting with files.

const fs = require('fs/promises');

async function example() {
  try {
    const data = await fs.readFile('/Users/joe/test.txt', { encoding: 'utf8' });

    console.log(data);
  } catch (err) {
    console.log(err);
  }
}

example();

Using the readFile method from fs/promises module, we are spared of using callbacks, while still having the asynchronous behavior. One thing to note here, the readFile method will read the full content of the file in memory before returning the data.

Writing files is just as easy as reading files when working with Node.js File System.

const fs = require('fs/promises');

async function example() {
  try {
    const content = 'Some content!';

    await fs.writeFile('/Users/joe/test.txt', content);
  } catch (err) {
    console.log(err);
  }
}

example();

writeFile method has 3 arguments: path, data (the content), and options.

fs.writeFile('/Users/joe/test.txt', content, { flag: 'a' });

The most commonly used flags are:

r opens the file for reading

w opens the file for writing and the file pointer is positioned at the beginning of the file

a opens the file for writing and the file pointer is positioned at the end of the file

If you are interested in reading more about all the available flags, here is a full list.

Alright, so, we’ve seen how to read a file and write a file and we also dived into some flags that are available when manipulating files. This is already more than enough for the vast majority of use cases. The only thing is, we’ve barely scratched the surface in the context of file manipulation capabilities with Node.js.

Streams

Streams are a core concept in Node.js when working with files. The entire fs module uses streams, which means the developer doesn’t have to reinvent the wheel for common tasks like reading a file. However, there are cases when fs is not enough, or streams can’t be avoided. Some common use cases are:

• handling large amounts of data
• data processing
• real-time communication
• video/audio streaming
• usages of 3rd parties that enforce streams

There are 4 types of streams.

• readable streams used to read data from a source in chunks
• writable streams used to write data to a source in chunks
• duplex streams used to read and write data in chunks and implement both, readable and writable streams, enabling bidirectional communication
• transform streams can read and write data, but are used for transforming/parsing/filtering data

Put everything into practice

Assuming we want to read data from a file, this would be an oversimplified example highlighting the key concepts:

We have a file representing binary data, data in the format of 0s and 1s. The binary data is encoded in such a manner that we will see it in a human-readable format like ASCII, UTF-8, or UTF-16.

A chunk is a small segment of data. A buffer is a class in Node.js that is the representation of a fixed-size chunk. For example, if we want to represent the string ‘Hello’ this would be its representation using a buffer <Buffer 48 65 6c 6c 6f> (can be checked using a console.log statement) and a chunk would be just the pure data, so 48 65 6c 6c 6f. When reading a file using a stream, the data is read in chunks, and as the readable stream emits a data event, a new chunk of data becomes ready for processing (represented as a buffer).

How to read the content of a file using streams:

const fs = require('fs/promises');

async function readFromFile(filename) {
  const readStream = fs.createReadStream(filename);

  let data = '';

  readStream.on('data', (chunk) => {
    data += chunk;
  });

  readStream.on('end', () => {
    console.log(data);
  });

  readStream.on('error', (err) => {
    console.error(err);
  });
}

readFromFile('example.txt');

Comparing this to the code snippet at the beginning where we read a file using fs, I think we can see the extra complexity that comes with the usage of streams. The only thing is, sometimes you don’t really have a choice. If you need performance or putting everything in memory at the same time is not an option, you can always go for a lower-level approach and use streams.

Let’s see another example…

As a web developer, a common scenario that you might run into is downloading an image from a cloud provider to your backend server and serving it to the client.

const http = require('http');

const AWS = require('aws-sdk');
const s3 = new AWS.S3();

http
  .createServer(function (req, res) {
    const s3params = {
      Bucket: 'my-bucket',

      Key: 'my-image.jpg'
    };

    const s3stream = s3.getObject(s3params).createReadStream();

    s3stream.pipe(res);
  })
  .listen(8080);

In this example, http was used, but the same applies to express. Make the API call, retrieve the data, create a readable stream, and pipe it into res parameter which is a stream. We didn’t cover piping, but it is a feature that simplifies transferring data from one stream to another. Didn’t know that? Don’t worry, you should just be aware of the concepts discussed and know that you can always google stuff or use Chat GPT, especially nowadays when documentation is so easily accessible with all the new tools.