Node.js Streams Demystified: Process Large Files Without Killing Your Server

Introduction

Here's a mistake many Node.js developers make at least once:

const data = fs.readFileSync("huge-file.csv", "utf8");

// Server runs out of memory. Crash.

When you load an entire file into memory, your application's RAM usage grows to at least the size of that file—and often much more after parsing. For very large files, this can easily exhaust available memory.

Streams solve this problem by processing data in small chunks, keeping memory usage nearly constant regardless of file size.

The Four Types of Streams

• Readable — A source of data (for example, reading a file).
• Writable — A destination for data (for example, writing to a file or HTTP response).
• Duplex — Both readable and writable (for example, a TCP socket).
• Transform — A duplex stream that modifies data as it passes through (for example, compression or encryption).

Example 1: Reading a Large File

import { createReadStream } from "fs";

const stream = createReadStream("huge-file.csv", {
  encoding: "utf8",
});

stream.on("data", (chunk) => {
  console.log("Received chunk:", chunk.length, "bytes");
});

stream.on("end", () => {
  console.log("Done reading file");
});

stream.on("error", (err) => {
  console.error("Stream error:", err);
});

By default, createReadStream() reads the file in chunks instead of loading the entire file into memory.

Example 2: Piping Streams

One of the biggest advantages of streams is piping them together.

import {
  createReadStream,
  createWriteStream,
} from "fs";
import { createGzip } from "zlib";

const source = createReadStream("huge-file.csv");
const gzip = createGzip();
const destination = createWriteStream("huge-file.csv.gz");

source.pipe(gzip).pipe(destination);

destination.on("finish", () => {
  console.log("File compressed successfully");
});

This pipeline:

1. Reads the file.
2. Compresses it.
3. Writes the compressed output.

All of this happens incrementally with minimal memory overhead.

Example 3: Transform Streams

Transform streams let you modify data while it flows through the pipeline.

import { Transform } from "stream";

const uppercaseTransform = new Transform({
  transform(chunk, encoding, callback) {
    this.push(chunk.toString().toUpperCase());
    callback();
  },
});

process.stdin
  .pipe(uppercaseTransform)
  .pipe(process.stdout);

In this example, everything typed into standard input is converted to uppercase before being written to standard output.

Example 4: Streaming an HTTP Response

Streams are also ideal for serving large downloads.

import express from "express";
import { createReadStream } from "fs";

const app = express();

app.get("/download", (req, res) => {
  res.setHeader("Content-Type", "text/csv");
  res.setHeader(
    "Content-Disposition",
    "attachment; filename=data.csv"
  );

  const fileStream = createReadStream("./data/huge-file.csv");

  fileStream.pipe(res);
});

Instead of loading the entire file into memory, the server streams it directly to the client.

Understanding Backpressure

Backpressure occurs when a readable stream produces data faster than a writable stream can consume it.

When you use pipe(), Node.js automatically manages backpressure for you.

If you're writing streams manually, always check the return value of writable.write():

• If it returns true, continue writing.
• If it returns false, pause the readable stream and wait for the writable stream's drain event before resuming.

Proper backpressure handling prevents excessive memory usage and improves application stability.

Conclusion

Streams are one of Node.js's most powerful features, especially when working with large files, uploads, downloads, CSV processing, or data transformations.

Instead of loading everything into memory, streams process data incrementally, making applications more memory-efficient and scalable. Whenever you're dealing with large amounts of data, streams are usually the right tool for the job.