Node.js Streams Demystified: Process Large Files Without Killing Your Server

Introduction

Here's a mistake almost every Node.js developer makes at least once:

const data = fs.readFileSync("huge-file.csv", "utf8");
// Server runs out of memory. Crash.

When you load an entire file into memory, your server RAM usage spikes to at least the size of that file — often much more after parsing. For a 2 GB CSV file, that's a guaranteed crash on most servers.

Streams fix this by processing data in small chunks, keeping memory usage constant no matter how large the file is.

The four types of streams

• Readable — source of data (e.g., reading a file)
• Writable — destination for data (e.g., writing a file or HTTP response)
• Duplex — both readable and writable (e.g., a TCP socket)
• Transform — duplex stream that modifies data as it passes through (e.g., compression)

Example 1: Reading a large file with a Readable stream

import { createReadStream } from "fs";

const stream = createReadStream("huge-file.csv", { encoding: "utf8" });

stream.on("data", (chunk) => {
  console.log("Received chunk:", chunk.length, "bytes");
});

stream.on("end", () => {
  console.log("Done reading file");
});

stream.on("error", (err) => {
  console.error("Stream error:", err);
});

This reads the file in 64KB chunks by default. Memory stays flat no matter how big the file is.

Example 2: Piping streams

The real power of streams is piping — connecting a readable to a writable:

import { createReadStream, createWriteStream } from "fs";
import { createGzip } from "zlib";

const source = createReadStream("huge-file.csv");
const gzip = createGzip();
const destination = createWriteStream("huge-file.csv.gz");

source.pipe(gzip).pipe(destination);

destination.on("finish", () => {
  console.log("File compressed successfully");
});

This reads, compresses, and writes a file — all in streaming chunks, with almost no memory overhead.

Example 3: Transform stream — process CSV rows

import { Transform } from "stream";

const uppercaseTransform = new Transform({
  transform(chunk, encoding, callback) {
    this.push(chunk.toString().toUpperCase());
    callback();
  },
});

process.stdin
  .pipe(uppercaseTransform)
  .pipe(process.stdout);

Example 4: Streaming an HTTP response

You can stream files directly as Express responses — perfect for large downloads:

import express from "express";
import { createReadStream } from "fs";

const app = express();

app.get("/download", (req, res) => {
  res.setHeader("Content-Type", "text/csv");
  res.setHeader("Content-Disposition", "attachment; filename=data.csv");
  
  const fileStream = createReadStream("./data/huge-file.csv");
  fileStream.pipe(res);
});

Backpressure — the thing most tutorials skip

Backpressure happens when a readable stream produces data faster than the writable stream can consume it. Node.js handles this automatically when you use pipe(). If you write streams manually, always check the return value of writable.write() — if it returns false, pause the readable and resume on the drain event.

Conclusion

Streams are one of Node.js's most powerful features — and one of the most underused. Any time you're reading files, processing CSVs, handling uploads, or sending large responses, reach for streams first. Your server's RAM will thank you.