Streams and Buffers

7 pages · ~5 min✓ Reviewed

Builds on The Node.js Event Loop. Next up: Express Routing and Middleware.

Node.jsPage 1

Buffers: Raw Bytes in Node

1. What a Buffer is

Fixed-size chunk of raw bytes; cannot be resized
A Uint8Array subclass
Allocated outside the V8 heap

2. Create and convert

Call	Result
`Buffer.from('hi', 'utf8')`	bytes of the string
`Buffer.alloc(n)`	zero-filled
`Buffer.allocUnsafe(n)`	faster, may hold old memory
`buf.toString('utf8'\|'hex'\|'base64')`	bytes to string
`Buffer.concat([a, b])`	new buffer, copies
`buf.subarray()` / `slice()`	shares memory, no copy

3. Gotchas

allocUnsafe: overwrite before use, or old data leaks
Writing to a slice changes the original
chunk.toString() per chunk splits multi-byte characters

javascript

r.setEncoding('utf8') // or a decoder:
const d = new StringDecoder('utf8')
text += d.write(chunk)

decoders hold back partial characters

4. Interview Point

Why is allocUnsafe faster? It skips zero-filling; the risk is leaking old memory
Does slice copy? No, it shares memory
Decode chunks safely? setEncoding('utf8') or StringDecoder

Node.jsPage 2

The Four Stream Types

1. What a Stream Is

Data moves in chunks, so memory stays flat at any size
Streams are EventEmitters
Chunk is a Buffer, or a string after setEncoding()

2. The Four Types

Type	Role	Example
Readable	source	`fs.createReadStream`, HTTP request
Writable	sink	`fs.createWriteStream`, HTTP response
Duplex	read + write, independent	TCP socket
Transform	Duplex, output from input	`zlib.createGzip`, crypto

r.on('data') and 'end' events
for await (const chunk of r)
r.pipe(w)

Paused until a data listener, pipe() or resume() starts flowing

3. Writable and Transform

write(chunk) returns false when the buffer is full
Then end(); listen for drain, finish, error
Custom Transform: transform(chunk, enc, cb), output via cb(null, out)

javascript

const up = new Transform({
  transform(chunk, enc, cb) { cb(null, chunk.toString().toUpperCase()) }
})

Interview

Transform is a Duplex. objectMode carries any value, not just Buffers, and is an option on every type.

Node.jsPage 3

pipe, pipeline and Backpressure

1. Backpressure

Slow consumer tells fast producer to stop
write() returns false → wait for 'drain'
Threshold is highWaterMark (buffer size)

Stream	highWaterMark default
Bytes, Node 22+	64 KiB
Bytes, Node 21 and earlier	16 KiB
`objectMode`	16 objects

2. pipe vs pipeline

	pipe	pipeline
Backpressure	yes	yes
Forwards errors	no	yes
Destroys streams	no	yes

javascript

import { pipeline } from 'node:stream/promises';
await pipeline(
  fs.createReadStream('in.log'),
  zlib.createGzip(),
  fs.createWriteStream('in.log.gz'));

Callback form: stream.pipeline(src, ...transforms, dest, cb)
Async generator functions work as middle stages

3. Interview Point

Backpressure? write() false → pause, resume on 'drain'; pipe automates it.
Why pipeline? pipe leaks on error; pipeline forwards and destroys.
highWaterMark? Buffer limit before write() returns false.
Ignore false? Buffer grows unbounded, memory balloons.

Prefer pipeline

pipe() leaves streams open on failure → leaked file descriptors.

Node.jsPage 4

Reading Files Efficiently

1. readFile Loads It All

fs.readFile / fs.promises.readFile load the whole file into memory
Fine for small files
Large files: memory spikes, may hit the Buffer size limit

2. Chunks and Lines

Default chunk of a file read stream: 64 KiB
A larger highWaterMark means fewer syscalls on big files
{ start, end } reads a byte range
Big JSON: use NDJSON, one object per line

javascript

// bigger chunks, fewer syscalls
fs.createReadStream(p, { highWaterMark: 256 * 1024, encoding: 'utf8' })
// byte range, a separate option
fs.createReadStream(p, { start: 0, end: 999 })

javascript

const rl = readline.createInterface({
  input: fs.createReadStream(p), crlfDelay: Infinity });
for await (const line of rl) {
  const obj = JSON.parse(line); // one NDJSON row
}

constant memory, one line at a time

3. Serve and Copy

Job	Use
Serve a big file	`pipeline(fs.createReadStream(p), res)`
Copy a file	`fs.copyFile(src, dest)`, no JS-land buffering

Remember

Never readFile a big file to serve it. Stream it with pipeline.

Node.jsPage 5

Common Stream Mistakes

1. pipe() and errors

pipe() forwards no errors and destroys nothing
One stage fails, the others stay open: file descriptor leak
Fix: pipeline() destroys every stage and rejects once

javascript

const { pipeline } = require('node:stream/promises');
try { await pipeline(src, gzip, dest); }
catch (err) { console.error(err); }

2. Ignored write()

If write() returns false, wait for 'drain' before writing more
Ignored in a loop: buffer grows without bound, memory balloons

javascript

const { once } = require('node:events');
for (const row of rows) {
  if (!w.write(row)) await once(w, 'drain');
}
w.end();

3. Errors and chunks

Mistake	Why	Instead
`'error'` listener late or missing	Unhandled `'error'` crashes the process	Attach before data flows
`data += chunk`	Defeats streaming, splits multi-byte characters	`setEncoding('utf8')` or `Buffer.concat(chunks)`

Remember

Use pipeline, honour write() and 'drain', handle 'error', never build strings from raw chunks.

Node.jsPage 6

Buffers and Streams Cheat Sheet

1. Buffers

Call	Does
`Buffer.from(x, enc)`	build from string or array
`Buffer.alloc(n)`	zero-filled
`Buffer.allocUnsafe(n)`	fast, may hold old memory, overwrite it
`Buffer.concat([...])`	join buffers
`buf.toString(enc)`	decode to string
`buf.subarray()`	shares memory, no copy

2. Stream types

Readable: source
Writable: sink
Duplex: both sides independent
Transform: output derived from input
objectMode: streams carry non-Buffer values

3. Backpressure

write() === false means wait for 'drain'

Stream	highWaterMark
Byte streams, Node 22+	64 KiB (16 KiB on Node 21 and earlier)
`objectMode`	16 objects

4. pipeline and files

javascript

import { pipeline } from 'node:stream/promises'
await pipeline(
  fs.createReadStream('in.log'),
  zlib.createGzip(),
  fs.createWriteStream('in.log.gz'))

errors propagate, all streams destroyed

Use pipeline(a, b, c) over .pipe(): errors propagate
Small file: readFile
Large file: createReadStream + pipeline
Lines: readline + for await
Text chunks: setEncoding('utf8')

Node.jsPage 7

Check yourself

A loop calls w.write(line) a million times on a file stream and ignores the result. What goes wrong?

Memory balloons: write() returns false when full, but data keeps queueing.
Fix: when false, await events.once(w, 'drain').

What does this print?

Hello: subarray shares memory, no copy.
72 is the code for 'H'.

const b = Buffer.from('hello');
const s = b.subarray(0, 2);
s[0] = 72;
console.log(b.toString());

In src.pipe(gzip).pipe(dest), dest fails with an error. What happens to src and gzip?

They stay open, leaking file descriptors: pipe forwards no errors.
Use pipeline, which destroys every stage.