Skip to content
Prompt Words
History

Streaming results

Also called: streaming, chunked processing.

The code reads a big result in small pieces, for example 1,000 rows at a time, and finishes each piece before it reads the next. Only one piece is in memory at once, so memory stays small and flat however big the data is.

Export 3,000,000 orders to a CSV file. The export job has a 2 GiB memory limit. Stream in chunks, then try loading everything first.

Rows read: 0 · Rows in the file: 0 · Peak memory: 200 MiB

Nothing exported yet. The job uses 200 MiB before it reads any rows.

    Idle

    Say it in a prompt

    Rewrite the orders CSV export to stream: read Postgres with a cursor (pg-cursor) 1,000 rows at a time, turn each chunk into CSV lines and write them to the file with stream.pipeline, then read the next chunk. Never load the whole table into an array. Memory must stay under 300 MiB for 3 million rows.

    Vague vs precise prompt

    Vague prompt

    the export crashes on big customers, fix it

    Typical resultRaises the container memory from 2 GiB to 8 GiB. It still runs SELECT * into one array, so the next, bigger customer crashes it again (in Node it may even die first with "JavaScript heap out of memory", when V8's own heap limit is lower than the container's).

    Precise prompt

    Stream the export: read orders with a Postgres cursor 1,000 rows at a time, write each chunk to the CSV file with stream.pipeline before reading the next. Keep memory under 300 MiB for any size.

    Typical resultMemory stays near 200 MiB for 3 million rows or 30 million. The export takes longer for bigger data, but it does not crash.

    Seen on

    • node-postgres: Its Cursor reads a large result set a few rows at a time, without loading the whole result into memory first.
    • PostgreSQL docs: FETCH reads the next rows from a cursor, for example FETCH FORWARD 1000, so a client can take a big query in pieces.
    • Node.js docs: Streams handle data chunk by chunk, and stream.pipeline connects a reader to a writer and manages the flow between them.

    You might describe it as

    • export a huge table without loading all of it
    • read a little, write a little, repeat
    • memory stays flat no matter how big the file is

    Not to be confused with

    • Server-sent events (SSE)

      Server-sent events stream messages to a browser over one open HTTP connection; streaming results means your code handles data in pieces so memory stays small.

    • OOM kill

      Streaming results keeps memory small by reading big data in pieces; an OOM kill is what happens when memory goes past the container's limit.

    • Backpressure

      Streaming results handles big data in small pieces so memory stays small; backpressure is the slow consumer making the fast producer wait, so the pieces don't pile up between them.