Skip to main content

Streams

What is a Stream in programming?​

At its core, a stream is like a conveyor belt in a factory. Items (or chunks of data) get placed on one end, and as the belt moves, each station along the belt may either inspect, modify, or utilize these items until they reach the other end.

Types of Streams (Conveyor Belts)

  • Readable Stream (Input Conveyor Belt): Think of it as a conveyor belt bringing raw material into the factory. You can only take items off of it. In code, you can only read data from it. Another way to think of this is Readable Streams are sources of data or generate data.

  • Writable Stream (Output Conveyor Belt): Imagine this as the conveyor belt that takes the final product out of the factory to be shipped. You can only place items onto it. In code, you can only write data to it. Another way to think of this is Writable Streams are sinks of data or consume data.

  • Duplex Stream (Two-way Conveyor Belt): This is like having a single conveyor belt that can carry items both into and out of the factory. In code, you can both read from and write to it. Another way to think of this is Duplex Streams are both sources and sinks of data or both generate and consume data.

  • Transform Stream (Processing Station): Think of this as a specialized station along the conveyor belt that can modify items as they pass by. It's a type of duplex stream, but the output is a transformation of the input.

Data Events (Factory Signals)

  • data: Signaled when a new item (chunk of data) arrives at a station.
  • end: Signaled when there are no more items coming, i.e., the factory shift is over.
  • error: Signaled if something goes wrong in the process, like the conveyor belt jamming.

Backpressure (Factory Overload)

  • If items arrive on the input conveyor belt too fast for the factory to process, you have a problem—this is analogous to backpressure. In streams, mechanisms exist to slow down the input if the process can't keep up.

How Do Streams Work?​

Chunk by Chunk Streams work with data in small pieces (or "chunks"). Just like a factory doesn't get all its raw material for the entire day at once, streams don't have to load the entire data set into memory; they work on small pieces at a time. This makes them highly memory-efficient.

Event-Driven Streams are event-driven. That means as soon as a chunk of data is available to be read or has been successfully written, an event is fired. Your code listens to these events and reacts accordingly.

Pipe The pipe method is like linking multiple conveyor belts together. The output from one becomes the input to the next. This way, you can form a whole processing chain for your data.

Simple Real-world Analogy​

Imagine a book printing factory. Raw text (Readable Stream) comes in at one end. The text goes through several stations where it gets formatted, spell-checked, and eventually printed (Transform Streams). The final printed pages (Writable Stream) then exit the factory.

  • Readable Stream: Raw text
  • Transform Streams: Formatting, spell-checking
  • Writable Stream: Printed pages

The factory doesn't read the entire text or print the whole book at once; it processes line by line, efficiently utilizing its resources. If the text comes in too fast, the factory slows it down to match its processing speed. For example, if the spell-checking station is overloaded, the formatting station will slow down to match the speed of the spell-checking station.