Skip to main content

Streams of Streams (Nested Conveyor Belts)

Let's return to our example of the book factory we discussed in the previous section. Imagine a factory that produces books in multiple languages. Each language has its own conveyor belt. The factory has a conveyor belt for each language, and each language conveyor belt has its own stations. The factory also has a conveyor belt for the final printed pages. The final printed pages conveyor belt is connected to each language conveyor belt. This way, the final printed pages conveyor belt receives the output from each language conveyor belt.

Another way to look at this is the factory has multiple departments. Each department has its own smaller conveyor belts (streams) that feed into the main factory belt. These sub-belts could themselves have more sub-belts. In code, this is analogous to having a readable stream where each chunk is itself another readable stream.

Flattening the Streams (Unifying Conveyor Belts)​

Having nested streams is complex and challenging to manage. You would ideally want to "flatten" these into a single, unified stream. This is like merging all those smaller conveyor belts into one main line that's easier to manage.

In Node.js, you can manually listen to the data events on the inner streams and then write that data to an output stream. In Highland.js, the sequence() and flatMap() methods are often used for this purpose.

Separate Pipelines (Parallel Conveyor Belts)​

In a more advanced factory, you could have multiple parallel conveyor belts, each doing a separate job. Similarly, you can split a stream into multiple separate pipelines that operate concurrently.

In Node.js, you'd manually manage these pipelines, but Highland.js provides a fork() method to create these separate pipelines from a single source stream.

Example: Advanced Factory Analogy​

Let's extend the book printing factory example.

  1. Readable Stream: Raw text comes in, but this time, each chunk is a separate book (each book is a stream of chapters, and each chapter is a stream of paragraphs).

  2. Transform Streams:

  • First, a "flattening station" takes each book and unwraps it so that all chapters are laid out linearly.
  • Next, another "flattening station" takes each chapter and unwraps the paragraphs.
  1. Separate Pipelines: Once flattened, the text is forked into three parallel lines:
  • Line 1: Formatting
  • Line 2: Spell-checking
  • Line 3: Plagiarism checking
  1. Writable Stream: Finally, all lines merge, and the fully processed paragraphs get printed.

Code Concepts Mapping​

  • Readable Stream of Streams: Your initial data source where each chunk is itself a stream.
const bookStream: Readable // A stream of book streams
  • Flattening Streams: Using sequence() in Highland to handle nested streams.
bookStream.sequence() // Flattens the nested book streams
  • Separate Pipelines: Using fork() in Highland to create parallel pipelines.
const fork1 = mainStream.fork()
const fork2 = mainStream.fork()