This package simplifies transformations and filtering of NodeJS streams. Think about it as Underscore.js for streams.
The basic use case I faced many times was a transformation of a large number of JSON objects that are finally stored in some database. Transformation is the quick part but then you have to then chunk data in size allowed by your database to limit the number of queries and control the flow of the whole stream based on how fast you are able to save the transformed data.
Hey guys, today we’re showing a new open-source library that we have been working on for almost a year. It incorporates lessons learned from scraping of thousands of websites over the last 4 years. We figured there was no such universal library for JavaScript, while for example Python has one (https://scrapy.org/). That wasn’t fair, because JavaScript is THE language of the web :)
Anyway, we hope you’ll give it a shot and we’re really looking forward to hear what you think about it. All feedback welcome!
mtrunkat 1932 days ago. link 1 point ▲ ▼