Using streamObject and streamArray together?

[faransajid]

Hi, Sorry if I'm missing something here but I am unable to get the desired results using the parser.

Here's my json data:

I need to get the following complete objects:

• personal_information
• profile_pictures
• contacts

And then I want chats>list array in streams (because that node is the one that can grow very big)

I was trying to do something as follows:
const pipeline = chain([ createReadStream(jsonFile), parser(), // to get complete objects excluding chats node pick({filter: 'personal_information'}), pick({filter: 'profile_pictures'}), pick({filter: 'contacts'}), ignore({filter: 'chats'}), streamObject(), // to stream the list array from chats node pick({filter: 'chats'}), pick({filter: 'list'}), streamArray(), ]);

Any help is appreciated, thanks.

[uhop]

It looks like you would need two streams:

• One stream to pick up complete objects. Ignore huge parts (chats) and convert the rest to a stream of objects with StreamArray.
• Another stream for huge items. Pick chats correlating them to complete objects from the previous stream.

Alternatively, you can always consume raw parser tokens to make your processing pipeline as flexible and as efficient as possible.

[faransajid]

Thank you @uhop . Could you please assist me a little more?

I am not sure how to combine these two streams together in a single file read. Or Do I need to read the file twice?
Whenever I chain both streams in a single pipeline, I don't get any data.

Also, could you provide me an example for raw parser tokens for my case?

Many thanks!