Skip to content

Possibility to tap into already executed data #207

@stychu

Description

@stychu

Hi I want to hit my external API with bunch of requests. API allows for 30 TPS. I need to execute 500 requests which each request returns 100 objects that I need to save to the database. Is it possible to tap into the queue while the queue is still running and access already resolved data? It will be a bit heavy process on memory where I need to wait till the whole queue resolve and only then I can access my in memory result of 50 000 objects to be saved into database.

If I have pqueue as

const queue = new PQueue({
    intervalCap: 30,
    interval: 1000,
    carryoverConcurrencyCount: true,
  });

const result = await queue.addAll([500requestPromises])

I would like to be able to tap into the data and every batch save to database so approx. 17 writes to database. Is it possible ??

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions