Skip to content

Add aliases #16

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 27 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -123,10 +123,17 @@ The package is available on npm as [@shutterstock/p-map-iterable](https://www.np
## Importing

```typescript
// Original class names
import {
IterableMapper,
IterableQueueMapper,
IterableQueueMapperSimple } from '@shutterstock/p-map-iterable';

// Or use the semantic aliases (recommended for new code)
import {
Prefetcher,
BackgroundFlusher,
SimpleBackgroundFlusher } from '@shutterstock/p-map-iterable';
```

## API Documentation
Expand All @@ -151,21 +158,23 @@ These diagrams illustrate the differences in operation betweeen `p-map`, `p-queu

# Features

- [IterableMapper](https://tech.shutterstock.com/p-map-iterable/classes/IterableMapper.html)
- [Prefetcher (IterableMapper)](https://tech.shutterstock.com/p-map-iterable/classes/IterableMapper.html)
- Interface and concept based on: [p-map](https://github.com/sindresorhus/p-map)
- Allows a sync or async iterable input
- User supplied sync or async mapper function
- Exposes an async iterable interface for consuming mapped items
- Allows a maximum queue depth of mapped items - if the consumer stops consuming, the queue will fill up, at which point the mapper will stop being invoked until an item is consumed from the queue
- This allows mapping with backpressure so that the mapper does not consume unlimited resources (e.g. memory, disk, network, event loop time) by racing ahead of the consumer
- [IterableQueueMapper](https://tech.shutterstock.com/p-map-iterable/classes/IterableQueueMapper.html)
- Also available as `IterableMapper`
- [BackgroundFlusher (IterableQueueMapper)](https://tech.shutterstock.com/p-map-iterable/classes/IterableQueueMapper.html)
- Wraps `IterableMapper`
- Adds items to the queue via the `enqueue` method
- [IterableQueueMapperSimple](https://tech.shutterstock.com/p-map-iterable/classes/IterableQueueMapperSimple.html)
- Also available as `IterableQueueMapper`
- [SimpleBackgroundFlusher (IterableQueueMapperSimple)](https://tech.shutterstock.com/p-map-iterable/classes/IterableQueueMapperSimple.html)
- Wraps `IterableQueueMapper`
- Discards results as they become available
- Exposes any accumulated errors through the `errors` property instead of throwing an `AggregateError`
- Not actually `Iterable` - May rename this before 1.0.0
- Also available as `IterableQueueMapperSimple`

## Lower Level Utilities
- [IterableQueue](https://tech.shutterstock.com/p-map-iterable/classes/IterableQueue.html)
Expand Down Expand Up @@ -204,6 +213,20 @@ See [examples/iterable-queue-mapper-simple.ts](./examples/iterable-queue-mapper-

Run the example with `npm run example:iterable-queue-mapper-simple`

# Semantic Aliases (Prefetcher, BackgroundFlusher, SimpleBackgroundFlusher)

The library now provides semantic aliases for the main classes to better describe their typical use cases:

- `Prefetcher` - Alias for `IterableMapper`: Processes items from an iterable source in the background before they're needed
- `BackgroundFlusher` - Alias for `IterableQueueMapper`: Processes items in the background with results accessible via iteration
- `SimpleBackgroundFlusher` - Alias for `IterableQueueMapperSimple`: Processes items in the background, automatically discarding results

These aliases make the code more intuitive and descriptive of the actual data flow patterns being implemented.

See [examples/semantic-aliases.ts](./examples/semantic-aliases.ts) for examples of using these aliases.

Run the example with `npm run example:semantic-aliases`

# Contributing - Setting up Build Environment

- `nvm use`
Expand Down
126 changes: 126 additions & 0 deletions examples/semantic-aliases.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,126 @@
/**
* Example demonstrating the semantic alias types
*/
import { Prefetcher, BackgroundFlusher, SimpleBackgroundFlusher } from '../src';

async function delay(ms: number): Promise<void> {
return new Promise((resolve) => setTimeout(resolve, ms));
}

/**
* Example showing how to use the Prefetcher (IterableMapper)
*/
async function prefetcherExample() {
console.log('\n--- Prefetcher Example ---');

Check warning on line 14 in examples/semantic-aliases.ts

View workflow job for this annotation

GitHub Actions / build

Unexpected console statement

// Source data - in real apps this could be database IDs, file paths, etc.
const sourceIds = [1, 2, 3, 4, 5];

// Simulate a slow source read operation (like a database or file read)
const readSource = async (id: number) => {
console.log(`Reading source ${id}...`);

Check warning on line 21 in examples/semantic-aliases.ts

View workflow job for this annotation

GitHub Actions / build

Unexpected console statement
await delay(300); // Simulate I/O operation
return { id, data: `Data for ${id}` };
};

// Create a prefetcher that will fetch items ahead of time
const prefetcher = new Prefetcher(sourceIds, readSource, {
concurrency: 2, // Process 2 reads at a time
maxUnread: 3, // Don't get more than 3 items ahead of consumer
});

// Process items as they become available
for await (const item of prefetcher) {
console.log(`Processing item: ${item.id}`);

Check warning on line 34 in examples/semantic-aliases.ts

View workflow job for this annotation

GitHub Actions / build

Unexpected console statement
await delay(200); // Simulate processing time
}

console.log('Prefetcher complete');

Check warning on line 38 in examples/semantic-aliases.ts

View workflow job for this annotation

GitHub Actions / build

Unexpected console statement
}

/**
* Example showing how to use the BackgroundFlusher (IterableQueueMapper)
*/
async function backgroundFlusherExample() {
console.log('\n--- BackgroundFlusher Example ---');

Check warning on line 45 in examples/semantic-aliases.ts

View workflow job for this annotation

GitHub Actions / build

Unexpected console statement

// Simulate a slow destination write operation
const writeDestination = async (item: { id: number; processed: boolean }) => {
console.log(`Flushing item ${item.id} to destination...`);

Check warning on line 49 in examples/semantic-aliases.ts

View workflow job for this annotation

GitHub Actions / build

Unexpected console statement
await delay(400); // Simulate I/O operation
return { id: item.id, status: 'written' };
};

// Create a background flusher that will handle writes in the background
const flusher = new BackgroundFlusher(writeDestination, {
concurrency: 2, // Process 2 writes at a time
});

// Simulate generating some data that needs to be flushed
for (let i = 1; i <= 5; i++) {
const item = { id: i, processed: true };
console.log(`Enqueueing item ${i} for flushing`);

Check warning on line 62 in examples/semantic-aliases.ts

View workflow job for this annotation

GitHub Actions / build

Unexpected console statement
await flusher.enqueue(item);
await delay(100); // Simulate time to prepare next item
}

// Mark the queue as done - no more items will be added
flusher.done();

// Iterate through the results if needed
console.log('Checking flush results:');

Check warning on line 71 in examples/semantic-aliases.ts

View workflow job for this annotation

GitHub Actions / build

Unexpected console statement
for await (const result of flusher) {
console.log(`Item ${result.id} flush status: ${result.status}`);

Check warning on line 73 in examples/semantic-aliases.ts

View workflow job for this annotation

GitHub Actions / build

Unexpected console statement
}

console.log('BackgroundFlusher complete');

Check warning on line 76 in examples/semantic-aliases.ts

View workflow job for this annotation

GitHub Actions / build

Unexpected console statement
}

/**
* Example showing how to use the SimpleBackgroundFlusher (IterableQueueMapperSimple)
*/
async function simpleBackgroundFlusherExample() {
console.log('\n--- SimpleBackgroundFlusher Example ---');

// Simulate a slow destination write operation where the result isn't needed
const writeDestination = async (item: { id: number; processed: boolean }) => {
console.log(`Writing item ${item.id} to destination...`);
await delay(300); // Simulate I/O operation
// No return value needed - operation is fire and forget
};

// Create a simple background flusher
const flusher = new SimpleBackgroundFlusher(writeDestination, {
concurrency: 2, // Process 2 writes at a time
});

// Simulate generating some data that needs to be written
for (let i = 1; i <= 5; i++) {
const item = { id: i, processed: true };
console.log(`Enqueueing item ${i} for writing`);
await flusher.enqueue(item);
await delay(100); // Simulate time to prepare next item
}

// Wait for all background operations to complete
console.log('Waiting for all writes to complete...');
await flusher.onIdle();

// Check if there were any errors
if (flusher.errors.length > 0) {
console.error('Errors occurred during background processing:', flusher.errors);
} else {
console.log('All writes completed successfully');
}

console.log('SimpleBackgroundFlusher complete');
}

// Run all examples
async function main() {
await prefetcherExample();
await backgroundFlusherExample();
await simpleBackgroundFlusherExample();
}

main().catch(console.error);
1 change: 1 addition & 0 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@
"example:iterable-mapper": "ts-node -r tsconfig-paths/register examples/iterable-mapper.ts",
"example:iterable-queue-mapper": "ts-node -r tsconfig-paths/register examples/iterable-queue-mapper.ts",
"example:iterable-queue-mapper-simple": "ts-node -r tsconfig-paths/register examples/iterable-queue-mapper-simple.ts",
"example:semantic-aliases": "ts-node -r tsconfig-paths/register examples/semantic-aliases.ts",
"test": "jest",
"lint": "eslint ./ --ext .ts --ext .tsx",
"lint-and-fix": "eslint ./ --ext .ts --ext .tsx --fix"
Expand Down
40 changes: 39 additions & 1 deletion src/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,16 +2,54 @@ import { Mapper, IterableMapper, IterableMapperOptions } from './iterable-mapper
import { BlockingQueue, BlockingQueueOptions } from './blocking-queue';
import { IterableQueue, IterableQueueOptions } from './iterable-queue';
import { IterableQueueMapper, IterableQueueMapperOptions } from './iterable-queue-mapper';
import { IterableQueueMapperSimple } from './iterable-queue-mapper-simple';
import {
IterableQueueMapperSimple,
IterableQueueMapperSimpleOptions,
} from './iterable-queue-mapper-simple';
import { Queue } from './queue';

// Create class aliases with more descriptive names
/**
* Prefetcher - Processes items from an iterable source in the background before they're needed.
* This is an alias for IterableMapper.
*/
export const Prefetcher = IterableMapper;
export type PrefetcherOptions = IterableMapperOptions;

/**
* BackgroundFlusher - Processes items in the background with results accessible via iteration.
* This is an alias for IterableQueueMapper.
*/
export const BackgroundFlusher = IterableQueueMapper;
export type BackgroundFlusherOptions = IterableQueueMapperOptions;

/**
* SimpleBackgroundFlusher - Processes items in the background, automatically discarding results.
* This is an alias for IterableQueueMapperSimple.
*/
export const SimpleBackgroundFlusher = IterableQueueMapperSimple;
export type SimpleBackgroundFlusherOptions = IterableQueueMapperSimpleOptions;

// For TypeScript type compatibility
export type Prefetcher<InputElement = unknown, OutputElement = unknown> = InstanceType<
typeof IterableMapper<InputElement, OutputElement>
>;
export type BackgroundFlusher<InputElement = unknown, OutputElement = unknown> = InstanceType<
typeof IterableQueueMapper<InputElement, OutputElement>
>;
export type SimpleBackgroundFlusher<Element = unknown> = InstanceType<
typeof IterableQueueMapperSimple<Element>
>;

// Export all types and classes
export {
Mapper,
IterableMapper,
IterableMapperOptions,
IterableQueueMapper,
IterableQueueMapperOptions,
IterableQueueMapperSimple,
IterableQueueMapperSimpleOptions,
BlockingQueue,
BlockingQueueOptions,
IterableQueue,
Expand Down