Skip to content

Latest commit

 

History

History
55 lines (36 loc) · 1.42 KB

README.md

File metadata and controls

55 lines (36 loc) · 1.42 KB

Batcher

Batcher is a base class that distributes dynamically sized batches of input data to forked child processes for parallel processing.

Install

CPAN/M

# TODO

Manual

  1. Clone the repository git clone https://github.com/selftaught/Batcher.git
  2. Cd into the repo root and generate a makefile: perl Makefile.pl
  3. Make it: make && make test && make install

Usage

Using Batcher is simple. Inherit Batcher in a subclass and then implement a few hooks that Batcher::run depends on.

  • batch_count() - provides the total number of batchs
  • batch_next($next_idx) - provides the next batch of data given the next index
  • batch_size() - provides the number of items in a batch
  • batch_result($result) - provides the next result to process

Browse examples for implementation examples.

Performance

Contributing

  1. Fork the repository
  2. Create feature branch (git checkout -b feature/adding-xyz)
  3. Create PR to main when feature branch is ready for review

Tests

Publishing to CPAN

perl Makefile.PL && make dist && make clean
cpan-upload -u <PAUSEUSERNAME> Batcher-$VERSION.tar.gz

Issues

If you're a mac user and encounter the following error: objc[62121]: +[NSString initialize] may have been in progress in another thread when fork() was called.

you may need to disable initialize fork safey for this to work.

export OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES