These data generators mocks several real-life situations. Purpose built for research and development of several big data use cases.
| Generator | Description | Schema | Scope |
|---|---|---|---|
| log | mocks the logs generated by apache httpd | Log Schema | 1. Real time data ingestion (using Flume & MapReduce) 2. Real time analytics (using Storm and kafka) 3. Click stream analytics 4. NoSql |
| odvs | mocks the data generated by a real life on-demand video service provider like netflix, hulu & amazon prime | ODVS Schema | 1. NoSql 2. Analytics using MapReduce/Hive |
| osge | mocks the data generated by a online social gaming entertainment provider like second life, imvu, onverse... | OSGE Schema | 1. NoSql 2. Analytics using MapReduce/Hive |
| retail | mocks the data generated by a large retail store with multiple locations like Target, Wal-Mart ... | [Retail Schema] (https://github.com/cloudwicklabs/generator/wiki/Retail-Schema) | 1. Analytics using MapReduce/Hive 2. Analytics using Spark, SparkSQL |
Can simulate random events and write to various destination's like local filesystem, Kafka, Kinesis in various formats such as text, avro and other
This project requires sbt, installation instructions found here
Once, sbt is installed use the assembly sbt task from the project path to build the jar with dependencies
git clone https://github.com/cloudwicklabs/generator.git
cd generator
sbt assembly
Use the built in shell wrapper to fire up the generator
bin/generator --help
All the data generators have a driver command line interface, following links show basic examples
- Log generator
- ODVS generator
- OSGE generator
- [Retail generator] (https://github.com/cloudwicklabs/generator/wiki/Retail-Generator-Driver)
For more generators, your specific use-cases or to leave feedback contact support
Apache 2.0. Please see LICENSE.txt. All contents copyright (c) 2013, Cloudwick Labs.