Skip to content

Latest commit

 

History

History
22 lines (16 loc) · 861 Bytes

README.md

File metadata and controls

22 lines (16 loc) · 861 Bytes
Raw

Benefault

License

A way for task preemption in Big data analytics platform

What we have done

  • a simple shell script for monitoring node's metadata (e.g. disk access, network Tx Rx etc) in a cluster
  • read and write for chekcpointing data (note: checkpointRead is private in spark, we need to package function into org.apache.spark)

We have already done some simulation about the JCT gain we can get using Benefault

The performance gain is 15-30%

We test latency in varied scenarios

  • Measure checkpoint latency using Spark
  • Word Count with checkpointing
  • Sorting with checkpoint
  • GroupByKey with Checkpointing
  • DecisionTree with periodic Checkpointing
  • We now design schemes for evaluate best gain we can get using Benefault
  • find sweet spot for whether kill or preempt