Support for execution plans

[sebhrusen]

#169

Execution plans will allow to run benchmarks for multiple frameworks, benchmark definitions, constraints, tasks, folds… and still benefit from full parallelism, especially in AWS mode.

An execution plan is represented as a csv file:

framework	benchmark	constraint	tasks	folds
tpot	medium	1h8c	car	1, 2, 6
autosklearn	small	1h8c	foo, bar	0-9

The failures.csv file generated in aws mode and collecting all the failed instances during a session is an extension of this execution plan format. As such, it can be concatenated with failures.csv from other sessions and fed back to runbenchmark.py to retry all failures with proper parallelism support.

[Innixma]

Would it make sense to add auto-retry support as an extra parameter? This way, if a failure occurs on framework A, task B, fold 3, then it is retried up-to the amount of retries specified by the user? This may be simpler than the user having to feed the failures.csv results into a follow-up run (and do it repeatedly until N retries have been performed).