spark-submit 1.4.0

Features:

• Report if framework not detected by spark_submit.system_info

Bug Fixes

• Fix quotes in spark-sumbit binary path in spark_submit.system_info

Misc/Internal

• Better way to construct the spark-submit command

spark-submit 1.3.0

Features:

• Addition of spark_submit.SparkJob.get_id() method to obtain the Spark submission ID
• Use all environment variables from os.environ if use_env_vars=True in spark_submit.SparkJob.submit()
• Additional environment variables can be used in spark_submit.SparkJob.submit() with extra_env_vars argument

Misc/Internal

• spark_submit.SparkJob.submit() now uses keyword arguments only
• Argument await_result is changed to poll_time in spark_submit.SparkJob.submit()
• No default environment variables to be used with spark_submit.SparkJob.submit()
• Code standards improvements and general refactoring

spark-submit 1.2.1

Misc/Internal

• Refactor to comply with pylint, isort and mypy checks
• Minor logic improvements

spark-submit 1.2.0

Features

• Added optional timeout argument for spark_submit.SparkJob.submit()

Improvements

• Improve support for Yarn and Kubernetes clusters on polling and killing submitted tasks

Misc/Internal

• Use dedicated file for default arguments and Spark states
• Use consistent path separators where possible

spark-submit 1.1.0

Features

• Addition of spark_submit.SparkJob.get_submit_cmd() method
• Introduction of spark_submit.SparkJob.concluded attribute to keep track of submitted job

Improvements

• Improve spark_submit.system_info() functionality
• More generic status polling
• Minor refactoring

spark-submit 1.0.1

Bug Fixes

• Fix return code bug in spark_submit.SparkJob.get_code()

Improvements

• Change argument order in spark_submit.SparkJob.submit()
• Minor improvements

spark-submit 1.0.0

First release