v0.9.0rc3

[qinxuye]

This is the release notes of v0.9.0rc3. See here for the complete list of solved issues and merged PRs.

New Features

• Tensor
  • Implementing Ellipsoidal Harmonics Functions (Implementing Ellipsoidal Harmonics Functions #2891, thanks @shantam-8!)
• Services
  • Support worker meta service (Support worker meta service #2909)
  • Basic Ray execution backend (Basic Ray execution backend #2921)

Enhancements

• Add execution API to enable custimization of Mars Task Service (Add execution API to enable custimization of Mars Task Service #2894)
• Optimize serialization performance (Optimize serialization performance #2914)
• Skip adding band in meta when fetch shuffle data (Skip adding band in meta when fetch shuffle data #2922)
• Store complete meta on worker and update supervisor meta via fetching from workers (Store complete meta on worker and update supervisor meta via fetching from workers #2912)
• Use cython to accelerate core serialization (Use cython to accelerate core serialization #2924)
• Refine lifecycle api to support incref or decref with ref counts (Refine lifecycle api to support incref or decref with ref counts #2926)
• Ignore fetch operands when assign initial nodes (Ignore fetch operands when assign initial nodes #2929)
• Use cython to accelerate message serialization (Use cython to accelerate message serialization #2932)
• Ignore broadcaster's locality when assign subtasks (Ignore broadcaster's locality when assign subtasks #2943)
• Allow spawning serialization to threads for large objects (Allow spawning serialization to threads for large objects #2944)
• Add metrics and event report for Ray channels (Add metrics and event report for Ray channels #2936)
• Add more logs about execution info (Add more logs about execution info #2940)
• Add support for dask.persist (Add support for dask.persist #2953, thanks @loopyme!)
• Remove should_be_monotonic property (Remove should_be_monotonic property #2949)
• Add metrics on operand and subtask executions ([Metrics] Add metrics on operand and subtask executions #2947, thanks @zhongchun!)
• [Ray] optimize ray fetcher by query in remote node ([Ray] optimize ray fetcher by query in remote node #2957)
• Improve deploy backend (Improve deploy backend #2958)
• Support reporting tile progress (Support reporting tile progress #2954)
• Add logic key for tileable graph (Add logic key for tileable graph #2961, thanks @zhongchun!)
• [Ray] Loads the subtask inputs from meta ([Ray] Loads the subtask inputs from meta #2976)
• New ExecutionConfig API (New ExecutionConfig API #2968)
• Fix speculative execution compatibility with coloring (Fix speculative execution compatibility with coloring #2995)
• Make functions that may take long run in thread for lifecycle tracker (Make functions that may take long run in thread for lifecycle tracker #2992)
• Optimize metric configs ([Metrics] Optimize metric configs #2996, thanks @zhongchun!)
• Expand the ability of resource evaluator ([Scheduling] Expand the ability of resource evaluator #2997, thanks @zhongchun!)
• Optimize gen subtask graph (Optimize gen subtask graph #3004)
• [Ray] Ray execution state ([Ray] Ray execution state #3002)

Bug fixes

• Fix paramter issue of worker actor pool (Fix paramter issue of worker actor pool #2911, thanks @zhongchun!)
• Fix default config to ensure storage backends configured (Fix default config to ensure storage backends configured #2935)
• Wrap errors in operand execution to protect scheduling service (Wrap errors in operand execution to protect scheduling service #2964)
• Fix dtype of series result for DataFrame.apply (Fix dtype of series result for DataFrame.apply #2978)
• Fix potential data leak for shuffle tasks (Fix potential data leak for shuffle tasks #2975)
• Fix potential empty chunks when creating DataFrame from pandas (Fix potential empty chunks when creating DataFrame from pandas #2987)
• [Ray] Support new ray cluster through ray client ([Ray] Support new ray cluster through ray client #2981)
• Fix missing extra_params when constructing operands (Fix missing extra_params when constructing operands #2999)
• Fix msg_to_simple_str in Ray backend and add tests (Fix msg_to_simple_str in Ray backend and add tests #3003)
• Fix incorrect result for df.sort_values when specifying multiple ascending (Fix incorrect result for df.sort_values when specifying multiple ascending #2984)

Documentation

• Add development documents for metrics (Add development documents for metrics #2955, thanks @zhongchun!)

Tests

• Add TPC-H benchmarks (Add TPC-H benchmarks #2937)
• Fix Ray cases (Fix Ray cases #2983)
• Fix version mismatch between kubernetes and minikube (Fix version mismatch between kubernetes and minikube #2986)
• Allow selecting TPC queries (Allow selecting TPC queries #3005)

---

This discussion was created from the release v0.9.0rc3.