v0.2.0

This is the release notes of v0.2.0. See here for the complete list of solved issues and merged PRs.

This release note only covers the difference from v0.2.0rc1; for all highlights and changes, please refer to the release notes of the pre-releases:

alpha1
alpha2
alpha3
beta1
beta2
rc1

New Features

• Tensor
  • Add F-order support (#637)
  • Implement scipy.special.gammaln for Mars tensor (#637)
  • Implements scipy.special.erf for Mars tensor (#637)
  • Add searchsorted support (#663)
• DataFrame
  • Support add scalar with dataframe (#610)
  • Implementation of Dataframe.merge/join by shuffle (#665)
  • Support DataFrame sum (#665)
  • Implements DataFrame.from_records. (#665)
  • Implements DataFrame.set_index, DataFrame.iloc. (#696)
  • Support getitem with labels for Series or column names for DataFrame (#696)
• Machine learning
  • Add PCA decomposition algorithm support (#687)
  • Add TruncatedSVD decomposition algorithm support (#687)

Enhancements

• Move optimizes out of tensor to make it universal (#606)
• Get more accurate memory stats from cgroup memory.stat in containers (#633)
• Support create mars session with given session_id. (#670)
• Extend the max_buffer_size of tornado server to 4G. (#683)
• Reuse sealer actor on the same worker. (#696)
• Remove conjugate in tile of Cholesky which effects performance heavily (#663)
• Generate seed instead of RandomState in tile of RandomOperand (#663)
• Use numpy only to calculate fft (#663)

Bug fixes

• Fix web connection error when running in Docker (#594)
• Fix proto-serialization for certain dtypes (#615)
• Backport serialization patch from pyarrow. (#614)
• Fix the default value parsing of open_browser (#613)
• Fix shuffle failover error in cases (#620)
• Fix infer_index_value when one of input indexes is RangeIndex (#626)
• Preserve table order while refreshing its contents (#662)
• Check disks before adding into spill targets (#664)
• Fix chunk shape in tile of SVD (#666)
• Fix compose issues when build chunk graph (#667)
• Fix readable_size accuracy with Python 2 (#668)
• Fix mutable tensor related bugs and some enhancement (#669)
• Raise an error when Python version between client and server is different (#672)
• Add missing size estimation function for TensorDot (#678)
• Support tokenize nested dict in mars.utils.tokenize (#683)
• Fix bug that set value to copied tensor still changed the original one (#681)
• Ensure remaining data removed on session close (#691)
• Fix bug in creating dataframe from identical tensor. (#696)

v0.3.0a1

This is the release notes of v0.3.0a1. See here for the complete list of solved issues and merged PRs.

Highlights

• Running on Kubernetes is supported now, for the usage please refer to docs.

New Features

• Add support for kubernetes (#644)
• Tensor
  • Implement scipy.special.gammaln for Mars tensor (#623)
  • Add F-order support for tensor (#595)
  • Implements scipy.special.erf for Mars tensor (#635)
  • Add searchsorted support (#628)
• DataFrame
  • Support add scalar with dataframe (#582)
  • Implements DataFrame.from_records. (#618)
  • Support DataFrame sum (#591)
  • Implementation of Dataframe.merge/join by shuffle (#574)
  • Implements DataFrame.set_index, DataFrame.iloc. (#638)
  • Support getitem with labels for Series or column names for DataFrame (#682)
• Machine learning
  • Add PCA decomposition algorithm support (#639)
  • Add TruncatedSVD decomposition algorithm support (#686)

Enhancements

• Move optimizes out of tensor to make it universal. (#598)
• Extract scheduler discoverer from cluster_info (#611)
• Support create mars session with given session_id. (#600)
• Add F-order support for tensor (#595)
• Get more accurate memory stats from cgroup memory.stat in containers (#632)
• Use numpy only to calculate fft (#646)
• Generate seed instead of RandomState in tile of RandomOperand (#648)
• Remove conjugate in tile of Cholesky which effects performance heavily (#658)
• Extend the max_buffer_size of tornado server to 4G. (#679)
• Refactor storage management in worker (#353)
• Reuse sealer actor on the same worker. (#692)

Bug fixes

• Backport patch from pyarrow. (#589)
• Fix proto-serialization for certain dtypes. (#592)
• Fix web connection error when running in Docker (#567)
• Fix the default value parsing of open_browser (#602)
• Fix shuffle failover error in cases (#616)
• Fix mutable tensor related bugs and some enhancement (#601)
• Fix infer_index_value when one of input indexes is RangeIndex (#624)
• Fix readable_size accuracy issue under python2 (#641)
• Fix compose issues when build chunk graph (#643)
• Preserve table order while refreshing its contents (#651)
• Fix the wrong shape of chunks in SVD (#656)
• Check disks before adding into spill targets (#654)
• Generate unique key for chunks of mutable tensors which have identical properties (#660)
• Raise an error when Python version between client and server is different (#671)
• Support tokenize nested dict in mars.utils.tokenize (#673)
• Fix bug that set value to copied tensor still changed the original one (#675)
• Add missing size estimation function for TensorDot (#677)
• Ensure remaining data removed on session close (#690)
• Fix bug in creating dataframe from identical tensor. (#695)

v0.2.0rc1

This is the release notes of v0.2.0rc1. See here for the complete list of solved issues and merged PRs.

Changes that break compatibility

• A huge refactor in #579 that merges expression and execution together both for tensor and DataFrame.

New Features

• Add a web page listing all running operands (#521)
• Support conversions between tensor and DataFrame (#524)
• Add distributed series support (#525)
• Add support for worker timeline (#529)
• Support view behavior in numpy and add tensor.view() as well as tensor.flat (#534)
• Mutable tensor for local session and web session (#536)
• Support fetch tensor data slices from client (#540)

Enhancements

• Do consolidation for totiledb (#403)
• Support failover for shuffle operands (#550)
• Allow specifying different calculation methods for CPU and memory (#560)
• Merge expressions and execution directories (#579)

Bug fixes

• Fix bug that redundant graphs are submitted unexpectedly when eager mode is on (#523)
• Fix unify chunks when add a scalar with out (#542)
• Fix execution error caused by fused ShuffleReduces (#546)
• Fix wrong result of negative indexing (#547)
• Fix error in importing new_client in Python 2.7 (#558)

Installation

• Fix build issue under cp27-cp27m-manylinux1 (#588)

Tests

• Separate scheduler integrated tests (#563)

v0.1.5

This is the release notes of v0.1.5. See here for the complete list of solved issues and merged PRs.

Enhancements

• Allow specifying different calculation methods for CPU and memory (#562)
• Do consolidation for totiledb (#565)

Bug fixes

• Fix assignment of structured data type arrays (#541)
• Fix error in importing new_client in Python 2.7 (#563)
• Fix unify chunks when add a scalar with out (#564)
• Fix wrong results when executing negative indexing (#547)

v0.1.4

This is the release notes of v0.1.4. See here for the complete list of solved issues and merged PRs.

Enhancements

• Remove handle_params and calc_shape in operand class (#462)
• Simplify building executable graph (#463)
• Replace default_name with default_uid (#467)
• Remove --ui-port option for Mars Web (#483)
• Add serialize support for complex type (#519)

Bug fixes

• Fix defunct compose option in web session (#437)
• Fix memory size estimation for Fuse nodes (#435)
• Fix compression types when transferring files (#472)
• Fix promise memory issues (#475)
• Return 404 when graph actor does not exist (#500)
• Specify an __order__ attribute in Enum class OperandState (#503)
• Callback on quota allocation failure & bump version (#518)

v0.2.0b2

This is the release notes of v0.2.0b2. See here for the complete list of solved issues and merged PRs.

New Features

• DataFrame
  • Support unary ops for DataFrame. (#432)
  • Add radd implementation for DataFrame (#506)
• Tensor
  • Add fancy indexing support (#444)
  • Implement randomized_svd (#388)
• Add mutable tensor support for local cluster (#464)

Enhancements

• Remove handle_params and calc_shape in operand class (#443)
• Simplify building executable graph (#455)
• Replace default_name with default_uid (#466)
• Remove --ui-port option for Mars Web (#482)
• Add serialize support for complex type (#488)
• Refactor web using standalone tornado handlers (#512)
• Define __str__ and __repr__ for DataFrame, Index, and Series. (#511)

Bug fixes

• Fix memory size estimation for Fuse nodes (#436)
• Avoid shuffle for adding two dataframes whose index and columns are identical (#456)
• Fix compression types when transferring files (#459)
• Fix promise reference counting issues (#474)
• Add resource heartbeat to avoid fake worker death (#471)
• Use lazy import for cupy and cudf (#480)
• Add placement_order as a priority item (#498)
• Specify an __order__ attribute in Enum class OperandState (#502)
• Move scheduler policy back to v0.1 (#445)
• Send callback on quota allocation failure (#516)

Installation

• Build wheel for tag cp27-cp27m (#457)

v0.2.0b1

This is the release notes of v0.2.0b1. See here for the complete list of solved issues and merged PRs.

Changes that break compatibility

• Serialization of chunks have been changed according to #412.

Highlights

• Distributed DataFrames are supported.

New Features

• Support concatenating DataFrame chunks into one (#396)
• Add support for distributed DataFrame (#399)

Enhancements

• Allow renaming allocation keys in QuotaActor (#336)
• Support **kwargs in new_session (#368)
• Enhance tile for DataFrame's add (#365)
• Provide a unified API to create tileable objects including tensor and DataFrame (#379)
• Support non-square matrix for lu decomposition (#383)
• Unify actor distributors for workers and schedulers (#386)
• Refactor operands and integrate the directory of operands into tensor (#393)
• Add FuseChunkData and FuseChunk to represent fused chunk (#412)
• Remove central ChunkMetaActor in schedulers (#421)
• Use Enum instead of numbers to specify compression types (#424)
• Support creating tensor from list of tensors (#426)

Bug fixes

• Add size estimator for sparse randint (#356)
• Pass an acceptable compression list when fetching results from server (#359)
• Adjust memory estimation for TensorDot (#375)
• Fix defunct hard_limit in worker (#378)
• Fix nsplits of tensors generated in the tile of QR decomposition (#387)
• Fix stuck web interface when tiling a graph (#390)
• Fix defunct compose option in web session (#395)
• Fix serialization of DataFrame's index_value or columns_value if its type is int64 (#401)
• Fix bug that min_val of index_value or columns could be nan for chunks of DataFrame (#405)
• Fix executing DataFrame with Fetch operand (#407)
• Check reference type in serialization (#414)
• Support execute dataframe with compose in both local and distributed mode (#410)

v0.1.3

This is the release notes of v0.1.3. See here for the complete list of solved issues and merged PRs.

Enhancements

• Unify actor distributors for workers and schedulers (#391)
• Support non-square matrix for lu decomposition (#397)
• Support **kwargs in new_session (#418)
• Remove central ChunkMetaActor in schedulers (#422)
• Use Enum instead of numbers to specify compression types (#427)
• Support creating tensor from list of tensors (#429)

Bug fixes

• Add size estimator for randint (#357)
• Pass an acceptable compression list when fetching results from server (#362)
• Remove DataTuple (#372)
• Adjust memory estimation for TensorDot & fix defunct hard_limit (#377)
• Fix nsplits of tensors generated in the tile of QR decomposition (#398)

v0.2.0a3

This is the release notes of v0.2.0a3. See here for the complete list of solved issues and merged PRs.

Changes that break compatibility

• Sparse arithmetic operations of tensor will only apply on non-zero elements.

Highlights

• Shuffle support for both local and distributed schedulers.

New Features

• Add shuffle capability in local mode (#236)
• Add shuffle capability in distributed environment (#260)
• Add support for detecting NVIDIA GPU resources (#317)
• Implement __array_function__ protocol (#332)
• Experimental add support for DataFrame (#351)

Enhancements

• Support mt.diag on sparse 2d tensor (#254)
• Move fetch into standalone package (#246)
• Remove rough_size from tensor & remove chunk registration when creation failed (#266)
• Remove chunk square limitation in lu decomposition (#272)
• Allow registering different operand actors for different operands (#278)
• Support execute graph without fusion (#294)
• Support arithmetic operators on only nonzero elements in a sparse matrix (#299)
• Implement operands for SparseVector (#313)
• Remove chunk square limitation in cholesky decomposition (#327)
• Allow batch rejecting peers in workers by endpoints (#328)
• Support replacing actor classes given registration (#348)

Bug fixes

• Fix TensorInv tile when input tensor has only one chunk (#253)
• Fix solve_triangular operand's sparse attributes (#257)
• Fix kernel_mode decorator when an exception occurred in wrapped function (#274)
• Pass sparse parameter when create fetch operand (#279)
• Add sparse property for fuse operand (#284)
• Use new allocation strategy to circumvent inflating granularity in arrow (#287)
• Fix create existing operand actor (#228)
• Validate axis for reduction (#304)
• Fix execution size for fuse nodes (#309)
• Fix arithmetic calculation with where (#314)
• Remove use_one_memory_mapped_file argument when starting plasma_store (#318)
• Fix errors caused by argument changes in tiledb (#324)
• Support execute tiled tensor in distributed mode (#322)
• Fix tiledb error when numpy int64 in shape (#339)
• Register worker when finish initialization (#343)

Documentation

• Add more Chinese translations (#285, thanks @chenquan!)
• Fix issues when creating generated docs && fix untranslated titles (#291)

Tests

• Use parallel build for coveralls (#261)

v0.1.2

This is the release notes of v0.1.2. See here for the complete list of solved issues and merged PRs.

Changes that break compatibility

• Sparse arithmetic operations of tensor will only apply on non-zero elements.

New Features

• Implement __array_function__ protocol (#333)

Enhancements

• Support mt.diag on sparse 2d tensor (#259)
• Remove rough_size from tensor (#273)
• Support execute graph without fusion (#295)
• Remove chunk square limitation in lu decomposition (#296)
• Fix codacy problems (#311)
• Support arithmetic operators on only nonzero elements in a sparse matrix (#312)
• Implement execution of operands for SparseVector (#315)
• Remove chunk square limitation in cholesky decomposition (#329)
• Sync promise.py in v0.1 with v0.2 to solve arg missing issue (#340)
• Decouple services from applications & unify worker start logic (#341)
• Support replacing actor classes given registration (#350）

Bug fixes

• Fix TensorInv tiles for input tensor with one chunk (#255)
• Fix solve_triangular operand's sparse attributes (#262)
• Add sparse property for fetch operand (#280)
• Add sparse property for fuse operand (#293)
• Use new allocation strategy to circumvent inflating granularity in arrow (#293)
• Validate axis for reduction (#305)
• Fix execution size for fuse nodes (#310)
• Resolve use_one_memory_mapped_file argument issue when starting plasma_store (#320)
• Fix arithmetic operator with where (#321)
• Fix errors caused by argument changes in tiledb (#325)
• Support execute tiled tensor in distributed mode (#338)
• Fix create existing operand actor (#345)
• Register worker when finish initialization (#346)
• Fix tiledb error when numpy int64 in shape (#347)

Documentation

• Add Chinese translations (#293, thanks @chenquan!)
• Fix issues when creating generated docs && fix untranslated titles (#293)

Tests

• Use parallel build for coveralls (#262)