Skip to content

Commit ecdb69f

Browse files
paulstaabsrowen
authored andcommitted
[SPARK-40154][PYTHON][DOCS] Correct storage level in Dataframe.cache docstring
### What changes were proposed in this pull request? Corrects the docstring `DataFrame.cache` to give the correct storage level after it changed with Spark 3.0. It seems that the docstring of `DataFrame.persist` was updated, but `cache` was forgotten. ### Why are the changes needed? The doctoring claims that `cache` uses serialised storage, but it actually uses deserialised storage. I confirmed that this is still the case with Spark 3.5.0 using the example code from the Jira ticket. ### Does this PR introduce _any_ user-facing change? Yes, the docstring changes. ### How was this patch tested? The Github actions workflow succeeded. ### Was this patch authored or co-authored using generative AI tooling? No Closes apache#43229 from paulstaab/SPARK-40154. Authored-by: Paul Staab <[email protected]> Signed-off-by: Sean Owen <[email protected]> (cherry picked from commit 94607dd) Signed-off-by: Sean Owen <[email protected]>
1 parent c730365 commit ecdb69f

File tree

1 file changed

+2
-2
lines changed

1 file changed

+2
-2
lines changed

python/pyspark/sql/dataframe.py

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1404,7 +1404,7 @@ def foreachPartition(self, f: Callable[[Iterator[Row]], None]) -> None:
14041404
self.rdd.foreachPartition(f) # type: ignore[arg-type]
14051405

14061406
def cache(self) -> "DataFrame":
1407-
"""Persists the :class:`DataFrame` with the default storage level (`MEMORY_AND_DISK`).
1407+
"""Persists the :class:`DataFrame` with the default storage level (`MEMORY_AND_DISK_DESER`).
14081408
14091409
.. versionadded:: 1.3.0
14101410
@@ -1413,7 +1413,7 @@ def cache(self) -> "DataFrame":
14131413
14141414
Notes
14151415
-----
1416-
The default storage level has changed to `MEMORY_AND_DISK` to match Scala in 2.0.
1416+
The default storage level has changed to `MEMORY_AND_DISK_DESER` to match Scala in 3.0.
14171417
14181418
Returns
14191419
-------

0 commit comments

Comments
 (0)