[SPARK-49741][DOCS] Add spark.shuffle.accurateBlockSkewedFactor to config docs page
#48189
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
JoshRosen
approved these changes
Sep 20, 2024
There was a problem hiding this comment.
+1 on documenting this, as this is a useful configuration. Looks like the docs generally match
spark/core/src/main/scala/org/apache/spark/internal/config/package.scala
Lines 1387 to 1397 in f3785fa
docs/configuration.md
Outdated
| @@ -1010,6 +1010,19 @@ Apart from these, the following properties are also available, and may be useful | |||
| </td> | |||
| <td>2.2.1</td> | |||
| </tr> | |||
| <tr> | |||
| <td><code>spark.shuffle.accurateBlockSkewedFactor</code></td> | |||
There was a problem hiding this comment.
I see that the spark.shuffle.accurateBlockThreshold configuration is already documented in this table. It looks like we're preexistingly inconsistent about alphabetizing this list.
What do you think about either moving this new configuration a bit further down so it's next to spark.shuffle.accurateBlockThreshold?
There was a problem hiding this comment.
Done, also updated the test section
dongjoon-hyun
changed the title
spark.sql.adaptive.skewJoin.skewedPartitionFactor to config docs page
| @@ -1222,6 +1222,19 @@ Apart from these, the following properties are also available, and may be useful | |||
| </td> | |||
| <td>2.2.1</td> | |||
| </tr> | |||
| <tr> | |||
| <td><code>spark.shuffle.accurateBlockSkewedFactor</code></td> | |||
There was a problem hiding this comment.
To @timlee0119 and @JoshRosen , shall we remove .internal() from the configuration definition together?
spark/core/src/main/scala/org/apache/spark/internal/config/package.scala
Lines 1387 to 1389 in bdea091
There was a problem hiding this comment.
In addition, the PR title looks wrong to me because we are touching spark.shuffle.accurateBlockSkewedFactor instead of spark.sql.adaptive.....
There was a problem hiding this comment.
Yeah, I would match the description
There was a problem hiding this comment.
Sorry for the typo, I've fixed the PR title and removed internal()
timlee0119
changed the title
spark.sql.adaptive.skewJoin.skewedPartitionFactor to config docs pagespark.shuffle.accurateBlockSkewedFactor to config docs page
dongjoon-hyun
changed the title
spark.shuffle.accurateBlockSkewedFactor to config docs pagespark.shuffle.accurateBlockSkewedFactor to config docs page
HyukjinKwon
changed the title
spark.shuffle.accurateBlockSkewedFactor to config docs pagespark.shuffle.accurateBlockSkewedFactor to config docs page
HyukjinKwon
approved these changes
Sep 22, 2024
|
Merged to master. |
attilapiros
pushed a commit
to attilapiros/spark
that referenced
this pull request
Oct 4, 2024
…config docs page ### What changes were proposed in this pull request? `spark.shuffle.accurateBlockSkewedFactor` was added in Spark 3.3.0 in https://issues.apache.org/jira/browse/SPARK-36967 and is a useful shuffle configuration to prevent issues where `HighlyCompressedMapStatus` wrongly estimates the shuffle block sizes when the block size distribution is skewed, which can cause the shuffle reducer to fetch too much data and OOM. This PR adds this config to the Spark config docs page to make it discoverable. ### Why are the changes needed? To make this useful config discoverable by users and make them able to resolve shuffle fetch OOM issues themselves. ### Does this PR introduce _any_ user-facing change? Yes, this is a documentation fix. Before this PR there's no `spark.sql.adaptive.skewJoin.skewedPartitionFactor` in the `Shuffle Behavior` section on [the Configurations page](https://spark.apache.org/docs/latest/configuration.html) and now there is. ### How was this patch tested? On the IDE: <img width="1633" alt="image" src="https://github.com/user-attachments/assets/616a94b9-2408-491c-a17b-c6dbdff14465"> Updated: <img width="1274" alt="image" src="https://github.com/user-attachments/assets/ba170e9a-eba2-4fdf-85eb-a3aebefc055e"> ### Was this patch authored or co-authored using generative AI tooling? No Closes apache#48189 from timlee0119/add-accurate-block-skewed-factor-to-doc. Authored-by: Tim Lee <[email protected]> Signed-off-by: Hyukjin Kwon <[email protected]>
himadripal
pushed a commit
to himadripal/spark
that referenced
this pull request
Oct 19, 2024
…config docs page ### What changes were proposed in this pull request? `spark.shuffle.accurateBlockSkewedFactor` was added in Spark 3.3.0 in https://issues.apache.org/jira/browse/SPARK-36967 and is a useful shuffle configuration to prevent issues where `HighlyCompressedMapStatus` wrongly estimates the shuffle block sizes when the block size distribution is skewed, which can cause the shuffle reducer to fetch too much data and OOM. This PR adds this config to the Spark config docs page to make it discoverable. ### Why are the changes needed? To make this useful config discoverable by users and make them able to resolve shuffle fetch OOM issues themselves. ### Does this PR introduce _any_ user-facing change? Yes, this is a documentation fix. Before this PR there's no `spark.sql.adaptive.skewJoin.skewedPartitionFactor` in the `Shuffle Behavior` section on [the Configurations page](https://spark.apache.org/docs/latest/configuration.html) and now there is. ### How was this patch tested? On the IDE: <img width="1633" alt="image" src="https://github.com/user-attachments/assets/616a94b9-2408-491c-a17b-c6dbdff14465"> Updated: <img width="1274" alt="image" src="https://github.com/user-attachments/assets/ba170e9a-eba2-4fdf-85eb-a3aebefc055e"> ### Was this patch authored or co-authored using generative AI tooling? No Closes apache#48189 from timlee0119/add-accurate-block-skewed-factor-to-doc. Authored-by: Tim Lee <[email protected]> Signed-off-by: Hyukjin Kwon <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
spark.shuffle.accurateBlockSkewedFactorwas added in Spark 3.3.0 in https://issues.apache.org/jira/browse/SPARK-36967 and is a useful shuffle configuration to prevent issues whereHighlyCompressedMapStatuswrongly estimates the shuffle block sizes when the block size distribution is skewed, which can cause the shuffle reducer to fetch too much data and OOM. This PR adds this config to the Spark config docs page to make it discoverable.Why are the changes needed?
To make this useful config discoverable by users and make them able to resolve shuffle fetch OOM issues themselves.
Does this PR introduce any user-facing change?
Yes, this is a documentation fix. Before this PR there's no
spark.sql.adaptive.skewJoin.skewedPartitionFactorin theShuffle Behaviorsection on the Configurations page and now there is.How was this patch tested?
On the IDE:
Updated:
Was this patch authored or co-authored using generative AI tooling?
No