Skip to content

Commit f4493fd

Browse files
committed
[MINOR][SQL][TESTS] Ignore messages below error level from HadoopFSUtils for sql module tests
### What changes were proposed in this pull request? This pr ignore messages below error level from `HadoopFSUtils` for sql module tests. ### Why are the changes needed? From the test logs of Github Actions, there are a large number of logs from `HadoopFSUtils`. For example, in the pipeline named `sql-other tests`, there are approximately 40,000 similar logs like the following: - https://github.com/apache/spark/actions/runs/13915102741/job/38936641387 ``` 04:22:51.192 WARN org.apache.spark.util.HadoopFSUtils: The directory /home/runner/work/spark/spark/target/tmp/spark-e3d8a446-2dd8-42fd-a64f-b96504d8cd69/deleted was not found. Was it deleted very recently? 04:22:51.193 WARN org.apache.spark.util.HadoopFSUtils: The directory /home/runner/work/spark/spark/target/tmp/spark-e3d8a446-2dd8-42fd-a64f-b96504d8cd69/deleted was not found. Was it deleted very recently? [info] - InMemoryFileIndex: root folders that don't exist don't throw exceptions (6 milliseconds) 04:22:51.244 WARN org.apache.spark.util.HadoopFSUtils: The directory mockFs:/rootDir/subDir was not found. Was it deleted very recently? 04:22:51.271 WARN org.apache.spark.util.HadoopFSUtils: The directory mockFs:/rootDir/subDir was not found. Was it deleted very recently? 04:22:51.280 WARN org.apache.spark.util.HadoopFSUtils: The directory mockFs:/rootDir/subDir was not found. Was it deleted very recently? 04:22:51.281 WARN org.apache.spark.util.HadoopFSUtils: The directory mockFs:/rootDir/subDir was not found. Was it deleted very recently? 04:22:51.307 WARN org.apache.spark.util.HadoopFSUtils: The directory mockFs:/rootDir/subDir was not found. Was it deleted very recently? 04:22:51.332 WARN org.apache.spark.util.HadoopFSUtils: The directory mockFs:/rootDir/subDir was not found. Was it deleted very recently? 04:22:51.334 WARN org.apache.spark.util.HadoopFSUtils: The directory mockFs:/rootDir/subDir was not found. Was it deleted very recently? 04:22:51.334 WARN org.apache.spark.util.HadoopFSUtils: The directory mockFs:/rootDir/subDir was not found. Was it deleted very recently? ``` Similar logs are more like noise, so the current pr makes `HadoopFSUtils` only print error-level logs during SQL module testing. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? - Pass Github Actions - Manual inspection shows that the new tests in the SQL module no longer print similar logs as mentioned above. ### Was this patch authored or co-authored using generative AI tooling? No Closes #50316 from LuciferYang/sql-log-level. Authored-by: yangjie01 <[email protected]> Signed-off-by: yangjie01 <[email protected]>
1 parent 35e0e08 commit f4493fd

File tree

1 file changed

+4
-0
lines changed

1 file changed

+4
-0
lines changed

sql/core/src/test/resources/log4j2.properties

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -84,3 +84,7 @@ logger.structured.name = org.apache.spark.sql.LogQuerySuite
8484
logger.structured.level = trace
8585
logger.structured.appenderRefs = structured
8686
logger.structured.appenderRef.structured.ref = structured
87+
88+
# Ignore messages below error level from HadoopFSUtils, because it's a bit verbose
89+
logger.hadoopfsutils.name = org.apache.spark.util.HadoopFSUtils
90+
logger.hadoopfsutils.level = error

0 commit comments

Comments
 (0)