Release v0.9.0

Apache Uniffle (Incubating) Release v0.9.0

Highlight

• Introduce dashboard.
• Introduce rust-based shuffle server.
• Add support for Spark 3.5.
• The data transportation Netty mode is production available.
• Reduce block id layout limitations and simplify layout configuration for Spark.

ChangeLog

• [#1751][0.9] improvement: support gluten (#1753)
• [#1764] fix(client): Fix timeout time unit for unregister requests (#1766)
• [#1149] fix: GC logs in JDK 11 do not include date and time stamps. (#1240)
• [#1675][FOLLOWUP] fix(test): Fix various flaky tests (#1730)
• [MINOR] fix: Update outdated config: rss.writer.send.check.timeout -> rss.client.send.check.timeout.ms (#1734)
• [#1721] fix(coordinator): classCastExpection of boolean->String with yaml style remote client conf (#1722)
• [#1673] fix(K8S): Fix the deployment of stable version K8S cluster (#1694)
• [#1675][FOLLOWUP] fix(test): Fix flaky tests which may cause port conflicts (#1696)
• [MINOR] fix(typo): Correct the removeShuffle method name (#1697)
• [MINOR] docs: modify the default value of rss.coordinator.select.partition.strategy in docs (#1692)
• [#1680] improvement(server): Remove partial HDFS files that written by server self for expired apps (#1681)
• [#1675] fix(test): Fix tests which may be flaky on different machines (#1676)
• [#1684] fix(server): use the diskSize obtained from periodic check to determine whether is writable (#1685)
• [#1678] fix(server): disk size leak on removing resources by AppPurgeEvent (#1679) (#1689)
• [#1657] build: Add license information after version 0.9.0 (#1671)
• [MINOR] chore(rust): disable flaky test of local_store_test (#1674)
• [#1459][FOLLOWUP] fix(server): Fix the issue of log variable printing (#1672)
• [#1459][FOLLOWUP] improvement(server): Print an error log when an event is dropped (#1643)
• [#1341] fix(mr): Fix MR Combiner ArrayIndexOutOfBoundsException Bug. (#1666)
• [#378][FOLLOWUP] fix(server): Fix huge_partition_num metric (#1669)
• [#1662] fix(test): Fix Netty related flaky tests (#1663)
• [#1629] fix(operator): Support parsing NaN float value in metrics (#1630)
• [#1634] fix(server): remove app folder if app is expired (#1635)
• [MINOR] chore(rust): disable flaky test of test_ticket_manager (#1637)
• [#1596][FOLLOWUP] fix(netty): Send failed responses only when the channel is writable (#1641)
• [#1626] fix(server): Remove the meaningless eventOfUnderStorageManagers cache (#1627)
• [#1631] fix(server): ShuffleTaskInfo may leak when app is removed. (#1632)
• [#1373][FOLLOWUP] fix(spark): register with incorrect partitionRanges after reassign (#1612)
• [#1608][part-2] fix(spark): avoid releasing block in advance when enable block resend (#1610)
• [#1606] feat(client): Add client retry mechanism for NO_BUFFER when reading data(memory/local/index) (#1616)
• [#1608][part-1] fix(spark): Only share the replacement servers for faulty servers in one stage (#1609)
• [#1373][FOLLOWUP] fix(spark): shuffle manager rpc service invalid when partition data reassign is enabled (#1583)
• [#1596] fix(netty): Use a ChannelFutureListener callback mechanism to release readMemory (#1605)
• [#1598] fix(server) Fix inaccurate used_direct_memory_size metric (#1599)
• [#1472][FOLLOWUP] improvement(server): Release memory more accurately when failing to cache shuffle data (#1597)
• [MINOR] refactor: Calling lock() method outside try block to avoid unnecessary errors (#1590)
• [#1591] feat(spark): Support Spark 3.5.1 (#1592)
• [#1586] improvement(netty): Allow Netty Worker thread pool size to dynamically adapt to the number of processor cores (#1587)
• [#1588] improvement(server): Add exception handling for the thread pool when flushing events (#1589)
• [#1576] feat(doc): server deploy guide without hadoop-home env (#1577)
• [#1571] fix(server): Memory may leak when EventInvalidException occurs (#1574)
• [#1373][FOLLOWUP] fix(spark): incorrect partition id type (#1582)
• [#1373][FOLLOWUP] fix(spark3):Add client type when request shuffle assignment (#1580)
• build(deps): bump google.golang.org/protobuf from 1.28.0 to 1.33.0 (#1575)
• [#1554] feat(spark): Fetch dynamic client conf as early as possible (#1557)
• [#1572] fix(spark): Exceptions might be discarded when spilling buffers (#1573)
• [#1564] fix(server): disk health check invalid when hang (#1568)
• [#731][FOLLOWUP] feat(Spark): Configure blockIdLayout for Spark based on max partitions (#1566)
• [#1567] fix(spark): Let Spark use its own NettyUtils (#1565)
• [#1569] fix(rust): flaky test for test_ticket_manager (#1570)
• [MINOR] improvement(test): A better computation logic for WriteAndReadMetricsTest without using reflection (#1563)
• [#731] feat(spark): Make blockid layout configurable for Spark clients (#1528)
• [#808] improvement(spark): Verify the number of written records to ensure data correctness (#1558)
• [MINOR] improvement(client): Override getClientInfo method in ShuffleServerGrpcNettyClient and remove unused getDesc method (#1559)
• [#1552] improvement: Migrate from log4j1 to log4j2 (#1553)
• [#1472][part-6] FOLLOWUP: Fix Netty transport time when sending shuffle data requests (#1551)
• [#134][FOLLOWUP] improvement(spark2): Use taskId and attemptNo as taskAttemptId (#1544)
• [#1549] fix(common): Uniformly throw RssException for external callers (#1550)
• [MINOR] test: Use sensible partition ids in ShuffleReadClientImplTest (#1545)
• [#1546] fix(spark): NPE could happen before uncompressing after #1360 (#1547)
• feat(docker): Add example docker compose Uniffle/Spark cluster (#1532)
• [#1472][part-6] fix(netty): Make UTs truly test Netty mode (#1540)
• [MINOR] improvement(tez): Only invoking LOG.debug when LOG.isDebugEnabled is true (#1541)
• [#1459] fix(server): Memory leak for exceptional scenarios when flushing events (#1537)
• [#1472] fix(client): IlegalReferenceCountException for clientReadHandler.readShuffleData (#1536)
• [#1472][part-5] Use UnpooledByteBufAllocator to fix inaccurate usedMemory issue causing OOM (#1534)
• [MINOR] refactor(common): Move blockId bit logic into common class (#1527)
• [#1373][part-1] feat(spark): partition write to multi servers leveraging from reassignment mechanism (#1445)
• [MINOR] Update dashboard pom.xml to take arguments for node and npm download locations (#1530)
• [#1316] improvement(spark): detect OutputTracker API version via Spark version (#1317)
• [#134] improvement(spark3): Use taskId and attemptNo as taskAttemptId (#1529)
• [MINOR] feat(build): Allow to build distribution without some modules (#1525)
• [#1407] fix(rust): use grpc runtime worker threads and adjust default runtime config (#1517)
• [#1407] feat(rust): fix + add total grpc request metrics (#1516)
• [#1407] chore(rust): add cpu profile doc (#1515)
• [#1472][part-2] fix(server): Reuse ByteBuf when decoding shuffle blocks instead of reallocating it (#1521)
• [MINOR] fix(CI): Improve dashboard across the CI (#1526)
• [#1472][part-3] fix(client): Fix occasional IllegalReferenceCountException issues in extremely rare scenarios (#1522)
• [MINOR] fix(pom): Add missing shuffle-server dependencies to work with -Ptez
• [#1472][part-4] feature(server): Add metrics for Netty's pinnedDirectMemory and usedDirectMemory (#1524)
• [#1472][part-1] fix(server): Upgrade Netty and GRPC (#1520)
• [MINOR] fix(deploy): Fix invocation of kubernetes bash scripts (#1513)
• [#1476] feat(rust): Provide dedicated unregister app rpc interface (#1511)
• [#1476] feat(spark): Provide dedicated unregister app rpc interface (#1510)
• [MINOR] improvement(CI): Rework build and rust workflow events (#1508)
• [#1407] fix(rust): drop events and release memory when errors happened (#1509)
• [#1267][FOLLOWUP] improvement(client): INFO log level should be used in RetryUtils (#1500)
• [MINOR] feat(CI): Report test results in github comments (#1506)
• [#1407] fix(rust): return error when getting data from hdfs by client (#1507)
• [#1501] fix(server): storage selection cache accidentally deleted when clearing stage level data. (#1505)
• [#1407] fix(rust): dont panic when no available local disks (#1504)
• fix(rust): avoid checking storage type in runtime (#1503)
• [MINOR] build: Move dashboard module into profile and disable it by default (#1498)
• [#1497] improvement(spark): flushing buffer if the memoryUsed of the first record of WriterBuffer larger than bufferSize (#1485)
• [MINOR] improvement(test): Identify duplicate blocks in TestUtils.validateResult (#1495)
• [MINOR] fix: Get and increment ATOMIC_LONG in that order everywhere (#1496)
• [MINOR] docs: Improve comment on blockId structure (#1492)
• [MINOR] fix(server): Assert actual number of bitmaps matches bitNum (#1493)
• [#1490] improvement(spark3): Disable dynamic allocation shuffle tracking by default (#1491)
• [#1407] feat(rust): support more metrics about disk and topN data size (#1488)
• [#1407] feat(rust): support multiple spill policies and simplify hdfs config (#1487)
• [#1356] feat(server): improve expired buffers metric and log (#1469)
• [#1464][FOLLOWUP] improvement(spark): print abnormal shuffle servers that blocks fail to send (#1473)
• [#1467] feat(server): introduce total hdfs write data size for huge partition (#1468)
• [#1355] fix(client): Netty client will leak when decoding responses (#1455)
• [#1462] fix(server): Memory may leak when flushQueue is full (#1463)
• [#1466] feat(server): introduce the JvmPauseMonitor to detect the gc pause (#1470)
• [#1459] improvement(server): refactor DefaultFlushEventHandler and support event retry into pending queue (#1461)
• [#1464] improvement(spark): print abnormal shuffle servers that blocks fail to send (#1465)
• [#1456] improvement(client): Better exception handling when calling requireBuffer using GRPC (#1457)
• [#1428] fix(server): fallback invalid when local storage can't write (#1429)
• [#1453] improvement: Force to use the UNIX line ending when using spotless-maven-plugin (#1454)
• [#1447] feat(client): Introduce configurations to control default behavior of RPC client (#1448)
• [#1267] improvement(client): throw the detailed stacktrace when exceptions happened (#1411)
• [#1189][FOLLOWUP] fix(server): Start NettyDirectMemoryTracker. (#1432)
• [#333] feat(server): expose metrics of TopN app bytes in one shuffle server (#1400)
• [#1433] fix(server): Race conditions with ShuffleServer state (#1434)
• [MINOR] refactor: avoid unnecessary bitmap clone and AND (#1442)
• [#532] fix: spotBugs of SC_START_IN_CTOR (#1440)
• [#1435] improvement: Improve log4j settings to avoid annoying messages (#1436)
• [MINOR] refactor: Avoid unnecessary recursion (#1441)
• [#1407] feat(rust): refactor localfile store to speed up writing (#1422)
• [#1416] feat(spark): support custom hadoop config in client side (#1417)
• [#1119] improvement(client): Explicitly throw BUFFER_LIMIT_OF_HUGE_PARTITION (#1425)
• [#974] fix(coordinator): Dynamic remote storage conf invalid for LegacyClientConfParser (#1424)
• [#1420] fix(client): reportShuffleWriteFailure failed because of IndexOutOfBoundsException (#1421)
• [#1356] improvement: add metric of total expired pre-allocated buffers (#1412)
• [#1414] feat(rust): introduce native hdfs client (#1415)
• [#1024] improvement(tez): Optimize user switch to shuffle mode local/remote. (#1397)
• [#1403] fix(client): RSS client configurations are not working. (#1404)
• [#1409] fix(client): Netty Epoll is unavailable for the RSS Client. (#1410)
• [#1407] improvement(rust): Critical bug fix of getting blockIds and some optimization (#1408)
• [#825][FOLLOWUP] fix(spark): Fix without returning an exception. (#1402)
• [#1385] improvement: Improve log4j appender layout pattern (#1386)
• [#851] improvement: Add a similar util method like ThreadUtils.parmap in the Spark (#1396)
• [#363] improvement(server): Make the coordinator client managed by CoordinatorClientFactory singleton (#1377)
• [#1391] fix(server): Direct memory may leak in exceptional scenarios in shuffle server. (#1392)
• [#1157] fix(tez): Container not exit because shuffle client is not closed
• [#460] improvement: Exit on OutOfMemoryError (#1390)
• [#1387] improvement: compatibility with jdk8 when call JavaUtils.newConcurrentMap (#1389)
• [#1369] feat: Provide distribution with Hadoop dependencies (#1379)
• [#1383] [DOCS] Improve Netty's documentation (#1384)
• [#1358] fix(spark): pre-check bytebuffer whether is direct before uncompress (#1360)
• [#1364] feat(client): introduce option to control whether to use local hadoop conf (#1370)
• [MINOR] chore(client): fix the incorrect partitionId (#1376)
• [#1189] feat(server): Add netty used direct memory size metric (#1363)
• [#960] fix(dashboard): simplify dependency and correct the startup script (#1347)
• [#1348] improvement(metrics): Unify tags generation for shuffle-server metrics reporter (#1349)
• [MINOR] chore: fix kubernetes ci pipeline (#1368)
• [MINOR] fix(spark): Fix NPE for ShuffleWriteClientImpl.unregisterShuffle (#1367)
• [#960][part-4] feat(dashboard): Fix some display bugs and optimize the display format. (#1326)
• [#1267] fix(client): fast fail without retry when oom occurs (#1344)
• [#1361] feat(netty): add netty metrics into reporter (#1362)
• [#1335] fix(server)(netty): release bytebuf explicitly when requiredId is expired or cache failed (#1357)
• [MINOR] chore(client): Specify name for data transfer thread pool (#1353)
• [#1319] fix(server): Add shaded com.google.guava:failureaccess dependency to prevent NoClassDefFoundError (#1352)
• [MINOR] improvement: use mvn wrapper in CI builds. (#1351)
• [#1191][FOLLOWUP] improvement(conf): use the unified name for hybrid storage in conf (#1350)
• [#960][FOLLOWUP] fix(dashboard): Fix get_pid_file_name function for the dashboard. (#1346)
• [MINOR] improvement: use mvn wrapper for builds (#1345)
• [#901] feat(server): respect disk capacity watermark rather than uniffle capacity (#1337)
• [#1342] improvement(server): dump appId when clearing resource fails (#1343)
• [#1110] improvement(coordinator): introduce pluggable remote storage config format (#1329)
• [#1330] improvement: optimize tips for checking replica settings (#1334)
• [#1187] feat(netty): Netty Encoder Support zero-copy. (#1313)
• [#960][part-3] feat(dashboard): Provides a start-stop script for the dashboard. (#1056)
• [#1308] improvement(rust): detect whether data has been purged in UT (#1323)
• [#1213] feat(rust): Support block filter by taskId when getting memory data (#1311)
• [#1290] improvement(operator): Avoid accidentally deleting data of other services when misconfiguring the mounting directory (#1291)
• [MINOR] fix: flaky test ShuffleTaskManagerTest#checkAndClearLeakShuffleDataTest (#1320)
• [MINOR] test: flaky test GrpcServerTest.testGrpcExecutorPool (#1321)
• [#960][part-2] feat(dashboard): Add a dashboard front-end module. (#1055)
• [#825][part-7] feat(spark): Write Stage resubmit and dynamic shuffle server assign integration tests. (#1148)
• [#1300] feat(mr): Support combine operation in map stage for mr engine. (#1301)
• [#1309] fix(spark): WriteBufferManager in Spark2 does not use a reassigned shuffle server. (#1310)
• [#1307] feat(rust): make each thread listen the socket to improve throughput in tonic (#1306)
• [#960][part-1] feat(dashboard): Add some dashboard interfaces. (#1053)
• [#825][part-6] feat(spark): Added logic that failed to send ShuffleServer. (#1147)
• [#1293] feat(rust): Add total_read_data metric (#1298)
• [#1094] docs: split client_guide.md (#1299)
• [#1221] feat(rust): Support grpc server graceful shutdown (#1292)
• [#1294] feat(rust): introduce the unified grpc latency metrics for all requests (#1295)
• [#1296] improvement(rust): use std.sync.lock to replace tokio lock for better performance (#1216)
• [#825][part-5] feat(spark): Adds the RPC interface to reassign the ShuffleServer list. (#1146)
• [MINOR] docs: update jar name for spark client (#1289)
• [MINOR] chore: add scripts for publishing tarballs to svn (#1284)
• [#1286] improvement(server): Add RemoveResourceTime Metric (#1288)
• [#1271] improvement(server): change transportTime and processTime summary to Thread Pool Instead of block (#1272)
• [#1269] fix(tez): uniqueMapId may be not unique when more than one fetcher are working. (#1270)
• [#1246] feat(tez): Support remote spill for unordered input. (#1250)
• [#825][part-4] feat(spark): Report write failures to ShuffleManager. (#1258)
• [MINOR] fix: missing to build spark shaded modules (#1282)
• [#1275] chore: add scripts for publishing maven releases (#1281)
• [#1274] feat: add shaded module for spark2 client (#1280)
• [#1273] feat: add shaded module for spark3 client (#1279)
• [#825][part-3] feat(spark): Get the ShuffleServer corresponding to the partition from ShuffleManager. (#1141)
• [#1277] chore: add flatten maven plugin (#1278)
• [#1252] fix(server): Incorrect storage write fail metric (#1253)
• [#825][FOLLOWUP] fix(spark): Apply a thread safety way to track the blocks sending result (#1260)
• [#1254][FOLLOWUP] fix(test): Fix the flaky test RssShuffleTest. (#1259)
• [#1261] fix(spark): Throw out InterruptedException for sleep in requestExecutorMemory #1262
• [#1256] refactor: optimize collections contruction (#1257)
• [#1254] fix(test): Fix the flaky test RssShuffleTest. (#1255)
• [#825][part-2] feat(spark): Report failed blocks and a list of ShuffleServer. (#1138)
• [#244][FOLLOWUP] test: CoordinatorGrpcTest.rpcMetricsTest. (#1251)
• [#1231] feat(tez): Support remote spill in merge stage. (#1245)
• [#1243] fix(test): Fix the flaky test SparkSQLTest and RepartitionTest (#1244)
• [#1089] feat(spark): Add dynamic allocation patch for Spark 2.3 (#1242)
• [#1237] feat(rust): support populating args by clap (#1236)
• [#1088] feat(spark): Add dynamic allocation patch for Spark 3.0 (#1241)
• [#1234] improvement(rust): separate runtimes for different overload (#1233)
• [#1090] refactor: Refactor the reader code with builder pattern (#1232)
• [#1219] fix(test): Fix the flaky test WriteAndReadMetricsTest (#1235)
• [#1206] chore(rust): ignore generated proto code in git (#1229)
• [#1091] refactor: Refactor the writer code with builder pattern (#1228)
• [MINOR] Fix kubernetest CI pipeline (#1227)
• [#802] feat(spark): Implement ShuffleDataIo (#1226)
• [#825][part-1] feat(spark): Add the RPC interface for reassigning ShuffleServer (#1137)
• [#1085] feat(spark): Add dynamic allocation patch for Spark 3.4 (#1225)
• [#1201] improvement: only invoking LOG.debug when LOG.isDebugEnabled() is true (#1217)
• [#1084] feat: Add dynamic allocation patch for Spark 3.3 (#1224)
• [#1083] feat(spark): Support Spark 3.5 (#1223)
• [#1211] fix(server): unexpectedly removing resources when app has re-registered shuffle later (#1212)
• [#1206] chore(rust): remove the auto-generated proto code (#1218)
• [#1209] improvement(server): Speed up cleanupStorageSelectionCache method in LocalStorageManager. (#1210)
• [#1206][part-2] feat(rust): introduce rust based shuffle-server (#1208)
• [#1206][part-1] feat(rust): create folder for rust-based shuffle server (#1207)
• [#1204] chores(ci): Fix the ci pipeline of Kubernetes #1205
• [#1202] improvement: Add HealthScriptChecker for execute special health check shell script (#1203)
• [#1198] improvement: zerocopy from Protobuf's ByteString to Netty's ByteBuf (#1199)
• [#1192] improvement(hdfs): Add RSS_SECURITY_HADOOP_KERBEROS_PROXY_USER_ENABLE conf for storing shuffle data (#1194)
• [MINOR] refactor: Rename MultiStorage to HybridStorage (#1191)
• [MINOR] Remove extra directory (#1190)
• [#1178] improvement: set rss.coordinator.quota.default.app.num default -1 to indicate no quota check (#1186)
• [#1182] fix(operator): The LeaderElectionNamespace of the rss-controller is hard-coded to kube-system. (#1183)
• [#1175] fix(netty): Retry failed with StacklessClosedChannelException after channel closed (#1181)
• [#1177] improvement: Reduce the write time of tasks (#1179)
• [MINOR] docs: Fix spark.serializer in README and client_guide (#1180)