Squashed commits:

- 1ca496b5505c7bcbea7651c6251aa644a039ca7f fix premerge tests by jingz-db <[email protected]> - 6d4e36523d2ced6704114f295e5195c9ba113c88 resolve comments by jingz-db <[email protected]> - 99a126a25bf12ab5bef63a09e9696c482194a51b resolve comments by jingz-db <[email protected]> - c0e9393f8109ac8b72baecf0f37fc8a49d01ed99 fix flag check for column-mapping mode by jingz-db <[email protected]> - 66ff54a2b4543872582a5fa73fe7b00415b1b55d add a flag by jingz-db <[email protected]> - da25d62fa2b4346652b4ccc7c3880b03eadbebef [LC-2927][LIQUID] Disallow set tblproperties (delta.featu... by Ami Oka <[email protected]> - 101e22404e1d3ee55f98bdc525c516a2157d0540 [DTS-2680] Fix Delta Sharing DataFrame not updated for OS... by Lin Zhou <[email protected]> - f2ced890a119fe823ffe0af3afaabf839d503eaf Revert "[ES-994299][SQL] Inline CURRENT_USER() and simila... by Xin Zhao <[email protected]> - a02baf8a2be83f4de0c838f2c3819e43416031dc [ES-998264][Follow-up][Liquid] Fix inconsistent "create e... by Jiaheng Tang <[email protected]> - 6e602faed8effcafa2309db5f2de963ba4812c28 [ES-993304] Fix a bug of Deep Clone Iceberg in UC by Ming DAI <[email protected]> - c2b1f48d32a35fd24ae014522271325c49e84439 [SC-147450][Delta] Support `_metadata` for Delta Streamin... by Yaohua Zhao <[email protected]> - a8518c05dadd2418e0e6470ac0da77007a22c2d1 [AUTO][SC-153803][SPARK-46644] Change add and merge in SQ... by Davin Tjong <[email protected]> - f7434fa843612dd1f7c98890a2358064c76a84f3 [DELTA-OSS-EXTERNAL] [Spark] Add messageParameters to Del... by Lars Kroll <[email protected]> - 9a7ef3be9ece561f9d86a42978ffdbb8057a7efb [DELTA][LC-2405] Apply DMQ tag in MetadatCleanup flow by Prakhar Jain <[email protected]> - 2ba43be1da767dfe4a185c6987c4cd4d787cf3b9 [ES-994299][SQL] Inline CURRENT_USER() and similar functi... by Daniel Tenedorio <[email protected]> - 017ae2075e0165d7231896621ac5f2d1e91490e5 [DELTA-OSS-EXTERNAL] Upgrade Hadoop version to 3.3.4 by Scott Sandre <[email protected]> - 26bf33ac2514ec42d9cc195f83c81ee05607d81a [LC-2865] Reorder checks in ConflictChecker by Christos Stavrakakis <[email protected]> - 2dfd2da0f6bb7a9b01199e20edbb17a49038a80a [ES-998264][Liquid] Allow creating liquid table from exte... by Jiaheng Tang <[email protected]> - 3e8fcfd01899410b68210461707cc82dec83de22 [LC-3002] Fix Delta benchmarks: unaccessible Utils.median by Johan Lasperas <[email protected]> - 92282898c139002146e7e1844089dc84df2ac8c3 [LC-2944][DELTA] Add Writer Protocol check in Vacuum Comm... by Prakhar Jain <[email protected]> - ac9d0bfb5b278bab3de8d2a27fea197b85650b8c [DTS-2637] Log more info in DeltaFormatSharingSource by Lin Zhou <[email protected]> - 851a127034b98fed886621491705e344b6f87c11 [ES-1006244] Always widen UDFs in Delta conflict checker by Christos Stavrakakis <[email protected]> - 0d1d16aad762608029d3f7c3972b620f183ba75f [SC-150028][WARMFIX] Improve Delta unsupported reader/wri... by Yaohua Zhao <[email protected]> - cd123efad2d4789710de4ce0b6bc472b48402442 [ES-956094][Delta] Handle CREATE OR REPLACE for existing ... by Dhruv Arya <[email protected]> - 808de7dabd61101b59f73ffc0a6b8a1149f1bd34 [LC-2995][DELTA] Add comment around why DeltaLog Vacuum l... by Prakhar Jain <[email protected]> - d125486836b167c63fa0b5a0f3535fc4cfc04274 [LC-2943] Add additional metrics to usage logging to get ... by Rajesh Parangi <[email protected]> - 99dc2eae14fa352a6ab03cccab173c72acf5b5d6 [Delta][LC-2153] Add no checksum (post-merge) variants of... by Dhruv Arya <[email protected]> - 7fc9b8aba880f2742a0e70057140b2c62941c0a9 [SC-153158] Fix for ES-990157 which is bug in the delta s... by Boyang Jerry Peng <[email protected]> - 9664dde1003c5717a9c8bc4403cafac748cb3e98 [DELTA-OSS-EXTERNAL] [Release] Update `run-integration-te... by Venki Korukanti <[email protected]> - 920fcd4deb27495cdf9c88ad20ca24f977ef5004 [AUTOSTATS][STATS-231] Add restrictions to collect stats ... by mohamedzait <[email protected]> - 3b8b7bf2a56ee0665c9c6d99fa4f3cb71a459b5d [DELTA-OSS-EXTERNAL] [Spark][Sharing][Build] Add release ... by Venki Korukanti <[email protected]> - df510b8cf441c8e1d149dd5d0509953ffae21095 [LC-2977] Add expression for Delta's canonical path by Menelaos Karavelas <[email protected]> - e9f1c17c482be35979a64b2ee5e72c4562c084d5 [LC-2892] Avoid redudant jobs in buildRowIndexSetsForFile... by Andreas Chatzistergiou <[email protected]> - 3ba616e632fe427057b02414546698d9b063d9a3 [Delta OSS][Kernel] Downgrade the SLF4J version used for ... by Allison Portis <[email protected]> - 8ec5560ebefde4b5c7edaac39569b404f9f79a8b needed changes for building flink with kernel by Nick Lanham <[email protected]> - 4753e54bacf2c076f1266c0a09c55c064e59c88d [AUTOSTATS][STATS-267] fix min max values before converti... by mohamedzait <[email protected]> - c489eecbcd5fa6f597dcedc17b0be4fb792c07d8 [bre][delta] Make sure files are writable in Evolvability... by Gabriel Russo <[email protected]> - d05beeb5e738bd072f200d67be6a6c871492b98c [LC-2695] Fix Exception when creating Liquid table and en... by Hao Jiang <[email protected]> - 895b82e456dd1cd26c7e6b13d84f4ad4428b9913 [LC-2801][Liquid OSS] Add Clustering example to Delta 3.1 by Jiaheng Tang <[email protected]> - 25d163dbd59c41ff4c5fb9c367d37e5669f60b21 [LC-2320] Abbreviate long strings in DML stats by Christos Stavrakakis <[email protected]> - 2cbdbcabd70730f5ab9348a0283e4c3a2e407cec [SC-153095] Refactor assume for better code sync in DBR c... by Hyukjin Kwon <[email protected]> - 4745ce2f36caad724171a282e22da87e0fc68594 [LC-2646][SC-150301] Widen non-deterministic predicates d... by Fredrik Klauss <[email protected]> - b893b8e9b3fd634b4c5f222fc8e78d83f1913dc3 [AUTOSTATS][SC-139947] Expose alterTableStats calls measu... by Pat Sukprasert <[email protected]> - 826a1aff5e86b1c31159f0b7820645e2a9179896 [AUTO][SC-149572][SPARK-45629][CORE][SQL][CONNECT][ML][ST... by tangjiafu <[email protected]> - d73e251739d3927aaa1c8774c191642c5a4989f4 [LC-2945] Rename row tracking tag in CommitInfo by Fredrik Klauss <[email protected]> - 3cc9847b0531f84815172cc067054cfa4b40844f [LC-226] Emphasize paths should (or should not) be escape... by Paddy Xu <[email protected]> - da50b2c37e24d90bd8bb7e4c714ea01c6c9a0bea [LC-2957] Remove edge code from WriteIntoDelta by sabir-akhadov <[email protected]> - c6ea78a12cc294d077ad75b6e17f88bc74100214 [Delta OSS] Change kernel CI to run integration tests by Venki Korukanti <[email protected]> - 816ae9b4c9409f301690e205621ed252848cbb5b [DTS-2381] Adds PrepareDeltaSharingScan and add it in Del... by Lin Zhou <[email protected]> - f1085d3a34775d93949b55ec52f2f2e0af09bcc6 [LC-2820] CreateOrReplace table command should not follow... by sabir-akhadov <[email protected]> - 93d502ca11e47a03d2be1a03a0821d45edad5836 [LC-2948] auto upgrade Delta protocol for REORG UPGRADE U... by Fred Storage Liu <[email protected]> - 54765a9285ef680dbecdd230a7fcac83c0da8f16 [LC-2949] Fix REORG APPLY UNIFORM by Hao Jiang <[email protected]> - b9ece910bcc4be8040b02a516b18fd41df47cff4 [LC-2940] Not enable DV by default when IcebergCompat is ... by Fred Storage Liu <[email protected]> - 05d1bba13436a563f7a80945e0350a0b57a73181 [LC-274] Add comments to DeltaTable.forName to clarify vi... by Jiaheng Tang <[email protected]> - c7ab3cbd2774eac1bee131b55e26d3a510e14e35 [DTS-2630] Export streaming and CM for delta format sharing by Lin Zhou <[email protected]> - ae894f0f90d0dec08bd6b0a4b2aeb6974c2fc647 [LC-92] Add separate metrics for DPP and DFP by Ole Sasse <[email protected]> - f82f2d50ff10df885f74aae0392d6c5991fe72e6 [Delta] Remove disclaimer about the Delta protocol still ... by Dhruv Arya <[email protected]> - f8e4cc7ea09a1bf14bfabdc71ca61764133e38b5 [DELTA-OSS-EXTERNAL] [Spark][UPDATE with DV] Let UPDATE c... by Paddy Xu <[email protected]> - 59f3382d2e8446e72a07ca4c2f219564bbbed4c6 [STATS-103][AUTO-STATS] Allow the StatsOnLoad tracker to ... by Desmond Cheong <[email protected]> - c46a3342115f86e40851c3444c4981f3afced093 [LC-2766][Delta] Temporarily disable two flaky tests from... by Dhruv Arya <[email protected]> - 4bd63ad92bcf534bb1bec95113e8009f41767bc3 [LC-2871][Delta][MC] Refactor conflict detection code so ... by Prakhar Jain <[email protected]> - b0fb8f09a5c13d2ac14a907e9e872646354e7539 Open source autocompact by Nick Lanham <[email protected]> - 3a906221689c4dcfed402746b13f8fa9a67f11fe [DELTA-OSS-EXTERNAL]Implement UPGRADE UNIFORM (ICEBERG_CO... by Kaiqi Jin <[email protected]> - 0627069552284f72cebb9ffd55259f5eba0404d9 [LC-2894][Delta][MC] Move lockInterruptibly helper method... by Prakhar Jain <[email protected]> - 821b91853cd229538e2f2e8e4439f5ce7a1438ce [Spark] Delete duplicate DeltaErrorsSuite test by Scott Sandre <[email protected]> - 9ecb7a64d0bcd8f77e411cb5a5b36c2b9ed8c2f3 [ES-966215] Capture a fake call site in Delta data skippi... by Wenchen Fan <[email protected]> - 2482f48c1344db14ecc79137ba1ea675820c83eb [DTS-2563] Export cdf support for delta format sharing by Lin Zhou <[email protected]> - 8161fc3fa4e4aa5c0e0192b336038439b2e435be [LS-965] Trigger synchronous Iceberg Conversion when enab... by Hao Jiang <[email protected]> - 9ab789acc9892daf95a8bbb0b71fcc3feb393d34 [LC-2922][Liquid OSS] Integrate domain metadata with OPTI... by Jiaheng Tang <[email protected]> - c142c673973388e7f1afa16432702537a35978f9 [LC-2218] Implement redaction of Arclight-managed entitie... by Mann Patel <[email protected]> - 984fa6f3304bf68e97757724c69a542205319b23 [LC-2766][Delta] Add support for syncing Delta table sche... by Dhruv Arya <[email protected]> - 143c6b383463a77da8afd188677f717215ee6a1c [AUTOSTATS][PO] Add runtime metrics to PO store for backg... by Satya Valluri <[email protected]> - 275cd6ec36d49bd078e5e4b7bd7c9bab871b0e1e [LC-2780][Delta] Support List and Map columns in Uniform by Lukas Rupprecht <[email protected]> - c61f3ea158811246dea0deef2a26d00866d46496 [LC-2697]Implement UPGRADE UNIFORM (ICEBERG_COMPAT_VERSIO... by Kaiqi Jin <[email protected]> - fc9ac35cf8be32c42e0c94c67ed11426d5500b63 [Delta OSS] Add a script to generate and upload docs to d... by Venki Korukanti <[email protected]> - a095445b6da809ee9a5b4ece7c38d04a172ff70f [DTS-2562][2nd export] Export snapshot support for "delta... by Lin Zhou <[email protected]> - 189b30448d606df335fa5c7af4b7a53d5208aefb [LC-2554][Liquid OSS] Store CLUSTER BY columns as DomainM... by Jiaheng Tang <[email protected]> - 0ded3b867705b5eb3921b936d45519df5dbe40e1 [LC-2852] Add DV Metrics in Classic Merge by Andreas Chatzistergiou <[email protected]> - 0f9eb1a02c52ed8d72a6ec8e57a2872de3e06c6e [Liquid][LC-2356] Repeat optimize until no optimize batch... by Wei Luo <[email protected]> - 930c22f0e0c8c376612af52d68a39c7e8f9ddb78 [LC-2896] Fix commit tags in DBR for Classic merge with DVs by Andreas Chatzistergiou <[email protected]> - 1406baa237a9bb439a0676fa3c5bd61ea23da0b2 [LS-969] Support Delta Uniform statistics conversion to I... by Fred Storage Liu <[email protected]> - bbcd1c5e46cd45727a519892cae22b57f458f544 [LS-965] Add unit tests for IcebergCompatV2 by Hao Jiang <[email protected]> - 563cfe78323b695e33bfa0286b8ab0810cdab517 [SC-147475][FOLLOW-UP] Add NestedColumnPruning telemetry ... by Anton Lykov <[email protected]> - 9b88f76bf99cc38bd4cf9d3397b7bb8ade822d0b [DELTA-OSS-EXTERNAL] Optimize Min/Max using Delta metadata by Venki Korukanti <[email protected]> - 42f9138f6c0bfc0fa8e98d4161b6ccb252482536 [LS-965] Update Delta Protocol for IcebergCompatV2 by Hao Jiang <[email protected]> - 86c0d510f1859bee252c0dfc1f51a3c64ff1642e [Liquid][LC-2864] Support `alter table cluster by` on unp... by Wei Luo <[email protected]> - f1eba9a0e2922a1521fc1e7b5e2bb2a06f197fbe [SC-151693] [LC-1311] DFP Null Safe Equality JOIN by Thang Long Vu <[email protected]> - 3596470844c8bb1f278fc2a53d9b7997ff60ae19 [LS-968] Use non-fate-sharing future for prefetched delta... by Fred Storage Liu <[email protected]> - 54c53c2b0fafe77afaf4ee6439b94b25b162ad2c [LC-2783][LIQUID] Reduce number of threads (concurrent ba... by Rahul Shivu Mahadev <[email protected]> - 9f0a8d40a3af45351f4544dde339fcb769d20f3e [LS-965] Use IcebergCompatV2 in UniForm by Hao Jiang <[email protected]> - 7bbd0c4d3f08d91df890f6900c76de3c5cc25748 [LC-1757] Add support for Deletion Vectors to Merge by Andreas Chatzistergiou <[email protected]> - dca825e3fd63c3e8c2d799f17ecf947dc548a56e [LC-2858][LIQUID] Add matching domain metadata as part of... by Ami Oka <[email protected]> - cf7233fee754fb116f75f7fef2bae9c50081e2b5 [ES-909313][Delta] Print correct error message when sourc... by Lukas Rupprecht <[email protected]> - f5a38dafbd7ffbb49fd87fbad3df13271d18f3fc [LC-2218] Implement redaction of Arclight-managed entitie... by Mann Patel <[email protected]> - ab2ddea623cf267eed305fd007a673c6d6be46c5 [DTS-2561] Export util functions for delta format sharing by Lin Zhou <[email protected]> - 2cabc99e153630e8be4eb65789513a1260a15464 [LC-2552][LIQUID] Phase out LiquidTableFeature by Ami Oka <[email protected]> - 68a36a595b312e4c4b9536c6fa8da79f4a432c2d [LC-2908][Liquid OSS] Move DDL functions/tests from Edge ... by Jiaheng Tang <[email protected]> - bd489a1b7b3a4adaaf3219dc0dfbd3e01abe675b [Liquid][LC-2858] Integrate ClusteringMetadataDomain by jintao shen <[email protected]> - 4882b1174045ef2732a08e9ee200fab104ab15bf [DELTA-OSS-EXTERNAL] [Spark] Discard scoverage from delta... by Fred Storage Liu <[email protected]> - e809c32c84534111f020c42a98d5e818960508a7 [ES-977873][DELTA] Add new checkpoint API which doesn't i... by Prakhar Jain <[email protected]> (And 3968 more changes) GitOrigin-RevId: 1ca496b5505c7bcbea7651c6251aa644a039ca7f
jingz-db · Feb 8, 2024 · 749faa6 · 749faa6
commit 749faa6
Show file tree

Hide file tree

Showing 837 changed files with 246,624 additions and 0 deletions.
diff --git a/.gitattributes b/.gitattributes
@@ -0,0 +1,3 @@
+*.bat text eol=crlf
+*.cmd text eol=crlf
+*.bin binary
diff --git a/.github/ISSUE_TEMPLATE/bug-issue.md b/.github/ISSUE_TEMPLATE/bug-issue.md
@@ -0,0 +1,60 @@
+---
+name: Bug Issue
+about: Use this template for reporting a bug
+labels: 'bug'
+title: '[BUG]'
+
+---
+
+## Bug
+
+#### Which Delta project/connector is this regarding?
+<!--
+Please add the component selected below to the beginning of the issue title
+For example: [BUG][Spark] Title of my issue
+-->
+
+- [ ] Spark
+- [ ] Standalone
+- [ ] Flink
+- [ ] Kernel
+- [ ] Other (fill in here)
+
+### Describe the problem
+
+#### Steps to reproduce
+
+<!--
+Please include copy-pastable code snippets if possible.
+1. _____
+2. _____
+3. _____
+-->
+
+#### Observed results
+
+<!-- What happened?  This could be a description, log output, etc. -->
+
+#### Expected results
+
+<!-- What did you expect to happen? -->
+
+#### Further details
+
+<!--
+Include any additional details that may be useful for diagnosing the problem here. If including tracebacks, please include the full traceback. Large logs and files should be attached.
+-->
+
+### Environment information
+
+* Delta Lake version:
+* Spark version:
+* Scala version:
+
+### Willingness to contribute
+
+The Delta Lake Community encourages bug fix contributions. Would you or another member of your organization be willing to contribute a fix for this bug to the Delta Lake code base?
+
+- [ ] Yes. I can contribute a fix for this bug independently.
+- [ ] Yes. I would be willing to contribute a fix for this bug with guidance from the Delta Lake community.
+- [ ] No. I cannot contribute a bug fix at this time.
diff --git a/.github/ISSUE_TEMPLATE/feature-request.md b/.github/ISSUE_TEMPLATE/feature-request.md
@@ -0,0 +1,44 @@
+---
+name: Feature Request
+about: Use this template for raising a feature request
+labels: 'enhancement'
+title: '[Feature Request]'
+
+---
+
+## Feature request
+
+#### Which Delta project/connector is this regarding?
+<!--
+Please add the component selected below to the beginning of the issue title
+For example: [Feature Request][Spark] Title of my issue
+-->
+
+- [ ] Spark
+- [ ] Standalone
+- [ ] Flink
+- [ ] Kernel
+- [ ] Other (fill in here)
+
+### Overview
+
+<!-- Provide a high-level description of the feature request. -->
+
+### Motivation
+
+<!-- How will this feature be used? Why is it important? Which users will benefit from it? -->
+
+### Further details
+
+<!--
+Use this section to include any additional information about the feature. If you have a proposal for how to implement this feature, please include it here. For implementation guidelines, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
+If there are any specific requirements for this feature that are not immediately obvious please outline them here.
+-->
+
+### Willingness to contribute
+
+The Delta Lake Community encourages new feature contributions. Would you or another member of your organization be willing to contribute an implementation of this feature?
+
+- [ ] Yes. I can contribute this feature independently.
+- [ ] Yes. I would be willing to contribute this feature with guidance from the Delta Lake community.
+- [ ] No. I cannot contribute this feature at this time.
diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md
@@ -0,0 +1,46 @@
+<!--
+Thanks for sending a pull request!  Here are some tips for you:
+  1. If this is your first time, please read our contributor guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
+  2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP] Your PR title ...'.
+  3. Be sure to keep the PR description updated to reflect all changes.
+  4. Please write your PR title to summarize what this PR proposes.
+  5. If possible, provide a concise example to reproduce the issue for a faster review.
+  6. If applicable, include the corresponding issue number in the PR title and link it in the body.
+-->
+
+#### Which Delta project/connector is this regarding?
+<!--
+Please add the component selected below to the beginning of the pull request title
+For example: [Spark] Title of my pull request
+-->
+
+- [ ] Spark
+- [ ] Standalone
+- [ ] Flink
+- [ ] Kernel
+- [ ] Other (fill in here)
+
+## Description
+
+<!--
+- Describe what this PR changes.
+- Describe why we need the change.
+ 
+If this PR resolves an issue be sure to include "Resolves #XXX" to correctly link and close the issue upon merge.
+-->
+
+## How was this patch tested?
+
+<!--
+If tests were added, say they were added here. Please make sure to test the changes thoroughly including negative and positive cases if possible.
+If the changes were tested in any way other than unit tests, please clarify how you tested step by step (ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future).
+If the changes were not tested, please explain why.
+-->
+
+## Does this PR introduce _any_ user-facing changes?
+
+<!--
+If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
+If possible, please also clarify if this is a user-facing change compared to the released Delta Lake versions or within the unreleased branches such as master.
+If no, write 'No'.
+-->
diff --git a/.github/workflows/connectors_test.yaml b/.github/workflows/connectors_test.yaml
@@ -0,0 +1,45 @@
+name: "Delta Connectors Tests"
+on: [push, pull_request]
+jobs:
+  build:
+    name: "Run tests"
+    runs-on: ubuntu-20.04
+    strategy:
+      matrix:
+        # These Scala versions must match those in the build.sbt
+        scala: [2.13.8, 2.12.17]
+    steps:
+      - uses: actions/checkout@v2
+      - name: install java
+        uses: actions/setup-java@v2
+        with:
+          distribution: 'zulu'
+          java-version: '8'
+      - name: Cache Scala, SBT
+        uses: actions/cache@v2
+        with:
+          path: |
+            ~/.sbt
+            ~/.ivy2
+            ~/.cache/coursier
+            ~/.m2
+          key: build-cache-3-with-scala_${{ matrix.scala }}
+      - name: Run Scala Style tests on test sources (Scala 2.12 only)
+        run: build/sbt "++ ${{ matrix.scala }}" testScalastyle
+        if: startsWith(matrix.scala, '2.12.')
+      - name: Run sqlDeltaImport tests (Scala 2.12 and 2.13 only)
+        run: build/sbt "++ ${{ matrix.scala }}" sqlDeltaImport/test
+        if: ${{ !startsWith(matrix.scala, '2.11.') }}
+        # These tests are not working yet
+        # - name: Run Delta Standalone Compatibility tests (Scala 2.12 only)
+        #  run: build/sbt "++ ${{ matrix.scala }}" compatibility/test
+        #  if: startsWith(matrix.scala, '2.12.')
+      - name: Run Delta Standalone tests
+        run: build/sbt "++ ${{ matrix.scala }}" standalone/test testStandaloneCosmetic/test standaloneParquet/test testParquetUtilsWithStandaloneCosmetic/test
+      - name: Run Hive 3 tests
+        run: build/sbt "++ ${{ matrix.scala }}" hiveMR/test hiveTez/test
+      - name: Run Hive 2 tests
+        run: build/sbt "++ ${{ matrix.scala }}" hive2MR/test hive2Tez/test
+      - name: Run Flink tests (Scala 2.12 only)
+        run: build/sbt -mem 3000 "++ ${{ matrix.scala }}" flink/test
+        if: ${{ startsWith(matrix.scala, '2.12.') }}
diff --git a/.github/workflows/kernel_docs.yaml b/.github/workflows/kernel_docs.yaml
@@ -0,0 +1,51 @@
+# Simple workflow for deploying static content to GitHub Pages
+name: Deploy static content to Pages
+
+on:
+  # Allows you to run this workflow manually from the Actions tab
+  workflow_dispatch:
+
+# Sets permissions of the GITHUB_TOKEN to allow deployment to GitHub Pages
+permissions:
+  contents: read
+  pages: write
+  id-token: write
+
+# Allow only one concurrent deployment, skipping runs queued between the run in-progress and latest queued.
+# However, do NOT cancel in-progress runs as we want to allow these production deployments to complete.
+concurrency:
+  group: "pages"
+  cancel-in-progress: false
+
+jobs:
+  # Single deploy job since we're just deploying
+  deploy_docs:
+    environment:
+      name: github-pages
+      url: ${{ steps.deployment.outputs.page_url }}
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v3
+      - name: install java
+        uses: actions/setup-java@v3
+        with:
+          distribution: "zulu"
+          java-version: "8"
+      - name: Generate docs
+        run: |
+          build/sbt kernelGroup/unidoc
+          mkdir -p kernel/docs/snapshot/kernel-api/java
+          mkdir -p kernel/docs/snapshot/kernel-defaults/java
+          cp -r kernel/kernel-api/target/javaunidoc/. kernel/docs/snapshot/kernel-api/java/
+          cp -r kernel/kernel-defaults/target/javaunidoc/. kernel/docs/snapshot/kernel-defaults/java/
+      - name: Setup Pages
+        uses: actions/configure-pages@v3
+      - name: Upload artifact
+        uses: actions/upload-pages-artifact@v1
+        with:
+          # Upload kernel docs
+          path: kernel/docs
+      - name: Deploy to GitHub Pages
+        id: deployment
+        uses: actions/deploy-pages@v2
diff --git a/.github/workflows/kernel_test.yaml b/.github/workflows/kernel_test.yaml
@@ -0,0 +1,20 @@
+name: "Delta Kernel Tests"
+on: [push, pull_request]
+jobs:
+  test:
+    runs-on: ubuntu-20.04
+    env:
+      SCALA_VERSION: 2.12.17
+    steps:
+      - uses: actions/checkout@v3
+      - name: install java
+        uses: actions/setup-java@v3
+        with:
+          distribution: "zulu"
+          java-version: "8"
+      - name: Run tests
+        run: |
+          python run-tests.py --group kernel --coverage
+      - name: Run integration tests
+        run: |
+          cd kernel/examples && python run-kernel-examples.py --use-local
diff --git a/.github/workflows/new_pull_request.yaml b/.github/workflows/new_pull_request.yaml
@@ -0,0 +1,16 @@
+name: Add new pull requests to Backlog (External)
+
+on:
+  pull_request_target:
+    types: [opened, reopened]
+
+jobs:
+  automate-new-pull-requests:
+    if: ${{ !contains('allisonport-db dennyglee scottsand-db tdas vkorukanti zsxwing', github.event.sender.login) }}
+    runs-on: ubuntu-latest
+    steps:
+      - uses: alex-page/[email protected]
+        with:
+          project: oss-delta-prs
+          column: Needs Review
+          repo-token: ${{ secrets.PROJECT_BOARD_AUTOMATION_TOKEN }}
diff --git a/.github/workflows/new_updated_issue.yaml b/.github/workflows/new_updated_issue.yaml
@@ -0,0 +1,32 @@
+name: Add new and updated issues to Needs Review
+
+env:
+  eng_usernames: allisonport-db scottsand-db tdas vkorukanti zsxwing
+  devrel_usernames: dennyglee MrPowers nkarpov vinijaiswal
+
+on:
+  issues:
+    types: [opened, reopened]
+  issue_comment:
+    types: [created]
+
+jobs:
+  automate-new-updated-issues:
+    if: ${{ !github.event.issue.pull_request &&
+      !contains('allisonport-db scottsand-db tdas vkorukanti zsxwing dennyglee MrPowers nkarpov vinijaiswal', github.event.sender.login) }}
+    runs-on: ubuntu-latest
+    steps:
+      - name: Move to Needs Review (ENG)
+        if: ${{ github.event.issue.assignee != null && contains(env.eng_usernames, github.event.issue.assignee.login) }}
+        uses: alex-page/[email protected]
+        with:
+          project: oss-delta-issues
+          column: Needs Review (ENG)
+          repo-token: ${{ secrets.PROJECT_BOARD_AUTOMATION_TOKEN }}
+      - name: Move to Needs Review
+        if: ${{ github.event.issue.assignee == null || contains(env.devrel_usernames, github.event.issue.assignee.login) }}
+        uses: alex-page/[email protected]
+        with:
+          project: oss-delta-issues
+          column: Needs Review
+          repo-token: ${{ secrets.PROJECT_BOARD_AUTOMATION_TOKEN }}
diff --git a/.github/workflows/spark_test.yaml b/.github/workflows/spark_test.yaml
@@ -0,0 +1,69 @@
+name: "Delta Spark Tests"
+on: [push, pull_request]
+jobs:
+  test:
+    runs-on: ubuntu-20.04
+    strategy:
+      matrix:
+        # These Scala versions must match those in the build.sbt
+        scala: [2.12.17, 2.13.8]
+    env:
+      SCALA_VERSION: ${{ matrix.scala }}
+    steps:
+      - uses: actions/checkout@v3
+      - uses: technote-space/get-diff-action@v4
+        id: git-diff
+        with:
+          PATTERNS: |
+            **
+            !kernel/**
+            !connectors/**
+      - name: install java
+        uses: actions/setup-java@v3
+        with:
+          distribution: "zulu"
+          java-version: "8"
+      - name: Cache Scala, SBT
+        uses: actions/cache@v3
+        with:
+          path: |
+            ~/.sbt
+            ~/.ivy2
+            ~/.cache/coursier
+          # Change the key if dependencies are changed. For each key, GitHub Actions will cache the
+          # the above directories when we use the key for the first time. After that, each run will
+          # just use the cache. The cache is immutable so we need to use a new key when trying to
+          # cache new stuff.
+          key: delta-sbt-cache-spark3.2-scala${{ matrix.scala }}
+      - name: Install Job dependencies
+        run: |
+          sudo apt-get update
+          sudo apt-get install -y make build-essential libssl-dev zlib1g-dev libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev libncursesw5-dev xz-utils tk-dev libffi-dev liblzma-dev python-openssl git
+          sudo apt install libedit-dev
+          sudo apt install python3-pip --fix-missing
+          sudo pip3 install pipenv==2021.5.29
+          curl https://pyenv.run | bash
+          export PATH="~/.pyenv/bin:$PATH"
+          eval "$(pyenv init -)"
+          eval "$(pyenv virtualenv-init -)"
+          pyenv install 3.8.18
+          pyenv global system 3.8.18
+          pipenv --python 3.8 install
+          pipenv run pip install pyspark==3.5.0
+          pipenv run pip install flake8==3.5.0 pypandoc==1.3.3
+          pipenv run pip install importlib_metadata==3.10.0
+          pipenv run pip install mypy==0.982
+          pipenv run pip install cryptography==37.0.4
+          pipenv run pip install twine==4.0.1
+          pipenv run pip install wheel==0.33.4
+          pipenv run pip install setuptools==41.0.1
+          pipenv run pip install pydocstyle==3.0.0
+          pipenv run pip install pandas==1.0.5
+          pipenv run pip install pyarrow==8.0.0
+          pipenv run pip install numpy==1.20.3
+        if: steps.git-diff.outputs.diff
+      - name: Run Scala/Java and Python tests
+        run: |
+          pipenv run python run-tests.py --group spark
+          cd examples/scala && build/sbt "++ $SCALA_VERSION compile"
+        if: steps.git-diff.outputs.diff