Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Pegasus spark3.1 #55

Open
wants to merge 91 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
91 commits
Select commit Hold shift + click to select a range
8c39bb0
feat: support auto-load table info from pegasus gateway (xiaomi#1)
foreverneverer Dec 23, 2020
214abd8
feat: support auto-load table version from pegasus gateway (xiaomi#2)
foreverneverer Dec 24, 2020
193ec37
feat: support auto-load table version from pegasus gateway (xiaomi#2)
hycdong Dec 30, 2020
a4c3175
fix: default auto load pegasus-gateway may faild(xiaomi#3)
foreverneverer Jan 7, 2021
459ffd3
fix: default auto load pegasus-gateway may faild(xiaomi#3)
Jan 20, 2021
03836cb
fix: get rdd partition count null pointer(xiaomi#4)
foreverneverer Jan 21, 2021
00b7090
refactor: merge master support mulit set load(xiaomi#5)
foreverneverer Feb 3, 2021
2db30f8
feat: update url match rule for pegasus v2.2.0 backup/bulkload version
Apr 16, 2021
f016f4c
Merge branch 'update-2.2.0' into 'xiaomi'
Apr 16, 2021
d453f3b
fix
foreverneverer Apr 22, 2021
ae554f5
update
foreverneverer Apr 23, 2021
abf651b
refactor: update data version init type
Apr 23, 2021
ff46764
Merge branch 'update-version' into 'xiaomi'
Apr 23, 2021
5cdb16f
Merge branch 'exist-table' into 'xiaomi'
Apr 23, 2021
1854913
doc: add comments on public user-facing api
Apr 23, 2021
9325f6b
Merge branch 'docs' into 'xiaomi'
Apr 23, 2021
6253b99
Merge branch 'fix-filesize' into 'xiaomi'
Apr 23, 2021
74d74d2
fix: metadata file retry operation is not right
Apr 23, 2021
48a3bd3
Merge branch 'fix-retry-size' into 'xiaomi'
Apr 23, 2021
24b1ae8
fix
foreverneverer Apr 25, 2021
da7d4cf
Merge branch 'fix-example' into 'xiaomi'
Apr 25, 2021
e460575
update
foreverneverer Apr 25, 2021
fb6b57d
Merge branch 'delete-useless-code' into 'xiaomi'
Apr 25, 2021
191125e
feat: support compress when writing data
May 6, 2021
a791e6e
Merge branch 'test-compress2' into 'xiaomi'
May 6, 2021
3ed8c8c
ignore not sst file size get
foreverneverer May 25, 2021
bde6b7f
fix: byte compare error when the data is negative
May 28, 2021
341f15c
Merge branch 'compare-fix' into 'xiaomi'
May 28, 2021
828af0f
fix: ignore not sst file size get when retry
May 28, 2021
ea080ae
refactor: update cold backup data path match rules for all version
Jul 26, 2021
8fb9a27
merge refactor: update cold backup data path match rules for all version
Jul 26, 2021
4869426
fix: latest timestamp may can't find the table
Oct 28, 2021
4217a02
feat: support auto detect the data version when read the dump data if…
Nov 1, 2021
e980aa1
Merge branch 'support-compatiable' into 'xiaomi'
Nov 1, 2021
995fec5
feat: support send backup request
Nov 16, 2021
75a0e3c
Merge branch 'support-backup' into 'xiaomi'
Nov 16, 2021
98bda2d
refactor: update threee module to one module
Nov 18, 2021
e08f63a
Merge branch 'one-module' into 'xiaomi'
Nov 18, 2021
b002c92
init
foreverneverer Nov 18, 2021
aaab07e
Merge branch 'resolve-package' into 'xiaomi'
Nov 18, 2021
9448f9e
fix: resolve the spark version compatibility with addTaskCompletionLi…
Nov 18, 2021
dddf3dd
fix: resolve the spark version compatibility with addTaskCompletionLi…
Nov 18, 2021
3c535b9
fix: fix hdfs backup info parser error
Nov 19, 2021
9b62d39
fix: fix hdfs backup info parser error
Nov 19, 2021
03b3904
fix: hdfs parser the backup path failed
Nov 19, 2021
82fee4d
fix: hdfs parser the backup path failed
Nov 19, 2021
5de9176
feat: support bulkload
Nov 22, 2021
232571e
feat: support bulkload
Nov 22, 2021
b162d09
feat: support compaction after bulkload
Nov 23, 2021
dce74b0
feat: support compaction after bulkload
Nov 23, 2021
3f40148
fix: fix some bug
Jan 4, 2022
a061fd4
Merge branch 'etl-snp' into 'xiaomi'
Jan 4, 2022
3052b47
fix: resp string length may be less 100
foreverneverer Jan 4, 2022
7873fee
feat: auto filter expired record
Jan 13, 2022
28b6070
Merge branch 'auto-filterExpired' into 'xiaomi'
Jan 13, 2022
06b597b
init
foreverneverer Feb 7, 2022
e214ffa
Merge branch 'update-bulkload-status' into 'xiaomi'
Feb 7, 2022
3076b92
init
foreverneverer Feb 14, 2022
15382a2
Merge branch 'compaction-limit' into 'xiaomi'
Feb 14, 2022
5c8dacf
init
foreverneverer Feb 15, 2022
a3e140f
Merge branch 'comopaction-fix' into 'xiaomi'
Feb 15, 2022
64850ae
fix next null
foreverneverer Feb 18, 2022
28abb9b
Merge branch 'fix-next-null' into 'xiaomi'
Feb 18, 2022
f265110
fix
foreverneverer Feb 18, 2022
4298e40
Merge branch 'fix-filter' into 'xiaomi'
Feb 18, 2022
4390a92
fix: avoid jni load system stdc++
Feb 21, 2022
db2c889
Merge branch 'fix-jni-env' into 'xiaomi'
Feb 21, 2022
4d7cd71
fix
foreverneverer Feb 23, 2022
7a9d0b0
fix: send backup request cluster is error
Feb 23, 2022
5952b8d
fix: update to support two partition count
Mar 16, 2022
d2fb9d4
Merge branch 'fix-partition-count' into 'xiaomi'
Mar 16, 2022
97e9534
refactor: support switch cluster via metaproxy
Mar 31, 2022
c53d802
refactor: support switch cluster via metaproxy
Mar 31, 2022
50de53d
init
foreverneverer Apr 13, 2022
ab49597
update version
foreverneverer Apr 19, 2022
f76545a
feature: add validate after create table data
Apr 19, 2022
a272a9d
Merge branch 'add_validate' into 'xiaomi'
Apr 19, 2022
35ff179
feat: support manual compaction periodicly
May 18, 2022
d850d6d
Merge branch 'comopaction-plus' into 'xiaomi'
May 18, 2022
4ae787c
update
foreverneverer May 18, 2022
74b82e4
Merge branch 'fix-empty' into 'xiaomi'
May 18, 2022
465d682
fix: jni lib path should put pwd
May 18, 2022
e192b24
Merge branch 'fix-jni-path' into 'xiaomi'
May 18, 2022
c22bdeb
fix: add jni load when validate data
May 19, 2022
29ca485
Merge branch 'fix-validate' into 'xiaomi'
May 19, 2022
663731f
feat: support manual compaction query
May 20, 2022
be51d71
Merge branch 'support-compaction-query' into 'xiaomi'
May 20, 2022
57de42f
Merge branch 'xiaomi' into spy3.0
foreverneverer May 27, 2022
42f4a82
fix compaction progress
foreverneverer Jun 22, 2022
bc147b4
Merge branch 'xiaomi' into 3.1
foreverneverer Jun 30, 2022
49d3fb0
update pegasus-spark version to 3.1
Apr 16, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -39,3 +39,9 @@ dependency-reduced-pom.xml
.project
.settings/
scalafmt
rolling_log

#vscode
.vscode
.metals
.bloop
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,4 +6,4 @@ manipulate your Pegasus data:
- Offline analysis of your Pegasus snapshot, see example: [count data](https://github.com/pegasus-kv/pegasus-spark/blob/8c585a47e4b618924275c5c1404bdaef9c26f40a/pegasus-spark-analyser/src/main/scala/com/xiaomi/infra/pegasus/spark/analyser/examples/basic/CountData.scala)
- Transform your Pegasus snapshot into Parquet files, see example: [convert parquet](https://github.com/pegasus-kv/pegasus-spark/tree/8c585a47e4b618924275c5c1404bdaef9c26f40a/pegasus-spark-analyser/src/main/scala/com/xiaomi/infra/pegasus/spark/analyser/examples/parquet).
- Compare your data which stored in two different pegasus clusters, see detail: [duplication verify](https://github.com/pegasus-kv/pegasus-spark/tree/8c585a47e4b618924275c5c1404bdaef9c26f40a/pegasus-spark-analyser/src/main/scala/com/xiaomi/infra/pegasus/spark/analyser/recipes/verify).
- pegasus-bulkloader: pegasus-bulkloader can convert source data to pegasus data and load into pegasus cluster with the [pegasus server 2.1](https://github.com/apache/incubator-pegasus/tree/v2.1) support, see example: [load csv data](https://github.com/pegasus-kv/pegasus-spark/blob/8c585a47e4b618924275c5c1404bdaef9c26f40a/pegasus-spark-bulkloader/src/main/scala/com/xiaomi/infra/pegasus/spark/bulkloader/examples/CSVBulkLoader.scala)
- pegasus-bulkloader: pegasus-bulkloader can transform a data set to pegasus files and load into pegasus cluster with the feature called "BulkLoad", see example: [load csv data](https://github.com/pegasus-kv/pegasus-spark/blob/8c585a47e4b618924275c5c1404bdaef9c26f40a/pegasus-spark-bulkloader/src/main/scala/com/xiaomi/infra/pegasus/spark/bulkloader/examples/CSVBulkLoader.scala). Since we are still actively improving this feature, we recommend that the Pegasus server version should be higher than v2.2.x.
32 changes: 0 additions & 32 deletions pegasus-spark-analyser/pom.xml

This file was deleted.

This file was deleted.

This file was deleted.

Loading