Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

support except table #74

Open
wants to merge 2 commits into
base: dev
Choose a base branch
from
Open

Conversation

lsy3993
Copy link
Collaborator

@lsy3993 lsy3993 commented May 7, 2024

post is

curl -X POST -H "Content-Type: application/json" -d '{
        "name": "ccr_test1",
        "src": {
        "host": "10.16.10.6",
        "port": "19035",
        "thrift_port": "19025",
        "user": "root",
        "password": "",
        "database": "test",
        "table": ""
        },
	"except_table":"example_range_tbl",
        "dest": {
        "host": "10.16.10.6",
        "port": "19035",
        "thrift_port": "19025",
        "user": "root",
        "password": "",
        "database": "test_bak",
        "table": ""
        }
    }' http://127.0.0.1:9190/create_ccr

You could specify except_table, that tables will not sync。
BTW, except_table could be "table1, table2,table3", separate by comma

Copy link
Contributor

@w41ter w41ter left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add a regression test for the except tables, and cover the below situations on the except tables:

  1. normal insert
  2. add/drop partition
  3. drop table
  4. alter/truncate table
  5. light schema change on the table

Src base.Spec `json:"src,required"`
Dest base.Spec `json:"dest,required"`
SkipError bool `json:"skip_error"`
ExceptTable string `json:"except_table"`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Define it to an array might be better.

Suggested change
ExceptTable string `json:"except_table"`
ExceptTables []string `json:"except_tables"`

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Then run strings.TrimSpace(exceptTable) before converting jobContext to Job.

dest base.Spec
db storage.DB
skipError bool
exceptTable string
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
exceptTable string
exceptTables []string

IDest base.Specer `json:"-"`
destMeta Metaer `json:"-"`
SkipError bool `json:"skip_error"`
ExceptTable string `json:"except_table"`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
ExceptTable string `json:"except_table"`
ExceptTables []string `json:"except_tables"`

}
}

func IsExceptTable(exceptTables string, targetTable string) bool {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Define it as a method of Job.

It likes:

func (j *Job) isExceptTable(targetTable string) bool {

backupTableList = append(backupTableList, table.Name)
}
log.Infof("fullsync table is : %s", backupTableList)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There may be thousands of tables here, so it is not suitable for printing logs.

@@ -918,6 +959,14 @@ func (j *Job) handleDropPartition(binlog *festruct.TBinlog) error {
if j.SyncType == TableSync {
destTableName = j.Dest.Table
} else if j.SyncType == DBSync {
// use sourceTableName to judge if table is except
var sourceTableName string
sourceTableName, err = j.srcMeta.GetTableNameById(dropPartition.TableId)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

GetTableNameById is a high-cost operation, it needs to connect doris FE and execute 'SHOW TABLE'.

One possible solution is to query the table ID of the except tables before adding the job to JobManager, see Job::FirstRun for details.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants