ci: force-push test-kit-latest regardless of branch/test result by donhardman · Pull Request #4609 · manticoresoftware/manticoresearch

donhardman · 2026-06-03T18:56:14Z

Temporary override of dist/test_kit_docker_push.sh so a PR branch refreshes the test-kit-latest image while main's CLT is red. Removes the branch==main gate and the "skip latest on test failure" gate. Revert before/after merging.

Temporary override of dist/test_kit_docker_push.sh so a PR branch refreshes the test-kit-latest image while main's CLT is red. Removes the branch==main gate and the "skip latest on test failure" gate. Revert before/after merging. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

github-actions · 2026-06-03T19:21:00Z

clt

❌ CLT tests in test/clt-tests/sharding/functional/functional-sharding-and-cluster-negative test/clt-tests/sharding/functional/functional-sharding-positive test/clt-tests/sharding/functional/test-dist-table-shards-5000 test/clt-tests/sharding/replication/create-table-1-node-10-shards test/clt-tests/sharding/replication/create-table-2-nodes-4-shards test/clt-tests/sharding/replication/create-table-3-nodes-6-shards test/clt-tests/sharding/replication/create-table-5-nodes-60-shards test/clt-tests/sharding/replication/test-data-manipulation test/clt-tests/sharding/replication/test-distributed-inserts-with-replication test/clt-tests/sharding/replication/test-multi-node-sharding-and-replication test/clt-tests/sharding/syntax/sharding-syntax-negative test/clt-tests/sharding/syntax/sharding-syntax-positive test/clt-tests/sharding/rollback/automatic-rollback-on-creation-failure
✅ OK: 12
❌ Failed: 1
⏳ Duration: 290s
👉 Check Action Results for commit f500ed5

Failed tests:

🔧 Edit failed tests in UI:

Edit test/clt-tests/sharding/rollback/automatic-rollback-on-creation-failure.rec

test/clt-tests/sharding/rollback/automatic-rollback-on-creation-failure.rec

––– input –––
export INSTANCE=1
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
export INSTANCE=2
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
export INSTANCE=3
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "CREATE TABLE c:test_products (id bigint, name string, price float) shards='2' rf='2'" & sleep 3; export INSTANCE=2; stdbuf -oL searchd --stopwait -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null; sleep 1; echo "Node 2 killed"
––– output –––
- Node 2 killed
+ ERROR 1064 (42000) at line 1: Cluster 'c' does not exist
+ Node 2 killed
––– input –––
sleep 15; echo "Wait completed"
––– output –––
OK
––– input –––
export INSTANCE=2
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
sleep 10; mysql -h0 -P2306 -e "SHOW STATUS LIKE 'cluster_c_status'\G"
––– output –––
- *************************** 1. row ***************************
- Counter: cluster_c_status
-   Value: primary
––– input –––
mysql -h0 -P1306 -e "SELECT COUNT(*) as shard_count FROM system.sharding_table WHERE table = 'test_products'"
––– output –––
- +-------------+
+ ERROR 1064 (42000) at line 1: unknown local table(s) 'system.sharding_table' in search request
- | shard_count |
- +-------------+
- |           2 |
- +-------------+

github-actions · 2026-06-03T19:28:52Z

clt

❌ CLT tests in test/clt-tests/sharding/rollback/data-consistency-multiple-failures test/clt-tests/sharding/rollback/master-node-failure test/clt-tests/sharding/rollback/master-takeover-while-rollback test/clt-tests/sharding/rollback/mixed-rf-tables-node-failure test/clt-tests/sharding/rollback/new-node-addition-rf-replication test/clt-tests/sharding/rollback/node-failure-mid-creation-recovery test/clt-tests/sharding/rollback/node-killed-rf-maintained test/clt-tests/sharding/rollback/node-rejoin-stale-cluster-cleanup test/clt-tests/sharding/rollback/node-removal-rf-maintained test/clt-tests/sharding/rollback/rebalance-master-killed-mid-operation test/clt-tests/sharding/rollback/rf1-no-rebalance-on-failure test/clt-tests/sharding/rollback/sequential-node-failures-high-rf test/clt-tests/sharding/regression/test-columnar-engine-shard
✅ OK: 1
❌ Failed: 12
⏳ Duration: 885s
👉 Check Action Results for commit f500ed5

Failed tests:

🔧 Edit failed tests in UI:

test/clt-tests/sharding/rollback/sequential-node-failures-high-rf.rec

––– input –––
export INSTANCE=1
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
export INSTANCE=2
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
export INSTANCE=3
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
export INSTANCE=4
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
export CLUSTER_NAME=c TABLE_NAME=t
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "create cluster ${CLUSTER_NAME}"
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "show status like 'cluster_${CLUSTER_NAME}_status'\G"
––– output –––
OK
––– input –––
for n in `seq 2 $INSTANCE`; do mysql -h0 -P${n}306 -e "join cluster ${CLUSTER_NAME} at '127.0.0.1:1312'"; done;
––– output –––
OK
––– input –––
mysql -h0 -P${INSTANCE}306 -e "show status like 'cluster_${CLUSTER_NAME}_status'\G"
––– output –––
OK
––– input –––
show_shards() {
	shards=$(mysql -h0 -P$1 -e "show create table $2 option force=1\G" | grep "Create Table" | cut -d: -f2-)

	(
		# Handle local shards
		echo $shards | grep -oP "local='[^']*'" | sed "s/local='//g; s/'//g" | tr ',' '\n' | while read -r tbl; do
			[ -n "$tbl" ] && echo "$tbl: local"
		done

		# Handle agent shards
		echo $shards | grep -oP "agent='[^']*'" | sed "s/agent='//g; s/'//g" | tr '|' '\n' | awk -F: '{agents[$3] = agents[$3] ? agents[$3] " " $1 ":" $2 : $1 ":" $2} END {for (t in agents) print t ": " agents[t]}'
	) | sort
}
––– output –––
OK
––– input –––
check_cluster_status() {
	mysql -h0 -P$1 -e "SHOW STATUS LIKE 'cluster_${2}_status'\\G" | grep -q "Value: primary" && echo "OK" || echo "FAIL"
}
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "CREATE TABLE ${CLUSTER_NAME}:${TABLE_NAME} (id bigint, value string) shards='3' rf='3'"; echo $?
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "INSERT INTO ${TABLE_NAME} (id, value) VALUES (1, 'Widget A'), (2, 'Widget B'), (3, 'Widget C')"; echo $?
––– output –––
OK
––– input –––
show_shards 1306 "$TABLE_NAME"
––– output –––
- system.t_s0: 127.0.0.1:1312 127.0.0.1:2312 127.0.0.1:3312
- system.t_s1: 127.0.0.1:2312 127.0.0.1:3312 127.0.0.1:4312
- system.t_s2: 127.0.0.1:1312 127.0.0.1:2312 127.0.0.1:4312
––– input –––
export INSTANCE=1; stdbuf -oL searchd --stopwait -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null; echo "Node 1 killed"
––– output –––
OK
––– input –––
timeout 30 bash -c 'while lsof -i :${INSTANCE}306 &>/dev/null; do sleep 1; done'
––– output –––
OK
––– input –––
timeout 60 grep -qm1 'Rebalancing completed' <(tail -n 100 -f /var/log/manticore-2/searchd.log); echo $?
––– output –––
OK
––– input –––
for i in 2 3 4; do check_cluster_status ${i}306 "$CLUSTER_NAME"; done | uniq
––– output –––
OK
––– input –––
show_shards 2306 "$TABLE_NAME"
––– output –––
- system.t_s0: 127.0.0.1:2312 127.0.0.1:3312 127.0.0.1:4312
- system.t_s1: 127.0.0.1:2312 127.0.0.1:3312 127.0.0.1:4312
- system.t_s2: 127.0.0.1:2312 127.0.0.1:3312 127.0.0.1:4312
––– input –––
mysql -h0 -P2306 -sN -e "SELECT COUNT(*) FROM ${TABLE_NAME}"
––– output –––
OK
––– input –––
export INSTANCE=2; stdbuf -oL searchd --stopwait -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null; echo "Node 2 killed"
––– output –––
OK
––– input –––
timeout 30 bash -c 'while lsof -i :${INSTANCE}306 &>/dev/null; do sleep 1; done'
––– output –––
OK
––– input –––
timeout 20 grep -qm1 'Skipping rebalance for table' <(tail -n 100 -f /var/log/manticore-{3,4}/searchd.log); echo $?
––– output –––
OK
––– input –––
for i in 3 4; do check_cluster_status ${i}306 "$CLUSTER_NAME"; done | uniq
––– output –––
OK
––– input –––
mysql -h0 -P3306 -sN -e "SELECT COUNT(*) FROM ${TABLE_NAME}"
––– output –––
OK
––– input –––
truncate -s 0 /var/log/manticore-3/searchd.log
––– output –––
OK
––– input –––
export INSTANCE=1
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
export INSTANCE=2
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
timeout 60 grep -qm1 'Rebalancing completed' <(tail -n 100 -F /var/log/manticore-3/searchd.log); echo $?
––– output –––
- 0
+ tail: /var/log/manticore-3/searchd.log: file truncated
+ 124
––– input –––
for i in 1 2 3 4; do check_cluster_status ${i}306 "$CLUSTER_NAME"; done | uniq
––– output –––
OK
––– input –––
show_shards 1306 "$TABLE_NAME"
––– output –––
- system.t_s0: 127.0.0.1:1312 127.0.0.1:2312 127.0.0.1:3312
- system.t_s1: 127.0.0.1:2312 127.0.0.1:3312 127.0.0.1:4312
- system.t_s2: 127.0.0.1:1312 127.0.0.1:2312 127.0.0.1:4312
––– input –––
mysql -h0 -P1306 -sN -e "SELECT COUNT(*) FROM ${TABLE_NAME}"
––– output –––
OK

test/clt-tests/sharding/rollback/master-takeover-while-rollback.rec

––– input –––
export INSTANCE=1
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
export INSTANCE=2
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
export INSTANCE=3
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
export CLUSTER_NAME=c TABLE_NAME=t
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "create cluster ${CLUSTER_NAME}"
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "show status like 'cluster_${CLUSTER_NAME}_status'\G"
––– output –––
OK
––– input –––
for n in `seq 2 $INSTANCE`; do mysql -h0 -P${n}306 -e "join cluster ${CLUSTER_NAME} at '127.0.0.1:1312'"; done;
––– output –––
OK
––– input –––
mysql -h0 -P${INSTANCE}306 -e "show status like 'cluster_${CLUSTER_NAME}_status'\G"
––– output –––
OK
––– input –––
show_shards() {
	shards=$(mysql -h0 -P$1 -e "show create table $2 option force=1\G" | grep "Create Table" | cut -d: -f2-)

	(
		# Handle local shards
		echo $shards | grep -oP "local='[^']*'" | sed "s/local='//g; s/'//g" | tr ',' '\n' | while read -r tbl; do
			[ -n "$tbl" ] && echo "$tbl: local"
		done

		# Handle agent shards
		echo $shards | grep -oP "agent='[^']*'" | sed "s/agent='//g; s/'//g" | tr '|' '\n' | awk -F: '{agents[$3] = agents[$3] ? agents[$3] " " $1 ":" $2 : $1 ":" $2} END {for (t in agents) print t ": " agents[t]}'
	) | sort
}
––– output –––
OK
––– input –––
check_cluster_status() {
	mysql -h0 -P$1 -e "SHOW STATUS LIKE 'cluster_${2}_status'\\G" | grep -q "Value: primary" && echo "OK" || echo "FAIL"
}
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "CREATE TABLE ${CLUSTER_NAME}:${TABLE_NAME} (id bigint, value string) shards='2' rf='2'"; echo $?
––– output –––
OK
––– input –––
show_shards 1306 "$TABLE_NAME"
––– output –––
- system.t_s0: 127.0.0.1:1312 127.0.0.1:3312
- system.t_s1: 127.0.0.1:1312 127.0.0.1:2312
––– input –––
export INSTANCE=1; stdbuf -oL searchd --stopwait -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null; echo "Node 1 killed"
––– output –––
OK
––– input –––
timeout 30 bash -c 'while lsof -i :${INSTANCE}306 &>/dev/null; do sleep 1; done'
––– output –––
OK
––– input –––
timeout 10 grep -qm1 'becoming master' <(tail -n 1000 -f /var/log/manticore-{2,3}/searchd.log); echo $?
––– output –––
OK
––– input –––
for i in 2 3; do check_cluster_status ${i}306 "$CLUSTER_NAME"; done | uniq
––– output –––
OK
––– input –––
show_shards 2306 "$TABLE_NAME"
––– output –––
- system.t_s0: 127.0.0.1:2312 127.0.0.1:3312
- system.t_s1: 127.0.0.1:2312 127.0.0.1:3312
––– input –––
export INSTANCE=1
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
sleep 2
––– output –––
OK
––– input –––
for i in 1 2 3; do check_cluster_status ${i}306 "$CLUSTER_NAME"; done | uniq
––– output –––
OK
––– input –––
show_shards 2306 "$TABLE_NAME"
––– output –––
- system.t_s0: 127.0.0.1:2312 127.0.0.1:3312
- system.t_s1: 127.0.0.1:2312 127.0.0.1:3312

test/clt-tests/sharding/rollback/node-failure-mid-creation-recovery.rec

––– input –––
export INSTANCE=1
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
export INSTANCE=2
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
export INSTANCE=3
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "CREATE TABLE c:orders (id bigint, total float) shards='2' rf='2'"; echo $?
––– output –––
- 0
+ ERROR 1064 (42000) at line 1: Cluster 'c' does not exist
+ 1
––– input –––
mysql -h0 -P1306 -e "INSERT INTO orders (id, total) VALUES (1, 100), (2, 200)"; echo $?
––– output –––
OK
––– input –––
export INSTANCE=1; stdbuf -oL searchd --stopwait -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null; echo "Node 1 killed"
––– output –––
OK
––– input –––
for port in 2306 3306; do timeout 30 mysql -h0 -P$port -e "SHOW STATUS LIKE 'cluster_c_status'\\G" > /tmp/status_$port.log && grep -q "Value: primary" /tmp/status_$port.log && echo "Port $port: synced"; done
––– output –––
- Port 2306: synced
- Port 3306: synced
––– input –––
export INSTANCE=1
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
sleep 15; for port in 1306 2306 3306; do timeout 30 mysql -h0 -P$port -e "SHOW STATUS LIKE 'cluster_c_status'\\G" > /tmp/status_$port.log && grep -q "Value: primary" /tmp/status_$port.log && echo "Port $port: synced"; done
––– output –––
- Port 1306: synced
- Port 2306: synced
- Port 3306: synced

test/clt-tests/sharding/rollback/data-consistency-multiple-failures.rec

––– input –––
export INSTANCE=1
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
export INSTANCE=2
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
export INSTANCE=3
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
export CLUSTER_NAME=c TABLE_NAME=t
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "create cluster ${CLUSTER_NAME}"
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "show status like 'cluster_${CLUSTER_NAME}_status'\G"
––– output –––
OK
––– input –––
for n in `seq 2 $INSTANCE`; do mysql -h0 -P${n}306 -e "join cluster ${CLUSTER_NAME} at '127.0.0.1:1312'"; done;
––– output –––
OK
––– input –––
mysql -h0 -P${INSTANCE}306 -e "show status like 'cluster_${CLUSTER_NAME}_status'\G"
––– output –––
OK
––– input –––
show_shards() {
	shards=$(mysql -h0 -P$1 -e "show create table $2 option force=1\G" | grep "Create Table" | cut -d: -f2-)

	(
		# Handle local shards
		echo $shards | grep -oP "local='[^']*'" | sed "s/local='//g; s/'//g" | tr ',' '\n' | while read -r tbl; do
			[ -n "$tbl" ] && echo "$tbl: local"
		done

		# Handle agent shards
		echo $shards | grep -oP "agent='[^']*'" | sed "s/agent='//g; s/'//g" | tr '|' '\n' | awk -F: '{agents[$3] = agents[$3] ? agents[$3] " " $1 ":" $2 : $1 ":" $2} END {for (t in agents) print t ": " agents[t]}'
	) | sort
}
––– output –––
OK
––– input –––
check_cluster_status() {
	mysql -h0 -P$1 -e "SHOW STATUS LIKE 'cluster_${2}_status'\\G" | grep -q "Value: primary" && echo "OK" || echo "FAIL"
}
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "CREATE TABLE ${CLUSTER_NAME}:${TABLE_NAME} (id bigint, account string, amount float, ts int) shards='2' rf='2'"; echo $?
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "INSERT INTO ${TABLE_NAME} (id, account, amount, ts) VALUES (1, 'ACC001', 100.50, 1000), (2, 'ACC002', 200.75, 2000), (3, 'ACC003', 150.25, 3000), (4, 'ACC001', 300.00, 4000), (5, 'ACC002', 250.50, 5000)"; echo $?
––– output –––
OK
––– input –––
mysql -h0 -P1306 -sN -e "SELECT COUNT(*), SUM(amount) FROM ${TABLE_NAME}"
––– output –––
- 5	1002.000000
+ +------+-------------+
+ |    5 | 1002.000000 |
+ +------+-------------+
––– input –––
export INSTANCE=1; stdbuf -oL searchd --stopwait -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null; echo "Node 1 killed"
––– output –––
OK
––– input –––
timeout 30 bash -c 'while lsof -i :${INSTANCE}306 &>/dev/null; do sleep 1; done'
––– output –––
OK
––– input –––
timeout 10 grep -qm1 'becoming master' <(tail -n 1000 -f /var/log/manticore-{2,3}/searchd.log); echo $?
––– output –––
OK
––– input –––
for i in 2 3; do check_cluster_status ${i}306 "$CLUSTER_NAME"; done | uniq
––– output –––
OK
––– input –––
mysql -h0 -P2306 -e "INSERT INTO ${TABLE_NAME} (id, account, amount, ts) VALUES (6, 'ACC003', 175.00, 6000), (7, 'ACC001', 225.75, 7000)"; echo $?
––– output –––
OK
––– input –––
export INSTANCE=1
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
sleep 2
––– output –––
OK
––– input –––
for i in 1 2 3; do check_cluster_status ${i}306 "$CLUSTER_NAME"; done | uniq
––– output –––
OK
––– input –––
export INSTANCE=2; stdbuf -oL searchd --stopwait -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null; echo "Node 2 killed"
––– output –––
OK
––– input –––
timeout 30 bash -c 'while lsof -i :${INSTANCE}306 &>/dev/null; do sleep 1; done'
––– output –––
OK
––– input –––
timeout 10 grep -qm1 'becoming master' <(tail -n 1000 -f /var/log/manticore-{1,3}/searchd.log); echo $?
––– output –––
OK
––– input –––
for i in 1 3; do check_cluster_status ${i}306 "$CLUSTER_NAME"; done | uniq
––– output –––
OK
––– input –––
export INSTANCE=2
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
sleep 2
––– output –––
OK
––– input –––
for i in 1 2 3; do check_cluster_status ${i}306 "$CLUSTER_NAME"; done | uniq
––– output –––
OK
––– input –––
mysql -h0 -P1306 -sN -e "SELECT COUNT(*), SUM(amount) FROM ${TABLE_NAME}"
––– output –––
- 7	1402.75
+ ERROR 1064 (42000) at line 1: unknown local table(s) 't' in search request
––– input –––
for port in 1306 2306 3306; do echo "Node $port:"; mysql -h0 -P$port -sN -e "SELECT COUNT(*), SUM(amount) FROM ${TABLE_NAME}"; done
––– output –––
Node 1306:
- 7	1402.75
+ ERROR 1064 (42000) at line 1: unknown local table(s) 't' in search request
Node 2306:
- 7	1402.75
+ +------+------------+
- Node 3306:
+ |    3 | 476.250000 |
- 7	1402.75
+ +------+------------+
+ Node 3306:
+ +------+-------------+
+ |    7 | 1402.750000 |
+ +------+-------------+
––– input –––
mysql -h0 -P1306 -e "DROP TABLE IF EXISTS ${TABLE_NAME}"; echo $?
––– output –––
OK

test/clt-tests/sharding/rollback/new-node-addition-rf-replication.rec

––– input –––
export INSTANCE=1
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
export INSTANCE=2
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "CREATE TABLE c:existing_table (id bigint, data string) shards='2' rf='2'"; echo $?
––– output –––
- 0
+ ERROR 1064 (42000) at line 1: Cluster 'c' does not exist
+ 1
––– input –––
for i in 1 2; do mysql -h0 -P${i}306 -sN -e "SELECT COUNT(*) FROM system.sharding_table WHERE table = 'existing_table'"; done
––– output –––
- 2
+ ERROR 1064 (42000) at line 1: unknown local table(s) 'system.sharding_table' in search request
- 2
+ ERROR 1064 (42000) at line 1: unknown local table(s) 'system.sharding_table' in search request
––– input –––
export INSTANCE=3
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
mysql -h0 -P3306 -e "JOIN CLUSTER c at '127.0.0.1:1312'"; echo $?
––– output –––
- 0
+ ERROR 1064 (42000) at line 1: cluster 'c', no nodes available(127.0.0.1:1312), error: '127.0.0.1:1312': retries limit exceeded
+ 1
––– input –––
for port in 1306 2306 3306; do timeout 30 mysql -h0 -P$port -e "SHOW STATUS LIKE 'cluster_c_status'\G" > /tmp/status_$port.log && grep -q "Value: primary" /tmp/status_$port.log && echo "Port $port: Node synced"; done
––– output –––
- Port 1306: Node synced
- Port 2306: Node synced
- Port 3306: Node synced
––– input –––
mysql -h0 -P1306 -sN -e "SELECT COUNT(*) FROM system.sharding_state"
––– output –––
- 3
+ ERROR 1064 (42000) at line 1: unknown local table(s) 'system.sharding_state' in search request
––– input –––
sleep 15; echo "Cluster stabilized"
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "DROP TABLE IF EXISTS existing_table"
––– output –––
OK

test/clt-tests/sharding/rollback/node-rejoin-stale-cluster-cleanup.rec

––– input –––
export INSTANCE=1
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
export INSTANCE=2
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
export INSTANCE=3
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
export CLUSTER_NAME=c TABLE_NAME=t
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "create cluster ${CLUSTER_NAME}"
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "show status like 'cluster_${CLUSTER_NAME}_status'\G"
––– output –––
OK
––– input –––
for n in `seq 2 $INSTANCE`; do mysql -h0 -P${n}306 -e "join cluster ${CLUSTER_NAME} at '127.0.0.1:1312'"; done;
––– output –––
OK
––– input –––
mysql -h0 -P${INSTANCE}306 -e "show status like 'cluster_${CLUSTER_NAME}_status'\G"
––– output –––
OK
––– input –––
show_shards() {
	shards=$(mysql -h0 -P$1 -e "show create table $2 option force=1\G" | grep "Create Table" | cut -d: -f2-)

	(
		# Handle local shards
		echo $shards | grep -oP "local='[^']*'" | sed "s/local='//g; s/'//g" | tr ',' '\n' | while read -r tbl; do
			[ -n "$tbl" ] && echo "$tbl: local"
		done

		# Handle agent shards
		echo $shards | grep -oP "agent='[^']*'" | sed "s/agent='//g; s/'//g" | tr '|' '\n' | awk -F: '{agents[$3] = agents[$3] ? agents[$3] " " $1 ":" $2 : $1 ":" $2} END {for (t in agents) print t ": " agents[t]}'
	) | sort
}
––– output –––
OK
––– input –––
check_cluster_status() {
	mysql -h0 -P$1 -e "SHOW STATUS LIKE 'cluster_${2}_status'\\G" | grep -q "Value: primary" && echo "OK" || echo "FAIL"
}
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "CREATE TABLE ${CLUSTER_NAME}:${TABLE_NAME} (id bigint, value string) shards='2' rf='2'"; echo $?
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "INSERT INTO ${TABLE_NAME} (id, value) VALUES (1, 'alpha'), (2, 'beta')"; echo $?
––– output –––
OK
––– input –––
show_shards 1306 "$TABLE_NAME"
––– output –––
- system.t_s0: 127.0.0.1:1312 127.0.0.1:2312
- system.t_s1: 127.0.0.1:2312 127.0.0.1:3312
––– input –––
export INSTANCE=1; stdbuf -oL searchd --stopwait -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null; echo "Node 1 killed"
––– output –––
OK
––– input –––
timeout 30 bash -c 'while lsof -i :${INSTANCE}306 &>/dev/null; do sleep 1; done'
––– output –––
OK
––– input –––
timeout 60 grep -qm1 'Rebalancing completed' <(tail -n 100 -f /var/log/manticore-2/searchd.log); echo $?
––– output –––
- 0
+ 124
––– input –––
for i in 2 3; do check_cluster_status ${i}306 "$CLUSTER_NAME"; done | uniq
––– output –––
OK
––– input –––
show_shards 2306 "$TABLE_NAME"
––– output –––
- system.t_s0: 127.0.0.1:2312 127.0.0.1:3312
- system.t_s1: 127.0.0.1:2312 127.0.0.1:3312
––– input –––
mysql -h0 -P2306 -sN -e "SELECT COUNT(*) FROM ${TABLE_NAME}"
––– output –––
OK
––– input –––
truncate -s 0 /var/log/manticore-2/searchd.log
––– output –––
OK
––– input –––
export INSTANCE=1
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
timeout 90 grep -qm1 'Rebalancing completed' <(tail -n 100 -F /var/log/manticore-2/searchd.log); echo $?
––– output –––
- 0
+ 124
––– input –––
for i in 1 2 3; do check_cluster_status ${i}306 "$CLUSTER_NAME"; done | uniq
––– output –––
OK
––– input –––
show_shards 1306 "$TABLE_NAME"
––– output –––
- system.t_s0: 127.0.0.1:1312 127.0.0.1:2312
+ ERROR 1064 (42000) at line 1: You have an error in your query. Please, double-check it.
- system.t_s1: 127.0.0.1:2312 127.0.0.1:3312
––– input –––
mysql -h0 -P1306 -sN -e "SELECT COUNT(*) FROM ${TABLE_NAME}"
––– output –––
- +------+
+ ERROR 1064 (42000) at line 1: unknown local table(s) 't' in search request
- |    2 |
- +------+

test/clt-tests/sharding/rollback/rf1-no-rebalance-on-failure.rec

––– input –––
export INSTANCE=1
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
export INSTANCE=2
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
export CLUSTER_NAME=c TABLE_NAME=t
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "create cluster ${CLUSTER_NAME}"
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "show status like 'cluster_${CLUSTER_NAME}_status'\G"
––– output –––
OK
––– input –––
for n in `seq 2 $INSTANCE`; do mysql -h0 -P${n}306 -e "join cluster ${CLUSTER_NAME} at '127.0.0.1:1312'"; done;
––– output –––
OK
––– input –––
mysql -h0 -P${INSTANCE}306 -e "show status like 'cluster_${CLUSTER_NAME}_status'\G"
––– output –––
OK
––– input –––
show_shards() {
	shards=$(mysql -h0 -P$1 -e "show create table $2 option force=1\G" | grep "Create Table" | cut -d: -f2-)

	(
		# Handle local shards
		echo $shards | grep -oP "local='[^']*'" | sed "s/local='//g; s/'//g" | tr ',' '\n' | while read -r tbl; do
			[ -n "$tbl" ] && echo "$tbl: local"
		done

		# Handle agent shards
		echo $shards | grep -oP "agent='[^']*'" | sed "s/agent='//g; s/'//g" | tr '|' '\n' | awk -F: '{agents[$3] = agents[$3] ? agents[$3] " " $1 ":" $2 : $1 ":" $2} END {for (t in agents) print t ": " agents[t]}'
	) | sort
}
––– output –––
OK
––– input –––
check_cluster_status() {
	mysql -h0 -P$1 -e "SHOW STATUS LIKE 'cluster_${2}_status'\\G" | grep -q "Value: primary" && echo "OK" || echo "FAIL"
}
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "CREATE TABLE ${CLUSTER_NAME}:${TABLE_NAME} (id bigint, value string) shards='2' rf='1'"; echo $?
––– output –––
OK
––– input –––
show_shards 1306 "$TABLE_NAME"
––– output –––
- system.t_s0: local
- system.t_s1: 127.0.0.1:2312
––– input –––
mysql -h0 -P1306 -e "INSERT INTO ${TABLE_NAME} (id, value) VALUES (1, 'Widget A'), (2, 'Widget B'), (3, 'Widget C'), (4, 'Widget D')"; echo $?
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "SELECT COUNT(*) as total FROM ${TABLE_NAME}\\G" | grep total
––– output –––
OK
––– input –––
export INSTANCE=2; stdbuf -oL searchd --stopwait -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null; echo "Node 2 killed"
––– output –––
OK
––– input –––
timeout 30 bash -c 'while lsof -i :${INSTANCE}306 &>/dev/null; do sleep 1; done'
––– output –––
OK
––– input –––
timeout 10 grep -qm1 'Skipping rebalance for table' <(tail -n 1000 -f /var/log/manticore-1/searchd.log); echo $?
––– output –––
OK
––– input –––
show_shards 1306 "$TABLE_NAME"
––– output –––
- system.t_s0: local
- system.t_s1: 127.0.0.1:2312
––– input –––
mysql -h0 -P1306 -e "SELECT COUNT(*) as cnt FROM ${TABLE_NAME}\\G" | grep -E 'cnt:' | awk '{print $2}'
––– output –––
OK
––– input –––
export INSTANCE=2
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
sleep 15; for port in 1306 2306; do timeout 30 mysql -h0 -P$port -e "SHOW STATUS LIKE 'cluster_c_status'\\G" > /tmp/status_$port.log && grep -q "Value: primary" /tmp/status_$port.log && echo "Port $port: synced"; done
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "SELECT COUNT(*) as cnt FROM ${TABLE_NAME}\\G" | grep -E 'cnt:' | awk '{print $2}'
––– output –––
OK

test/clt-tests/sharding/rollback/node-removal-rf-maintained.rec

––– input –––
export INSTANCE=1
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
export INSTANCE=2
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
export INSTANCE=3
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "CREATE TABLE c:items (id bigint, name string) shards='2' rf='2'"; echo $?
––– output –––
- 0
+ ERROR 1064 (42000) at line 1: Cluster 'c' does not exist
+ 1
––– input –––
mysql -h0 -P1306 -e "INSERT INTO items (id, name) VALUES (1, 'Item1'), (2, 'Item2'), (3, 'Item3')"; echo $?
––– output –––
OK
––– input –––
for port in 1306 2306 3306; do timeout 30 mysql -h0 -P$port -e "SHOW STATUS LIKE 'cluster_c_status'\\G" > /tmp/status_$port.log && grep -q "Value: primary" /tmp/status_$port.log && echo "Port $port: synced"; done
––– output –––
- Port 1306: synced
- Port 2306: synced
- Port 3306: synced
––– input –––
mysql -h0 -P1306 -e "show tables from system\\G" | grep -E 'items_s[0-9]' | wc -l
––– output –––
- 2
+ 0

test/clt-tests/sharding/rollback/node-killed-rf-maintained.rec

––– input –––
export INSTANCE=1
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
export INSTANCE=2
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "CREATE TABLE c:data (id bigint, value string) shards='2' rf='2'"; echo $?
––– output –––
- 0
+ ERROR 1064 (42000) at line 1: Cluster 'c' does not exist
+ 1
––– input –––
mysql -h0 -P1306 -e "INSERT INTO data (id, value) VALUES (1, 'test1'), (2, 'test2')"; echo $?
––– output –––
OK
––– input –––
export INSTANCE=2; stdbuf -oL searchd --stopwait -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null; echo "Node 2 killed"
––– output –––
OK
––– input –––
sleep 10; mysql -h0 -P1306 -e "SHOW STATUS LIKE 'cluster_c_status'\G"
––– output –––
- *************************** 1. row ***************************
- Counter: cluster_c_status
-   Value: primary
––– input –––
mysql -h0 -P1306 -e "INSERT INTO data (id, value) VALUES (3, 'test3')"; echo $?
––– output –––
OK
––– input –––
export INSTANCE=2
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
sleep 20
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "SHOW STATUS LIKE 'cluster_c_status'\G"
––– output –––
- *************************** 1. row ***************************
- Counter: cluster_c_status
-   Value: primary
––– input –––
mysql -h0 -P2306 -e "SHOW STATUS LIKE 'cluster_c_status'\G"
––– output –––
- *************************** 1. row ***************************
- Counter: cluster_c_status
-   Value: primary
––– input –––
mysql -h0 -P1306 -sN -e "SELECT COUNT(*) FROM data"
––– output –––
- 3
+ +------+
+ |    3 |
+ +------+
––– input –––
mysql -h0 -P2306 -sN -e "SELECT COUNT(*) FROM data"
––– output –––
- 3
+ ERROR 1064 (42000) at line 1: unknown local table(s) 'data' in search request
––– input –––
mysql -h0 -P1306 -e "DROP TABLE IF EXISTS data"; echo $?
––– output –––
OK

test/clt-tests/sharding/rollback/master-node-failure.rec

––– input –––
export INSTANCE=1
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
export INSTANCE=2
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
export INSTANCE=3
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "CREATE TABLE c:data (id bigint, value string) shards='2' rf='2'"; echo $?
––– output –––
- 0
+ ERROR 1064 (42000) at line 1: Cluster 'c' does not exist
+ 1
––– input –––
sleep 5; mysql -h0 -P1306 -e "INSERT INTO data (id, value) VALUES (1, 'test_value')"; echo $?
––– output –––
OK
––– input –––
export INSTANCE=1; stdbuf -oL searchd --stopwait -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null; echo "Node 1 killed"
––– output –––
OK
––– input –––
timeout 30 bash -c 'while lsof -i :${INSTANCE}306 &>/dev/null; do sleep 1; done'
––– output –––
OK
––– input –––
for port in 2306 3306; do timeout 30 mysql -h0 -P$port -e "SHOW STATUS LIKE 'cluster_c_status'\\G" > /tmp/status_$port.log && grep -q "Value: primary" /tmp/status_$port.log && echo "Port $port: synced"; done
––– output –––
- Port 2306: synced
- Port 3306: synced
––– input –––
export INSTANCE=1
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
sleep 15; mysql -h0 -P1306 -e "SHOW STATUS LIKE 'cluster_c_status'\\G"
––– output –––
- *************************** 1. row ***************************
- Counter: cluster_c_status
-   Value: primary
––– input –––
for port in 1306 2306 3306; do timeout 30 mysql -h0 -P$port -e "SHOW STATUS LIKE 'cluster_c_status'\\G" > /tmp/status_$port.log && grep -q "Value: primary" /tmp/status_$port.log && echo "Port $port: synced"; done
––– output –––
- Port 1306: synced
- Port 2306: synced
- Port 3306: synced

test/clt-tests/sharding/rollback/rebalance-master-killed-mid-operation.rec

––– input –––
export INSTANCE=1
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
export INSTANCE=2
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
export INSTANCE=3
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
export INSTANCE=4
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
export CLUSTER_NAME=c TABLE_NAME=t
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "create cluster ${CLUSTER_NAME}"
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "show status like 'cluster_${CLUSTER_NAME}_status'\G"
––– output –––
OK
––– input –––
for n in `seq 2 $INSTANCE`; do mysql -h0 -P${n}306 -e "join cluster ${CLUSTER_NAME} at '127.0.0.1:1312'"; done;
––– output –––
OK
––– input –––
mysql -h0 -P${INSTANCE}306 -e "show status like 'cluster_${CLUSTER_NAME}_status'\G"
––– output –––
OK
––– input –––
show_shards() {
	shards=$(mysql -h0 -P$1 -e "show create table $2 option force=1\G" | grep "Create Table" | cut -d: -f2-)

	(
		# Handle local shards
		echo $shards | grep -oP "local='[^']*'" | sed "s/local='//g; s/'//g" | tr ',' '\n' | while read -r tbl; do
			[ -n "$tbl" ] && echo "$tbl: local"
		done

		# Handle agent shards
		echo $shards | grep -oP "agent='[^']*'" | sed "s/agent='//g; s/'//g" | tr '|' '\n' | awk -F: '{agents[$3] = agents[$3] ? agents[$3] " " $1 ":" $2 : $1 ":" $2} END {for (t in agents) print t ": " agents[t]}'
	) | sort
}
––– output –––
OK
––– input –––
check_cluster_status() {
	mysql -h0 -P$1 -e "SHOW STATUS LIKE 'cluster_${2}_status'\\G" | grep -q "Value: primary" && echo "OK" || echo "FAIL"
}
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "CREATE TABLE ${CLUSTER_NAME}:${TABLE_NAME} (id bigint, value string) shards='3' rf='2'"; echo $?
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "INSERT INTO ${TABLE_NAME} (id, value) VALUES (1, 'one'), (2, 'two'), (3, 'three')"; echo $?
––– output –––
OK
––– input –––
show_shards 1306 "$TABLE_NAME"
––– output –––
- system.t_s0: 127.0.0.1:1312 127.0.0.1:2312
- system.t_s1: 127.0.0.1:2312 127.0.0.1:3312
- system.t_s2: 127.0.0.1:3312 127.0.0.1:4312
––– input –––
export INSTANCE=4; stdbuf -oL searchd --stopwait -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null; echo "Node 4 killed"
––– output –––
OK
––– input –––
timeout 30 bash -c 'while lsof -i :${INSTANCE}306 &>/dev/null; do sleep 1; done'
––– output –––
OK
––– input –––
timeout 20 grep -qm1 'Rebalancing due to inactive nodes' <(tail -n 100 -f /var/log/manticore-1/searchd.log); echo $?
––– output –––
OK
––– input –––
export INSTANCE=1; stdbuf -oL searchd --stopwait -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null; echo "Master killed"
––– output –––
OK
––– input –––
timeout 30 bash -c 'while lsof -i :${INSTANCE}306 &>/dev/null; do sleep 1; done'
––– output –––
OK
––– input –––
timeout 30 grep -qm1 'becoming master' <(tail -n 100 -f /var/log/manticore-{2,3}/searchd.log); echo $?
––– output –––
OK
––– input –––
sleep 5
––– output –––
OK
––– input –––
for i in 2 3; do check_cluster_status ${i}306 "$CLUSTER_NAME"; done | uniq
––– output –––
OK
––– input –––
mysql -h0 -P2306 -sN -e "SELECT COUNT(*) FROM ${TABLE_NAME}"
––– output –––
OK
––– input –––
truncate -s 0 /var/log/manticore-{2,3}/searchd.log
––– output –––
OK
––– input –––
export INSTANCE=4
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
export INSTANCE=1
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
timeout 90 grep -qm1 'Rebalancing completed' <(tail -n 100 -F /var/log/manticore-{2,3}/searchd.log); echo $?
––– output –––
- 0
+ 124
––– input –––
for i in 1 2 3 4; do check_cluster_status ${i}306 "$CLUSTER_NAME"; done | uniq
––– output –––
OK
––– input –––
show_shards 2306 "$TABLE_NAME"
––– output –––
- system.t_s0: 127.0.0.1:3312 127.0.0.1:2312
- system.t_s1: 127.0.0.1:3312 127.0.0.1:2312
- system.t_s2: 127.0.0.1:3312 127.0.0.1:2312
––– input –––
mysql -h0 -P2306 -sN -e "SELECT COUNT(*) FROM ${TABLE_NAME}"
––– output –––
OK

test/clt-tests/sharding/rollback/mixed-rf-tables-node-failure.rec

––– input –––
export INSTANCE=1
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
export INSTANCE=2
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
export INSTANCE=3
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
export CLUSTER_NAME=c TABLE_NAME=t
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "create cluster ${CLUSTER_NAME}"
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "show status like 'cluster_${CLUSTER_NAME}_status'\G"
––– output –––
OK
––– input –––
for n in `seq 2 $INSTANCE`; do mysql -h0 -P${n}306 -e "join cluster ${CLUSTER_NAME} at '127.0.0.1:1312'"; done;
––– output –––
OK
––– input –––
mysql -h0 -P${INSTANCE}306 -e "show status like 'cluster_${CLUSTER_NAME}_status'\G"
––– output –––
OK
––– input –––
show_shards() {
	shards=$(mysql -h0 -P$1 -e "show create table $2 option force=1\G" | grep "Create Table" | cut -d: -f2-)

	(
		# Handle local shards
		echo $shards | grep -oP "local='[^']*'" | sed "s/local='//g; s/'//g" | tr ',' '\n' | while read -r tbl; do
			[ -n "$tbl" ] && echo "$tbl: local"
		done

		# Handle agent shards
		echo $shards | grep -oP "agent='[^']*'" | sed "s/agent='//g; s/'//g" | tr '|' '\n' | awk -F: '{agents[$3] = agents[$3] ? agents[$3] " " $1 ":" $2 : $1 ":" $2} END {for (t in agents) print t ": " agents[t]}'
	) | sort
}
––– output –––
OK
––– input –––
check_cluster_status() {
	mysql -h0 -P$1 -e "SHOW STATUS LIKE 'cluster_${2}_status'\\G" | grep -q "Value: primary" && echo "OK" || echo "FAIL"
}
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "CREATE TABLE ${CLUSTER_NAME}:norep (id bigint, value string) shards='2' rf='1'"; echo $?
––– output –––
- 0
+ ERROR 1064 (42000) at line 1: Waiting timeout exceeded.
+ 1
––– input –––
mysql -h0 -P1306 -e "CREATE TABLE ${CLUSTER_NAME}:rep (id bigint, value string) shards='2' rf='2'"; echo $?
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "INSERT INTO norep (id, value) VALUES (1, 'a'), (2, 'b')"; echo $?
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "INSERT INTO rep (id, value) VALUES (1, 'x'), (2, 'y')"; echo $?
––– output –––
OK
––– input –––
show_shards 1306 "norep"
––– output –––
- system.norep_s0: 127.0.0.1:1312
- system.norep_s1: 127.0.0.1:2312
––– input –––
show_shards 1306 "rep"
––– output –––
- system.rep_s0: 127.0.0.1:1312 127.0.0.1:2312
- system.rep_s1: 127.0.0.1:2312 127.0.0.1:3312
––– input –––
export INSTANCE=1; stdbuf -oL searchd --stopwait -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null; echo "Node 1 killed"
––– output –––
OK
––– input –––
timeout 30 bash -c 'while lsof -i :${INSTANCE}306 &>/dev/null; do sleep 1; done'
––– output –––
OK
––– input –––
timeout 60 grep -qm1 'Skipping rebalance for table norep' <(tail -n 100 -f /var/log/manticore-2/searchd.log); echo $?
––– output –––
- 0
+ 124
––– input –––
timeout 60 grep -qm1 'Rebalancing completed' <(tail -n 100 -f /var/log/manticore-2/searchd.log); echo $?
––– output –––
- 0
+ 124
––– input –––
for i in 2 3; do check_cluster_status ${i}306 "$CLUSTER_NAME"; done | uniq
––– output –––
OK
––– input –––
show_shards 2306 "norep"
––– output –––
- system.norep_s0: 127.0.0.1:1312
+ ERROR 1064 (42000) at line 1: You have an error in your query. Please, double-check it.
- system.norep_s1: 127.0.0.1:2312
––– input –––
show_shards 2306 "rep"
––– output –––
- system.rep_s0: 127.0.0.1:2312 127.0.0.1:3312
- system.rep_s1: 127.0.0.1:2312 127.0.0.1:3312
––– input –––
mysql -h0 -P2306 -sN -e "SELECT COUNT(*) FROM rep"
––– output –––
OK
––– input –––
truncate -s 0 /var/log/manticore-2/searchd.log
––– output –––
OK
––– input –––
export INSTANCE=1
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
timeout 90 grep -qm1 'Rebalancing completed' <(tail -n 100 -F /var/log/manticore-2/searchd.log); echo $?
––– output –––
- 0
+ 124
––– input –––
for i in 1 2 3; do check_cluster_status ${i}306 "$CLUSTER_NAME"; done | uniq
––– output –––
OK
––– input –––
show_shards 1306 "norep"
––– output –––
- system.norep_s0: 127.0.0.1:1312
- system.norep_s1: 127.0.0.1:2312
––– input –––
show_shards 1306 "rep"
––– output –––
- system.rep_s0: 127.0.0.1:1312 127.0.0.1:2312
+ ERROR 1064 (42000) at line 1: You have an error in your query. Please, double-check it.
- system.rep_s1: 127.0.0.1:2312 127.0.0.1:3312
––– input –––
mysql -h0 -P1306 -sN -e "SELECT COUNT(*) FROM norep"
––– output –––
OK
––– input –––
mysql -h0 -P1306 -sN -e "SELECT COUNT(*) FROM rep"
––– output –––
- +------+
+ ERROR 1064 (42000) at line 1: unknown local table(s) 'rep' in search request
- |    2 |
- +------+

github-actions · 2026-06-03T20:21:53Z

clt

❌ CLT tests in test/clt-tests/sharding/functional/functional-sharding-and-cluster-negative test/clt-tests/sharding/functional/functional-sharding-positive test/clt-tests/sharding/functional/test-dist-table-shards-5000 test/clt-tests/sharding/replication/create-table-1-node-10-shards test/clt-tests/sharding/replication/create-table-2-nodes-4-shards test/clt-tests/sharding/replication/create-table-3-nodes-6-shards test/clt-tests/sharding/replication/create-table-5-nodes-60-shards test/clt-tests/sharding/replication/test-data-manipulation test/clt-tests/sharding/replication/test-distributed-inserts-with-replication test/clt-tests/sharding/replication/test-multi-node-sharding-and-replication test/clt-tests/sharding/syntax/sharding-syntax-negative test/clt-tests/sharding/syntax/sharding-syntax-positive test/clt-tests/sharding/rollback/automatic-rollback-on-creation-failure
✅ OK: 12
❌ Failed: 1
⏳ Duration: 290s
👉 Check Action Results for commit f500ed5

Failed tests:

🔧 Edit failed tests in UI:

Edit test/clt-tests/sharding/rollback/automatic-rollback-on-creation-failure.rec

test/clt-tests/sharding/rollback/automatic-rollback-on-creation-failure.rec

––– input –––
export INSTANCE=1
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
export INSTANCE=2
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
export INSTANCE=3
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "CREATE TABLE c:test_products (id bigint, name string, price float) shards='2' rf='2'" & sleep 3; export INSTANCE=2; stdbuf -oL searchd --stopwait -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null; sleep 1; echo "Node 2 killed"
––– output –––
- Node 2 killed
+ ERROR 1064 (42000) at line 1: Cluster 'c' does not exist
+ Node 2 killed
––– input –––
sleep 15; echo "Wait completed"
––– output –––
OK
––– input –––
export INSTANCE=2
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
sleep 10; mysql -h0 -P2306 -e "SHOW STATUS LIKE 'cluster_c_status'\G"
––– output –––
- *************************** 1. row ***************************
- Counter: cluster_c_status
-   Value: primary
––– input –––
mysql -h0 -P1306 -e "SELECT COUNT(*) as shard_count FROM system.sharding_table WHERE table = 'test_products'"
––– output –––
- +-------------+
+ ERROR 1064 (42000) at line 1: unknown local table(s) 'system.sharding_table' in search request
- | shard_count |
- +-------------+
- |           2 |
- +-------------+

github-actions · 2026-06-03T20:31:22Z

clt

❌ CLT tests in test/clt-tests/sharding/rollback/data-consistency-multiple-failures test/clt-tests/sharding/rollback/master-node-failure test/clt-tests/sharding/rollback/master-takeover-while-rollback test/clt-tests/sharding/rollback/mixed-rf-tables-node-failure test/clt-tests/sharding/rollback/new-node-addition-rf-replication test/clt-tests/sharding/rollback/node-failure-mid-creation-recovery test/clt-tests/sharding/rollback/node-killed-rf-maintained test/clt-tests/sharding/rollback/node-rejoin-stale-cluster-cleanup test/clt-tests/sharding/rollback/node-removal-rf-maintained test/clt-tests/sharding/rollback/rebalance-master-killed-mid-operation test/clt-tests/sharding/rollback/rf1-no-rebalance-on-failure test/clt-tests/sharding/rollback/sequential-node-failures-high-rf test/clt-tests/sharding/regression/test-columnar-engine-shard
✅ OK: 1
❌ Failed: 12
⏳ Duration: 905s
👉 Check Action Results for commit f500ed5

Failed tests:

🔧 Edit failed tests in UI:

test/clt-tests/sharding/rollback/sequential-node-failures-high-rf.rec

––– input –––
export INSTANCE=1
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
export INSTANCE=2
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
export INSTANCE=3
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
export INSTANCE=4
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
export CLUSTER_NAME=c TABLE_NAME=t
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "create cluster ${CLUSTER_NAME}"
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "show status like 'cluster_${CLUSTER_NAME}_status'\G"
––– output –––
OK
––– input –––
for n in `seq 2 $INSTANCE`; do mysql -h0 -P${n}306 -e "join cluster ${CLUSTER_NAME} at '127.0.0.1:1312'"; done;
––– output –––
OK
––– input –––
mysql -h0 -P${INSTANCE}306 -e "show status like 'cluster_${CLUSTER_NAME}_status'\G"
––– output –––
OK
––– input –––
show_shards() {
	shards=$(mysql -h0 -P$1 -e "show create table $2 option force=1\G" | grep "Create Table" | cut -d: -f2-)

	(
		# Handle local shards
		echo $shards | grep -oP "local='[^']*'" | sed "s/local='//g; s/'//g" | tr ',' '\n' | while read -r tbl; do
			[ -n "$tbl" ] && echo "$tbl: local"
		done

		# Handle agent shards
		echo $shards | grep -oP "agent='[^']*'" | sed "s/agent='//g; s/'//g" | tr '|' '\n' | awk -F: '{agents[$3] = agents[$3] ? agents[$3] " " $1 ":" $2 : $1 ":" $2} END {for (t in agents) print t ": " agents[t]}'
	) | sort
}
––– output –––
OK
––– input –––
check_cluster_status() {
	mysql -h0 -P$1 -e "SHOW STATUS LIKE 'cluster_${2}_status'\\G" | grep -q "Value: primary" && echo "OK" || echo "FAIL"
}
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "CREATE TABLE ${CLUSTER_NAME}:${TABLE_NAME} (id bigint, value string) shards='3' rf='3'"; echo $?
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "INSERT INTO ${TABLE_NAME} (id, value) VALUES (1, 'Widget A'), (2, 'Widget B'), (3, 'Widget C')"; echo $?
––– output –––
OK
––– input –––
show_shards 1306 "$TABLE_NAME"
––– output –––
- system.t_s0: 127.0.0.1:1312 127.0.0.1:2312 127.0.0.1:3312
- system.t_s1: 127.0.0.1:2312 127.0.0.1:3312 127.0.0.1:4312
- system.t_s2: 127.0.0.1:1312 127.0.0.1:2312 127.0.0.1:4312
––– input –––
export INSTANCE=1; stdbuf -oL searchd --stopwait -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null; echo "Node 1 killed"
––– output –––
OK
––– input –––
timeout 30 bash -c 'while lsof -i :${INSTANCE}306 &>/dev/null; do sleep 1; done'
––– output –––
OK
––– input –––
timeout 60 grep -qm1 'Rebalancing completed' <(tail -n 100 -f /var/log/manticore-2/searchd.log); echo $?
––– output –––
OK
––– input –––
for i in 2 3 4; do check_cluster_status ${i}306 "$CLUSTER_NAME"; done | uniq
––– output –––
OK
––– input –––
show_shards 2306 "$TABLE_NAME"
––– output –––
- system.t_s0: 127.0.0.1:2312 127.0.0.1:3312 127.0.0.1:4312
- system.t_s1: 127.0.0.1:2312 127.0.0.1:3312 127.0.0.1:4312
- system.t_s2: 127.0.0.1:2312 127.0.0.1:3312 127.0.0.1:4312
––– input –––
mysql -h0 -P2306 -sN -e "SELECT COUNT(*) FROM ${TABLE_NAME}"
––– output –––
OK
––– input –––
export INSTANCE=2; stdbuf -oL searchd --stopwait -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null; echo "Node 2 killed"
––– output –––
OK
––– input –––
timeout 30 bash -c 'while lsof -i :${INSTANCE}306 &>/dev/null; do sleep 1; done'
––– output –––
OK
––– input –––
timeout 20 grep -qm1 'Skipping rebalance for table' <(tail -n 100 -f /var/log/manticore-{3,4}/searchd.log); echo $?
––– output –––
OK
––– input –––
for i in 3 4; do check_cluster_status ${i}306 "$CLUSTER_NAME"; done | uniq
––– output –––
OK
––– input –––
mysql -h0 -P3306 -sN -e "SELECT COUNT(*) FROM ${TABLE_NAME}"
––– output –––
OK
––– input –––
truncate -s 0 /var/log/manticore-3/searchd.log
––– output –––
OK
––– input –––
export INSTANCE=1
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
export INSTANCE=2
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
timeout 60 grep -qm1 'Rebalancing completed' <(tail -n 100 -F /var/log/manticore-3/searchd.log); echo $?
––– output –––
- 0
+ tail: /var/log/manticore-3/searchd.log: file truncated
+ 124
––– input –––
for i in 1 2 3 4; do check_cluster_status ${i}306 "$CLUSTER_NAME"; done | uniq
––– output –––
OK
––– input –––
show_shards 1306 "$TABLE_NAME"
––– output –––
- system.t_s0: 127.0.0.1:1312 127.0.0.1:2312 127.0.0.1:3312
- system.t_s1: 127.0.0.1:2312 127.0.0.1:3312 127.0.0.1:4312
- system.t_s2: 127.0.0.1:1312 127.0.0.1:2312 127.0.0.1:4312
––– input –––
mysql -h0 -P1306 -sN -e "SELECT COUNT(*) FROM ${TABLE_NAME}"
––– output –––
OK

test/clt-tests/sharding/rollback/master-takeover-while-rollback.rec

––– input –––
export INSTANCE=1
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
export INSTANCE=2
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
export INSTANCE=3
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
export CLUSTER_NAME=c TABLE_NAME=t
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "create cluster ${CLUSTER_NAME}"
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "show status like 'cluster_${CLUSTER_NAME}_status'\G"
––– output –––
OK
––– input –––
for n in `seq 2 $INSTANCE`; do mysql -h0 -P${n}306 -e "join cluster ${CLUSTER_NAME} at '127.0.0.1:1312'"; done;
––– output –––
OK
––– input –––
mysql -h0 -P${INSTANCE}306 -e "show status like 'cluster_${CLUSTER_NAME}_status'\G"
––– output –––
OK
––– input –––
show_shards() {
	shards=$(mysql -h0 -P$1 -e "show create table $2 option force=1\G" | grep "Create Table" | cut -d: -f2-)

	(
		# Handle local shards
		echo $shards | grep -oP "local='[^']*'" | sed "s/local='//g; s/'//g" | tr ',' '\n' | while read -r tbl; do
			[ -n "$tbl" ] && echo "$tbl: local"
		done

		# Handle agent shards
		echo $shards | grep -oP "agent='[^']*'" | sed "s/agent='//g; s/'//g" | tr '|' '\n' | awk -F: '{agents[$3] = agents[$3] ? agents[$3] " " $1 ":" $2 : $1 ":" $2} END {for (t in agents) print t ": " agents[t]}'
	) | sort
}
––– output –––
OK
––– input –––
check_cluster_status() {
	mysql -h0 -P$1 -e "SHOW STATUS LIKE 'cluster_${2}_status'\\G" | grep -q "Value: primary" && echo "OK" || echo "FAIL"
}
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "CREATE TABLE ${CLUSTER_NAME}:${TABLE_NAME} (id bigint, value string) shards='2' rf='2'"; echo $?
––– output –––
OK
––– input –––
show_shards 1306 "$TABLE_NAME"
––– output –––
- system.t_s0: 127.0.0.1:1312 127.0.0.1:3312
- system.t_s1: 127.0.0.1:1312 127.0.0.1:2312
––– input –––
export INSTANCE=1; stdbuf -oL searchd --stopwait -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null; echo "Node 1 killed"
––– output –––
OK
––– input –––
timeout 30 bash -c 'while lsof -i :${INSTANCE}306 &>/dev/null; do sleep 1; done'
––– output –––
OK
––– input –––
timeout 10 grep -qm1 'becoming master' <(tail -n 1000 -f /var/log/manticore-{2,3}/searchd.log); echo $?
––– output –––
OK
––– input –––
for i in 2 3; do check_cluster_status ${i}306 "$CLUSTER_NAME"; done | uniq
––– output –––
OK
––– input –––
show_shards 2306 "$TABLE_NAME"
––– output –––
- system.t_s0: 127.0.0.1:2312 127.0.0.1:3312
- system.t_s1: 127.0.0.1:2312 127.0.0.1:3312
––– input –––
export INSTANCE=1
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
sleep 2
––– output –––
OK
––– input –––
for i in 1 2 3; do check_cluster_status ${i}306 "$CLUSTER_NAME"; done | uniq
––– output –––
OK
––– input –––
show_shards 2306 "$TABLE_NAME"
––– output –––
- system.t_s0: 127.0.0.1:2312 127.0.0.1:3312
- system.t_s1: 127.0.0.1:2312 127.0.0.1:3312

test/clt-tests/sharding/rollback/node-failure-mid-creation-recovery.rec

––– input –––
export INSTANCE=1
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
export INSTANCE=2
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
export INSTANCE=3
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "CREATE TABLE c:orders (id bigint, total float) shards='2' rf='2'"; echo $?
––– output –––
- 0
+ ERROR 1064 (42000) at line 1: Cluster 'c' does not exist
+ 1
––– input –––
mysql -h0 -P1306 -e "INSERT INTO orders (id, total) VALUES (1, 100), (2, 200)"; echo $?
––– output –––
OK
––– input –––
export INSTANCE=1; stdbuf -oL searchd --stopwait -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null; echo "Node 1 killed"
––– output –––
OK
––– input –––
for port in 2306 3306; do timeout 30 mysql -h0 -P$port -e "SHOW STATUS LIKE 'cluster_c_status'\\G" > /tmp/status_$port.log && grep -q "Value: primary" /tmp/status_$port.log && echo "Port $port: synced"; done
––– output –––
- Port 2306: synced
- Port 3306: synced
––– input –––
export INSTANCE=1
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
sleep 15; for port in 1306 2306 3306; do timeout 30 mysql -h0 -P$port -e "SHOW STATUS LIKE 'cluster_c_status'\\G" > /tmp/status_$port.log && grep -q "Value: primary" /tmp/status_$port.log && echo "Port $port: synced"; done
––– output –––
- Port 1306: synced
- Port 2306: synced
- Port 3306: synced

test/clt-tests/sharding/rollback/data-consistency-multiple-failures.rec

––– input –––
export INSTANCE=1
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
export INSTANCE=2
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
export INSTANCE=3
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
export CLUSTER_NAME=c TABLE_NAME=t
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "create cluster ${CLUSTER_NAME}"
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "show status like 'cluster_${CLUSTER_NAME}_status'\G"
––– output –––
OK
––– input –––
for n in `seq 2 $INSTANCE`; do mysql -h0 -P${n}306 -e "join cluster ${CLUSTER_NAME} at '127.0.0.1:1312'"; done;
––– output –––
OK
––– input –––
mysql -h0 -P${INSTANCE}306 -e "show status like 'cluster_${CLUSTER_NAME}_status'\G"
––– output –––
OK
––– input –––
show_shards() {
	shards=$(mysql -h0 -P$1 -e "show create table $2 option force=1\G" | grep "Create Table" | cut -d: -f2-)

	(
		# Handle local shards
		echo $shards | grep -oP "local='[^']*'" | sed "s/local='//g; s/'//g" | tr ',' '\n' | while read -r tbl; do
			[ -n "$tbl" ] && echo "$tbl: local"
		done

		# Handle agent shards
		echo $shards | grep -oP "agent='[^']*'" | sed "s/agent='//g; s/'//g" | tr '|' '\n' | awk -F: '{agents[$3] = agents[$3] ? agents[$3] " " $1 ":" $2 : $1 ":" $2} END {for (t in agents) print t ": " agents[t]}'
	) | sort
}
––– output –––
OK
––– input –––
check_cluster_status() {
	mysql -h0 -P$1 -e "SHOW STATUS LIKE 'cluster_${2}_status'\\G" | grep -q "Value: primary" && echo "OK" || echo "FAIL"
}
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "CREATE TABLE ${CLUSTER_NAME}:${TABLE_NAME} (id bigint, account string, amount float, ts int) shards='2' rf='2'"; echo $?
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "INSERT INTO ${TABLE_NAME} (id, account, amount, ts) VALUES (1, 'ACC001', 100.50, 1000), (2, 'ACC002', 200.75, 2000), (3, 'ACC003', 150.25, 3000), (4, 'ACC001', 300.00, 4000), (5, 'ACC002', 250.50, 5000)"; echo $?
––– output –––
OK
––– input –––
mysql -h0 -P1306 -sN -e "SELECT COUNT(*), SUM(amount) FROM ${TABLE_NAME}"
––– output –––
- 5	1002.000000
+ +------+-------------+
+ |    5 | 1002.000000 |
+ +------+-------------+
––– input –––
export INSTANCE=1; stdbuf -oL searchd --stopwait -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null; echo "Node 1 killed"
––– output –––
OK
––– input –––
timeout 30 bash -c 'while lsof -i :${INSTANCE}306 &>/dev/null; do sleep 1; done'
––– output –––
OK
––– input –––
timeout 10 grep -qm1 'becoming master' <(tail -n 1000 -f /var/log/manticore-{2,3}/searchd.log); echo $?
––– output –––
OK
––– input –––
for i in 2 3; do check_cluster_status ${i}306 "$CLUSTER_NAME"; done | uniq
––– output –––
OK
––– input –––
mysql -h0 -P2306 -e "INSERT INTO ${TABLE_NAME} (id, account, amount, ts) VALUES (6, 'ACC003', 175.00, 6000), (7, 'ACC001', 225.75, 7000)"; echo $?
––– output –––
OK
––– input –––
export INSTANCE=1
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
sleep 2
––– output –––
OK
––– input –––
for i in 1 2 3; do check_cluster_status ${i}306 "$CLUSTER_NAME"; done | uniq
––– output –––
OK
––– input –––
export INSTANCE=2; stdbuf -oL searchd --stopwait -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null; echo "Node 2 killed"
––– output –––
OK
––– input –––
timeout 30 bash -c 'while lsof -i :${INSTANCE}306 &>/dev/null; do sleep 1; done'
––– output –––
OK
––– input –––
timeout 10 grep -qm1 'becoming master' <(tail -n 1000 -f /var/log/manticore-{1,3}/searchd.log); echo $?
––– output –––
OK
––– input –––
for i in 1 3; do check_cluster_status ${i}306 "$CLUSTER_NAME"; done | uniq
––– output –––
OK
––– input –––
export INSTANCE=2
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
sleep 2
––– output –––
OK
––– input –––
for i in 1 2 3; do check_cluster_status ${i}306 "$CLUSTER_NAME"; done | uniq
––– output –––
OK
––– input –––
mysql -h0 -P1306 -sN -e "SELECT COUNT(*), SUM(amount) FROM ${TABLE_NAME}"
––– output –––
- 7	1402.75
+ ERROR 1064 (42000) at line 1: unknown local table(s) 't' in search request
––– input –––
for port in 1306 2306 3306; do echo "Node $port:"; mysql -h0 -P$port -sN -e "SELECT COUNT(*), SUM(amount) FROM ${TABLE_NAME}"; done
––– output –––
Node 1306:
- 7	1402.75
+ ERROR 1064 (42000) at line 1: unknown local table(s) 't' in search request
Node 2306:
- 7	1402.75
+ +------+------------+
- Node 3306:
+ |    3 | 476.250000 |
- 7	1402.75
+ +------+------------+
+ Node 3306:
+ +------+-------------+
+ |    7 | 1402.750000 |
+ +------+-------------+
––– input –––
mysql -h0 -P1306 -e "DROP TABLE IF EXISTS ${TABLE_NAME}"; echo $?
––– output –––
OK

test/clt-tests/sharding/rollback/new-node-addition-rf-replication.rec

––– input –––
export INSTANCE=1
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
export INSTANCE=2
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "CREATE TABLE c:existing_table (id bigint, data string) shards='2' rf='2'"; echo $?
––– output –––
- 0
+ ERROR 1064 (42000) at line 1: Cluster 'c' does not exist
+ 1
––– input –––
for i in 1 2; do mysql -h0 -P${i}306 -sN -e "SELECT COUNT(*) FROM system.sharding_table WHERE table = 'existing_table'"; done
––– output –––
- 2
+ ERROR 1064 (42000) at line 1: unknown local table(s) 'system.sharding_table' in search request
- 2
+ ERROR 1064 (42000) at line 1: unknown local table(s) 'system.sharding_table' in search request
––– input –––
export INSTANCE=3
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
mysql -h0 -P3306 -e "JOIN CLUSTER c at '127.0.0.1:1312'"; echo $?
––– output –––
- 0
+ ERROR 1064 (42000) at line 1: cluster 'c', no nodes available(127.0.0.1:1312), error: '127.0.0.1:1312': retries limit exceeded
+ 1
––– input –––
for port in 1306 2306 3306; do timeout 30 mysql -h0 -P$port -e "SHOW STATUS LIKE 'cluster_c_status'\G" > /tmp/status_$port.log && grep -q "Value: primary" /tmp/status_$port.log && echo "Port $port: Node synced"; done
––– output –––
- Port 1306: Node synced
- Port 2306: Node synced
- Port 3306: Node synced
––– input –––
mysql -h0 -P1306 -sN -e "SELECT COUNT(*) FROM system.sharding_state"
––– output –––
- 3
+ ERROR 1064 (42000) at line 1: unknown local table(s) 'system.sharding_state' in search request
––– input –––
sleep 15; echo "Cluster stabilized"
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "DROP TABLE IF EXISTS existing_table"
––– output –––
OK

test/clt-tests/sharding/rollback/node-rejoin-stale-cluster-cleanup.rec

––– input –––
export INSTANCE=1
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
export INSTANCE=2
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
export INSTANCE=3
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
export CLUSTER_NAME=c TABLE_NAME=t
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "create cluster ${CLUSTER_NAME}"
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "show status like 'cluster_${CLUSTER_NAME}_status'\G"
––– output –––
OK
––– input –––
for n in `seq 2 $INSTANCE`; do mysql -h0 -P${n}306 -e "join cluster ${CLUSTER_NAME} at '127.0.0.1:1312'"; done;
––– output –––
OK
––– input –––
mysql -h0 -P${INSTANCE}306 -e "show status like 'cluster_${CLUSTER_NAME}_status'\G"
––– output –––
OK
––– input –––
show_shards() {
	shards=$(mysql -h0 -P$1 -e "show create table $2 option force=1\G" | grep "Create Table" | cut -d: -f2-)

	(
		# Handle local shards
		echo $shards | grep -oP "local='[^']*'" | sed "s/local='//g; s/'//g" | tr ',' '\n' | while read -r tbl; do
			[ -n "$tbl" ] && echo "$tbl: local"
		done

		# Handle agent shards
		echo $shards | grep -oP "agent='[^']*'" | sed "s/agent='//g; s/'//g" | tr '|' '\n' | awk -F: '{agents[$3] = agents[$3] ? agents[$3] " " $1 ":" $2 : $1 ":" $2} END {for (t in agents) print t ": " agents[t]}'
	) | sort
}
––– output –––
OK
––– input –––
check_cluster_status() {
	mysql -h0 -P$1 -e "SHOW STATUS LIKE 'cluster_${2}_status'\\G" | grep -q "Value: primary" && echo "OK" || echo "FAIL"
}
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "CREATE TABLE ${CLUSTER_NAME}:${TABLE_NAME} (id bigint, value string) shards='2' rf='2'"; echo $?
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "INSERT INTO ${TABLE_NAME} (id, value) VALUES (1, 'alpha'), (2, 'beta')"; echo $?
––– output –––
OK
––– input –––
show_shards 1306 "$TABLE_NAME"
––– output –––
- system.t_s0: 127.0.0.1:1312 127.0.0.1:2312
- system.t_s1: 127.0.0.1:2312 127.0.0.1:3312
––– input –––
export INSTANCE=1; stdbuf -oL searchd --stopwait -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null; echo "Node 1 killed"
––– output –––
OK
––– input –––
timeout 30 bash -c 'while lsof -i :${INSTANCE}306 &>/dev/null; do sleep 1; done'
––– output –––
OK
––– input –––
timeout 60 grep -qm1 'Rebalancing completed' <(tail -n 100 -f /var/log/manticore-2/searchd.log); echo $?
––– output –––
- 0
+ 124
––– input –––
for i in 2 3; do check_cluster_status ${i}306 "$CLUSTER_NAME"; done | uniq
––– output –––
OK
––– input –––
show_shards 2306 "$TABLE_NAME"
––– output –––
- system.t_s0: 127.0.0.1:2312 127.0.0.1:3312
- system.t_s1: 127.0.0.1:2312 127.0.0.1:3312
––– input –––
mysql -h0 -P2306 -sN -e "SELECT COUNT(*) FROM ${TABLE_NAME}"
––– output –––
OK
––– input –––
truncate -s 0 /var/log/manticore-2/searchd.log
––– output –––
OK
––– input –––
export INSTANCE=1
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
timeout 90 grep -qm1 'Rebalancing completed' <(tail -n 100 -F /var/log/manticore-2/searchd.log); echo $?
––– output –––
- 0
+ 124
––– input –––
for i in 1 2 3; do check_cluster_status ${i}306 "$CLUSTER_NAME"; done | uniq
––– output –––
OK
––– input –––
show_shards 1306 "$TABLE_NAME"
––– output –––
- system.t_s0: 127.0.0.1:1312 127.0.0.1:2312
+ ERROR 1064 (42000) at line 1: You have an error in your query. Please, double-check it.
- system.t_s1: 127.0.0.1:2312 127.0.0.1:3312
––– input –––
mysql -h0 -P1306 -sN -e "SELECT COUNT(*) FROM ${TABLE_NAME}"
––– output –––
- +------+
+ ERROR 1064 (42000) at line 1: unknown local table(s) 't' in search request
- |    2 |
- +------+

test/clt-tests/sharding/rollback/rf1-no-rebalance-on-failure.rec

––– input –––
export INSTANCE=1
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
export INSTANCE=2
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
export CLUSTER_NAME=c TABLE_NAME=t
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "create cluster ${CLUSTER_NAME}"
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "show status like 'cluster_${CLUSTER_NAME}_status'\G"
––– output –––
OK
––– input –––
for n in `seq 2 $INSTANCE`; do mysql -h0 -P${n}306 -e "join cluster ${CLUSTER_NAME} at '127.0.0.1:1312'"; done;
––– output –––
OK
––– input –––
mysql -h0 -P${INSTANCE}306 -e "show status like 'cluster_${CLUSTER_NAME}_status'\G"
––– output –––
OK
––– input –––
show_shards() {
	shards=$(mysql -h0 -P$1 -e "show create table $2 option force=1\G" | grep "Create Table" | cut -d: -f2-)

	(
		# Handle local shards
		echo $shards | grep -oP "local='[^']*'" | sed "s/local='//g; s/'//g" | tr ',' '\n' | while read -r tbl; do
			[ -n "$tbl" ] && echo "$tbl: local"
		done

		# Handle agent shards
		echo $shards | grep -oP "agent='[^']*'" | sed "s/agent='//g; s/'//g" | tr '|' '\n' | awk -F: '{agents[$3] = agents[$3] ? agents[$3] " " $1 ":" $2 : $1 ":" $2} END {for (t in agents) print t ": " agents[t]}'
	) | sort
}
––– output –––
OK
––– input –––
check_cluster_status() {
	mysql -h0 -P$1 -e "SHOW STATUS LIKE 'cluster_${2}_status'\\G" | grep -q "Value: primary" && echo "OK" || echo "FAIL"
}
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "CREATE TABLE ${CLUSTER_NAME}:${TABLE_NAME} (id bigint, value string) shards='2' rf='1'"; echo $?
––– output –––
OK
––– input –––
show_shards 1306 "$TABLE_NAME"
––– output –––
- system.t_s0: local
- system.t_s1: 127.0.0.1:2312
––– input –––
mysql -h0 -P1306 -e "INSERT INTO ${TABLE_NAME} (id, value) VALUES (1, 'Widget A'), (2, 'Widget B'), (3, 'Widget C'), (4, 'Widget D')"; echo $?
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "SELECT COUNT(*) as total FROM ${TABLE_NAME}\\G" | grep total
––– output –––
OK
––– input –––
export INSTANCE=2; stdbuf -oL searchd --stopwait -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null; echo "Node 2 killed"
––– output –––
OK
––– input –––
timeout 30 bash -c 'while lsof -i :${INSTANCE}306 &>/dev/null; do sleep 1; done'
––– output –––
OK
––– input –––
timeout 10 grep -qm1 'Skipping rebalance for table' <(tail -n 1000 -f /var/log/manticore-1/searchd.log); echo $?
––– output –––
OK
––– input –––
show_shards 1306 "$TABLE_NAME"
––– output –––
- system.t_s0: local
- system.t_s1: 127.0.0.1:2312
––– input –––
mysql -h0 -P1306 -e "SELECT COUNT(*) as cnt FROM ${TABLE_NAME}\\G" | grep -E 'cnt:' | awk '{print $2}'
––– output –––
OK
––– input –––
export INSTANCE=2
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
sleep 15; for port in 1306 2306; do timeout 30 mysql -h0 -P$port -e "SHOW STATUS LIKE 'cluster_c_status'\\G" > /tmp/status_$port.log && grep -q "Value: primary" /tmp/status_$port.log && echo "Port $port: synced"; done
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "SELECT COUNT(*) as cnt FROM ${TABLE_NAME}\\G" | grep -E 'cnt:' | awk '{print $2}'
––– output –––
OK

test/clt-tests/sharding/rollback/node-removal-rf-maintained.rec

––– input –––
export INSTANCE=1
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
export INSTANCE=2
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
export INSTANCE=3
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "CREATE TABLE c:items (id bigint, name string) shards='2' rf='2'"; echo $?
––– output –––
- 0
+ ERROR 1064 (42000) at line 1: Cluster 'c' does not exist
+ 1
––– input –––
mysql -h0 -P1306 -e "INSERT INTO items (id, name) VALUES (1, 'Item1'), (2, 'Item2'), (3, 'Item3')"; echo $?
––– output –––
OK
––– input –––
for port in 1306 2306 3306; do timeout 30 mysql -h0 -P$port -e "SHOW STATUS LIKE 'cluster_c_status'\\G" > /tmp/status_$port.log && grep -q "Value: primary" /tmp/status_$port.log && echo "Port $port: synced"; done
––– output –––
- Port 1306: synced
- Port 2306: synced
- Port 3306: synced
––– input –––
mysql -h0 -P1306 -e "show tables from system\\G" | grep -E 'items_s[0-9]' | wc -l
––– output –––
- 2
+ 0

test/clt-tests/sharding/rollback/node-killed-rf-maintained.rec

––– input –––
export INSTANCE=1
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
export INSTANCE=2
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "CREATE TABLE c:data (id bigint, value string) shards='2' rf='2'"; echo $?
––– output –––
- 0
+ ERROR 1064 (42000) at line 1: Cluster 'c' does not exist
+ 1
––– input –––
mysql -h0 -P1306 -e "INSERT INTO data (id, value) VALUES (1, 'test1'), (2, 'test2')"; echo $?
––– output –––
OK
––– input –––
export INSTANCE=2; stdbuf -oL searchd --stopwait -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null; echo "Node 2 killed"
––– output –––
OK
––– input –––
sleep 10; mysql -h0 -P1306 -e "SHOW STATUS LIKE 'cluster_c_status'\G"
––– output –––
- *************************** 1. row ***************************
- Counter: cluster_c_status
-   Value: primary
––– input –––
mysql -h0 -P1306 -e "INSERT INTO data (id, value) VALUES (3, 'test3')"; echo $?
––– output –––
OK
––– input –––
export INSTANCE=2
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
sleep 20
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "SHOW STATUS LIKE 'cluster_c_status'\G"
––– output –––
- *************************** 1. row ***************************
- Counter: cluster_c_status
-   Value: primary
––– input –––
mysql -h0 -P2306 -e "SHOW STATUS LIKE 'cluster_c_status'\G"
––– output –––
- *************************** 1. row ***************************
- Counter: cluster_c_status
-   Value: primary
––– input –––
mysql -h0 -P1306 -sN -e "SELECT COUNT(*) FROM data"
––– output –––
- 3
+ +------+
+ |    3 |
+ +------+
––– input –––
mysql -h0 -P2306 -sN -e "SELECT COUNT(*) FROM data"
––– output –––
- 3
+ ERROR 1064 (42000) at line 1: unknown local table(s) 'data' in search request
––– input –––
mysql -h0 -P1306 -e "DROP TABLE IF EXISTS data"; echo $?
––– output –––
OK

test/clt-tests/sharding/rollback/master-node-failure.rec

––– input –––
export INSTANCE=1
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
export INSTANCE=2
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
export INSTANCE=3
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "CREATE TABLE c:data (id bigint, value string) shards='2' rf='2'"; echo $?
––– output –––
- 0
+ ERROR 1064 (42000) at line 1: Cluster 'c' does not exist
+ 1
––– input –––
sleep 5; mysql -h0 -P1306 -e "INSERT INTO data (id, value) VALUES (1, 'test_value')"; echo $?
––– output –––
OK
––– input –––
export INSTANCE=1; stdbuf -oL searchd --stopwait -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null; echo "Node 1 killed"
––– output –––
OK
––– input –––
timeout 30 bash -c 'while lsof -i :${INSTANCE}306 &>/dev/null; do sleep 1; done'
––– output –––
OK
––– input –––
for port in 2306 3306; do timeout 30 mysql -h0 -P$port -e "SHOW STATUS LIKE 'cluster_c_status'\\G" > /tmp/status_$port.log && grep -q "Value: primary" /tmp/status_$port.log && echo "Port $port: synced"; done
––– output –––
- Port 2306: synced
- Port 3306: synced
––– input –––
export INSTANCE=1
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
sleep 15; mysql -h0 -P1306 -e "SHOW STATUS LIKE 'cluster_c_status'\\G"
––– output –––
- *************************** 1. row ***************************
- Counter: cluster_c_status
-   Value: primary
––– input –––
for port in 1306 2306 3306; do timeout 30 mysql -h0 -P$port -e "SHOW STATUS LIKE 'cluster_c_status'\\G" > /tmp/status_$port.log && grep -q "Value: primary" /tmp/status_$port.log && echo "Port $port: synced"; done
––– output –––
- Port 1306: synced
- Port 2306: synced
- Port 3306: synced

test/clt-tests/sharding/rollback/rebalance-master-killed-mid-operation.rec

––– input –––
export INSTANCE=1
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
export INSTANCE=2
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
export INSTANCE=3
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
export INSTANCE=4
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
export CLUSTER_NAME=c TABLE_NAME=t
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "create cluster ${CLUSTER_NAME}"
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "show status like 'cluster_${CLUSTER_NAME}_status'\G"
––– output –––
OK
––– input –––
for n in `seq 2 $INSTANCE`; do mysql -h0 -P${n}306 -e "join cluster ${CLUSTER_NAME} at '127.0.0.1:1312'"; done;
––– output –––
OK
––– input –––
mysql -h0 -P${INSTANCE}306 -e "show status like 'cluster_${CLUSTER_NAME}_status'\G"
––– output –––
OK
––– input –––
show_shards() {
	shards=$(mysql -h0 -P$1 -e "show create table $2 option force=1\G" | grep "Create Table" | cut -d: -f2-)

	(
		# Handle local shards
		echo $shards | grep -oP "local='[^']*'" | sed "s/local='//g; s/'//g" | tr ',' '\n' | while read -r tbl; do
			[ -n "$tbl" ] && echo "$tbl: local"
		done

		# Handle agent shards
		echo $shards | grep -oP "agent='[^']*'" | sed "s/agent='//g; s/'//g" | tr '|' '\n' | awk -F: '{agents[$3] = agents[$3] ? agents[$3] " " $1 ":" $2 : $1 ":" $2} END {for (t in agents) print t ": " agents[t]}'
	) | sort
}
––– output –––
OK
––– input –––
check_cluster_status() {
	mysql -h0 -P$1 -e "SHOW STATUS LIKE 'cluster_${2}_status'\\G" | grep -q "Value: primary" && echo "OK" || echo "FAIL"
}
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "CREATE TABLE ${CLUSTER_NAME}:${TABLE_NAME} (id bigint, value string) shards='3' rf='2'"; echo $?
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "INSERT INTO ${TABLE_NAME} (id, value) VALUES (1, 'one'), (2, 'two'), (3, 'three')"; echo $?
––– output –––
OK
––– input –––
show_shards 1306 "$TABLE_NAME"
––– output –––
- system.t_s0: 127.0.0.1:1312 127.0.0.1:2312
- system.t_s1: 127.0.0.1:2312 127.0.0.1:3312
- system.t_s2: 127.0.0.1:3312 127.0.0.1:4312
––– input –––
export INSTANCE=4; stdbuf -oL searchd --stopwait -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null; echo "Node 4 killed"
––– output –––
OK
––– input –––
timeout 30 bash -c 'while lsof -i :${INSTANCE}306 &>/dev/null; do sleep 1; done'
––– output –––
OK
––– input –––
timeout 20 grep -qm1 'Rebalancing due to inactive nodes' <(tail -n 100 -f /var/log/manticore-1/searchd.log); echo $?
––– output –––
OK
––– input –––
export INSTANCE=1; stdbuf -oL searchd --stopwait -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null; echo "Master killed"
––– output –––
OK
––– input –––
timeout 30 bash -c 'while lsof -i :${INSTANCE}306 &>/dev/null; do sleep 1; done'
––– output –––
OK
––– input –––
timeout 30 grep -qm1 'becoming master' <(tail -n 100 -f /var/log/manticore-{2,3}/searchd.log); echo $?
––– output –––
OK
––– input –––
sleep 5
––– output –––
OK
––– input –––
for i in 2 3; do check_cluster_status ${i}306 "$CLUSTER_NAME"; done | uniq
––– output –––
OK
––– input –––
mysql -h0 -P2306 -sN -e "SELECT COUNT(*) FROM ${TABLE_NAME}"
––– output –––
OK
––– input –––
truncate -s 0 /var/log/manticore-{2,3}/searchd.log
––– output –––
OK
––– input –––
export INSTANCE=4
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
export INSTANCE=1
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
timeout 90 grep -qm1 'Rebalancing completed' <(tail -n 100 -F /var/log/manticore-{2,3}/searchd.log); echo $?
––– output –––
- 0
+ 124
––– input –––
for i in 1 2 3 4; do check_cluster_status ${i}306 "$CLUSTER_NAME"; done | uniq
––– output –––
OK
––– input –––
show_shards 2306 "$TABLE_NAME"
––– output –––
- system.t_s0: 127.0.0.1:3312 127.0.0.1:2312
- system.t_s1: 127.0.0.1:3312 127.0.0.1:2312
- system.t_s2: 127.0.0.1:3312 127.0.0.1:2312
––– input –––
mysql -h0 -P2306 -sN -e "SELECT COUNT(*) FROM ${TABLE_NAME}"
––– output –––
OK

test/clt-tests/sharding/rollback/mixed-rf-tables-node-failure.rec

––– input –––
export INSTANCE=1
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
export INSTANCE=2
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
export INSTANCE=3
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
export CLUSTER_NAME=c TABLE_NAME=t
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "create cluster ${CLUSTER_NAME}"
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "show status like 'cluster_${CLUSTER_NAME}_status'\G"
––– output –––
OK
––– input –––
for n in `seq 2 $INSTANCE`; do mysql -h0 -P${n}306 -e "join cluster ${CLUSTER_NAME} at '127.0.0.1:1312'"; done;
––– output –––
OK
––– input –––
mysql -h0 -P${INSTANCE}306 -e "show status like 'cluster_${CLUSTER_NAME}_status'\G"
––– output –––
OK
––– input –––
show_shards() {
	shards=$(mysql -h0 -P$1 -e "show create table $2 option force=1\G" | grep "Create Table" | cut -d: -f2-)

	(
		# Handle local shards
		echo $shards | grep -oP "local='[^']*'" | sed "s/local='//g; s/'//g" | tr ',' '\n' | while read -r tbl; do
			[ -n "$tbl" ] && echo "$tbl: local"
		done

		# Handle agent shards
		echo $shards | grep -oP "agent='[^']*'" | sed "s/agent='//g; s/'//g" | tr '|' '\n' | awk -F: '{agents[$3] = agents[$3] ? agents[$3] " " $1 ":" $2 : $1 ":" $2} END {for (t in agents) print t ": " agents[t]}'
	) | sort
}
––– output –––
OK
––– input –––
check_cluster_status() {
	mysql -h0 -P$1 -e "SHOW STATUS LIKE 'cluster_${2}_status'\\G" | grep -q "Value: primary" && echo "OK" || echo "FAIL"
}
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "CREATE TABLE ${CLUSTER_NAME}:norep (id bigint, value string) shards='2' rf='1'"; echo $?
––– output –––
- 0
+ ERROR 1064 (42000) at line 1: Waiting timeout exceeded.
+ 1
––– input –––
mysql -h0 -P1306 -e "CREATE TABLE ${CLUSTER_NAME}:rep (id bigint, value string) shards='2' rf='2'"; echo $?
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "INSERT INTO norep (id, value) VALUES (1, 'a'), (2, 'b')"; echo $?
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "INSERT INTO rep (id, value) VALUES (1, 'x'), (2, 'y')"; echo $?
––– output –––
OK
––– input –––
show_shards 1306 "norep"
––– output –––
- system.norep_s0: 127.0.0.1:1312
- system.norep_s1: 127.0.0.1:2312
––– input –––
show_shards 1306 "rep"
––– output –––
- system.rep_s0: 127.0.0.1:1312 127.0.0.1:2312
- system.rep_s1: 127.0.0.1:2312 127.0.0.1:3312
––– input –––
export INSTANCE=1; stdbuf -oL searchd --stopwait -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null; echo "Node 1 killed"
––– output –––
OK
––– input –––
timeout 30 bash -c 'while lsof -i :${INSTANCE}306 &>/dev/null; do sleep 1; done'
––– output –––
OK
––– input –––
timeout 60 grep -qm1 'Skipping rebalance for table norep' <(tail -n 100 -f /var/log/manticore-2/searchd.log); echo $?
––– output –––
- 0
+ 124
––– input –––
timeout 60 grep -qm1 'Rebalancing completed' <(tail -n 100 -f /var/log/manticore-2/searchd.log); echo $?
––– output –––
- 0
+ 124
––– input –––
for i in 2 3; do check_cluster_status ${i}306 "$CLUSTER_NAME"; done | uniq
––– output –––
OK
––– input –––
show_shards 2306 "norep"
––– output –––
- system.norep_s0: 127.0.0.1:1312
+ ERROR 1064 (42000) at line 1: You have an error in your query. Please, double-check it.
- system.norep_s1: 127.0.0.1:2312
––– input –––
show_shards 2306 "rep"
––– output –––
- system.rep_s0: 127.0.0.1:2312 127.0.0.1:3312
- system.rep_s1: 127.0.0.1:2312 127.0.0.1:3312
––– input –––
mysql -h0 -P2306 -sN -e "SELECT COUNT(*) FROM rep"
––– output –––
OK
––– input –––
truncate -s 0 /var/log/manticore-2/searchd.log
––– output –––
OK
––– input –––
export INSTANCE=1
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
timeout 90 grep -qm1 'Rebalancing completed' <(tail -n 100 -F /var/log/manticore-2/searchd.log); echo $?
––– output –––
- 0
+ 124
––– input –––
for i in 1 2 3; do check_cluster_status ${i}306 "$CLUSTER_NAME"; done | uniq
––– output –––
OK
––– input –––
show_shards 1306 "norep"
––– output –––
- system.norep_s0: 127.0.0.1:1312
- system.norep_s1: 127.0.0.1:2312
––– input –––
show_shards 1306 "rep"
––– output –––
- system.rep_s0: 127.0.0.1:1312 127.0.0.1:2312
+ ERROR 1064 (42000) at line 1: You have an error in your query. Please, double-check it.
- system.rep_s1: 127.0.0.1:2312 127.0.0.1:3312
––– input –––
mysql -h0 -P1306 -sN -e "SELECT COUNT(*) FROM norep"
––– output –––
OK
––– input –––
mysql -h0 -P1306 -sN -e "SELECT COUNT(*) FROM rep"
––– output –––
- +------+
+ ERROR 1064 (42000) at line 1: unknown local table(s) 'rep' in search request
- |    2 |
- +------+

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

ci: force-push test-kit-latest regardless of branch/test result#4609

ci: force-push test-kit-latest regardless of branch/test result#4609
donhardman wants to merge 1 commit into
mainfrom
ci/force-push-test-kit-latest

donhardman commented Jun 3, 2026

Uh oh!

github-actions Bot commented Jun 3, 2026

Uh oh!

github-actions Bot commented Jun 3, 2026

Uh oh!

github-actions Bot commented Jun 3, 2026

Uh oh!

github-actions Bot commented Jun 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

donhardman commented Jun 3, 2026

Uh oh!

github-actions Bot commented Jun 3, 2026

clt

Failed tests:

Uh oh!

github-actions Bot commented Jun 3, 2026

clt

Failed tests:

Uh oh!

github-actions Bot commented Jun 3, 2026

clt

Failed tests:

Uh oh!

github-actions Bot commented Jun 3, 2026

clt

Failed tests:

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant