BUG(go client):when go client is writing to one partition and the replica node core dump, go client will finish after timeout without updating the configuration. #1856
Labels
type/bug
This issue reports a bug.
Comments
|
@lengyuexuexuan Thanks for the feedback, could you please submit a patch to fix it? |
OK. No problem. |
empiredan
pushed a commit
that referenced
this issue
Jun 20, 2024
…to primary meta server if it was changed (#1916) #1880 #1856 As for #1856: when go client is writing to one partition and the replica node core dump, go client will finish after timeout without updating the configuration. In this case, the go client only restart to solve the problem. In this pr, the client would update configuration of table automatically when someone replica core dump. After testing, we found that the the replica error is "context.DeadlineExceeded" (incubator-pegasus/go-client/pegasus/table_connector.go) when the replica core dump. Therefore, when client meets the error, the go client will update configuration automatically. Besides, this request will not retry. Because only in the case of timeout, the configuration will be automatically updated. If you try again before then, it will still fail. There is also the risk of infinite retries. Therefore, it is better to directly return the request error to the user and let the user try again. As for #1880: When the client sends an RPC message "RPC_CM_QUERY_PARTITION_CONFIG_BY_INDEX" to the meta server, if the meta server isn't primary, the response that forward to the primary meta server will return. According to the above description, assuming that the client does not have a primary meta server configured, we can connect to the primary meta server in this way. About tests: 1. Start onebox, and the primary meta server is not added to the go client configuration. 2. The go client writes data to a certain partition and then kills the replica process.
ruojieranyishen
pushed a commit
to ruojieranyishen/incubator-pegasus
that referenced
this issue
Jul 17, 2024
…to primary meta server if it was changed (apache#1916) apache#1880 apache#1856 As for apache#1856: when go client is writing to one partition and the replica node core dump, go client will finish after timeout without updating the configuration. In this case, the go client only restart to solve the problem. In this pr, the client would update configuration of table automatically when someone replica core dump. After testing, we found that the the replica error is "context.DeadlineExceeded" (incubator-pegasus/go-client/pegasus/table_connector.go) when the replica core dump. Therefore, when client meets the error, the go client will update configuration automatically. Besides, this request will not retry. Because only in the case of timeout, the configuration will be automatically updated. If you try again before then, it will still fail. There is also the risk of infinite retries. Therefore, it is better to directly return the request error to the user and let the user try again. As for apache#1880: When the client sends an RPC message "RPC_CM_QUERY_PARTITION_CONFIG_BY_INDEX" to the meta server, if the meta server isn't primary, the response that forward to the primary meta server will return. According to the above description, assuming that the client does not have a primary meta server configured, we can connect to the primary meta server in this way. About tests: 1. Start onebox, and the primary meta server is not added to the go client configuration. 2. The go client writes data to a certain partition and then kills the replica process.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
In the code, when replcia core dump, the function loopForResponse() will return "nil".
Then, the process will be blocked in function CallWithGpid() until the time exceeds the timeout.
why not update the configuration of table and retry previous operation when the above situation occurs?
The text was updated successfully, but these errors were encountered: