MRConnector doesn't refresh tikv hosts when a node is rotated in a cluster

[ankita25]

It tries to connect to old tikv nodes even after tidb makes them "tombstone". When running TiDBMRDemo , I see following errors

Feb 01, 2022 12:53:06 AM org.tikv.shade.io.grpc.internal.ManagedChannelImpl$NameResolverListener handleErrorInSyncContext
WARNING: [Channel<21>: (XXX-YYY-ZZZ.com:20160)] Failed to resolve name. status=Status{code=UNAVAILABLE, description=Unable to resolve host XXX-YYY-ZZZ..com, cause=java.lang.RuntimeException: java.net.UnknownHostException: XXX-YYY-ZZZ..com: Name or service not known
at org.tikv.shade.io.grpc.internal.DnsNameResolver.resolveAddresses(DnsNameResolver.java:223)
at org.tikv.shade.io.grpc.internal.DnsNameResolver.doResolve(DnsNameResolver.java:282)
at org.tikv.shade.io.grpc.internal.DnsNameResolver$Resolve.run(DnsNameResolver.java:318)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.UnknownHostException: XXX-YYY-ZZZ.com: Name or service not known
at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:929)
at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1324)
at java.net.InetAddress.getAllByName0(InetAddress.java:1277)
at java.net.InetAddress.getAllByName(InetAddress.java:1193)
at java.net.InetAddress.getAllByName(InetAddress.java:1127)
at org.tikv.shade.io.grpc.internal.DnsNameResolver$JdkAddressResolver.resolveAddress(DnsNameResolver.java:631)
at org.tikv.shade.io.grpc.internal.DnsNameResolver.resolveAddresses(DnsNameResolver.java:219)
... 5 more
}
Feb 01, 2022 12:53:06 AM org.tikv.shade.io.grpc.internal.ManagedChannelImpl$NameResolverListener handleErrorInSyncContext
WARNING: [Channel<31>: (XXX-YYY-ZZZ.com:20160)] Failed to resolve name. status=Status{code=UNAVAILABLE, description=Unable to resolve host XXX-YYY-ZZZ.com, cause=java.lang.RuntimeException: java.net.UnknownHostException: XXX-YYY-ZZZ.com: Name or service not known
at org.tikv.shade.io.grpc.internal.DnsNameResolver.resolveAddresses(DnsNameResolver.java:223)
at org.tikv.shade.io.grpc.internal.DnsNameResolver.doResolve(DnsNameResolver.java:282)
at org.tikv.shade.io.grpc.internal.DnsNameResolver$Resolve.run(DnsNameResolver.java:318)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.UnknownHostException: XXX-YYY-ZZZ.com: Name or service not known
at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:929)
at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1324)
at java.net.InetAddress.getAllByName0(InetAddress.java:1277)
at java.net.InetAddress.getAllByName(InetAddress.java:1193)
at java.net.InetAddress.getAllByName(InetAddress.java:1127)
at org.tikv.shade.io.grpc.internal.DnsNameResolver$JdkAddressResolver.resolveAddress(DnsNameResolver.java:631)
at org.tikv.shade.io.grpc.internal.DnsNameResolver.resolveAddresses(DnsNameResolver.java:219)
... 5 more
}

[ankita25]

Steps to reproduce -

1. Replace one of the tikv nodes in the cluster
2. Run TiDbMapReduceDemo
3. Error logs show that job is still trying to connect to old node

[sunxiaoguang]

Does it cause tasks to fail? We can't tell who is the caller of resolver from stack.

[sunxiaoguang]

Do we have more logs about the failed task? For example the entire log of failed process after redaction.

[ankita25]

This stack trace was from hadoop logs. The job does not fail due to it but due to tons of noise, this caused debugging of another issue very hard. Is it possible to fix it ?