TSan reports thread leaks on ctests

[bjacob]

In a build with TSan, ctest says:

The following tests FAILED:
         28 - BlockingExecutorTest.concurrent_tasks (Failed)
         29 - BlockingExecutorTest.inhibit_when_shutdown (Failed)
         30 - BlockingExecutorTest.warn_deadline (Failed)
         31 - BlockingExecutorTest.threads_recycle (Failed)

All four have reports of thread leaks, i.e. threads are not joined by the main thread (one of them, threads_recycle, also has a data race, see #847).

Sample TSan report:

WARNING: ThreadSanitizer: thread leak (pid=2202513)
  Thread T1 'blocking-0' (tid=2202515, finished) created by main thread at:
    #0 pthread_create <null> (shortfin_support_test+0x9c5cd) (BuildId: e2b494448416a64d)
    #1 iree_thread_create /home/benjacob/iree/runtime/src/iree/base/internal/threading_pthreads.c:166:10 (libshortfin.so.1+0x209423) (BuildId: 0132f7b140c0704d)
    #2 shortfin::BlockingExecutor::Schedule(std::function<void ()>) /home/benjacob/shark-ai/shortfin/src/shortfin/support/blocking_executor.cc:118:21 (libshortfin.so.1+0xff8e6) (BuildId: 0132f7b140c0704d)
    #3 shortfin::BlockingExecutorTest_inhibit_when_shutdown_Test::TestBody() /home/benjacob/shark-ai/shortfin/src/shortfin/support/blocking_executor_test.cc:48:16 (shortfin_support_test+0x127ca3) (BuildId: e2b494448416a64d)
    #4 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/benjacob/iree/third_party/googletest/googletest/src/gtest.cc:2635:10 (shortfin_support_test+0x141370) (BuildId: e2b494448416a64d)
    #5 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/benjacob/iree/third_party/googletest/googletest/src/gtest.cc:2671:14 (shortfin_support_test+0x141370)
    #6 testing::Test::Run() /home/benjacob/iree/third_party/googletest/googletest/src/gtest.cc:2710:5 (shortfin_support_test+0x15ad30) (BuildId: e2b494448416a64d)
    #7 testing::TestInfo::Run() /home/benjacob/iree/third_party/googletest/googletest/src/gtest.cc:2856:11 (shortfin_support_test+0x15ad30)
    #8 testing::TestSuite::Run() /home/benjacob/iree/third_party/googletest/googletest/src/gtest.cc:3034:30 (shortfin_support_test+0x15ad30)
    #9 testing::internal::UnitTestImpl::RunAllTests() /home/benjacob/iree/third_party/googletest/googletest/src/gtest.cc:5964:44 (shortfin_support_test+0x15ad30)
    #10 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/benjacob/iree/third_party/googletest/googletest/src/gtest.cc:2635:10 (shortfin_support_test+0x157909) (BuildId: e2b494448416a64d)
    #11 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/benjacob/iree/third_party/googletest/googletest/src/gtest.cc:2671:14 (shortfin_support_test+0x157909)
    #12 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/benjacob/iree/third_party/googletest/googletest/src/gtest.cc:2635:10 (shortfin_support_test+0x157909) (BuildId: e2b494448416a64d)
    #13 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/benjacob/iree/third_party/googletest/googletest/src/gtest.cc:2671:14 (shortfin_support_test+0x157909)
    #14 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/benjacob/iree/third_party/googletest/googletest/src/gtest.cc:2635:10 (shortfin_support_test+0x157909) (BuildId: e2b494448416a64d)
    #15 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/benjacob/iree/third_party/googletest/googletest/src/gtest.cc:2671:14 (shortfin_support_test+0x157909)
    #16 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/benjacob/iree/third_party/googletest/googletest/src/gtest.cc:2635:10 (shortfin_support_test+0x157909) (BuildId: e2b494448416a64d)
    #17 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/benjacob/iree/third_party/googletest/googletest/src/gtest.cc:2671:14 (shortfin_support_test+0x157909)
    #18 testing::UnitTest::Run() /home/benjacob/iree/third_party/googletest/googletest/src/gtest.cc:5543:10 (shortfin_support_test+0x1576d3) (BuildId: e2b494448416a64d)
    #19 RUN_ALL_TESTS() /home/benjacob/iree/third_party/googletest/googletest/include/gtest/gtest.h:2334:73 (shortfin_support_test+0x12b86a) (BuildId: e2b494448416a64d)
    #20 main /home/benjacob/iree/third_party/googletest/googletest/src/gtest_main.cc:64:10 (shortfin_support_test+0x12b86a)

[bjacob]

This existing TODO comment suggests this is a known issue and it's tied to an already-planned future refactoring, so I'll leave it there:

shark-ai/shortfin/src/shortfin/support/blocking_executor.cc

Lines 87 to 88 in 1f50538

	// TODO: Introduce iree_thread_try_join and refactor the shutdown sequence
	// to be based on thread joining with a warn deadline.

Note that, intentionally or not, pthread_join is in fact already happening... but it's happening on the worker thread, thus waiting for itself, whatever that means. Here:

shark-ai/shortfin/src/shortfin/support/blocking_executor.cc

Line 169 in 1f50538

delete this;

Stack:

Thread 3 "blocking-1" hit Breakpoint 1, ___pthread_join (threadid=140737335313984, thread_return=0x0) at ./nptl/pthread_join.c:23
23      ./nptl/pthread_join.c: No such file or directory.
(gdb) bt
#0  ___pthread_join (threadid=140737335313984, thread_return=0x0) at ./nptl/pthread_join.c:23
#1  0x00007ffff7f67efd in iree_thread_delete (thread=0x55555562fd70)
    at /home/benjacob/iree/runtime/src/iree/base/internal/threading_pthreads.c:194
#2  0x00007ffff7f67d11 in iree_thread_release (thread=0x55555562fd70)
    at /home/benjacob/iree/runtime/src/iree/base/internal/threading_pthreads.c:211
#3  0x00007ffff7d8ef55 in shortfin::iree::detail::thread_ptr_helper::release (obj=0x55555562fd70)
    at /home/benjacob/shark-ai/shortfin/src/shortfin/support/iree_concurrency.h:31
#4  0x00007ffff7d8d40a in shortfin::iree::object_ptr<iree_thread_t, shortfin::iree::detail::thread_ptr_helper>::reset (this=0x55555562c6f8) at /home/benjacob/shark-ai/shortfin/src/shortfin/support/iree_helpers.h:127
#5  0x00007ffff7d8d3a5 in shortfin::iree::object_ptr<iree_thread_t, shortfin::iree::detail::thread_ptr_helper>::~object_ptr (this=0x55555562c6f8) at /home/benjacob/shark-ai/shortfin/src/shortfin/support/iree_helpers.h:83
#6  0x00007ffff7daaaa7 in shortfin::BlockingExecutor::ThreadInstance::~ThreadInstance (this=0x55555562c6f0)
    at /home/benjacob/shark-ai/shortfin/src/shortfin/support/blocking_executor.h:47
#7  0x00007ffff7daa94a in shortfin::BlockingExecutor::ThreadInstance::RunOnThread (this=0x55555562c6f0)
    at /home/benjacob/shark-ai/shortfin/src/shortfin/support/blocking_executor.cc:173
#8  0x00007ffff7daa9c9 in shortfin::BlockingExecutor::Schedule(std::function<void ()>)::$_0::operator()(void*) const (this=0x55555562c6f0, self=0x55555562c6f0)
    at /home/benjacob/shark-ai/shortfin/src/shortfin/support/blocking_executor.cc:116
#9  0x00007ffff7daa9a5 in shortfin::BlockingExecutor::Schedule(std::function<void ()>)::$_0::__invoke(void*) (
    self=0x55555562c6f0) at /home/benjacob/shark-ai/shortfin/src/shortfin/support/blocking_executor.cc:115
#10 0x00007ffff7f67cbc in iree_thread_start_routine (param=0x55555562fd70)
    at /home/benjacob/iree/runtime/src/iree/base/internal/threading_pthreads.c:119
#11 0x00007ffff76a7ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#12 0x00007ffff7739850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81