Skip to content

Race between procedure destruction and procedure application #946

@jeremybarnes

Description

@jeremybarnes
MLDBFB-724_classifier_exp_segfault_test.py FAILED
. virtualenv/bin/activate; PYTHONPATH=build/arm64/bin build/arm64/bin/mldb_runner -h localhost -p '11700-12700'  --run-script mldb/testing/MLDBFB-724_classifier_exp_segfault_test.py --mute-final-output --config-path mldb/container_files/mldb.conf --watchdog-timeout=120 
reading configuration from file: 'mldb/container_files/mldb.conf'


MLDB ready


creating SYMLINK /var/folders/ch/fv_rwtr97j55bhh8lz6p2dl80000gp/T/REVv50/main.py -> /Users/jeremy/projects/mldb/mldb/testing/MLDBFB-724_classifier_exp_segfault_test.py
loading from: /var/folders/ch/fv_rwtr97j55bhh8lz6p2dl80000gp/T/REVv50/main.py
2021-08-30 10:00:41.042 script runner plugin SELECT {_inception.* EXCLUDING(image_url)} AS features,
                 ds.cei AS label
                 FROM _inception
                 INNER JOIN ds ON _inception.image_url=ds.image_url
2021-08-30 10:00:41.045 script runner plugin [
    [
        "_rowName"
    ]
]
2021-08-30 10:00:41.059 script runner plugin test_it (__main__.Mldbfb724ClassifierExpSegfaultTest) ... ok

----------------------------------------------------------------------
Ran 1 test in 0.152s

OK

exception in accept: Operation canceled
exception in accept: Operation canceled
ServicePeer [2021-08-30T10:00:41.059-4:00] warning WARNING: peer mldb lost its own entry in discovery.  Letting it come back
peer mldb connection to mldb changed state to 3
peer mldb connection to mldb changed state to 3
=================================================================
==67440==ERROR: AddressSanitizer: heap-use-after-free on address 0x00011834aa08 at pc 0x0001053dd2cc bp 0x00016c5030c0 sp 0x00016c5030b8
READ of size 8 at 0x00011834aa08 thread T34
    #0 0x1053dd2c8 in MLDB::SelectExpressionDescription::printJsonTyped(MLDB::SelectExpression const*, MLDB::JsonPrintingContext&) const sql_expression.cc:3616
    #1 0x1081408cc in MLDB::StructureDescriptionBase::printJson(void const*, MLDB::JsonPrintingContext&) const value_description.cc:583
    #2 0x1053f7150 in MLDB::InputQueryDescription::printJsonTyped(MLDB::InputQuery const*, MLDB::JsonPrintingContext&) const sql_expression.cc:4429
    #3 0x1081408cc in MLDB::StructureDescriptionBase::printJson(void const*, MLDB::JsonPrintingContext&) const value_description.cc:583
    #4 0x10d7fab20 in Json::Value MLDB::jsonEncode<MLDB::ExperimentProcedureConfig>(MLDB::ExperimentProcedureConfig const&) value_description.h:650
    #5 0x10d7cb33c in MLDB::ExperimentProcedureConfig MLDB::Procedure::applyRunConfOverProcConf<MLDB::ExperimentProcedureConfig>(MLDB::ExperimentProcedureConfig const&, MLDB::ProcedureRunConfig const&) const procedure.h:161
    #6 0x10d7c0b54 in MLDB::ExperimentProcedure::run(MLDB::ProcedureRunConfig const&, std::__1::function<bool (Json::Value const&)> const&) const experiment_procedure.cc:308
    #7 0x106c74730 in MLDB::ProcedureRun::ProcedureRun(MLDB::Procedure*, MLDB::ProcedureRunConfig, std::__1::function<bool (Json::Value const&)> const&) procedure.cc:85
    #8 0x10721bfe4 in MLDB::ProcedureRunCollection::construct(MLDB::ProcedureRunConfig, std::__1::function<bool (Json::Value const&)> const&) const procedure_run_collection.cc:262
    #9 0x1071f3e44 in MLDB::RestConfigurableCollection<MLDB::Utf8String, MLDB::ProcedureRun, MLDB::ProcedureRunConfig, MLDB::ProcedureRunStatus>::constructCancellable(MLDB::ProcedureRunConfig, std::__1::function<bool (Json::Value const&)> const&, MLDB::WatchT<bool>) const rest_collection_impl.h:1415
    #10 0x1072e0f6c in std::__1::shared_ptr<MLDB::ProcedureRun> std::__1::__invoke_void_return_wrapper<std::__1::shared_ptr<MLDB::ProcedureRun> >::__call<std::__1::__bind<std::__1::shared_ptr<MLDB::ProcedureRun> (MLDB::RestConfigurableCollection<MLDB::Utf8String, MLDB::ProcedureRun, MLDB::ProcedureRunConfig, MLDB::ProcedureRunStatus>::*)(MLDB::ProcedureRunConfig, std::__1::function<bool (Json::Value const&)> const&, MLDB::WatchT<bool>) const, MLDB::RestConfigurableCollection<MLDB::Utf8String, MLDB::ProcedureRun, MLDB::ProcedureRunConfig, MLDB::ProcedureRunStatus>*, MLDB::ProcedureRunConfig, std::__1::placeholders::__ph<1> const&, std::__1::placeholders::__ph<2> const&>&, std::__1::function<bool (Json::Value const&)> const&, MLDB::WatchT<bool> >(std::__1::__bind<std::__1::shared_ptr<MLDB::ProcedureRun> (MLDB::RestConfigurableCollection<MLDB::Utf8String, MLDB::ProcedureRun, MLDB::ProcedureRunConfig, MLDB::ProcedureRunStatus>::*)(MLDB::ProcedureRunConfig, std::__1::function<bool (Json::Value const&)> const&, MLDB::WatchT<bool>) const, MLDB::RestConfigurableCollection<MLDB::Utf8String, MLDB::ProcedureRun, MLDB::ProcedureRunConfig, MLDB::ProcedureRunStatus>*, MLDB::ProcedureRunConfig, std::__1::placeholders::__ph<1> const&, std::__1::placeholders::__ph<2> const&>&, std::__1::function<bool (Json::Value const&)> const&, MLDB::WatchT<bool>&&) __functional_base:317
    #11 0x107258400 in MLDB::RestCollection<MLDB::Utf8String, MLDB::ProcedureRun>::addBackgroundJobInThread(MLDB::Utf8String, std::__1::function<std::__1::shared_ptr<MLDB::ProcedureRun> (std::__1::function<bool (Json::Value const&)> const&, MLDB::WatchT<bool>)> const&, std::__1::function<bool (Json::Value const&)> const&, std::__1::function<void (std::__1::shared_ptr<MLDB::ProcedureRun>)> const&, bool, MLDB::Any)::'lambda'()::operator()() const rest_collection_impl.h:586
    #12 0x107257a58 in void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, MLDB::RestCollection<MLDB::Utf8String, MLDB::ProcedureRun>::addBackgroundJobInThread(MLDB::Utf8String, std::__1::function<std::__1::shared_ptr<MLDB::ProcedureRun> (std::__1::function<bool (Json::Value const&)> const&, MLDB::WatchT<bool>)> const&, std::__1::function<bool (Json::Value const&)> const&, std::__1::function<void (std::__1::shared_ptr<MLDB::ProcedureRun>)> const&, bool, MLDB::Any)::'lambda'()> >(void*) thread:291
    #13 0x18e887874 in _pthread_start+0x13c (libsystem_pthread.dylib:arm64e+0x7874)
    #14 0x18e8825dc in thread_start+0x4 (libsystem_pthread.dylib:arm64e+0x25dc)

0x00011834aa08 is located 72 bytes inside of 360-byte region [0x00011834a9c0,0x00011834ab28)
freed by thread T0 here:
    #0 0x109b18f64 in wrap__ZdlPv+0x74 (libclang_rt.asan_osx_dynamic.dylib:arm64e+0x4cf64)
    #1 0x10d7d9fe8 in MLDB::ExperimentProcedureConfig::~ExperimentProcedureConfig() experiment_procedure.h:60
    #2 0x10d7d410c in MLDB::ExperimentProcedure::~ExperimentProcedure() experiment_procedure.h:133
    #3 0x1089dbcf8 in std::__1::__tree<std::__1::__value_type<MLDB::Utf8String, MLDB::RestCollection<MLDB::Utf8String, MLDB::PolyEntity>::Entry>, std::__1::__map_value_compare<MLDB::Utf8String, std::__1::__value_type<MLDB::Utf8String, MLDB::RestCollection<MLDB::Utf8String, MLDB::PolyEntity>::Entry>, std::__1::less<MLDB::Utf8String>, true>, std::__1::allocator<std::__1::__value_type<MLDB::Utf8String, MLDB::RestCollection<MLDB::Utf8String, MLDB::PolyEntity>::Entry> > >::destroy(std::__1::__tree_node<std::__1::__value_type<MLDB::Utf8String, MLDB::RestCollection<MLDB::Utf8String, MLDB::PolyEntity>::Entry>, void*>*) __tree:1836
    #4 0x1089424d4 in MLDB::RestCollection<MLDB::Utf8String, MLDB::PolyEntity>::clear() rest_collection_impl.h:195
    #5 0x1049fae6c in MLDB::MldbServer::shutdown() mldb_server.cc:436
    #6 0x1049fa690 in MLDB::MldbServer::~MldbServer() mldb_server.cc:114
    #7 0x1049630b4 in main mldb_runner.cc:429
    #8 0x18e8a544c in start+0x0 (libdyld.dylib:arm64e+0x1844c)

previously allocated by thread T33 here:
    #0 0x109b18b4c in wrap__Znwm+0x74 (libclang_rt.asan_osx_dynamic.dylib:arm64e+0x4cb4c)
    #1 0x1053f6284 in MLDB::InputQueryDescription::parseJsonTyped(MLDB::InputQuery*, MLDB::JsonParsingContext&) const sql_expression.cc:4409
    #2 0x10814cfe4 in std::__1::__function::__func<MLDB::StructureDescriptionBase::parseJson(void*, MLDB::JsonParsingContext&) const::$_0, std::__1::allocator<MLDB::StructureDescriptionBase::parseJson(void*, MLDB::JsonParsingContext&) const::$_0>, void ()>::operator()() functional:1727
    #3 0x1081a9d08 in MLDB::StructuredJsonParsingContext::forEachMember(std::__1::function<void ()> const&) json_parsing.cc:1894
    #4 0x10813fcd4 in MLDB::StructureDescriptionBase::parseJson(void*, MLDB::JsonParsingContext&) const value_description.cc:555
    #5 0x10d7bf6e8 in MLDB::ExperimentProcedureConfig MLDB::Any::convert<MLDB::ExperimentProcedureConfig>(MLDB::ValueDescription const&) const any_impl.h:68
    #6 0x10d7bf180 in MLDB::ExperimentProcedure::ExperimentProcedure(MLDB::MldbEngine*, MLDB::PolyConfig, std::__1::function<bool (Json::Value const&)> const&) experiment_procedure.cc:289
    #7 0x10d7fdea4 in std::__1::shared_ptr<MLDB::EntityType<MLDB::Procedure> > MLDB::registerProcedureType<MLDB::ExperimentProcedure, MLDB::ExperimentProcedureConfig>(MLDB::Package const&, MLDB::Utf8String const&, MLDB::Utf8String const&, std::__1::function<MLDB::RestRequestMatchResult (MLDB::RestDirectory*, MLDB::RestConnection&, MLDB::RestRequest const&, MLDB::RestRequestParsingContext&)>, std::__1::set<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > >)::'lambda'(MLDB::RestDirectory*, MLDB::PolyConfig, std::__1::function<bool (Json::Value)> const&)::operator()(MLDB::RestDirectory*, MLDB::PolyConfig, std::__1::function<bool (Json::Value)> const&) const procedure.h:241
    #8 0x10d7fd6e0 in MLDB::Procedure* std::__1::__invoke_void_return_wrapper<MLDB::Procedure*>::__call<std::__1::shared_ptr<MLDB::EntityType<MLDB::Procedure> > MLDB::registerProcedureType<MLDB::ExperimentProcedure, MLDB::ExperimentProcedureConfig>(MLDB::Package const&, MLDB::Utf8String const&, MLDB::Utf8String const&, std::__1::function<MLDB::RestRequestMatchResult (MLDB::RestDirectory*, MLDB::RestConnection&, MLDB::RestRequest const&, MLDB::RestRequestParsingContext&)>, std::__1::set<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > >)::'lambda'(MLDB::RestDirectory*, MLDB::PolyConfig, std::__1::function<bool (Json::Value)> const&)&, MLDB::RestDirectory*, MLDB::PolyConfig, std::__1::function<bool (Json::Value)> const&>(MLDB::ExperimentProcedure&&...) __functional_base:317
    #9 0x10719dff8 in MLDB::PolyCollection<MLDB::Procedure>::doConstruct(MLDB::RestDirectory*, MLDB::PolyConfig, std::__1::function<bool (Json::Value const&)> const&) poly_collection_impl.h:365
    #10 0x10719ca38 in MLDB::PolyCollection<MLDB::Procedure>::construct(MLDB::PolyConfig, std::__1::function<bool (Json::Value const&)> const&) const poly_collection_impl.h:114
    #11 0x10897b8d4 in MLDB::RestConfigurableCollection<MLDB::Utf8String, MLDB::PolyEntity, MLDB::PolyConfig, MLDB::PolyStatus>::constructCancellable(MLDB::PolyConfig, std::__1::function<bool (Json::Value const&)> const&, MLDB::WatchT<bool>) const rest_collection_impl.h:1415
    #12 0x108a22c48 in std::__1::shared_ptr<MLDB::PolyEntity> std::__1::__invoke_void_return_wrapper<std::__1::shared_ptr<MLDB::PolyEntity> >::__call<std::__1::__bind<std::__1::shared_ptr<MLDB::PolyEntity> (MLDB::RestConfigurableCollection<MLDB::Utf8String, MLDB::PolyEntity, MLDB::PolyConfig, MLDB::PolyStatus>::*)(MLDB::PolyConfig, std::__1::function<bool (Json::Value const&)> const&, MLDB::WatchT<bool>) const, MLDB::RestConfigurableCollection<MLDB::Utf8String, MLDB::PolyEntity, MLDB::PolyConfig, MLDB::PolyStatus>*, MLDB::PolyConfig, std::__1::placeholders::__ph<1> const&, std::__1::placeholders::__ph<2> const&>&, std::__1::function<bool (Json::Value const&)> const&, MLDB::WatchT<bool> >(std::__1::__bind<std::__1::shared_ptr<MLDB::PolyEntity> (MLDB::RestConfigurableCollection<MLDB::Utf8String, MLDB::PolyEntity, MLDB::PolyConfig, MLDB::PolyStatus>::*)(MLDB::PolyConfig, std::__1::function<bool (Json::Value const&)> const&, MLDB::WatchT<bool>) const, MLDB::RestConfigurableCollection<MLDB::Utf8String, MLDB::PolyEntity, MLDB::PolyConfig, MLDB::PolyStatus>*, MLDB::PolyConfig, std::__1::placeholders::__ph<1> const&, std::__1::placeholders::__ph<2> const&>&, std::__1::function<bool (Json::Value const&)> const&, MLDB::WatchT<bool>&&) __functional_base:317
    #13 0x1089f2310 in MLDB::RestCollection<MLDB::Utf8String, MLDB::PolyEntity>::addBackgroundJobInThread(MLDB::Utf8String, std::__1::function<std::__1::shared_ptr<MLDB::PolyEntity> (std::__1::function<bool (Json::Value const&)> const&, MLDB::WatchT<bool>)> const&, std::__1::function<bool (Json::Value const&)> const&, std::__1::function<void (std::__1::shared_ptr<MLDB::PolyEntity>)> const&, bool, MLDB::Any)::'lambda'()::operator()() const rest_collection_impl.h:586
    #14 0x1089f1968 in void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, MLDB::RestCollection<MLDB::Utf8String, MLDB::PolyEntity>::addBackgroundJobInThread(MLDB::Utf8String, std::__1::function<std::__1::shared_ptr<MLDB::PolyEntity> (std::__1::function<bool (Json::Value const&)> const&, MLDB::WatchT<bool>)> const&, std::__1::function<bool (Json::Value const&)> const&, std::__1::function<void (std::__1::shared_ptr<MLDB::PolyEntity>)> const&, bool, MLDB::Any)::'lambda'()> >(void*) thread:291
    #15 0x18e887874 in _pthread_start+0x13c (libsystem_pthread.dylib:arm64e+0x7874)
    #16 0x18e8825dc in thread_start+0x4 (libsystem_pthread.dylib:arm64e+0x25dc)

Thread T34 created by T7 here:
    #0 0x109b055b0 in wrap_pthread_create+0x54 (libclang_rt.asan_osx_dynamic.dylib:arm64e+0x395b0)
    #1 0x107257650 in std::__1::thread::thread<MLDB::RestCollection<MLDB::Utf8String, MLDB::ProcedureRun>::addBackgroundJobInThread(MLDB::Utf8String, std::__1::function<std::__1::shared_ptr<MLDB::ProcedureRun> (std::__1::function<bool (Json::Value const&)> const&, MLDB::WatchT<bool>)> const&, std::__1::function<bool (Json::Value const&)> const&, std::__1::function<void (std::__1::shared_ptr<MLDB::ProcedureRun>)> const&, bool, MLDB::Any)::'lambda'()&, void>(MLDB::Utf8String&&, MLDB::ProcedureRun&&...) thread:307
    #2 0x1071c6a78 in MLDB::RestCollection<MLDB::Utf8String, MLDB::ProcedureRun>::addBackgroundJobInThread(MLDB::Utf8String, std::__1::function<std::__1::shared_ptr<MLDB::ProcedureRun> (std::__1::function<bool (Json::Value const&)> const&, MLDB::WatchT<bool>)> const&, std::__1::function<bool (Json::Value const&)> const&, std::__1::function<void (std::__1::shared_ptr<MLDB::ProcedureRun>)> const&, bool, MLDB::Any) rest_collection_impl.h:621
    #3 0x1071e3fc0 in MLDB::RestConfigurableCollection<MLDB::Utf8String, MLDB::ProcedureRun, MLDB::ProcedureRunConfig, MLDB::ProcedureRunStatus>::handlePutItl(MLDB::Utf8String, MLDB::ProcedureRunConfig, std::__1::function<void (std::__1::shared_ptr<MLDB::ProcedureRun>)> const&, bool) rest_collection_impl.h:1767
    #4 0x1071e3324 in MLDB::RestConfigurableCollection<MLDB::Utf8String, MLDB::ProcedureRun, MLDB::ProcedureRunConfig, MLDB::ProcedureRunStatus>::handlePut(MLDB::Utf8String, MLDB::ProcedureRunConfig, bool) rest_collection_impl.h:1795
    #5 0x1071e66fc in MLDB::RestConfigurableCollection<MLDB::Utf8String, MLDB::ProcedureRun, MLDB::ProcedureRunConfig, MLDB::ProcedureRunStatus>::handlePost(MLDB::Utf8String, MLDB::ProcedureRunConfig, bool) rest_collection_impl.h:1852
    #6 0x1072b1d3c in MLDB::RestConfigurableCollection<MLDB::Utf8String, MLDB::ProcedureRun, MLDB::ProcedureRunConfig, MLDB::ProcedureRunStatus>::RouteManager::addPostRoute()::'lambda0'(MLDB::RestConnection&, MLDB::RestRequest const&, MLDB::RestRequestParsingContext const&)::operator()(MLDB::RestConnection&, MLDB::RestRequest const&, MLDB::RestRequestParsingContext const&) const rest_collection_impl.h:1649
    #7 0x107c0109c in MLDB::RestRequestRouter::processRequest(MLDB::RestConnection&, MLDB::RestRequest const&, MLDB::RestRequestParsingContext&) const rest_request_router.cc:386
    #8 0x107c036c4 in MLDB::RestRequestRouter::Route::process(MLDB::RestRequest const&, MLDB::RestRequestParsingContext&, MLDB::RestConnection&) const rest_request_router.cc:521
    #9 0x107c00b04 in MLDB::RestRequestRouter::processRequest(MLDB::RestConnection&, MLDB::RestRequest const&, MLDB::RestRequestParsingContext&) const rest_request_router.cc:393
    #10 0x107c036c4 in MLDB::RestRequestRouter::Route::process(MLDB::RestRequest const&, MLDB::RestRequestParsingContext&, MLDB::RestConnection&) const rest_request_router.cc:521
    #11 0x107c00b04 in MLDB::RestRequestRouter::processRequest(MLDB::RestConnection&, MLDB::RestRequest const&, MLDB::RestRequestParsingContext&) const rest_request_router.cc:393
    #12 0x107c036c4 in MLDB::RestRequestRouter::Route::process(MLDB::RestRequest const&, MLDB::RestRequestParsingContext&, MLDB::RestConnection&) const rest_request_router.cc:521
    #13 0x107c00b04 in MLDB::RestRequestRouter::processRequest(MLDB::RestConnection&, MLDB::RestRequest const&, MLDB::RestRequestParsingContext&) const rest_request_router.cc:393
    #14 0x107c036c4 in MLDB::RestRequestRouter::Route::process(MLDB::RestRequest const&, MLDB::RestRequestParsingContext&, MLDB::RestConnection&) const rest_request_router.cc:521
    #15 0x107c00b04 in MLDB::RestRequestRouter::processRequest(MLDB::RestConnection&, MLDB::RestRequest const&, MLDB::RestRequestParsingContext&) const rest_request_router.cc:393
    #16 0x107c036c4 in MLDB::RestRequestRouter::Route::process(MLDB::RestRequest const&, MLDB::RestRequestParsingContext&, MLDB::RestConnection&) const rest_request_router.cc:521
    #17 0x107c00b04 in MLDB::RestRequestRouter::processRequest(MLDB::RestConnection&, MLDB::RestRequest const&, MLDB::RestRequestParsingContext&) const rest_request_router.cc:393
    #18 0x107bffeec in MLDB::RestRequestRouter::handleRequest(MLDB::RestConnection&, MLDB::RestRequest const&) const rest_request_router.cc:326
    #19 0x1071ac7b4 in MLDB::ProcedureCollection::handlePutWithFirstRun(MLDB::Utf8String, MLDB::PolyConfig, bool, bool) procedure_collection.cc:176
    #20 0x1071ab344 in MLDB::ProcedureCollection::handlePut(MLDB::Utf8String, MLDB::PolyConfig, bool) procedure_collection.cc:148
    #21 0x108a060b8 in MLDB::RestConfigurableCollection<MLDB::Utf8String, MLDB::PolyEntity, MLDB::PolyConfig, MLDB::PolyStatus>::RouteManager::addPutRoute()::'lambda'(MLDB::RestConnection&, MLDB::RestRequest const&, MLDB::RestRequestParsingContext const&)::operator()(MLDB::RestConnection&, MLDB::RestRequest const&, MLDB::RestRequestParsingContext const&) const rest_collection_impl.h:1508
    #22 0x107c0109c in MLDB::RestRequestRouter::processRequest(MLDB::RestConnection&, MLDB::RestRequest const&, MLDB::RestRequestParsingContext&) const rest_request_router.cc:386
    #23 0x107c036c4 in MLDB::RestRequestRouter::Route::process(MLDB::RestRequest const&, MLDB::RestRequestParsingContext&, MLDB::RestConnection&) const rest_request_router.cc:521
    #24 0x107c00b04 in MLDB::RestRequestRouter::processRequest(MLDB::RestConnection&, MLDB::RestRequest const&, MLDB::RestRequestParsingContext&) const rest_request_router.cc:393
    #25 0x107c036c4 in MLDB::RestRequestRouter::Route::process(MLDB::RestRequest const&, MLDB::RestRequestParsingContext&, MLDB::RestConnection&) const rest_request_router.cc:521
    #26 0x107c00b04 in MLDB::RestRequestRouter::processRequest(MLDB::RestConnection&, MLDB::RestRequest const&, MLDB::RestRequestParsingContext&) const rest_request_router.cc:393
    #27 0x107c036c4 in MLDB::RestRequestRouter::Route::process(MLDB::RestRequest const&, MLDB::RestRequestParsingContext&, MLDB::RestConnection&) const rest_request_router.cc:521
    #28 0x107c00b04 in MLDB::RestRequestRouter::processRequest(MLDB::RestConnection&, MLDB::RestRequest const&, MLDB::RestRequestParsingContext&) const rest_request_router.cc:393
    #29 0x107c036c4 in MLDB::RestRequestRouter::Route::process(MLDB::RestRequest const&, MLDB::RestRequestParsingContext&, MLDB::RestConnection&) const rest_request_router.cc:521
    #30 0x107c00b04 in MLDB::RestRequestRouter::processRequest(MLDB::RestConnection&, MLDB::RestRequest const&, MLDB::RestRequestParsingContext&) const rest_request_router.cc:393
    #31 0x107bffeec in MLDB::RestRequestRouter::handleRequest(MLDB::RestConnection&, MLDB::RestRequest const&) const rest_request_router.cc:326
    #32 0x1115a4cb8 in MLDB::MldbPythonContext::perform(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MLDB::RestParams const&, Json::Value, MLDB::RestParams const&) python_plugin_context.cc:730
    #33 0x11156a8ec in _object* boost::python::detail::invoke<boost::python::to_python_value<Json::Value const&>, Json::Value (MLDB::MldbPythonContext::*)(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MLDB::RestParams const&, Json::Value, MLDB::RestParams const&), boost::python::arg_from_python<MLDB::MldbPythonContext&>, boost::python::arg_from_python<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&>, boost::python::arg_from_python<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&>, boost::python::arg_from_python<MLDB::RestParams const&>, boost::python::arg_from_python<Json::Value>, boost::python::arg_from_python<MLDB::RestParams const&> >(boost::python::detail::invoke_tag_<false, true>, boost::python::to_python_value<Json::Value const&> const&, Json::Value (MLDB::MldbPythonContext::*&)(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MLDB::RestParams const&, Json::Value, MLDB::RestParams const&), boost::python::arg_from_python<MLDB::MldbPythonContext&>&, boost::python::arg_from_python<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&>&, boost::python::arg_from_python<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&>&, boost::python::arg_from_python<MLDB::RestParams const&>&, boost::python::arg_from_python<Json::Value>&, boost::python::arg_from_python<MLDB::RestParams const&>&) invoke.hpp:86
    #34 0x111569ff8 in boost::python::detail::caller_arity<6u>::impl<Json::Value (MLDB::MldbPythonContext::*)(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MLDB::RestParams const&, Json::Value, MLDB::RestParams const&), boost::python::default_call_policies, boost::mpl::vector7<Json::Value, MLDB::MldbPythonContext&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MLDB::RestParams const&, Json::Value, MLDB::RestParams const&> >::operator()(_object*, _object*) caller.hpp:233
    #35 0x11392eebc in boost::python::objects::function::call(_object*, _object*) const+0x2b8 (libboost_python39.dylib:arm64+0xeebc)
    #36 0x113931718 in boost::detail::function::void_function_ref_invoker0<boost::python::objects::(anonymous namespace)::bind_return, void>::invoke(boost::detail::function::function_buffer&)+0x18 (libboost_python39.dylib:arm64+0x11718)
    #37 0x1139363a8 in boost::python::handle_exception_impl(boost::function0<void>)+0x5c (libboost_python39.dylib:arm64+0x163a8)
    #38 0x113931200 in boost::python::objects::function_call(_object*, _object*, _object*)+0x44 (libboost_python39.dylib:arm64+0x11200)
    #39 0x1135eba54 in _PyObject_MakeTpCall+0x80 (Python:arm64+0x63a54)
    #40 0x1135ef35c in method_vectorcall+0x278 (Python:arm64+0x6735c)
    #41 0x1136e414c in _PyEval_EvalFrameDefault+0x9d5c (Python:arm64+0x15c14c)
    #42 0x1136d93ec in _PyEval_EvalCode+0x1b8 (Python:arm64+0x1513ec)
    #43 0x1135ec778 in _PyFunction_Vectorcall+0x168 (Python:arm64+0x64778)
    #44 0x1136e658c in call_function+0x7c (Python:arm64+0x15e58c)
    #45 0x1136e3f04 in _PyEval_EvalFrameDefault+0x9b14 (Python:arm64+0x15bf04)
    #46 0x1136d93ec in _PyEval_EvalCode+0x1b8 (Python:arm64+0x1513ec)
    #47 0x1135ec778 in _PyFunction_Vectorcall+0x168 (Python:arm64+0x64778)
    #48 0x1135ef15c in method_vectorcall+0x78 (Python:arm64+0x6715c)
    #49 0x1135ec304 in PyVectorcall_Call+0xb4 (Python:arm64+0x64304)
    #50 0x113780aa0 in partial_call+0x64 (Python:arm64+0x1f8aa0)
    #51 0x1135eba54 in _PyObject_MakeTpCall+0x80 (Python:arm64+0x63a54)
    #52 0x1136e658c in call_function+0x7c (Python:arm64+0x15e58c)
    #53 0x1136e3f80 in _PyEval_EvalFrameDefault+0x9b90 (Python:arm64+0x15bf80)
    #54 0x1135ec6c0 in _PyFunction_Vectorcall+0xb0 (Python:arm64+0x646c0)
    #55 0x1135ef15c in method_vectorcall+0x78 (Python:arm64+0x6715c)
    #56 0x1136e658c in call_function+0x7c (Python:arm64+0x15e58c)
    #57 0x1136e3fa4 in _PyEval_EvalFrameDefault+0x9bb4 (Python:arm64+0x15bfa4)
    #58 0x1135ec6c0 in _PyFunction_Vectorcall+0xb0 (Python:arm64+0x646c0)
    #59 0x1136e658c in call_function+0x7c (Python:arm64+0x15e58c)
    #60 0x1136e3f04 in _PyEval_EvalFrameDefault+0x9b14 (Python:arm64+0x15bf04)
    #61 0x1136d93ec in _PyEval_EvalCode+0x1b8 (Python:arm64+0x1513ec)
    #62 0x1135ec778 in _PyFunction_Vectorcall+0x168 (Python:arm64+0x64778)
    #63 0x1135ef200 in method_vectorcall+0x11c (Python:arm64+0x67200)
    #64 0x1136e414c in _PyEval_EvalFrameDefault+0x9d5c (Python:arm64+0x15c14c)
    #65 0x1136d93ec in _PyEval_EvalCode+0x1b8 (Python:arm64+0x1513ec)
    #66 0x1135ec778 in _PyFunction_Vectorcall+0x168 (Python:arm64+0x64778)
    #67 0x1135ebd00 in _PyObject_FastCallDictTstate+0x5c (Python:arm64+0x63d00)
    #68 0x11365ea38 in slot_tp_call+0xb8 (Python:arm64+0xd6a38)
    #69 0x1135eba54 in _PyObject_MakeTpCall+0x80 (Python:arm64+0x63a54)
    #70 0x1136e6618 in call_function+0x108 (Python:arm64+0x15e618)
    #71 0x1136e3fa4 in _PyEval_EvalFrameDefault+0x9bb4 (Python:arm64+0x15bfa4)
    #72 0x1136d93ec in _PyEval_EvalCode+0x1b8 (Python:arm64+0x1513ec)
    #73 0x1135ec778 in _PyFunction_Vectorcall+0x168 (Python:arm64+0x64778)
    #74 0x1135ef200 in method_vectorcall+0x11c (Python:arm64+0x67200)
    #75 0x1136e414c in _PyEval_EvalFrameDefault+0x9d5c (Python:arm64+0x15c14c)
    #76 0x1136d93ec in _PyEval_EvalCode+0x1b8 (Python:arm64+0x1513ec)
    #77 0x1135ec778 in _PyFunction_Vectorcall+0x168 (Python:arm64+0x64778)
    #78 0x1135ebd00 in _PyObject_FastCallDictTstate+0x5c (Python:arm64+0x63d00)
    #79 0x11365ea38 in slot_tp_call+0xb8 (Python:arm64+0xd6a38)
    #80 0x1135eba54 in _PyObject_MakeTpCall+0x80 (Python:arm64+0x63a54)
    #81 0x1136e6618 in call_function+0x108 (Python:arm64+0x15e618)
    #82 0x1136e3fa4 in _PyEval_EvalFrameDefault+0x9bb4 (Python:arm64+0x15bfa4)
    #83 0x1136d93ec in _PyEval_EvalCode+0x1b8 (Python:arm64+0x1513ec)
    #84 0x1135ec778 in _PyFunction_Vectorcall+0x168 (Python:arm64+0x64778)
    #85 0x1135ef200 in method_vectorcall+0x11c (Python:arm64+0x67200)
    #86 0x1136e414c in _PyEval_EvalFrameDefault+0x9d5c (Python:arm64+0x15c14c)
    #87 0x1136d93ec in _PyEval_EvalCode+0x1b8 (Python:arm64+0x1513ec)
    #88 0x1135ec778 in _PyFunction_Vectorcall+0x168 (Python:arm64+0x64778)
    #89 0x1135ebd00 in _PyObject_FastCallDictTstate+0x5c (Python:arm64+0x63d00)
    #90 0x11365ea38 in slot_tp_call+0xb8 (Python:arm64+0xd6a38)
    #91 0x1135eba54 in _PyObject_MakeTpCall+0x80 (Python:arm64+0x63a54)
    #92 0x1136e6618 in call_function+0x108 (Python:arm64+0x15e618)
    #93 0x1136e3fa4 in _PyEval_EvalFrameDefault+0x9bb4 (Python:arm64+0x15bfa4)
    #94 0x1135ec6c0 in _PyFunction_Vectorcall+0xb0 (Python:arm64+0x646c0)
    #95 0x1136e658c in call_function+0x7c (Python:arm64+0x15e58c)
    #96 0x1136e3f04 in _PyEval_EvalFrameDefault+0x9b14 (Python:arm64+0x15bf04)
    #97 0x1135ec6c0 in _PyFunction_Vectorcall+0xb0 (Python:arm64+0x646c0)
    #98 0x1136e658c in call_function+0x7c (Python:arm64+0x15e58c)
    #99 0x1136e3f04 in _PyEval_EvalFrameDefault+0x9b14 (Python:arm64+0x15bf04)
    #100 0x1136d93ec in _PyEval_EvalCode+0x1b8 (Python:arm64+0x1513ec)
    #101 0x1135ec778 in _PyFunction_Vectorcall+0x168 (Python:arm64+0x64778)
    #102 0x1135ebd70 in _PyObject_FastCallDictTstate+0xcc (Python:arm64+0x63d70)
    #103 0x11366005c in slot_tp_init+0xb8 (Python:arm64+0xd805c)
    #104 0x113665cb8 in type_call+0x128 (Python:arm64+0xddcb8)
    #105 0x1135eba54 in _PyObject_MakeTpCall+0x80 (Python:arm64+0x63a54)
    #106 0x1136e6618 in call_function+0x108 (Python:arm64+0x15e618)
    #107 0x1136e4028 in _PyEval_EvalFrameDefault+0x9c38 (Python:arm64+0x15c028)
    #108 0x1135ec6c0 in _PyFunction_Vectorcall+0xb0 (Python:arm64+0x646c0)
    #109 0x1136e658c in call_function+0x7c (Python:arm64+0x15e58c)
    #110 0x1136e3f04 in _PyEval_EvalFrameDefault+0x9b14 (Python:arm64+0x15bf04)
    #111 0x1136d93ec in _PyEval_EvalCode+0x1b8 (Python:arm64+0x1513ec)
    #112 0x113730ce0 in run_eval_code_obj+0x84 (Python:arm64+0x1a8ce0)
    #113 0x113730bf4 in run_mod+0x6c (Python:arm64+0x1a8bf4)
    #114 0x11372e430 in pyrun_file+0xa4 (Python:arm64+0x1a6430)
    #115 0x113731d28 in PyRun_FileExFlags+0x6c (Python:arm64+0x1a9d28)
    #116 0x11398f418 in MLDB::PythonThread::exec(MLDB::EnterThreadToken const&, MLDB::Utf8String const&, MLDB::Utf8String const&, boost::python::api::object, boost::python::api::object) python_interpreter.cc:318
    #117 0x11159aadc in MLDB::MldbPythonInterpreter::runPythonScript(MLDB::EnterThreadToken const&, MLDB::Utf8String, MLDB::Utf8String, boost::python::api::object, boost::python::api::object) python_plugin_context.cc:360
    #118 0x111547f64 in MLDB::PythonPlugin::handleTypeRoute(MLDB::RestDirectory*, MLDB::RestConnection&, MLDB::RestRequest const&, MLDB::RestRequestParsingContext&) python_plugin.cc:356
    #119 0x1074ce55c in MLDB::PolyCollection<MLDB::Plugin>::Registry::handleCustomRequest(MLDB::RestDirectory*, MLDB::Utf8String const&, MLDB::RestConnection&, MLDB::RestRequest const&, MLDB::RestRequestParsingContext const&) poly_collection_impl.h:311
    #120 0x1074a04c8 in MLDB::RestRequestMatchResult std::__1::__invoke_void_return_wrapper<MLDB::RestRequestMatchResult>::__call<MLDB::TypeCollection<MLDB::Plugin>::initRoutes(MLDB::RestCollection<MLDB::Utf8String, MLDB::TypeEntry>::RouteManager&)::'lambda1'(MLDB::RestConnection&, MLDB::RestRequest const&, MLDB::RestRequestParsingContext const&)&, MLDB::RestConnection&, MLDB::RestRequest const&, MLDB::RestRequestParsingContext&>(MLDB::Plugin&&...) __functional_base:317
    #121 0x107c0109c in MLDB::RestRequestRouter::processRequest(MLDB::RestConnection&, MLDB::RestRequest const&, MLDB::RestRequestParsingContext&) const rest_request_router.cc:386
    #122 0x107c036c4 in MLDB::RestRequestRouter::Route::process(MLDB::RestRequest const&, MLDB::RestRequestParsingContext&, MLDB::RestConnection&) const rest_request_router.cc:521
    #123 0x107c00b04 in MLDB::RestRequestRouter::processRequest(MLDB::RestConnection&, MLDB::RestRequest const&, MLDB::RestRequestParsingContext&) const rest_request_router.cc:393
    #124 0x107c036c4 in MLDB::RestRequestRouter::Route::process(MLDB::RestRequest const&, MLDB::RestRequestParsingContext&, MLDB::RestConnection&) const rest_request_router.cc:521
    #125 0x107c00b04 in MLDB::RestRequestRouter::processRequest(MLDB::RestConnection&, MLDB::RestRequest const&, MLDB::RestRequestParsingContext&) const rest_request_router.cc:393
    #126 0x107c036c4 in MLDB::RestRequestRouter::Route::process(MLDB::RestRequest const&, MLDB::RestRequestParsingContext&, MLDB::RestConnection&) const rest_request_router.cc:521
    #127 0x107c00b04 in MLDB::RestRequestRouter::processRequest(MLDB::RestConnection&, MLDB::RestRequest const&, MLDB::RestRequestParsingContext&) const rest_request_router.cc:393
    #128 0x107c036c4 in MLDB::RestRequestRouter::Route::process(MLDB::RestRequest const&, MLDB::RestRequestParsingContext&, MLDB::RestConnection&) const rest_request_router.cc:521
    #129 0x107c00b04 in MLDB::RestRequestRouter::processRequest(MLDB::RestConnection&, MLDB::RestRequest const&, MLDB::RestRequestParsingContext&) const rest_request_router.cc:393
    #130 0x107c036c4 in MLDB::RestRequestRouter::Route::process(MLDB::RestRequest const&, MLDB::RestRequestParsingContext&, MLDB::RestConnection&) const rest_request_router.cc:521
    #131 0x107c00b04 in MLDB::RestRequestRouter::processRequest(MLDB::RestConnection&, MLDB::RestRequest const&, MLDB::RestRequestParsingContext&) const rest_request_router.cc:393
    #132 0x107bffeec in MLDB::RestRequestRouter::handleRequest(MLDB::RestConnection&, MLDB::RestRequest const&) const rest_request_router.cc:326
    #133 0x107c7b338 in std::__1::__function::__func<MLDB::HttpRestService::init()::$_1, std::__1::allocator<MLDB::HttpRestService::init()::$_1>, void (std::__1::shared_ptr<MLDB::HttpRestEndpoint::RestConnectionHandler>, MLDB::HttpHeader const&, MLDB::Utf8String const&)>::operator()(std::__1::shared_ptr<MLDB::HttpRestEndpoint::RestConnectionHandler>&&, MLDB::HttpHeader const&, MLDB::Utf8String const&) functional:1727
    #134 0x107c54658 in MLDB::HttpRestEndpoint::RestConnectionHandler::handleHttpPayload(MLDB::HttpHeader const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) http_rest_endpoint.cc:151
    #135 0x109801610 in MLDB::HttpLegacySocketHandler::onDone(bool) http_socket_handler.cc:269
    #136 0x10983a8a0 in MLDB::HttpParser::feed(char const*, unsigned long) http_parsers.cc:204
    #137 0x1097fe388 in MLDB::HttpSocketHandler::onReceivedData(char const*, unsigned long) http_socket_handler.cc:106
    #138 0x10999cc7c in boost::asio::detail::reactive_socket_recv_op<boost::asio::mutable_buffers_1, std::__1::function<void (boost::system::error_code const&, unsigned long)>, boost::asio::any_io_executor>::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long) reactive_socket_recv_op.hpp:145
    #139 0x10990f16c in boost::asio::detail::scheduler::do_run_one(boost::asio::detail::conditionally_enabled_mutex::scoped_lock&, boost::asio::detail::scheduler_thread_info&, boost::system::error_code const&) scheduler.ipp:486
    #140 0x10990e1cc in boost::asio::detail::scheduler::run(boost::system::error_code&) scheduler.ipp:204
    #141 0x10990da7c in MLDB::AsioThreadPool::Impl::run(int) asio_thread_pool.cc:84
    #142 0x109911e88 in void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, MLDB::AsioThreadPool::Impl::ensureThreads(int)::'lambda'()> >(void*) thread:291
    #143 0x18e887874 in _pthread_start+0x13c (libsystem_pthread.dylib:arm64e+0x7874)
    #144 0x18e8825dc in thread_start+0x4 (libsystem_pthread.dylib:arm64e+0x25dc)

Thread T7 created by T0 here:
    #0 0x109b055b0 in wrap_pthread_create+0x54 (libclang_rt.asan_osx_dynamic.dylib:arm64e+0x395b0)
    #1 0x109911d34 in std::__1::thread::thread<MLDB::AsioThreadPool::Impl::ensureThreads(int)::'lambda'(), void>(MLDB::AsioThreadPool::Impl::ensureThreads(int)::'lambda'()&&) thread:307
    #2 0x1099054b8 in MLDB::AsioThreadPool::Impl::ensureThreads(int) asio_thread_pool.cc:66
    #3 0x104961a44 in main mldb_runner.cc:348
    #4 0x18e8a544c in start+0x0 (libdyld.dylib:arm64e+0x1844c)

Thread T33 created by T7 here:
    #0 0x109b055b0 in wrap_pthread_create+0x54 (libclang_rt.asan_osx_dynamic.dylib:arm64e+0x395b0)
    #1 0x1089f1560 in std::__1::thread::thread<MLDB::RestCollection<MLDB::Utf8String, MLDB::PolyEntity>::addBackgroundJobInThread(MLDB::Utf8String, std::__1::function<std::__1::shared_ptr<MLDB::PolyEntity> (std::__1::function<bool (Json::Value const&)> const&, MLDB::WatchT<bool>)> const&, std::__1::function<bool (Json::Value const&)> const&, std::__1::function<void (std::__1::shared_ptr<MLDB::PolyEntity>)> const&, bool, MLDB::Any)::'lambda'()&, void>(MLDB::Utf8String&&, MLDB::PolyEntity&&...) thread:307
    #2 0x10894d008 in MLDB::RestCollection<MLDB::Utf8String, MLDB::PolyEntity>::addBackgroundJobInThread(MLDB::Utf8String, std::__1::function<std::__1::shared_ptr<MLDB::PolyEntity> (std::__1::function<bool (Json::Value const&)> const&, MLDB::WatchT<bool>)> const&, std::__1::function<bool (Json::Value const&)> const&, std::__1::function<void (std::__1::shared_ptr<MLDB::PolyEntity>)> const&, bool, MLDB::Any) rest_collection_impl.h:621
    #3 0x10896aaa8 in MLDB::RestConfigurableCollection<MLDB::Utf8String, MLDB::PolyEntity, MLDB::PolyConfig, MLDB::PolyStatus>::handlePutItl(MLDB::Utf8String, MLDB::PolyConfig, std::__1::function<void (std::__1::shared_ptr<MLDB::PolyEntity>)> const&, bool) rest_collection_impl.h:1767
    #4 0x10896c76c in MLDB::RestConfigurableCollection<MLDB::Utf8String, MLDB::PolyEntity, MLDB::PolyConfig, MLDB::PolyStatus>::handlePutSync(MLDB::Utf8String, MLDB::PolyConfig, bool) rest_collection_impl.h:1807
    #5 0x1071ac128 in MLDB::ProcedureCollection::handlePutWithFirstRun(MLDB::Utf8String, MLDB::PolyConfig, bool, bool) procedure_collection.cc:164
    #6 0x1071ab344 in MLDB::ProcedureCollection::handlePut(MLDB::Utf8String, MLDB::PolyConfig, bool) procedure_collection.cc:148
    #7 0x108a060b8 in MLDB::RestConfigurableCollection<MLDB::Utf8String, MLDB::PolyEntity, MLDB::PolyConfig, MLDB::PolyStatus>::RouteManager::addPutRoute()::'lambda'(MLDB::RestConnection&, MLDB::RestRequest const&, MLDB::RestRequestParsingContext const&)::operator()(MLDB::RestConnection&, MLDB::RestRequest const&, MLDB::RestRequestParsingContext const&) const rest_collection_impl.h:1508
    #8 0x107c0109c in MLDB::RestRequestRouter::processRequest(MLDB::RestConnection&, MLDB::RestRequest const&, MLDB::RestRequestParsingContext&) const rest_request_router.cc:386
    #9 0x107c036c4 in MLDB::RestRequestRouter::Route::process(MLDB::RestRequest const&, MLDB::RestRequestParsingContext&, MLDB::RestConnection&) const rest_request_router.cc:521
    #10 0x107c00b04 in MLDB::RestRequestRouter::processRequest(MLDB::RestConnection&, MLDB::RestRequest const&, MLDB::RestRequestParsingContext&) const rest_request_router.cc:393
    #11 0x107c036c4 in MLDB::RestRequestRouter::Route::process(MLDB::RestRequest const&, MLDB::RestRequestParsingContext&, MLDB::RestConnection&) const rest_request_router.cc:521
    #12 0x107c00b04 in MLDB::RestRequestRouter::processRequest(MLDB::RestConnection&, MLDB::RestRequest const&, MLDB::RestRequestParsingContext&) const rest_request_router.cc:393
    #13 0x107c036c4 in MLDB::RestRequestRouter::Route::process(MLDB::RestRequest const&, MLDB::RestRequestParsingContext&, MLDB::RestConnection&) const rest_request_router.cc:521
    #14 0x107c00b04 in MLDB::RestRequestRouter::processRequest(MLDB::RestConnection&, MLDB::RestRequest const&, MLDB::RestRequestParsingContext&) const rest_request_router.cc:393
    #15 0x107c036c4 in MLDB::RestRequestRouter::Route::process(MLDB::RestRequest const&, MLDB::RestRequestParsingContext&, MLDB::RestConnection&) const rest_request_router.cc:521
    #16 0x107c00b04 in MLDB::RestRequestRouter::processRequest(MLDB::RestConnection&, MLDB::RestRequest const&, MLDB::RestRequestParsingContext&) const rest_request_router.cc:393
    #17 0x107bffeec in MLDB::RestRequestRouter::handleRequest(MLDB::RestConnection&, MLDB::RestRequest const&) const rest_request_router.cc:326
    #18 0x1115a4cb8 in MLDB::MldbPythonContext::perform(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MLDB::RestParams const&, Json::Value, MLDB::RestParams const&) python_plugin_context.cc:730
    #19 0x11156a8ec in _object* boost::python::detail::invoke<boost::python::to_python_value<Json::Value const&>, Json::Value (MLDB::MldbPythonContext::*)(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MLDB::RestParams const&, Json::Value, MLDB::RestParams const&), boost::python::arg_from_python<MLDB::MldbPythonContext&>, boost::python::arg_from_python<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&>, boost::python::arg_from_python<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&>, boost::python::arg_from_python<MLDB::RestParams const&>, boost::python::arg_from_python<Json::Value>, boost::python::arg_from_python<MLDB::RestParams const&> >(boost::python::detail::invoke_tag_<false, true>, boost::python::to_python_value<Json::Value const&> const&, Json::Value (MLDB::MldbPythonContext::*&)(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MLDB::RestParams const&, Json::Value, MLDB::RestParams const&), boost::python::arg_from_python<MLDB::MldbPythonContext&>&, boost::python::arg_from_python<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&>&, boost::python::arg_from_python<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&>&, boost::python::arg_from_python<MLDB::RestParams const&>&, boost::python::arg_from_python<Json::Value>&, boost::python::arg_from_python<MLDB::RestParams const&>&) invoke.hpp:86
    #20 0x111569ff8 in boost::python::detail::caller_arity<6u>::impl<Json::Value (MLDB::MldbPythonContext::*)(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MLDB::RestParams const&, Json::Value, MLDB::RestParams const&), boost::python::default_call_policies, boost::mpl::vector7<Json::Value, MLDB::MldbPythonContext&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, MLDB::RestParams const&, Json::Value, MLDB::RestParams const&> >::operator()(_object*, _object*) caller.hpp:233
    #21 0x11392eebc in boost::python::objects::function::call(_object*, _object*) const+0x2b8 (libboost_python39.dylib:arm64+0xeebc)
    #22 0x113931718 in boost::detail::function::void_function_ref_invoker0<boost::python::objects::(anonymous namespace)::bind_return, void>::invoke(boost::detail::function::function_buffer&)+0x18 (libboost_python39.dylib:arm64+0x11718)
    #23 0x1139363a8 in boost::python::handle_exception_impl(boost::function0<void>)+0x5c (libboost_python39.dylib:arm64+0x163a8)
    #24 0x113931200 in boost::python::objects::function_call(_object*, _object*, _object*)+0x44 (libboost_python39.dylib:arm64+0x11200)
    #25 0x1135eba54 in _PyObject_MakeTpCall+0x80 (Python:arm64+0x63a54)
    #26 0x1135ef35c in method_vectorcall+0x278 (Python:arm64+0x6735c)
    #27 0x1136e414c in _PyEval_EvalFrameDefault+0x9d5c (Python:arm64+0x15c14c)
    #28 0x1136d93ec in _PyEval_EvalCode+0x1b8 (Python:arm64+0x1513ec)
    #29 0x1135ec778 in _PyFunction_Vectorcall+0x168 (Python:arm64+0x64778)
    #30 0x1136e658c in call_function+0x7c (Python:arm64+0x15e58c)
    #31 0x1136e3f04 in _PyEval_EvalFrameDefault+0x9b14 (Python:arm64+0x15bf04)
    #32 0x1136d93ec in _PyEval_EvalCode+0x1b8 (Python:arm64+0x1513ec)
    #33 0x1135ec778 in _PyFunction_Vectorcall+0x168 (Python:arm64+0x64778)
    #34 0x1135ef15c in method_vectorcall+0x78 (Python:arm64+0x6715c)
    #35 0x1135ec304 in PyVectorcall_Call+0xb4 (Python:arm64+0x64304)
    #36 0x113780aa0 in partial_call+0x64 (Python:arm64+0x1f8aa0)
    #37 0x1135eba54 in _PyObject_MakeTpCall+0x80 (Python:arm64+0x63a54)
    #38 0x1136e658c in call_function+0x7c (Python:arm64+0x15e58c)
    #39 0x1136e3f80 in _PyEval_EvalFrameDefault+0x9b90 (Python:arm64+0x15bf80)
    #40 0x1135ec6c0 in _PyFunction_Vectorcall+0xb0 (Python:arm64+0x646c0)
    #41 0x1135ef15c in method_vectorcall+0x78 (Python:arm64+0x6715c)
    #42 0x1136e658c in call_function+0x7c (Python:arm64+0x15e58c)
    #43 0x1136e3fa4 in _PyEval_EvalFrameDefault+0x9bb4 (Python:arm64+0x15bfa4)
    #44 0x1135ec6c0 in _PyFunction_Vectorcall+0xb0 (Python:arm64+0x646c0)
    #45 0x1136e658c in call_function+0x7c (Python:arm64+0x15e58c)
    #46 0x1136e3f04 in _PyEval_EvalFrameDefault+0x9b14 (Python:arm64+0x15bf04)
    #47 0x1136d93ec in _PyEval_EvalCode+0x1b8 (Python:arm64+0x1513ec)
    #48 0x1135ec778 in _PyFunction_Vectorcall+0x168 (Python:arm64+0x64778)
    #49 0x1135ef200 in method_vectorcall+0x11c (Python:arm64+0x67200)
    #50 0x1136e414c in _PyEval_EvalFrameDefault+0x9d5c (Python:arm64+0x15c14c)
    #51 0x1136d93ec in _PyEval_EvalCode+0x1b8 (Python:arm64+0x1513ec)
    #52 0x1135ec778 in _PyFunction_Vectorcall+0x168 (Python:arm64+0x64778)
    #53 0x1135ebd00 in _PyObject_FastCallDictTstate+0x5c (Python:arm64+0x63d00)
    #54 0x11365ea38 in slot_tp_call+0xb8 (Python:arm64+0xd6a38)
    #55 0x1135eba54 in _PyObject_MakeTpCall+0x80 (Python:arm64+0x63a54)
    #56 0x1136e6618 in call_function+0x108 (Python:arm64+0x15e618)
    #57 0x1136e3fa4 in _PyEval_EvalFrameDefault+0x9bb4 (Python:arm64+0x15bfa4)
    #58 0x1136d93ec in _PyEval_EvalCode+0x1b8 (Python:arm64+0x1513ec)
    #59 0x1135ec778 in _PyFunction_Vectorcall+0x168 (Python:arm64+0x64778)
    #60 0x1135ef200 in method_vectorcall+0x11c (Python:arm64+0x67200)
    #61 0x1136e414c in _PyEval_EvalFrameDefault+0x9d5c (Python:arm64+0x15c14c)
    #62 0x1136d93ec in _PyEval_EvalCode+0x1b8 (Python:arm64+0x1513ec)
    #63 0x1135ec778 in _PyFunction_Vectorcall+0x168 (Python:arm64+0x64778)
    #64 0x1135ebd00 in _PyObject_FastCallDictTstate+0x5c (Python:arm64+0x63d00)
    #65 0x11365ea38 in slot_tp_call+0xb8 (Python:arm64+0xd6a38)
    #66 0x1135eba54 in _PyObject_MakeTpCall+0x80 (Python:arm64+0x63a54)
    #67 0x1136e6618 in call_function+0x108 (Python:arm64+0x15e618)
    #68 0x1136e3fa4 in _PyEval_EvalFrameDefault+0x9bb4 (Python:arm64+0x15bfa4)
    #69 0x1136d93ec in _PyEval_EvalCode+0x1b8 (Python:arm64+0x1513ec)
    #70 0x1135ec778 in _PyFunction_Vectorcall+0x168 (Python:arm64+0x64778)
    #71 0x1135ef200 in method_vectorcall+0x11c (Python:arm64+0x67200)
    #72 0x1136e414c in _PyEval_EvalFrameDefault+0x9d5c (Python:arm64+0x15c14c)
    #73 0x1136d93ec in _PyEval_EvalCode+0x1b8 (Python:arm64+0x1513ec)
    #74 0x1135ec778 in _PyFunction_Vectorcall+0x168 (Python:arm64+0x64778)
    #75 0x1135ebd00 in _PyObject_FastCallDictTstate+0x5c (Python:arm64+0x63d00)
    #76 0x11365ea38 in slot_tp_call+0xb8 (Python:arm64+0xd6a38)
    #77 0x1135eba54 in _PyObject_MakeTpCall+0x80 (Python:arm64+0x63a54)
    #78 0x1136e6618 in call_function+0x108 (Python:arm64+0x15e618)
    #79 0x1136e3fa4 in _PyEval_EvalFrameDefault+0x9bb4 (Python:arm64+0x15bfa4)
    #80 0x1135ec6c0 in _PyFunction_Vectorcall+0xb0 (Python:arm64+0x646c0)
    #81 0x1136e658c in call_function+0x7c (Python:arm64+0x15e58c)
    #82 0x1136e3f04 in _PyEval_EvalFrameDefault+0x9b14 (Python:arm64+0x15bf04)
    #83 0x1135ec6c0 in _PyFunction_Vectorcall+0xb0 (Python:arm64+0x646c0)
    #84 0x1136e658c in call_function+0x7c (Python:arm64+0x15e58c)
    #85 0x1136e3f04 in _PyEval_EvalFrameDefault+0x9b14 (Python:arm64+0x15bf04)
    #86 0x1136d93ec in _PyEval_EvalCode+0x1b8 (Python:arm64+0x1513ec)
    #87 0x1135ec778 in _PyFunction_Vectorcall+0x168 (Python:arm64+0x64778)
    #88 0x1135ebd70 in _PyObject_FastCallDictTstate+0xcc (Python:arm64+0x63d70)
    #89 0x11366005c in slot_tp_init+0xb8 (Python:arm64+0xd805c)
    #90 0x113665cb8 in type_call+0x128 (Python:arm64+0xddcb8)
    #91 0x1135eba54 in _PyObject_MakeTpCall+0x80 (Python:arm64+0x63a54)
    #92 0x1136e6618 in call_function+0x108 (Python:arm64+0x15e618)
    #93 0x1136e4028 in _PyEval_EvalFrameDefault+0x9c38 (Python:arm64+0x15c028)
    #94 0x1135ec6c0 in _PyFunction_Vectorcall+0xb0 (Python:arm64+0x646c0)
    #95 0x1136e658c in call_function+0x7c (Python:arm64+0x15e58c)
    #96 0x1136e3f04 in _PyEval_EvalFrameDefault+0x9b14 (Python:arm64+0x15bf04)
    #97 0x1136d93ec in _PyEval_EvalCode+0x1b8 (Python:arm64+0x1513ec)
    #98 0x113730ce0 in run_eval_code_obj+0x84 (Python:arm64+0x1a8ce0)
    #99 0x113730bf4 in run_mod+0x6c (Python:arm64+0x1a8bf4)
    #100 0x11372e430 in pyrun_file+0xa4 (Python:arm64+0x1a6430)
    #101 0x113731d28 in PyRun_FileExFlags+0x6c (Python:arm64+0x1a9d28)
    #102 0x11398f418 in MLDB::PythonThread::exec(MLDB::EnterThreadToken const&, MLDB::Utf8String const&, MLDB::Utf8String const&, boost::python::api::object, boost::python::api::object) python_interpreter.cc:318
    #103 0x11159aadc in MLDB::MldbPythonInterpreter::runPythonScript(MLDB::EnterThreadToken const&, MLDB::Utf8String, MLDB::Utf8String, boost::python::api::object, boost::python::api::object) python_plugin_context.cc:360
    #104 0x111547f64 in MLDB::PythonPlugin::handleTypeRoute(MLDB::RestDirectory*, MLDB::RestConnection&, MLDB::RestRequest const&, MLDB::RestRequestParsingContext&) python_plugin.cc:356
    #105 0x1074ce55c in MLDB::PolyCollection<MLDB::Plugin>::Registry::handleCustomRequest(MLDB::RestDirectory*, MLDB::Utf8String const&, MLDB::RestConnection&, MLDB::RestRequest const&, MLDB::RestRequestParsingContext const&) poly_collection_impl.h:311
    #106 0x1074a04c8 in MLDB::RestRequestMatchResult std::__1::__invoke_void_return_wrapper<MLDB::RestRequestMatchResult>::__call<MLDB::TypeCollection<MLDB::Plugin>::initRoutes(MLDB::RestCollection<MLDB::Utf8String, MLDB::TypeEntry>::RouteManager&)::'lambda1'(MLDB::RestConnection&, MLDB::RestRequest const&, MLDB::RestRequestParsingContext const&)&, MLDB::RestConnection&, MLDB::RestRequest const&, MLDB::RestRequestParsingContext&>(MLDB::Plugin&&...) __functional_base:317
    #107 0x107c0109c in MLDB::RestRequestRouter::processRequest(MLDB::RestConnection&, MLDB::RestRequest const&, MLDB::RestRequestParsingContext&) const rest_request_router.cc:386
    #108 0x107c036c4 in MLDB::RestRequestRouter::Route::process(MLDB::RestRequest const&, MLDB::RestRequestParsingContext&, MLDB::RestConnection&) const rest_request_router.cc:521
    #109 0x107c00b04 in MLDB::RestRequestRouter::processRequest(MLDB::RestConnection&, MLDB::RestRequest const&, MLDB::RestRequestParsingContext&) const rest_request_router.cc:393
    #110 0x107c036c4 in MLDB::RestRequestRouter::Route::process(MLDB::RestRequest const&, MLDB::RestRequestParsingContext&, MLDB::RestConnection&) const rest_request_router.cc:521
    #111 0x107c00b04 in MLDB::RestRequestRouter::processRequest(MLDB::RestConnection&, MLDB::RestRequest const&, MLDB::RestRequestParsingContext&) const rest_request_router.cc:393
    #112 0x107c036c4 in MLDB::RestRequestRouter::Route::process(MLDB::RestRequest const&, MLDB::RestRequestParsingContext&, MLDB::RestConnection&) const rest_request_router.cc:521
    #113 0x107c00b04 in MLDB::RestRequestRouter::processRequest(MLDB::RestConnection&, MLDB::RestRequest const&, MLDB::RestRequestParsingContext&) const rest_request_router.cc:393
    #114 0x107c036c4 in MLDB::RestRequestRouter::Route::process(MLDB::RestRequest const&, MLDB::RestRequestParsingContext&, MLDB::RestConnection&) const rest_request_router.cc:521
    #115 0x107c00b04 in MLDB::RestRequestRouter::processRequest(MLDB::RestConnection&, MLDB::RestRequest const&, MLDB::RestRequestParsingContext&) const rest_request_router.cc:393
    #116 0x107c036c4 in MLDB::RestRequestRouter::Route::process(MLDB::RestRequest const&, MLDB::RestRequestParsingContext&, MLDB::RestConnection&) const rest_request_router.cc:521
    #117 0x107c00b04 in MLDB::RestRequestRouter::processRequest(MLDB::RestConnection&, MLDB::RestRequest const&, MLDB::RestRequestParsingContext&) const rest_request_router.cc:393
    #118 0x107bffeec in MLDB::RestRequestRouter::handleRequest(MLDB::RestConnection&, MLDB::RestRequest const&) const rest_request_router.cc:326
    #119 0x107c7b338 in std::__1::__function::__func<MLDB::HttpRestService::init()::$_1, std::__1::allocator<MLDB::HttpRestService::init()::$_1>, void (std::__1::shared_ptr<MLDB::HttpRestEndpoint::RestConnectionHandler>, MLDB::HttpHeader const&, MLDB::Utf8String const&)>::operator()(std::__1::shared_ptr<MLDB::HttpRestEndpoint::RestConnectionHandler>&&, MLDB::HttpHeader const&, MLDB::Utf8String const&) functional:1727
    #120 0x107c54658 in MLDB::HttpRestEndpoint::RestConnectionHandler::handleHttpPayload(MLDB::HttpHeader const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) http_rest_endpoint.cc:151
    #121 0x109801610 in MLDB::HttpLegacySocketHandler::onDone(bool) http_socket_handler.cc:269
    #122 0x10983a8a0 in MLDB::HttpParser::feed(char const*, unsigned long) http_parsers.cc:204
    #123 0x1097fe388 in MLDB::HttpSocketHandler::onReceivedData(char const*, unsigned long) http_socket_handler.cc:106
    #124 0x10999cc7c in boost::asio::detail::reactive_socket_recv_op<boost::asio::mutable_buffers_1, std::__1::function<void (boost::system::error_code const&, unsigned long)>, boost::asio::any_io_executor>::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long) reactive_socket_recv_op.hpp:145
    #125 0x10990f16c in boost::asio::detail::scheduler::do_run_one(boost::asio::detail::conditionally_enabled_mutex::scoped_lock&, boost::asio::detail::scheduler_thread_info&, boost::system::error_code const&) scheduler.ipp:486
    #126 0x10990e1cc in boost::asio::detail::scheduler::run(boost::system::error_code&) scheduler.ipp:204
    #127 0x10990da7c in MLDB::AsioThreadPool::Impl::run(int) asio_thread_pool.cc:84
    #128 0x109911e88 in void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, MLDB::AsioThreadPool::Impl::ensureThreads(int)::'lambda'()> >(void*) thread:291
    #129 0x18e887874 in _pthread_start+0x13c (libsystem_pthread.dylib:arm64e+0x7874)
    #130 0x18e8825dc in thread_start+0x4 (libsystem_pthread.dylib:arm64e+0x25dc)

SUMMARY: AddressSanitizer: heap-use-after-free sql_expression.cc:3616 in MLDB::SelectExpressionDescription::printJsonTyped(MLDB::SelectExpression const*, MLDB::JsonPrintingContext&) const
Shadow bytes around the buggy address:
  0x0070230894f0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x007023089500: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x007023089510: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x007023089520: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x007023089530: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd
=>0x007023089540: fd[fd]fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x007023089550: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x007023089560: fd fd fd fd fd fa fa fa fa fa fa fa fa fa fa fa
  0x007023089570: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x007023089580: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x007023089590: 00 00 00 00 00 00 00 00 00 00 00 00 00 fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==67440==ABORTING
                       MLDBFB-724_classifier_exp_segfault_test.py FAILED

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions