fix(*) various fixes, several pertaining to FFI & Gateway integration

[thibaultcha]

2 segfault fixes:

• 672070b - fix(ffi) correctly set filter->log during chain loading
• 82a3c86 - fix(proxy-wasm) always unset instance ctx filter chain

[flrgh]

👍 I've been taking this for a spin this afternoon. Tested all runtimes with FILTER and UNSET isolation inputs.

Wasmer and Wasmtime are working flawlessly now--not a single segfault or unexplained error from new()/load()/attach()/start() FFI calls. With V8 there does seem to be an improvement (no more failed loading filters: unknown error), though I am still encountering occasional segfaults. I am not sure that they are related to what this PR aims at fixing though, because I get a nearly identical trace on the main branch:

#0  0x0000000000e6edb2 in v8::internal::WeakArrayList::Append(v8::internal::Isolate*, v8::internal::Handle<v8::internal::WeakArrayList>, v8::internal::MaybeObjectHandle const&, v8::internal::AllocationType) ()
#1  0x000000000108901d in v8::internal::WasmInstanceObject::New(v8::internal::Isolate*, v8::internal::Handle<v8::internal::WasmModuleObject>) ()
#2  0x000000000186f907 in v8::internal::wasm::InstanceBuilder::Build() ()
#3  0x0000000001871142 in v8::internal::wasm::InstantiateToInstanceObject(v8::internal::Isolate*, v8::internal::wasm::ErrorThrower*, v8::internal::Handle<v8::internal::WasmModuleObject>, v8::internal::MaybeHandle<v8::internal::JSReceiver>, v8::internal::MaybeHandle<v8::internal::JSArrayBuffer>) ()
#4  0x0000000001057cc3 in v8::internal::wasm::WasmEngine::SyncInstantiate(v8::internal::Isolate*, v8::internal::wasm::ErrorThrower*, v8::internal::Handle<v8::internal::WasmModuleObject>, v8::internal::MaybeHandle<v8::internal::JSReceiver>, v8::internal::MaybeHandle<v8::internal::JSArrayBuffer>) ()
#5  0x0000000000ac6fb5 in wasm::Instance::make(wasm::Store*, wasm::Module const*, wasm::Extern const* const*, std::unique_ptr<wasm::Trap, std::default_delete<wasm::Trap> >*) ()
#6  0x0000000000acca65 in wasm_instance_new ()
#7  0x0000000000aaa100 in ngx_v8_init_instance ()
#8  0x0000000000a9e0a8 in ngx_wavm_instance_create ()
#9  0x0000000000aa36c2 in ngx_proxy_wasm_get_instance ()
#10 0x0000000000aa39d3 in ngx_proxy_wasm_ctx ()
#11 0x0000000000a9aab5 in ngx_wasm_op_proxy_wasm_handler ()
#12 0x0000000000a9b02d in ngx_wasm_ops_resume ()
#13 0x0000000000a9f5ac in ngx_http_wasm_ffi_start ()
#14 0x00007f4f5ff45f65 in lj_vm_ffi_call ()
   from /home/michaelm/git/kong/kong/bazel-bin/build/kong-dev/openresty/luajit/lib/libluajit-5.1.so.2
#15 0x00007f4f5ff92bb7 in lj_ccall_func (L=<optimized out>, cd=<optimized out>) at lj_ccall.c:1382
#16 0x00007f4f5ffaa11d in lj_cf_ffi_meta___call (L=0x7f4f48939da0) at lib_ffi.c:230
#17 0x00007f4f5ff43b1b in lj_BC_FUNCC ()
   from /home/michaelm/git/kong/kong/bazel-bin/build/kong-dev/openresty/luajit/lib/libluajit-5.1.so.2
#18 0x0000000000a47acf in ngx_http_lua_run_thread ()
#19 0x0000000000a4d15b in ngx_http_lua_access_by_chunk ()
#20 0x00000000009bb3cd in ngx_http_core_access_phase ()
#21 0x00000000009b7285 in ngx_http_core_run_phases ()
#22 0x00000000009c1f87 in ngx_http_process_request_headers ()
#23 0x00000000009c237c in ngx_http_process_request_line ()
#24 0x00000000009aa202 in ngx_epoll_process_events ()
#25 0x00000000009a0446 in ngx_process_events_and_timers ()
#26 0x00000000009a7fde in ngx_worker_process_cycle ()
#27 0x00000000009a6564 in ngx_spawn_process ()
#28 0x00000000009a733c in ngx_start_worker_processes ()
#29 0x00000000009a8a1f in ngx_master_process_cycle ()
#30 0x000000000096e911 in main ()

[thibaultcha]

Hmm this looks like a third thing at first glance... This is with V8 and the same cache invalidation test suite? By the way, does that suite pass entirely for you? Am I missing anything besides my local postgres/redis instances?

[thibaultcha]

Oh, actually all I needed to do was to build the binary without debug mode; makes sense, especially with the no-pool patch enabled. Great.

[thibaultcha]

Oh, I spoke too soon. I got a number of these:

Error → spec/02-integration/20-wasm/05-cache-invalidation_spec.lua @ 285
#wasm filter chain cache (strategy: #off) (#eventual consistency) is invalidated on filter creation and removal
./spec/helpers.lua:188: Error: 'kong config db_export' only works with a database.

  Run with --v (verbose) or --vv (debug) for more details


stack traceback:
        ./spec/helpers.lua:188: in function 'make_yaml_file'
        spec/02-integration/20-wasm/05-cache-invalidation_spec.lua:25: in function 'update_yaml_config'
        spec/02-integration/20-wasm/05-cache-invalidation_spec.lua:46: in function 'insert'
        spec/02-integration/20-wasm/05-cache-invalidation_spec.lua:288: in function <spec/02-integration/20-wasm/05-cache-invalidation_spec.lua:285>

I encountered it last week too but forgot about it.

[flrgh]

Error → spec/02-integration/20-wasm/05-cache-invalidation_spec.lua @ 285
#wasm filter chain cache (strategy: #off) (#eventual consistency) is invalidated on filter creation and removal
./spec/helpers.lua:188: Error: 'kong config db_export' only works with a database.

  Run with --v (verbose) or --vv (debug) for more details


stack traceback:
        ./spec/helpers.lua:188: in function 'make_yaml_file'
        spec/02-integration/20-wasm/05-cache-invalidation_spec.lua:25: in function 'update_yaml_config'
        spec/02-integration/20-wasm/05-cache-invalidation_spec.lua:46: in function 'insert'
        spec/02-integration/20-wasm/05-cache-invalidation_spec.lua:288: in function <spec/02-integration/20-wasm/05-cache-invalidation_spec.lua:285>

Ah, so I was mistaken about this before. You (currently) do need a postgres instance for this test, even when selecting #off. In dbless mode, this test creates all the entities in the DB and then performs kong config db_export to generate a yaml file just before starting kong. It's a pattern used in many places throughout the test suite because it makes testing both dbless and db strategies easy with very little extra code.

[thibaultcha]

Yes, I came up with that testing pattern in the Gateway lol :)

I guess there are two questions, the first of which relates to successfully running all 20-wasm tests locally:

I do have a Postgres instance running, and I am trying to run the whole suite (all DB modes) as it is in the feat/wasmx branch. So I spin up kong-tests-compose (Postgres/C*/Redis) and run:

$ bin/busted spec/02-integration/20-wasm/05-cache-invalidation_spec.lua -v

Which works to a certain point and then the last few tests fail:

$ bin/busted spec/02-integration/20-wasm/05-cache-invalidation_spec.lua -v
2023/06/27 14:16:58 [warn] 2491026#0: 4096 worker_connections exceed open file resource limit: 1024
●●●●●●●●●●●●✱✱✱✱✱✱
12 successes / 0 failures / 7 errors / 0 pending : 73.173543 seconds

Error → spec/02-integration/20-wasm/05-cache-invalidation_spec.lua @ 285
#wasm filter chain cache (strategy: #off) (#eventual consistency) is invalidated on filter creation and removal
./spec/helpers.lua:188: Error: 'kong config db_export' only works with a database.

  Run with --v (verbose) or --vv (debug) for more details


stack traceback:
        ./spec/helpers.lua:188: in function 'make_yaml_file'
        spec/02-integration/20-wasm/05-cache-invalidation_spec.lua:25: in function 'update_yaml_config'
        spec/02-integration/20-wasm/05-cache-invalidation_spec.lua:46: in function 'insert'
        spec/02-integration/20-wasm/05-cache-invalidation_spec.lua:288: in function <spec/02-integration/20-wasm/05-cache-invalidation_spec.lua:285>

So as I was saying above, I encountered this when investigating the coredumps, and remember going down a Kong spec utils/yaml rabbit hole. Do you not experience this problem, and if you don't, are there any other steps I may have missed before running the tests?

And then, my other question was about the V8 coredump:

This is with V8 and the same cache invalidation test suite?

[flrgh]

[slacking you about the db_export thing]

This is with V8 and the same cache invalidation test suite?

Yup! It causes a bunch of churn on the kong filter chain entities, so it just seems the easiest way to uncover issues right now.