ERTS calling driver callback through incompatible fp

[jgonet]

Describe the bug
Driver's start callback is always called with three arguments, user defined drivers are created with two arguments. This works in most architectures (but not in Wasm, notably) and it's UB in C.

Example:
callback definition

#ifndef ERL_SYS_DRV
    ErlDrvData (*start)(ErlDrvPort port, char *command);
				/* called when open_port/2 is invoked.
				   return value -1 means failure. */
#else
    ErlDrvData (*start)(ErlDrvPort port, char *command, SysDriverOpts* opts);
				/* special options, only for system driver */
#endif

udp inet driver start callback

static ErlDrvData udp_inet_start(ErlDrvPort port, char *args)

call site

drv_data = (*driver->start)(ERTS_Port2ErlDrvPort(port), name, opts);

To Reproduce
Use ubsan to detect it, -fsanitize=function.

Bash script for checking (macOS, change LLVM_PREFIX for linux)

#!/usr/bin/env bash
set -euo pipefail

OTP_REF="${OTP_REF:-master}"
OTP_DIR="${OTP_DIR:-otp-function-sanitizer-repro}"

git clone --depth 1 \
  --single-branch \
  --branch "$OTP_REF" \
  https://github.com/erlang/otp.git \
  "$OTP_DIR"

cd "$OTP_DIR"

LLVM_PREFIX="$(brew --prefix llvm)"
export PATH="$LLVM_PREFIX/bin:$PATH"
export CC="$LLVM_PREFIX/bin/clang"
export CXX="$LLVM_PREFIX/bin/clang++"

SAN_FLAGS="-O1 -g -fno-omit-frame-pointer -fsanitize=function -fsanitize-recover=function"
export CFLAGS="$SAN_FLAGS"
export CXXFLAGS="$SAN_FLAGS"
export LDFLAGS="-fsanitize=function"

export UBSAN_OPTIONS="halt_on_error=0:print_stacktrace=0"
./configure \
  --disable-jit \
  --without-javac \
  --without-wx \
  --without-odbc

BUILD_STDERR="$PWD/build-ubsan.stderr.log"
echo "Building OTP; stderr is being written to:"
echo "  $BUILD_STDERR"

: >"$BUILD_STDERR"

build_target() {
  if ! make -j"$(nproc)" "$@" 2>>"$BUILD_STDERR"; then
    echo "OTP build failed. Last 80 stderr lines:"
    tail -n 80 "$BUILD_STDERR"
    exit 1
  fi
}

build_target erl_interface
build_target emulator
build_target bootstrap_setup

RUNTIME_STDERR="$PWD/runtime-ubsan.stderr.log"
UBSAN_OPTIONS="halt_on_error=0:print_stacktrace=1" \
  ./bootstrap/bin/erl -noshell -eval 'halt().' \
  >/dev/null \
  2>"$RUNTIME_STDERR"

echo
echo "Driver start mismatch:"
grep -n -B2 -A4 \
  'call to function udp_inet_start' \
  "$RUNTIME_STDERR"

echo
echo "Complete runtime sanitizer output:"
echo "  $RUNTIME_STDERR"

Excerpt:

Driver start mismatch:
203-
204-SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior beam/hash.c:218:5
205:beam/io.c:713:13: runtime error: call to function udp_inet_start through pointer to incorrect function type 'struct _erl_drv_data *(*)(struct _erl_drv_port *, char *, struct _SysDriverOpts *)'
206-inet_drv.c:14507: note: udp_inet_start defined here
207-    #0 0x000104a94988 in erts_open_driver io.c:747
208-    #1 0x000104b3be68 in erts_internal_open_port_2 erl_bif_port.c:69
209-    #2 0x0001049ffa84 in process_main beam_hot.h:291

Expected behavior
ERTS doesn't call functions through incompatible pointers.

Affected versions
Newest master (6efeea8), OTP-29 and older are affected.

Additional context
Popcorn started compiling OTP/BEAM for Wasm, we needed to use EMULATE_FUNCTION_POINTER_CASTS which creates trampolines with non-compliant C code. This adds measurable amount of code in the final binary (90kB) and we'd like to drop it.

While ubsan reports a lot of findings, only driver one seems to be actually important (Wasm doesn't have a lot of types in ABI and most of incompatible types are lowered to i32; it's stricter on number of arguments, though).