feat(compat): coerce Arrow types DuckDB cannot consume (Float16 -> Float32)

[jiaoew1991]

Summary

Teach the Lance → DuckDB reader boundary to transparently coerce Arrow primitives DuckDB has no mapping for, starting with Float16 (FloatingPoint(HALF)).

Before this PR, any SELECT against a Lance dataset with a float16 column — scalar, fixed_size_list<float16>, or nested inside struct/list/map — fails at bind time:

[UNSUPPORTED_ARROWTYPE] Unsupported arrow type FloatingPoint(HALF).

DuckDB's bundled Arrow primitive decoder (arrow_duck_schema.cpp::GetTypeFromFormat) is hardcoded and has no plugin API for adding primitive formats. ArrowTypeExtension is only for named Arrow extension types (ARROW:extension:name), not primitive format codes. So a reader-boundary shim on our side is the right layer.

Design

New src/lance_arrow_compat.{hpp,cpp} with a tiny rule registry — adding a future unsupported primitive is one row:

struct CoercionRule {
  bool (*matches)(const char *format);
  const char *coerced_format;
  size_t src_element_size;
  size_t coerced_element_size;
  void (*convert)(const void *src, void *dst, int64_t count);
};

constexpr CoercionRule kRules[] = {
    {MatchesFloat16, kFloat32FormatLiteral, sizeof(uint16_t), sizeof(float),
     ConvertFloat16ToFloat32},
    // Future types: add one row here. Call sites don't change.
};

Two umbrella helpers walk schema + array (including children[] and dictionary):

• LanceCoerceArrowSchemaForDuckDB(ArrowSchema *) — rewrites format pointers via a wrapping release so the producer's release still frees the original string. Returns top-level column names that were coerced.
• LanceCoerceArrowArrayForDuckDB(const ArrowSchema *, ArrowArray *) — allocates new buffers, converts element-wise (HalfToFloat matches lance-spark's Float16Utils.halfToFloat bit-for-bit), installs a wrapping release.

Wired at every Lance → DuckDB boundary: scan / exec-IR / KNN / FTS binds, batch loads, deferred-take, MERGE take, catalog schema generators.

Write-path guard

The coercion is intentionally asymmetric: if we let INSERT / UPDATE / MERGE flow, DuckDB would hand back FLOAT and we'd silently widen the on-disk HALF storage. Instead:

• LanceTableEntry now carries a vector<string> coerced_column_names (populated when the catalog bind sees any rule fire).
• PlanLanceInsertAppend, PlanLanceUpdateOverwrite, PlanLanceMergeInto throw NotImplementedException naming the affected columns — no silent storage corruption.

INSERT into Lance table 'videos' is not supported: column(s) [raw_video_index]
have Arrow types DuckDB cannot represent natively, so the catalog exposes a
coerced type. Writing in the coerced type would silently change the on-disk
storage.

Tests

test/sql/float16_widening.test exercises every recursion branch against a pylance fixture:

• scalar float16
• fixed_size_list<float16, 3>
• variable-size list<float16>
• struct<float16, float32> + struct field projection
• list<struct<float16>> (struct-in-list)
• map<varchar, float16>
• pushdown filter + projection-only scan
• write guards (INSERT / UPDATE error assertions)

Fixture generator at test/scripts/gen_float16_fixture.py (pylance + pyarrow).

Related

• No prior lance-duckdb issue/PR for Float16 — first report. Related prior work (CREATE INDEX fails on common list<float> embedding schema written from Python/Arrow #184, UX issue: Type casting error when using float[] or double[] #117) addressed list<float> vs fixed_size_list<float> shape mismatches, not unsupported primitives.
• Mirrors lance-spark's existing Float16Utils.halfToFloat widening (commit 821a436), so cross-engine reads surface identical values.

Test plan

• Run test/scripts/gen_float16_fixture.py (requires pylance + pyarrow) to generate test/data/float16_fixture.lance and commit it, so CI picks up the new sqllogic test.
• make test (or equivalent) to run the new float16_widening.test locally.
• Reproduce the original select raw_video_index from lance.all_video_shot limit 5 against a real Float16-carrying dataset and confirm it now returns rows.
• Try INSERT INTO <table-with-half-col> and confirm the clear NotImplementedException fires (no silent corruption).

🤖 Generated with Claude Code

[Xuanwo]

LanceTableEntry::Copy() should preserve coerced_column_names (see src/lance_scan.cpp, around the copied entry construction). The new INSERT/UPDATE/MERGE guards rely on HasCoercedColumns(), so a copied catalog entry can expose the widened FLOAT schema while losing the write-protection state. Please copy the coerced column list into the returned entry.

[jiaoew1991]

@Xuanwo I have made the changes as requested, pls take a look again.