Describe the problem
parquet-variant-json currently converts every non-integer JSON number to Variant::Double. As a result, fixed-point values lose both decimal type semantics and potentially precision:
1.23 -> Double (expected Decimal4(123, 2))
9999999999999999.99 -> Double (expected Decimal8(..., 2))
9999999999999999999 -> Double (expected Decimal16(..., 0))
The crate already contains ignored tests for Decimal4/8/16 covering these cases. The original implementation discussion in #7783 also identified this as follow-up work.
This matters for Parquet Variant interoperability. Engines such as Snowflake distinguish fixed-point Variant numbers (DECIMAL) from scientific or floating-point numbers (DOUBLE).
Proposed semantics
Use the existing mapping described in #7783:
- integers fitting in
i64: preserve the current smallest integer encoding;
- fixed-point values: use the smallest fitting
VariantDecimal4/8/16;
- integers wider than
i64 and at most 38 digits: use VariantDecimal16;
- exponent notation or values outside Variant decimal range: use
Double;
- apply the same rules recursively inside arrays and objects.
Implementation constraint
Enabling serde_json's arbitrary_precision feature is not a suitable fix. It is unified across the Arrow workspace and changes unrelated serde_json users; this was already observed during #7783. It also retains the intermediate serde_json::Value tree.
A performance-first implementation should consume exact numeric lexemes and append directly to VariantBuilderExt, avoiding an intermediate JSON value tree and avoiding a second per-value conversion pass. Arrow JSON's tape parser already retains number text in TapeElement::Number, but TapeDecoder is intentionally private. Two possible directions are:
- expose a narrow, non-public-implementation-specific JSON token callback API from
arrow-json; or
- add a focused streaming parser in
parquet-variant-json that writes directly to the Variant builder.
I can implement either direction. Maintainer guidance on the preferred API boundary would avoid introducing a second JSON tokenizer or prematurely exposing TapeDecoder.
Validation
The existing ignored decimal tests can be enabled, with added nested object/list cases. A benchmark should compare integer-only, mixed-object, and decimal-heavy JSON against the current serde path to ensure ordinary inputs do not regress.
Describe the problem
parquet-variant-jsoncurrently converts every non-integer JSON number toVariant::Double. As a result, fixed-point values lose both decimal type semantics and potentially precision:The crate already contains ignored tests for Decimal4/8/16 covering these cases. The original implementation discussion in #7783 also identified this as follow-up work.
This matters for Parquet Variant interoperability. Engines such as Snowflake distinguish fixed-point Variant numbers (
DECIMAL) from scientific or floating-point numbers (DOUBLE).Proposed semantics
Use the existing mapping described in #7783:
i64: preserve the current smallest integer encoding;VariantDecimal4/8/16;i64and at most 38 digits: useVariantDecimal16;Double;Implementation constraint
Enabling serde_json's
arbitrary_precisionfeature is not a suitable fix. It is unified across the Arrow workspace and changes unrelated serde_json users; this was already observed during #7783. It also retains the intermediateserde_json::Valuetree.A performance-first implementation should consume exact numeric lexemes and append directly to
VariantBuilderExt, avoiding an intermediate JSON value tree and avoiding a second per-value conversion pass. Arrow JSON's tape parser already retains number text inTapeElement::Number, butTapeDecoderis intentionally private. Two possible directions are:arrow-json; orparquet-variant-jsonthat writes directly to the Variant builder.I can implement either direction. Maintainer guidance on the preferred API boundary would avoid introducing a second JSON tokenizer or prematurely exposing
TapeDecoder.Validation
The existing ignored decimal tests can be enabled, with added nested object/list cases. A benchmark should compare integer-only, mixed-object, and decimal-heavy JSON against the current serde path to ensure ordinary inputs do not regress.