Skip to content

[Variant] Preserve fixed-point JSON numbers as decimals #11169

Description

@osipovartem

Describe the problem

parquet-variant-json currently converts every non-integer JSON number to Variant::Double. As a result, fixed-point values lose both decimal type semantics and potentially precision:

1.23                    -> Double (expected Decimal4(123, 2))
9999999999999999.99     -> Double (expected Decimal8(..., 2))
9999999999999999999     -> Double (expected Decimal16(..., 0))

The crate already contains ignored tests for Decimal4/8/16 covering these cases. The original implementation discussion in #7783 also identified this as follow-up work.

This matters for Parquet Variant interoperability. Engines such as Snowflake distinguish fixed-point Variant numbers (DECIMAL) from scientific or floating-point numbers (DOUBLE).

Proposed semantics

Use the existing mapping described in #7783:

  • integers fitting in i64: preserve the current smallest integer encoding;
  • fixed-point values: use the smallest fitting VariantDecimal4/8/16;
  • integers wider than i64 and at most 38 digits: use VariantDecimal16;
  • exponent notation or values outside Variant decimal range: use Double;
  • apply the same rules recursively inside arrays and objects.

Implementation constraint

Enabling serde_json's arbitrary_precision feature is not a suitable fix. It is unified across the Arrow workspace and changes unrelated serde_json users; this was already observed during #7783. It also retains the intermediate serde_json::Value tree.

A performance-first implementation should consume exact numeric lexemes and append directly to VariantBuilderExt, avoiding an intermediate JSON value tree and avoiding a second per-value conversion pass. Arrow JSON's tape parser already retains number text in TapeElement::Number, but TapeDecoder is intentionally private. Two possible directions are:

  1. expose a narrow, non-public-implementation-specific JSON token callback API from arrow-json; or
  2. add a focused streaming parser in parquet-variant-json that writes directly to the Variant builder.

I can implement either direction. Maintainer guidance on the preferred API boundary would avoid introducing a second JSON tokenizer or prematurely exposing TapeDecoder.

Validation

The existing ignored decimal tests can be enabled, with added nested object/list cases. A benchmark should compare integer-only, mixed-object, and decimal-heavy JSON against the current serde path to ensure ordinary inputs do not regress.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions