Excessive `TokenBuffer` allocations when deserializing records

[phraktle]

Search before asking

• I searched in the issues and found nothing similar.

Describe the bug

When deserializing objects whose construction goes through a property-based creator – which includes every Java record, since records always use the canonical constructor – BeanDeserializer._deserializeUsingPropertyBased allocates a fresh TokenBuffer on every call. Each TokenBuffer allocates at least one Segment (an Object[] plus a token-type array). For workloads that stream many small records, this shows up prominently in allocation profiles.

Code path:

  BeanDeserializer.deserialize
    → BeanDeserializerBase.deserializeFromObjectUsingNonDefault
      → BeanDeserializer._deserializeUsingPropertyBased
        → DeserializationContext.bufferForInputBuffering
          → TokenBuffer.forBuffering   // fresh allocation per object

Format-independent — reproduces with JSON, CSV, or any other backend. Since records are now the idiomatic way to model immutable DTOs in Java and unconditionally force this path, users migrating DTOs from classes-with-setters to records see a regression in allocation rate with no change in code shape.

The buffer's lifetime is bounded to _deserializeUsingPropertyBased, so reusing it (with RecyclerPool?) may be feasible.

Version Information

3.1.1

Reproduction

No response

Expected behavior

No response

Additional context

No response

[cowtowncoder]

My first though is that I don't think this should be changed. Avoiding complexity and possible concurrency issue with pooling (esp. RecyclerPool) is the main benefit.

But leaving open just in case.

[phraktle]

Attaching a small benchmark showing the excess allocation. Turns out there's a fast path too, that isn't taken here: @JsonIgnoreProperties(ignoreUnknown=true) short-circuits to p.skipChildren() via _ignoreAllUnknown before the TokenBuffer is allocated, but the global FAIL_ON_UNKNOWN_PROPERTIES=false flag doesn't – it only filters after the buffer has been built and populated.

Three targets, same input (5 known + 3 unknown props), same mapper:

• record plain: allocates TokenBuffer per call in _deserializeUsingPropertyBased, which handleUnknownProperties then silently drops.
• record with @JsonIgnoreProperties(ignoreUnknown=true): fast-path skip, no allocation.
• class (no-arg ctor): routes through deserializeFromObject

  record plain                   1376  bytes/op    520 ns/op
  record @JsonIgnoreProperties   1024  bytes/op    455 ns/op
  class (no-arg ctor)             712  bytes/op    317 ns/op

Benchmark:

import java.lang.management.ManagementFactory;

import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
import com.sun.management.ThreadMXBean;
import tools.jackson.databind.DeserializationFeature;
import tools.jackson.databind.json.JsonMapper;

// Repro for jackson-databind #5897 -- wasted TokenBuffer allocation when
// deserializing via property-based creator with unknown properties in input.
public class TokenBufferBench {

    public record RecordPlain(long a, long b, long c, long d, long e) {}

    @JsonIgnoreProperties(ignoreUnknown = true)
    public record RecordAnnotated(long a, long b, long c, long d, long e) {}

    public static class ClassPlain {
        public long a, b, c, d, e;
    }

    private static final String JSON =
            "{\"a\":1,\"b\":2,\"c\":3,\"d\":4,\"e\":5,\"x1\":11,\"x2\":12,\"x3\":13}";

    private static final int WARMUP  = 10_000_000;
    private static final int MEASURE = 10_000_000;

    public static void main(String[] args) throws Exception {
        JsonMapper mapper = JsonMapper.builder()
                .disable(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES)
                .build();
        ThreadMXBean tmx = (ThreadMXBean) ManagementFactory.getThreadMXBean();
        tmx.setThreadAllocatedMemoryEnabled(true);

        for (var k : new Class<?>[] { RecordPlain.class, RecordAnnotated.class, ClassPlain.class }) {
            run(mapper, k, WARMUP);
        }

        System.out.printf("%-32s %12s %12s%n", "scenario", "bytes/op", "ns/op");
        bench(mapper, tmx, RecordPlain.class,     "record plain");
        bench(mapper, tmx, RecordAnnotated.class, "record @JsonIgnoreProperties");
        bench(mapper, tmx, ClassPlain.class,      "class (no-arg ctor)");
    }

    private static void bench(JsonMapper mapper, ThreadMXBean tmx, Class<?> klass, String label) throws Exception {
        long tid = Thread.currentThread().threadId();
        long b0 = tmx.getThreadAllocatedBytes(tid);
        long t0 = System.nanoTime();
        run(mapper, klass, MEASURE);
        long t1 = System.nanoTime();
        long b1 = tmx.getThreadAllocatedBytes(tid);
        System.out.printf("%-32s %12.1f %12.1f%n",
                label, (double) (b1 - b0) / MEASURE, (double) (t1 - t0) / MEASURE);
    }

    static volatile Object sink;

    private static void run(JsonMapper mapper, Class<?> klass, int iters) throws Exception {
        for (int i = 0; i < iters; i++) {
            sink = mapper.readValue(JSON, klass);
        }
    }
}

[phraktle]

@cowtowncoder see this partial improvement (improves the fast-path check, but records remains slower than a POJO)

[cowtowncoder]

@phraktle To some degree non-default ctor will always have some overhead. But perhaps simple TokenBuffer reuse could work; keeping one in DeserializationContext, deserializers can ask for one, return -- at most 1 instance (so nested ones wouldn't benefit).
I'd rather not try to do RecyclerPool style with all the complications. But more like how JsonReadContext and related work (altho not quite the same) -- or more so , how DeserializationContext._dateFormat reuse works.

[phraktle]

@cowtowncoder I guess with PR #5907 the TokenBuffer part is improved, but records are still more expensive than a POJO because it also needs to allocate a PropertyValueBuffer – would local reuse in the context make sense for these?

[cowtowncoder]

It may be trickier as PropertyValueBuffer is, I think, wrapper around Object[] allocated to exact size needed for N creator arguments.

[phraktle]

If PR #5907 is accepted, I think we can close this for now. It can be a separate request to improve record deserialization performance (to be closer to a POJO), but the originally profiled problem of TokenBuffer allocations was addressed.

[cowtowncoder]

@phraktle Agreed. I am still working through testing aspects but should be able to merge the PR today