enhancement(codecs): allow configurable null handling in Arrow encoder

[benjamin-awd]

Summary

This PR is a follow-up from #24124 (comment), allowing for sinks using the Arrow encoder to mark fields as nullable.

This is useful when working with downstream systems that handle null values through defaults or computed columns.

For example, in Clickhouse MATERIALIZED/DEFAULT columns are often used to provide metadata for an event:

CREATE TABLE app_events (
    event_id UInt64,
    ingested_at DateTime64(9) DEFAULT now64(9),
)

In this case, it isn't desirable to use a Nullable(DateTime64(9)) type in Clickhouse, since it creates an additional UInt8 column that is never used. This change gives flexibility to users who may not necessarily want the overhead of schema management.

In general, this also makes schema evolution much easier since we won't need to worry about potential breakage during transitions. Dealing with highly nested data also becomes easier, since we won't need to define all possible nulls within a data structure.

Vector configuration

N/A

How did you test this PR?

Unit tests

Change Type

• Bug fix
• New feature
• Non-functional (chore, refactoring, docs)
• Performance

Is this a breaking change?

• Yes
• No

Does this PR include user facing changes?

• Yes. Please add a changelog fragment based on our guidelines.
• No. A maintainer will apply the no-changelog label to this PR.

References

Notes

• Please read our Vector contributor resources.
• Do not hesitate to use @vectordotdev/vector to reach out to us regarding this PR.
• Some CI checks run only after we manually approve them.
  • We recommend adding a pre-push hook, please see this template.
  • Alternatively, we recommend running the following locally before pushing to the remote branch:
    • make fmt
    • make check-clippy (if there are failures it's possible some of them can be fixed with make clippy-fix)
    • make test
• After a review is requested, please avoid force pushes to help us review incrementally.
  • Feel free to push as many commits as you want. They will be squashed into one before merging.
  • For example, you can run git merge origin master and git push.
• If this PR introduces changes Vector dependencies (modifies Cargo.lock), please
  run make build-licenses to regenerate the license inventory and commit the changes (if any). More details here.