[data] fix bug that _align_struct_fields fails if there are unaligned scalar…

[947132885]

Description

ray fails to merge blocks when there are both unaligned scalar and struct fields.

Minimal code to reproduce:

`
import pyarrow as pa
from ray.data._internal.arrow_ops.transform_pyarrow import concat

table1 = pa.table({
"a": [1, 2, 3],
"s": [
{"x": 7,},
{"x": 8,},
{"x": 9,},
],
})

table2 = pa.table({
"b": [4, 5, 6],
})
concat([table1, table2])
`

Fix:

create null array in _align_struct_fields instead of assertion

[gemini-code-assist]

Code Review

This pull request addresses a bug in _align_struct_fields that caused a failure when processing blocks with missing scalar fields alongside struct fields. The fix correctly replaces an overly strict assertion with logic to pad missing columns with nulls. This change is correct, well-targeted, and resolves the issue described. The implementation is clean and follows standard practices for handling missing data in pyarrow tables.

[goutamvenkat-anyscale]

Thanks @947132885 for your contribution. Can you please add some tests for this change?

[947132885]

ok, I added a simple test