Skip to content

Conversation

@Alex-PLACET
Copy link
Collaborator

No description provided.

@Alex-PLACET Alex-PLACET self-assigned this Dec 22, 2025
@Alex-PLACET Alex-PLACET marked this pull request as ready for review December 23, 2025 10:17
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds support for the Arrow canonical extension type arrow.fixed_shape_tensor, which enables storing fixed-shape tensors in Arrow format. The implementation follows the Apache Arrow specification for fixed shape tensor arrays.

Key Changes

  • Introduces fixed_shape_tensor_array class wrapping fixed_sized_list_array with tensor-specific metadata
  • Implements JSON serialization/deserialization for tensor metadata (shape, dimension names, permutation)
  • Replaces nlohmann_json dependency with simdjson for improved JSON parsing performance
  • Adds comprehensive test coverage with 752 lines of tests

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 7 comments.

Show a summary per file
File Description
include/sparrow_extensions/fixed_shape_tensor.hpp Defines the fixed_shape_tensor_array class and metadata structures with comprehensive documentation
src/fixed_shape_tensor.cpp Implements metadata validation, JSON serialization, and array construction logic
tests/test_fixed_shape_tensor.cpp Provides extensive test coverage for metadata operations, array construction, and spec compliance
cmake/external_dependencies.cmake Adds simdjson dependency and updates doctest version
environment-dev.yml Replaces nlohmann_json with simdjson in conda dependencies
CMakeLists.txt Registers new source files and headers in build system
include/sparrow_extensions.hpp Includes the new fixed_shape_tensor header in main library header
tests/CMakeLists.txt Adds test file to test suite

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@Alex-PLACET Alex-PLACET linked an issue Jan 5, 2026 that may be closed by this pull request
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 3 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@Alex-PLACET Alex-PLACET force-pushed the add_fixed_shape_tensor branch from 6011091 to e518d22 Compare January 6, 2026 15:49
@Alex-PLACET
Copy link
Collaborator Author

I add some fixes on the variable shape tensor as the layout concept satisfaction was missing

@JohanMabille
Copy link
Contributor

The failure on OSX are unrelated to the changes.

@JohanMabille JohanMabille merged commit 0b52884 into sparrow-org:main Jan 7, 2026
38 of 42 checks passed
@Alex-PLACET Alex-PLACET deleted the add_fixed_shape_tensor branch January 7, 2026 14:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add fixed shape tensor

2 participants