Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Structured zarr dtype as bytes #176

Open
wants to merge 46 commits into
base: master
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
46 commits
Select commit Hold shift + click to select a range
4aa8cae
Implemented concept of "open as byte array" for any dtype
brian-michell Jun 20, 2024
ac6ba5c
Added open option for byte array
brian-michell Jun 21, 2024
6b43911
Removed open option for byte array
brian-michell Jun 21, 2024
c4beb11
Allow modification of ParseDTypeOptions
brian-michell Jun 21, 2024
43cc7c8
Update default behavior
brian-michell Jun 21, 2024
a76363a
Emplace instead of replace
brian-michell Jun 21, 2024
6342417
Update field selection to use emplaced field
brian-michell Jun 21, 2024
f684f17
Stopped synthetic dimension from getting seralized
brian-michell Jun 24, 2024
93e92ce
Pushback synthetic field rather than make it the only field
brian-michell Jun 24, 2024
874d581
Skip the synthetic field as part of the number of bytes
brian-michell Jun 24, 2024
c7408d4
Fix const qualifier
brian-michell Jun 24, 2024
83acfa6
Properly modify const object
brian-michell Jun 24, 2024
7ac53a4
Properly modify has_fields logic
brian-michell Jun 24, 2024
2c0bdc3
Pulled check outside loop and fixed logic error
brian-michell Jun 24, 2024
442daa0
Explicit initilization of values. Ensured has_fields for structured d…
brian-michell Jun 24, 2024
92145fc
Implemented proper field_shape for synthetic field
brian-michell Jun 24, 2024
77ce1ca
Added valid case for an empty named field
brian-michell Jun 24, 2024
1113ec7
Disabled setting has_fields to false
brian-michell Jun 24, 2024
9d9a555
Fix encoding
brian-michell Jun 24, 2024
1d597ad
Fixed chunkgrid to use the true size
brian-michell Jun 25, 2024
ea1ae5c
Temporary drop of synthetic field for metadata validation
brian-michell Jun 25, 2024
5355201
Fix single field dtype without name incorrectly passing
brian-michell Jun 25, 2024
5c9bd16
Fix multiple synthetic fields getting added
brian-michell Jun 25, 2024
c4f0178
Removed extra argument
brian-michell Jun 25, 2024
288b175
Catch case where there was no field due to being a struct array of si…
brian-michell Jun 25, 2024
de830ef
Ignore the synthetic dimension when chunking
brian-michell Jun 25, 2024
619a03a
Remove the synthetic dimension when it is not needed
brian-michell Jun 25, 2024
bb68abf
Update tests to reflect DType without going through the spec driver p…
brian-michell Jun 25, 2024
e40dfdd
Update test to reflect metadata containing synthetic dimension. (Hand…
brian-michell Jun 25, 2024
8fc4fa6
Remove check that is no-longer true
brian-michell Jun 25, 2024
9912b32
Remove test that is no longer true
brian-michell Jun 25, 2024
6015c31
Only use the actual fields for fill values
brian-michell Jun 25, 2024
5ae3d36
Cleaning up convention
brian-michell Jun 25, 2024
8ca6c74
Fixed incorrect assertion
brian-michell Jun 25, 2024
211beab
Add beginning of new test for field-free opening
brian-michell Jun 26, 2024
a4d7e73
Fix num_bytes not getting set
brian-michell Jun 26, 2024
e51b3fe
Allow the metadata to be one size higher in the event of structured d…
brian-michell Jun 26, 2024
f30948c
Properly specify the chunk grid for synthetic data field
brian-michell Jun 26, 2024
824d5bc
Added fallback for zarr synthetic field
brian-michell Jun 26, 2024
364a569
Fix test to reflect num_bytes getting properly set
brian-michell Jun 26, 2024
620af12
Disabled ignoring 'true size' for chunk layout
brian-michell Jul 1, 2024
60060a7
Fixed metadata caching causing extra dimension getting added
brian-michell Jul 1, 2024
5b56a98
Fix hardcoded case for opening existing zarr without field
brian-michell Jul 5, 2024
08fbd48
Implement naieve fix for fill_value assertion failure on no field str…
brian-michell Jul 5, 2024
c1c0ccd
Remove incorrect decrement of struct data
brian-michell Jul 5, 2024
96144b7
Added test for opening existing store as struct array
brian-michell Jul 9, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Fixed chunkgrid to use the true size
brian-michell committed Jun 25, 2024
commit 1d597ad1018474c0e4b8f53c67387b93272f6f50
8 changes: 6 additions & 2 deletions tensorstore/driver/zarr/driver.cc
Original file line number Diff line number Diff line change
@@ -295,13 +295,17 @@ Result<std::shared_ptr<const void>> DataCache::GetResizedMetadata(

internal::ChunkGridSpecification DataCache::GetChunkGridSpecification(
const ZarrMetadata& metadata) {
size_t true_size = metadata.dtype.fields.size();
if (true_size > 1 && metadata.dtype.fields.back().name.empty()) {
--true_size;
}
internal::ChunkGridSpecification::ComponentList components;
components.reserve(metadata.dtype.fields.size());
components.reserve(true_size);
std::vector<DimensionIndex> chunked_to_cell_dimensions(
metadata.chunks.size());
std::iota(chunked_to_cell_dimensions.begin(),
chunked_to_cell_dimensions.end(), static_cast<DimensionIndex>(0));
for (size_t field_i = 0; field_i < metadata.dtype.fields.size(); ++field_i) {
for (size_t field_i = 0; field_i < true_size; ++field_i) {
const auto& field = metadata.dtype.fields[field_i];
const auto& field_layout = metadata.chunk_layout.fields[field_i];
auto fill_value = metadata.fill_value[field_i];