-
|
Thank you for the excellent work everyone does on duckdb; I have an optimization question. I am struggling with and wondering if folks had opinions. Thank you Context
ogr2ogr -f "Parquet" reference_flowline.parquet reference_flowline.gpkg -t_srs EPSG:4326 -lco COMPRESSION=ZSTD -lco ROW_GROUP_SIZE=122880 EXPLAIN SELECT fid
FROM read_parquet(
'gcs://MY_FILE.parquet'
) AS catchments
WHERE ST_Intersects(geom, ST_Point(-108.50231860661755, 39.05108882481538))
LIMIT 1Geoparquet MetadataWhat worksI am finding that I can use the following, but this seems to just download all the data directly and put it in memory. So at this point it seems like it makes more sense to just download the parquet file on disk. This is the output of the macos system monitor after running that then building the rtree spatial index. |
Beta Was this translation helpful? Give feedback.
Replies: 3 comments 1 reply
-
|
@C-Loftus Pretty sure you just want to create a |
Beta Was this translation helpful? Give feedback.
-
|
Thank you very much for your response. I tried to do that but perhaps I am missing something. It seems quite slow since my understanding is that I cannot build the rtree index on the I haven't timed this yet but the intersection is currently taking >4 minutes (EDIT: It appears to have completed in 4 minutes 30 seconds) DROP VIEW IF EXISTS flowline;
CREATE VIEW flowline as SELECT fid, geom
FROM read_parquet(
'gcs://national-hydrologic-geospatial-fabric-reference-hydrofabric/reference_flowline.parquet'
)
SELECT * FROM flowline
WHERE ST_Intersects(geom, ST_Point(-108.50231860661755, 39.05108882481538))
LIMIT 1 |
Beta Was this translation helpful? Give feedback.
-
|
From discussing with a coworker it seems like a way to solve this might be to partition the parquet dataset into 50 separate parquet files by their geometry, i.e. a separate parquet file for each state and have another table with the geometry of each state. Then a user could
This would likely work although I was hoping to find a way to do this without needing to have an extra states table which adds a bit of overhead. |
Beta Was this translation helpful? Give feedback.
-
|
I had some great help on this issue here: geoparquet/geoparquet-io#95 First, I tried h3 partitioning but I was told that the size of my dataset (few gigabytes) is relatively small and wouldn't really need it. I got some great advice and, it does seem like there IS in fact predicate pushdown on a query like this. My guess is that SELECT *
FROM read_parquet(
'gcs://dataset.parquet'
)
WHERE ST_Intersects(
geom,
ST_MakeEnvelope(-69.0877, 47.0479, -69.0453, 47.0889)
);As such I am closing this since this query completes in under a second even when querying to a file on my object store. |
Beta Was this translation helpful? Give feedback.
I had some great help on this issue here: geoparquet/geoparquet-io#95
First, I tried h3 partitioning but I was told that the size of my dataset (few gigabytes) is relatively small and wouldn't really need it.
I got some great advice and, it does seem like there IS in fact predicate pushdown on a query like this. My guess is that
ST_MakeEnvelopecauses it to implicitly filter against the geometry bbox column which I was not doing above.As such I am closing this since this query completes in under a second even when querying to …