Skip to content

timestamp errors working with Delta Lake tables (and presumably Parquet?) #6396

Discussion options

You must be logged in to vote

solving this led me here: https://stackoverflow.com/questions/59682833/pyarrow-lib-arrowinvalid-casting-from-timestampns-to-timestampms-would-los

and here: https://delta-io.github.io/delta-rs/python/api_reference.html#writing-deltatables

with good examples in here: https://github.com/delta-io/delta-rs/blob/5dc89b389d830cf72f68ace54adab85c81c26a69/python/tests/test_writer.py#L462-L488

with the solution being to, on writing, do:

import ibis
from pyarrow.dataset import ParquetFileFormat

t = ... # get t however

t.to_delta(
    "path/to/delta", 
    mode="overwrite", 
    file_options=ParquetFileFormat().make_write_options().update(coerce_timestamps="us")
)

it was a little hard to figure out…

Replies: 1 comment

Comment options

You must be logged in to vote
0 replies
Answer selected by lostmygithubaccount
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Category
Q&A
Labels
None yet
1 participant