Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: tfrecordio does not work with snappy >= 0.7 #33407

Open
1 of 17 tasks
cnsgsz opened this issue Dec 17, 2024 · 1 comment · May be fixed by #34225
Open
1 of 17 tasks

[Bug]: tfrecordio does not work with snappy >= 0.7 #33407

cnsgsz opened this issue Dec 17, 2024 · 1 comment · May be fixed by #34225

Comments

@cnsgsz
Copy link

cnsgsz commented Dec 17, 2024

What happened?

import snappy succeeds but both getattr in tfrecordio.py return None. snappy < 0.7 does not have the issue.

Marking it as P1 as it silently but dramatically slows down reading & writing. It would also be nice to add a test for this or make the error less silent.

Issue Priority

Priority: 1 (data loss / total loss of function)

Issue Components

  • Component: Python SDK
  • Component: Java SDK
  • Component: Go SDK
  • Component: Typescript SDK
  • Component: IO connector
  • Component: Beam YAML
  • Component: Beam examples
  • Component: Beam playground
  • Component: Beam katas
  • Component: Website
  • Component: Infrastructure
  • Component: Spark Runner
  • Component: Flink Runner
  • Component: Samza Runner
  • Component: Twister2 Runner
  • Component: Hazelcast Jet Runner
  • Component: Google Cloud Dataflow Runner
@liferoad
Copy link
Contributor

liferoad commented Mar 9, 2025

Beam indeed reports this as a warning message if snappy cannot be used here. I think the right fix is to get rid of snappy and switch to other crc32c packages. Created #34226 to track this as a new feature.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants