[Bug]: beam.io.WriteToCsv ignores num_shards argument.

[ganesh4991]

What happened?

beam.io.WriteToCsv does not take respect the num_shards argument while writing the output. The workaround for now is to use beam.io.WriteToText instead.

Issue Priority

Priority: 2 (default / most bugs should be filed as P2)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam YAML
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Infrastructure
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[Rupsa1234]

Thanks for reporting this issue! I have also encountered the same problem where WriteToCsv does not respect the num_shards parameter, causing all data to be written to a single file instead of multiple shards.

I tested this with Apache Beam Python SDK 2.49.0 on Google Dataflow Runner and observed:

• Expected: output-00000-of-00005.csv, output-00001-of-00005.csv, etc.
• Actual: Only one output.csv file is generated.

Workaround:
Using WriteToText instead solves the issue:
p | "Write to Text" >> beam.io.WriteToText("output", file_name_suffix=".csv", num_shards=5)

[Rupsa1234]

can i work upon this?

[ganesh4991]

cc @tvalentyn

[liferoad]

@Rupsa1234 feel free to take it. https://beam.apache.org/contribute/