[Feature Request]: Add InsertRetryPolicy for non-successful BigQuery insertAll responses.

[baeminbo]

What would you like to happen?

BigQueryIO's InsertRetryPolicy only applies to per-element errors within 200 OK responses. Non-200 responses (e.g., 400 Bad Request) are not handled by the policy and cause indefinite retries in Dataflow streaming jobs, even with InsertRetryPolicy.neverRetry(). #34118 was filed to clarify this behavior in Javadoc.

However, I believe we can extend the InsertRetryPoilcy to handle non-successful responses easily. To address this, InsertRetryPolicy's Context, which currently only contains per-element error information, would need to be extended to include an exception. The elements that fail with non-a successful response should then be routed to Dead Letter Queue (DLQ) within BigQueryServicesImpl#insertAll

Issue Priority

Priority: 2 (default / most feature requests should be filed as P2)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam YAML
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Infrastructure
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner