Using the dataflow runner, log messages always show up in stackdriver with the step_id as the empty string, so filtering log messages for a step doesn't work.
resource: {
labels: {
job_id: "<job id>"
job_name: "<job name>"
project_id: "<project
id>"
region: "<region>"
step_id: ""
}
type: "dataflow_step"
}
Another user seems to have posted in the old github repo and appears to be seeing the same problem based on their output:
GoogleCloudPlatform/DataflowPythonSDK#62
From what I can tell is only affecting streaming pipelines
Imported from Jira BEAM-7934. Original Jira may contain additional context.
Reported by: jimpremise.