Skip to content

Dataflow Python SDK logging: step_id is always empty string #19711

@damccorm

Description

@damccorm

Using the dataflow runner, log messages always show up in stackdriver with the step_id as the empty string, so filtering log messages for a step doesn't work.


resource: {
  labels: {
    job_id: "<job id>" 
    job_name: "<job name>" 
    project_id: "<project
id>" 
    region: "<region>" 
    step_id: "" 
  }
  type: "dataflow_step" 
}

Another user seems to have posted in the old github repo and appears to be seeing the same problem based on their output:

GoogleCloudPlatform/DataflowPythonSDK#62

From what I can tell is only affecting streaming pipelines

Imported from Jira BEAM-7934. Original Jira may contain additional context.
Reported by: jimpremise.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions