Skip to content

Commit

Permalink
cloudpickle support is no longer experimental.
Browse files Browse the repository at this point in the history
  • Loading branch information
tvalentyn committed Mar 7, 2025
1 parent c1d0fa4 commit d59cc77
Showing 1 changed file with 1 addition and 1 deletion.
Original file line number Diff line number Diff line change
Expand Up @@ -163,7 +163,7 @@ Dataflow, see [Pre-building the python SDK custom container image with extra dep

When the Python SDK submits the pipeline for execution to a remote runner, the pipeline contents, such as transform user code, is serialized (or pickled) into a bytecode using
libraries that perform the serialization (also called picklers). The default pickler library used by Beam is `dill`.
To use the `cloudpickle` pickler, supply the `--pickle_library=cloudpickle` pipeline option. The `cloudpickle` support is currently [experimental](https://github.com/apache/beam/issues/21298).
To use the `cloudpickle` pickler, supply the `--pickle_library=cloudpickle` pipeline option.

By default, global imports, functions, and variables defined in the main pipeline module are not saved during the serialization of a Beam job.
Thus, one might encounter an unexpected `NameError` when running a `DoFn` on any remote runner. To resolve this, supply the main session content with the pipeline by
Expand Down

0 comments on commit d59cc77

Please sign in to comment.