Skip to content

Conversation

@DavisVaughan
Copy link
Collaborator

Closes #253

@HenrikBengtsson does this make sense to you?

Here's the setup:

  • Locally, you have purrr 1.0.0
  • On your worker, you have purrr 0.3.5

The API hasn't changed between the two versions, but the internals did.

We used to capture the local version of purrr::map and ship that over to the worker. The captured version of purrr::map() had internals that looked for purrr:::map_(), which is an internal function that exists in purrr 1.0.0 but not 0.3.5, so it could fail. That resulted in #253 and https://stackoverflow.com/questions/75660223/r-furure-map-across-multiple-servers.

We now ship the name of the purrr function over to the worker, but look up the actual function from the purrr namespace while on the worker itself. That should prevent this issue (though I don't have a very easy way to test this particular issue).

So `expr_seed_setup` and `expr_seed_update` still live close together
Comment on lines +48 to +53
...furrr_purrr_fn <- get(
!!purrr_fn_name,
envir = asNamespace("purrr"),
mode = "function",
inherits = FALSE
)
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here's where we now splice in purrr_fn_name, which is just "map" or "map_dbl". Then we let that be looked up on the worker itself.

We're building an expression to send to the worker here, so after the splicing this part of the expression should just look like:

    ...furrr_purrr_fn <- get(
      "map",
      envir = asNamespace("purrr"),
      mode = "function",
      inherits = FALSE
    )

and the worker gets that

@HenrikBengtsson
Copy link
Contributor

Yes, this update makes sense to me.

Inconsistent setups between main R session and workers is a problem to all parallel frameworks, including futureverse. Right now, we're crossing our fingers and hope it'll work, and if it doesn't, hopefully there's a run-time error catching it. The worst case scenario is that the error is silent and we get incorrect results.

What the future ecosystem could do is to report on package version discrepancies, e.g. produce a FutureWarning, or even an FutureError if there are major version differences. This is to be explored. This also fall under the plan to introduce resource specifications, i.e. package version requirements are in one way similar to OS and GPU requirements.

The bottom line is that, furrr should not have to worry about this, but rely on the future core to handle this.

@DavisVaughan DavisVaughan merged commit 3b67165 into main Jan 23, 2026
12 of 13 checks passed
@DavisVaughan DavisVaughan deleted the feature/get-purrr-on-worker branch January 23, 2026 20:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Export a wrapper around purrr map() to workers rather than map() itself

3 participants