Unbalanced Load distribution on first run

[mfakaehler]

Hello,

I'm experiencing an issue with the load distribution of parallel jobs executed with future_map. In particular, I observe that the first time I call future_map the workload is happening on 2-3 workers only, while on consecutive runs of the same call the workload is shared evenly across the workers.
I tried to narrow it down into a reprex:

library(future)
library(tictoc)

plan(multisession, workers = 10)

tic()
res <- purrr::map(
  .x = 1:1e6, .f = ~.x +1
)
toc()
# 1.92 sec not in paralle

tic()
furrr::future_map(
  .x = 1:1e6, .f = ~.x +1
)
toc()
# 3.462 sec on first run

microbenchmark::microbenchmark(
  {
    furrr::future_map(
      .x = 1:1e6, .f = ~.x +1
    )
  },
  times = 20
)
# 1.2 secs on average on consecutive runs

In my "real-world" applications, where there is also a considerable amount of data to be passed to the workers, this tends to be more extreme.

This might be related to this previous issue:
#3

I'm working on the following system:

R version 4.2.2 Patched (2022-11-10 r83330)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 22.04.1 LTS

Any thoughts appreciated.

Best,
Maximilian

[DavisVaughan]

I have a feeling that in this particular example you are just seeing some "startup costs" on the first run that aren't there on the other runs.

In particular, on the first run the purrr package is loaded on the workers, but on the subsequent runs it should already be loaded.

library(future)
library(tictoc)

plan(multisession, workers = 10)

tic()
res <- purrr::map(
  .x = 1:1e6,
  .f = ~ .x + 1
)
toc()
#> 0.637 sec elapsed

tic()
res <- furrr::future_map(
  .x = 1:1e6,
  .f = ~ .x + 1
)
toc()
#> 0.635 sec elapsed

tic()
res <- furrr::future_map(
  .x = 1:1e6,
  .f = ~ .x + 1
)
toc()
#> 0.33 sec elapsed

tic()
res <- furrr::future_map(
  .x = 1:1e6,
  .f = ~ .x + 1
)
toc()
#> 0.221 sec elapsed

tic()
res <- furrr::future_map(
  .x = 1:1e6,
  .f = ~ .x + 1
)
toc()
#> 0.361 sec elapsed

tic()
res <- furrr::future_map(
  .x = 1:1e6,
  .f = ~ .x + 1
)
toc()
#> 0.24 sec elapsed

If you use entirely new workers between the furrr runs, things stabilize:

library(future)
library(tictoc)

tic()
res <- purrr::map(
  .x = 1:1e6,
  .f = ~ .x + 1
)
toc()
#> 0.64 sec elapsed

plan(multisession, workers = 10)
tic()
res <- furrr::future_map(
  .x = 1:1e6,
  .f = ~ .x + 1
)
toc()
#> 0.478 sec elapsed
plan(sequential)

plan(multisession, workers = 10)
tic()
res <- furrr::future_map(
  .x = 1:1e6,
  .f = ~ .x + 1
)
toc()
#> 0.45 sec elapsed
plan(sequential)

plan(multisession, workers = 10)
tic()
res <- furrr::future_map(
  .x = 1:1e6,
  .f = ~ .x + 1
)
toc()
#> 0.435 sec elapsed
plan(sequential)

plan(multisession, workers = 10)
tic()
res <- furrr::future_map(
  .x = 1:1e6,
  .f = ~ .x + 1
)
toc()
#> 0.452 sec elapsed
plan(sequential)

plan(multisession, workers = 10)
tic()
res <- furrr::future_map(
  .x = 1:1e6,
  .f = ~ .x + 1
)
toc()
#> 0.433 sec elapsed
plan(sequential)

The future package tries to "clean" as much state as possible between runs, but packages do stay loaded between runs.

[mfakaehler]

Thanks for your thoughts. That makes sense. When I use plan(multicore) (i.e. forking of processes) instead of plan(multisessions) the issue does not appear, which supports your start-up explanation.