POC: Signal progress via progressr

[DavisVaughan]

Closes #138
Closes #263

This implements the ideas in futureverse/progressr#85, all of the arguments there are still valid, and the global calling handler part of progressr makes this way easier to swallow as well.

• Remove TODO! in docs
• Rewrite progress vignette entirely
  • Show how to customize handler
  • Show how to set handler "globally"
  • Show how to "bring your own" progressr if you don't want to use .progress = TRUE (for advanced cases only)
• NEWS bullet
• Can we resolve the handle_cli() slowness? It is very bad! Do not force cli to refresh the display progressr#190

---

@HenrikBengtsson this is an experimental ideal that emits progressr::progressor() backed progress updates after each .f invocation when .progress = TRUE. We create a progressr::progressor() in the main R process which has as many steps as there are elements to iterate over, and that is then exported to all of the workers. The workers just call p() after each .f call.

I know we've had various conversations about where the user vs developer split is on progress bars. I feel like this is actually a pretty reasonable solution, but I'm very interested to hear your thoughts!

• The user is still in charge of displaying progress. They must opt in with with_progress() or (my preference) handlers(global = TRUE).
• The user can customize progress output with handlers(handler_cli()) and friends.
• Interactive users of furrr retain an easy way to add a progress bar to furrr functions.
• Developers building with furrr get an easy way to access the workflow of: 1) create a scoped progressr::progressor() 2) call it at some interval.
  • IMO developers should expose a progress = FALSE/TRUE flag of their own, which their users should opt into for progression updated. I feel this is particularly important because progression signaling is definitely not "free" and should not really be used with short running tasks.

I noticed that handler_cli() is extremely slow. I see futureverse/progressr#167 and would love to see if we can get to the root of that. The progress bar in purrr doesn't seem to have this issue and is backed by cli. Something weird must be happening here.

---

I imagine user scripts to look something like this

library(furrr)

progressr::handlers(global = TRUE)
progressr::handlers(progressr::handler_cli())

plan(multisession, workers = 2)

xs <- seq_len(100)

model_fn <- function(x) {
  Sys.sleep(.1)
  x
}

out <- future_map(xs, model_fn, .progress = TRUE)

[HenrikBengtsson]

Yes, this all makes sense to me. Providing a .progress = TRUE/FALSE will also cover parts what users expect coming from purrr.

Also, analogously to the new futurize::futurize(), I'm working on adding support for progressify(), e.g.

ys <- furrr::future_map(xs, fcn) |> progressify()
ys <- purrr::map(xs, fcn) |> progressify() |> futurize()

can simply transpile info:

ys <- furrr::future_map(xs, fcn, .progress = TRUE)

IMO developers should expose a progress = FALSE/TRUE flag of their own, which their users should opt into for progression updated. I feel this is particularly important because progression signaling is definitely not "free" and should not really be used with short running tasks.

Agree - this is tricky. I've been hoping something would emerge over time. If a progressify() transpiler could become a standard, the one could disable progressify() with a global settings - which would be very cheap. Anyway, more on that later.