Feature request: A simple and less-reproducible version of caching

[pdeffebach]

I was frustrated with my first foray into quarto today, partly because I had unreasonable expectations using it.

My quarto document looks something like

using MyPackage 

out = a_very_long_and_heavy_function()

and then

lots_of_graphs(out)

even_more_graphs(out)

I was trying a workflow where I had Revise.jl running and would modify the function lots_of_graphs. Unfortunately, whenever I added a new graph, quarto would re-run a_very_long_and_heavy_function() again. I wanted to change what I wrote based on the output of the graphs (and fixing stylistic things in the graphs located in MyPackage), but I could do that because very time I did anything with a graph I would have to wait 5 minutes for a_ver_long_and_heavy_function.

One proposal would be that quarto caches the literally text of each code block, and then only executes the code blocks that have been edited. More like a jupyter notebook. If I create a code block and hit "save", then it only re-runs the cells that were modified, without paying attention to the dependency graph in any way.

This is obviously a bad workflow for creating reproducible documents, but it will allow me to work iteratively. I'm not sure how to use quarto in julia otherwise, though. Do people just write a whole document without seeing the graphs, and then click "run" to fill it all in?

Perhaps there is a feature that I am missing?

[ajinkya-k]

I could be wrong but this particular issue is easily solved by QuartoTools caching.

```{julia}
import QuartoTools
```

```{julia}
#| julia:
#|   cache:
#|     enabled: true
out = a_very_long_and_heavy_function()
```

[MichaelHatherly]

Yes, see whether the caching in QuartoTools helps you with this particular use case. If not then I'm sure we can come up with some improvements to either the built in caching in QNR, or to the caching in QT.

[pdeffebach]

Thanks,

The caching does help with this particular use-case, but even the lag of running the plotting cells, along with the loading from the cache, is too much for me to like using Quarto. In the preview mode, I was expecting it to run only the code that was absolutely necessary, and function more similar to a jupyter notebook, where you can build up code linearly, and add on piece by piece.

Re-running a large number of cells, whether cached or not, is simply too high latency. Not sure what tool is best, because I don't like jupyter notebooks or in-line evaluation in general. I like keeping formatted output separate from my plain text writing.

[ajinkya-k]

You can just use jupyter notebooks which do have julia kernels

[pdeffebach]

I certainly can, and I'm working on installing IJulia (and jupyter) now. But still think that a lower latency, more interactive, version of Quarto is something I would really appreciate.

[pdeffebach]

This is basically what I expected quarto preview to be with the caveats that (1) I can't install this but it doesn't look like it is re-evaluating all cells (2) The output on the RHS would be a static html, not a notebook.

[MichaelHatherly]

We'd be happy to have improvements to the default caching behaviour where it makes sense. The current behaviour was just the lowest hanging fruit. It would certainly be nice to have the preview render be closer to instant.

[jkrumbiegel]

I guess what would be a bit annoying even if we introduced a second caching strategy that only invalidates cells if they have been edited, is that you'd have to add arbitrary edits to the cells you want to rerun if you are actually making changes somewhere else via Revise. Sounded like that was your idea with the plotting code?

[pdeffebach]

That's a really good point, yeah. I would like to re-run later cells without editing them, but don't know how ctrl-enter would be registered from sublime text or something. You are right that this is a flaw in what I've described above.

Trying out a jupyter notebook the past few days has made me realize that the pain point in the notebook is that the plots and output just take up too much space. Its impossible to edit code when you are scrolling through a bunch of plots all the time.

A key benefit to the workflow I'm describing is that all the code (and text) is on a separate editor (sublime text for me) that's easy to navigate, then the output is separate. Basically a generic workflow with LaTeX, which is what I think I will go back to for producing outputs for the time being, and just not bother with letting the reader see what code is run.