Persistent Jupyter Kernels - Restore/Re-connect to an existing local/remote kernel (do not shutdown kernel upon closing/reloading vscode/vscode.dev)

[DonJayamanne]

Problem

Local

• User starts a notebook, the kernel is now running on the local machine
• Assume the computer goes to sleep,
• After a while if we go back into the notebook, the Notebook is unable to re-connect to the same kernel (kernel state is lost)
• Similarly if user re-loads VS Code, the notebook is unable to re-connect to the same kernel (kernel state is lost)
• Similarly if a user is using Remote SSH, and connection is reset and user re-connects and opens the same notebook, then the user is unable to re-connect to the same kernel (kernel state is lost)

Remote

• User opens a notebook and runs a cell against a remote kernel
• Similarly if user re-loads VS Code (or vscode.dev), the notebook is unable to re-connect to the same kernel (kernel state is lost)

Investigation Running Server & JupyterLab API for extensibility
Goals:

• Long running kernels
  • User can open a notebook with a cell thats still running & see the output being generated
  • Same as User can open notebooks related to kernels that are still running.
• Extensibility for extension authors
  This is a by-product of the long running kernels (i.e. you get this for free - almost)

Planned (related) Prototypes

• Long running kernels
  Solve problems related to kernel/session being lost due to :
  • VS Code Shutdown
  • VS Code Restart
  • Computer sleeping
  • SSH connection issues
  • AML Compute will benefit
• By product of extensiblity
  • Julia Widgets (i might end up doing this first, might be easier)
  • IPyWidget outside notebooks
  • Variable viewer using the new api
  • Data Frame viewer using the new api

Technical details

• Server
  Background process
  Manages kernels & sessions
  Expose kernel socket connection over this connection (we already have the code/technology for this) - proxy socket (dummy kernel in UI layer, by creating a dummy socket connection)
  Security - how do we secure this web server (will need to be addressed, but i'm leaving that for later)
• Expose Jupyter extension extensibility over Jupyter Lab API
  I wont be exposing a connection, instead will just expose the SessionManager, KernelManager & other class instances from extension API

Also related #300

[matifali]

Is there a milestone issue to see the progress of the update?

[DonJayamanne]

Unfortunately this issue has not yet been prioritized at our end, please do vote on this issue though

[matifali]

What do you suggest as a workaround if one wants to run long 10+ hours sessions using Jupyter notebooks in vscode when connected to a remote kernel over SSH (using vscode remote extension)?
after some hours the connection gets disconnected and there is no way to see the progress or output of running cells.

[DonJayamanne]

@matifali unfortunately at this stage we have no work around for this, let me see if i can get an udpate within a week.

[DonJayamanne]

@matifali
I'm trying to understand your expectations, hence the following questions

• Assume you have 1 cell
  *Code in this cell prints numbers from 1 to 100, printing a number every hour
• Assume you run this cell and saw the number 1 printed out.
• Now you run this cell, and close vscode and come back tomorrow and open vscode and open this asme notebook
• Would you expect to see the numbers 1, 2, 3, 4 and then slowly the number going up to 100 while vscode is open (as the execution is still in progress)
• Or would you expect to see 1, 80, 81, 82 and then the number will keep going up while vscode is open (as the execution is still in progress)
• Assume you have opened vscode after a few hours and you know all 100 would have been printed out and vscode was closed. Would you expect to see all 1, 2, ... 100 in the output or just expect to be able to connect to the kernel and see the fact that execution has completed

I ask this because the easiest thing to get working is:

• if the cell is still running then we display 1, 80, 81, 82 (where 1 was from the first instance of vscode and 80, 81 and so on after vscode is opened again. I.e. all of the output generated while vscode was closed will not becaptured and not stored in teh notebook)
• I.e. we will only allow connecting to a kernel and you can see whether exeuction has comlpeted {or not, and if it is still going on then the data will be appened to what was stored previously

Thanks

[matifali]

• Would you expect to see the numbers 1, 2, 3, 4 and then slowly the number going up to 100 while vscode is open (as the execution is still in progress)

I would prefer this output as my use case is to train deep learning models and its better if we can see the full history.

Assume you have opened vscode after a few hours and you know all 100 would have been printed out and vscode was closed. Would you expect to see all 1, 2, ... 100 in the output or just expect to be able to connect to the kernel and see the fact that execution has completed

This is preferred,

Or would you expect to see 1, 80, 81, 82 and then the number will keep going up while vscode is open (as the execution is still in progress)

This is also OK but the problem is vscode is unable to connect to a running remote kernel and show any outputs. Yes, the process is running but we do not see anything printed. There is no indication if the losses are actually decreasing.

[ma-ji]

+1, and echoing above points that this is a breaking issue, and the number 1 pain point for me developing on VS code.

Can't agree more!

[jmodlis]

+1

[jmodlis]

I'm wondering if this could be handled the same way that persistent R sessions are handled? I utilize the extension below, that has a "session watcher feature". I'm able to attach to R sessions from the previous day, with objects intact. It is just as reliable and sometimes even more so, than Rstudio IDE.

https://github.com/REditorSupport/vscode-R?tab=readme-ov-file

I was hoping that I could do something similar with python by utilizing a python script, the #%% feature, and by including this in my settings.json file:
"jupyter.interactiveWindow.textEditor.executeSelection": false

While I am able to send code to the python terminal within a single session using the above solution, when I log in the next day and try to send code again, it creates a new python terminal and my old one is ignored. The only way to interact with the old python terminal is to manually copy/paste code into the terminal which is hugely problematic. Perhaps it is possible to reconnect my python "session" with this particular terminal, but I have not figured out how to do so.

For reference, I am using CodeServer/VSCode in an OnDemand interface, with the session being supported by a slurm job on HPC cluster.

ETA, this suggestion is coming from my complete lack of knowledge on how things work in the backend. :)

[sol401430]

+1

[Art-MC]

+1 this is by far the biggest issue I have with VS code

[lheinke]

+1

[NakerTheFirst]

+1

On a side note, what's the convention for upvoting an issue? Are top issues determined by the number of reactions or by the number of comments? Sure spamming +1 does bump the issue, but also sacrifices on its readability.

[alopez247]

+1 this is by far the biggest issue I have with VS code

Mine as well

[3f6a]

This is the sole reason why I don't use VS code notebooks (except in the rare occasions I'm running everything locally).
Please, add this. It would be a major improvement.

Related: #15543

See also jupyterlab/jupyter-collaboration#279, that got recently merged, and is trying to solve the same kind of problem (on JupyterLab)

[yunho-c]

+1

[Butanium]

+1, so frustrating when the Jupyter environment is still running when I reconnect, but I can't access it

[custard-pi]

+1

[peterkilfeather]

Please fix this. VSCode is unusable because of this.

[yunho-c]

+1

[thewh1teagle]

+1

[pcmoritz]

+1, this would be great to fix :)

[sijieamoy]

+1

[brucez-anyscale]

+1

[alexr-anyscale]

+1

[PawaritL]

+1

[GokuMohandas]

+1 this would improve my vscode experience 10x

[KoljaFrahm]

+1

[gargj-aws]

+1 This is a very important issue that is impacting our productivity daily and made us switch to JupyterLab (hopefully only temporarily)

[3f6a]

@gargj-aws not to derail this topic, but does JupyterLab currently support persistent remote sessions, where you can reconnect without losing output ?

[gargj-aws]

@gargj-aws not to derail this topic, but does JupyterLab currently support persistent remote sessions, where you can reconnect without losing output ?

Yes and no: yes on persistent remote sessions (e.g. on SageMaker Studio), but no on the output that is generated while the browser is closed.

[xgdgsc]

Call the command reload from disk in jupyterlab after restarting browser or change device seems working fine for me.