Enable smart code intelligence in external editor through jupytext

[jvican]

Love the project. Been a user for the past months.

I'm a big user of the external editor, since Euporie's editor experience is somewhat frustrating even with vim keybindings enabled.

One problem the external editor approach has is that if it's set up to work with ty or another LSP server, it won't work because imports are missing.

Would it be possible to add some changes to the external editor mode so all existing imports in the notebook (up to the given cell) are added at the beginning of the code in the editor, separated by a comment marker? Something like this:

# DON'T CHANGE, IMPORTS FROM NOTEBOOK
import torch
# WRITE YOUR CODE UNDER THIS LINE FOR CELL CHANGES
...

The idea is that the marker would help Euporie strip those imports when writing back the edited contents.

Thoughts?

[joouha]

Hi,

Thanks for reporting this. I've noticed this issue too when using an external editor.

The problem with fixing it in the way you suggest is that as a Jupyter client, euporie is language agnostic, and this fix would only work when working with Python notebooks. I'm keen to avoid writing adding special language-specific behaviour to euporie which involves parsing the code being edited.

There's another issue - you'll also get LSP errors for variables defined in other cells, so they'd need to be copied over too, among other things. It would become a bit of a minefield.

It would be better to be able to tell the external editor's LSP that it's editing a single cell from a larger notebook, but I'm not sure if there's a way to do that. Euporie has it's own LSP client capabilities which works with notebooks, but I'm not sure it's possible to connect an external editor to an already running LSP server. This might be possible is the LSP server is served over pipes or sockets, instead of stdio - I'll have to do some research and have a think about what could be done.

These links look interesting:

• Plans for supporting multiple clients from a single LSP server? microsoft/language-server-protocol#1160
• https://codeberg.org/p2502/lspmux

In the meantime, I'd be very interested to hear more about your frustrations with euporie's editing experience, as that's something I'm keen to improve.

[jvican]

I agree that's an interesting way of solving the problem. It could work. My only concern would be that it'd require LSP-specific configurations on the editor side. Most of the times, my driver LSP server for Python for nvim/helix is ty, and configuring both would require avoiding failures when the notebook environment isn't used, or finding ways it could be configured that way.

Another less Python-specific way of supporting this workflow could be to automatically convert the whole jupyter notebook into a Python file (I know there are specific Jupyter tools that do this out of the box) and then load that Python file with comments guarding/fencing the contents that will be copied back into the cell.

This way, both the problems with imports and variables could be fixed. Still not ideal, but as far as simple solutions go, it's a simple solution that could also work for other languages out of the box.

[jvican]

By the way, jupytext was the solution that converts notebooks to Python files directly and I imagined could be used for language-agnostic support. It's nice that there's bidirectional support for it in VS Code.
https://jupytext.readthedocs.io/en/latest/vs-code.html

[joouha]

Euporie already supports jupytext - just install it in the same environment as euporie, then you'll be able to open Python scripts as notebooks, or save your notebooks as Python scripts. If you edit the script externally, you can reload the file in euporie using F5 to see the changes.

[jvican]

Right, jupytext works out of the box. I could go down that route with an external editor.
I understand if this is not a priority for you right now, but I'd also see value in supporting jupytext as part of the external editor shortcut. That would likely provide a better experience.

[jvican]

In the meantime, I'd be very interested to hear more about your frustrations with euporie's editing experience, as that's something I'm keen to improve.

By the way, I forgot to reply to this. My main frustrations, right now, are UX and speed. I find that

• opening medium-size notebooks creates UI lag (clicking buttons, moving around cells, etc.)
• when switching around ghostty windows, Euporie introduces weird control codes into buffers and sometimes actuates some actions (a common complaint is it annoyingly turns code cells to markdown cells when my mouse changes focus to another ghostty window)
• whenever I resize the terminal windows, euporie takes a long time to rerender everything and sometimes it freezes (weirdly, mouse movement also gets unaligned, so when you click on things nothing happens because coords are wrongly recomputed)

I'm loving it, but those are the pain points I'm experiencing.

[joouha]

Oh I see what you mean, you mean use jupytext to send the whole notebook file to the external editor instead of just the current cell, then sync with the notebook when editing is complete. That should be possible, I'll do some experimenting.

Regarding the other issues.

• The UI responsiveness when opening large notebooks should be much improved in the development version
• I've seen the escape code leakage issue, but I'm not sure what triggers it. Do you know a way of triggering it reliably?
• I am able to reproduce the slow re-rendering on resize and mouse-movement misalignment. I'll look into this.

Thanks!

[joouha]

The dev branch now contains fixes for the issues you reported:

• The UI lag when opening large notebooks is fixed by 3ac6fe0
• The unrecognize escape codes when switching frames in ghostty is fixed by 212390f
• The slow resizing is fixed by dbb8bf0
• The mouse position unalignment on resize is fixed by d4f86eb

[jvican]

The UI responsiveness when opening large notebooks should be much improved in the development version
The slow resizing is fixed by dbb8bf0

Interestingly, I ran the dev version yesterday on a medium-sized notebook and still had rendering issues and sluggishness. These two UX issues remained.

I noticed image support improved. That's also much welcome. However, some images were rendering as corrupted. Weirdly enough, the image was blurred and had some weird symptoms, yet the overall shape of the image was rendered. I was hoping the bug I filed fixes it, but it's possible it might be unrelated. Rendering happens on ghostty.

I'll do more experimentation with the development version and get back to you.

Oh I see what you mean, you mean use jupytext to send the whole notebook file to the external editor instead of just the current cell, then sync with the notebook when editing is complete. That should be possible, I'll do some experimenting.

Yes, in my opinion this would be the best of both worlds! You get language-agnostic code intelligence, you make euporie completely editor-agnostic, etc.

[jvican]

By the way, with the external editor solution, I'd expect the experience to be this one:

• User clicks on E key on a cell they want to edit
  • Euporie runs jupytext on virtual ipynb, obtains new source file (.py), and obtains line position for the cell
• Editor is launched with both file name and line number, so the editor jumps straight into the line to be edited
  • Can be achieved by passing in +line_number for vim/emacs, and by appending line number to end of file with the rest of the editors, e.g. edit file:line_number
• User makes changes in the cell code, which is surrounded by special euporie delimiters, then saves them and quits the editor
• Euporie UI shows up, changes are applied to the cell, then user can continue with the euporie experience

[jvican]

By the way, my bad. I hadn't realized those bug fixes above were indeed done a few hours ago.
I'm going to pull them in and try them out. Will report back.

The bug with the rendered images still remains. This is how they're rendered:

The weird thing is that a few larger images do get rendered well, but 10 images like the one above do not. No logs were produced. The right image is this one:

In this case, it was rendered through an image link, not a base64 image.

[joouha]

The "blurred" images you are seeing are text renderings, which use unicode characters to represent the image. Which rendering method gets used will depend on many factors, including which graphics protocol if any your terminal supports, which applications and python packages you have installed, and the format of the images. Euporie will always try to render images in the best way it can, depending on what methods are available.

If you want to learn more about the external tools / packages euporie can use, you can read the documentation here:
https://euporie.readthedocs.io/en/latest/pages/installation.html#optional-dependencies

If some images are rendering using terminal graphics and some are rendering with text, I suspect that some of them might be SVG images, and euporie can't find a method to rasterize them.

[jvican]

If some images are rendering using terminal graphics and some are rendering with text, I suspect that some of them might be SVG images, and euporie can't find a method to rasterize them.

These images are all png images (I've tried this both in base64-encoded images and regular image URLs). Some of them get rendered, some others don't. Another issue I've noticed is that resizing the terminal window where Euporie runs would black out images.